J83: stabilize Rust tests under VM load

kayg commented

2026-11-02 00:08:52 +00:00

Owner

J83: make the five reported Rust test failures deterministic under VM load without masking defects.

Plan (estimate: 3 hours, stop and report if exceeding 6 hours):

Branch test-stability-load from origin/dev in an isolated control-host worktree. Read relevant test and production paths; reproduce each named test/suite 20 times under bounded artificial CPU load on a development VM, retaining individual logs.
Fix proven root causes separately: executable publication/write descriptor lifecycle, readiness/event synchronization for restart and socket tests, private service/runtime environment for status tests. No guessed deadline increases or production contract relaxation.
Run the same 20-iteration load sweep after each fix. One atomic commit per test or shared root cause with issue reference and gate evidence.
Run scripts/verify fast under load, record static sizes/deltas and verbatim output, push only test-stability-load, report head SHA and before/after counts. Clean up owned processes and the single target directory.

Initial queues: dev-02 has 1/3 slots occupied and 68G free; compute01 has 3/4 occupied and 115G free. Use dev-02 unless queues change before launch. All heavy commands use build-slot; no builds on the control host. No new dependencies, foundational choices, merge or deploy.

J83: make the five reported Rust test failures deterministic under VM load without masking defects. Plan (estimate: 3 hours, stop and report if exceeding 6 hours): 1. Branch test-stability-load from origin/dev in an isolated control-host worktree. Read relevant test and production paths; reproduce each named test/suite 20 times under bounded artificial CPU load on a development VM, retaining individual logs. 2. Fix proven root causes separately: executable publication/write descriptor lifecycle, readiness/event synchronization for restart and socket tests, private service/runtime environment for status tests. No guessed deadline increases or production contract relaxation. 3. Run the same 20-iteration load sweep after each fix. One atomic commit per test or shared root cause with issue reference and gate evidence. 4. Run scripts/verify fast under load, record static sizes/deltas and verbatim output, push only test-stability-load, report head SHA and before/after counts. Clean up owned processes and the single target directory. Initial queues: dev-02 has 1/3 slots occupied and 68G free; compute01 has 3/4 occupied and 115G free. Use dev-02 unless queues change before launch. All heavy commands use build-slot; no builds on the control host. No new dependencies, foundational choices, merge or deploy.

kayg commented

2026-11-02 00:09:17 +00:00

Author

Owner

started: branch test-stability-load, base ffebd0561 (origin/dev), worktree /home/kayg/Developer/Worktrees/infraternal-j83. plan and 3-hour estimate are in the issue body. initial VM queues checked; reproductions will use dev-02 and one target at /home/claude/worktrees/j83/target.

started: branch test-stability-load, base ffebd0561 (origin/dev), worktree /home/kayg/Developer/Worktrees/infraternal-j83. plan and 3-hour estimate are in the issue body. initial VM queues checked; reproductions will use dev-02 and one target at /home/claude/worktrees/j83/target.

kayg commented

2026-11-02 00:16:18 +00:00

Author

Owner

finding: status_exits_5_when_something_that_should_run_is_down fails 20/20 under the existing six-worker / three-core load when DBUS_SESSION_BUS_ADDRESS points at an unavailable Unix socket and XDG_RUNTIME_DIR is private/unavailable. Each failure is expected exit 5 versus actual 0 (Unknown is deliberately not confirmed-down). Evidence: dev-02 ~/.cache/codex-scratch/j83/status-private-bus/{1..20}.log. The original loaded sweep passed status 20/20 with the live host manager, proving host-state dependence. Fix will isolate environment/system-unit discovery and provide an immutable inactive service-manager fixture, retaining production deadlines and Unknown semantics.

kayg commented

2026-11-02 00:20:23 +00:00

Author

Owner

status fix validated under the same load: unavailable inherited bus before 20/20 failures, after 0/20. All 8 cli_envelope tests PASS; cargo fmt --all --check PASS; targeted cargo clippy --locked -p infraternal --test cli_envelope -- -D warnings PASS. Receipt: dev-02 ~/.cache/codex-scratch/j83/status-check2.log plus status-private-bus-after/*.log. Separate atomic status commit created. Restart exact test and full 18-test restart suite both passed their baseline load loops 20/20; no deadline changes are justified by those observations. Daemon sweep remains active; it will be waited through completion.

kayg commented

2026-11-02 00:20:50 +00:00

Author

Owner

finding: added a controlled real-kernel ETXTBSY regression to ssh_askpass. It holds an executable writer open until the initial exec future is polled, then releases it. With the current direct Command::output path it fails 20/20 while the CPU-load workers are active. This demonstrates the transient writer/exec ordering without a timing sleep. Receipt: dev-02 ~/.cache/codex-scratch/j83/askpass-red/{1..20}.log. The ordinary askpass load loop was 0/20 failures; these counts will be distinguished in the final report. Fix is test-only retry of ETXTBSY; other spawn/auth/protocol failures remain fatal.

kayg commented

2026-11-02 00:22:36 +00:00

Author

Owner

askpass fix validated: controlled writable-fd/exec race before 20/20 failures, after 0/20 under load. The complete ssh_askpass binary passes 20/20 loaded runs (5 passed, 1 pre-existing ignored controlling-terminal test per run). Unrelated NotFound spawn errors are preserved and existing private-pipe authentication/refusal assertions remain. cargo fmt --all --check and targeted clippy PASS. Receipts: dev-02 ~/.cache/codex-scratch/j83/askpass-check.log and askpass-green/*.log. Separate atomic ETXTBSY commit created; no production code or deadlines changed.

kayg commented

2026-11-02 00:25:50 +00:00

Author

Owner

baseline loaded sweep finished: restart exact 0/20 failures; full restart unit suite additionally 0/20; agent_daemon (the sole 24-abrupt-stop test) 0/20; ordinary ssh_askpass 0/20; live-host status 0/20. Controlled regressions separately demonstrated status unavailable-bus 20/20 and ETXTBSY held-writer 20/20, both now 0/20. Final head candidate 2691f69f13. scripts/verify fast is running on this head in dev-02 with six CPU workers and command affinity 0-2; a full 20-run after sweep is also running. No timeout failure reproduced yet, so no production or test deadline has been widened. Logs/one target remain /home/claude/worktrees/j83/target and ~/.cache/codex-scratch/j83/.

baseline loaded sweep finished: restart exact 0/20 failures; full restart unit suite additionally 0/20; agent_daemon (the sole 24-abrupt-stop test) 0/20; ordinary ssh_askpass 0/20; live-host status 0/20. Controlled regressions separately demonstrated status unavailable-bus 20/20 and ETXTBSY held-writer 20/20, both now 0/20. Final head candidate 2691f69f1305b4fd7353e9bc8cd31e7e4529ce7d. scripts/verify fast is running on this head in dev-02 with six CPU workers and command affinity 0-2; a full 20-run after sweep is also running. No timeout failure reproduced yet, so no production or test deadline has been widened. Logs/one target remain /home/claude/worktrees/j83/target and ~/.cache/codex-scratch/j83/.

kayg commented

2026-11-02 00:27:17 +00:00

Author

Owner

checkpoint: pushed only test-stability-load at 2691f69f13 to preserve the two validated atomic changes while the final loaded gate runs. This is not a completion claim. No merge or deploy. Baseline sweep is complete; final after sweep and scripts/verify fast are being blocked on in this same turn. Targets: dev-02 /home/claude/worktrees/j83/target (one directory); receipts ~/.cache/codex-scratch/j83/.

checkpoint: pushed only test-stability-load at 2691f69f1305b4fd7353e9bc8cd31e7e4529ce7d to preserve the two validated atomic changes while the final loaded gate runs. This is not a completion claim. No merge or deploy. Baseline sweep is complete; final after sweep and scripts/verify fast are being blocked on in this same turn. Targets: dev-02 /home/claude/worktrees/j83/target (one directory); receipts ~/.cache/codex-scratch/j83/.

kayg commented

2026-11-02 00:32:21 +00:00

Author

Owner

additional finding: the default-restart test executes a 410 MiB debug unit image and retains two sealed copies. A private systemd scope capped at 512 MiB (no swap), with the existing CPU load, kills the exact test with exit 137. This is a controlled resource-pressure reproduction, distinct from the reported timeout. A 20-run before loop is queued through build-slot. I stopped the superseded first fast-gate process tree by its owned PIDs before editing this fixture; its interrupted log is retained, not counted as a gate pass. Proposed narrow repair: reuse the existing stripped_test_image fixture for this exact test, preserving the real receiver/default-target path, one-open assertion and all production limits.

kayg commented

2026-11-02 00:35:30 +00:00

Author

Owner

checkpoint: pushed 2691f69f13. controlled host-bus failure: 20/20 before, 0/20 after; controlled open-writer ETXTBSY: 20/20 before, 0/20 after. ordinary CPU-load baseline: all four suites 0/20. after sweep is still running, daemon iteration 10 passed, other suites complete. restart memory probe at 512 MiB exits 137 because the 410 MiB debug image is retained twice; the 20-run confirmation is queued. preparing reuse of existing stripped-test-image fixture, keeping production and test deadlines unchanged. first gate stopped before tests to avoid compiling superseded test fixtures; will rerun full gate on final head. VM target retained for active job: /home/claude/worktrees/j83/target. logs: /home/claude/.cache/codex-scratch/j83.

checkpoint: pushed 2691f69f1305b4fd7353e9bc8cd31e7e4529ce7d. controlled host-bus failure: 20/20 before, 0/20 after; controlled open-writer ETXTBSY: 20/20 before, 0/20 after. ordinary CPU-load baseline: all four suites 0/20. after sweep is still running, daemon iteration 10 passed, other suites complete. restart memory probe at 512 MiB exits 137 because the 410 MiB debug image is retained twice; the 20-run confirmation is queued. preparing reuse of existing stripped-test-image fixture, keeping production and test deadlines unchanged. first gate stopped before tests to avoid compiling superseded test fixtures; will rerun full gate on final head. VM target retained for active job: /home/claude/worktrees/j83/target. logs: /home/claude/.cache/codex-scratch/j83.

kayg commented

2026-11-02 00:40:43 +00:00

Author

Owner

restart resource reproduction is confirmed: 20/20 OOM kills (exit 137) before, 0/20 failures after in private MemoryMax=512M/MemorySwapMax=0 scopes, concurrent with CPU load. Existing stripped executable cache reused through a child running the same exact test; cold cache creation passed too. Production admission and all deadlines unchanged. Third atomic commit pushed: 0b0d9690a0. fmt, targeted clippy, and 20 resource loops pass; full restart-suite repetitions and ordinary daemon after sweep still active. Final verify fast under six CPU burners is queued on dev02 (both VMs checked, compute01 full). This memory reproduction does not establish the cause of the historical restart timeout.

restart resource reproduction is confirmed: 20/20 OOM kills (exit 137) before, 0/20 failures after in private MemoryMax=512M/MemorySwapMax=0 scopes, concurrent with CPU load. Existing stripped executable cache reused through a child running the same exact test; cold cache creation passed too. Production admission and all deadlines unchanged. Third atomic commit pushed: 0b0d9690a047d390dea378bc6b7a7ec9ffba0aa8. fmt, targeted clippy, and 20 resource loops pass; full restart-suite repetitions and ordinary daemon after sweep still active. Final verify fast under six CPU burners is queued on dev02 (both VMs checked, compute01 full). This memory reproduction does not establish the cause of the historical restart timeout.

kayg commented

2026-11-02 00:44:36 +00:00

Author

Owner

owner direction: no raw daemon-timeout traceback survives; j82-gate.log retains only ssh_askpass ETXTBSY, and j82-gate2.log has no Rust failures. Stop investigating the unreproduced daemon timeout. Recorded evidence: agent_daemon::real_binary_preserves_input_and_output_across_24_abrupt_stops completed 20 baseline + 20 follow-up runs with 0/40 failures on dev02, six Python CPU burners, all test/load processes pinned to CPUs 0–2, four named test suites swept concurrently. Follow-up daemon runs also overlapped the restart resource/full-suite loops and initial gate work. Per-run logs and durations are in j83/before/agent_daemon-* and j83/after/agent_daemon-*; no daemon deadline or implementation changed. The two reported daemon labels identify this single test. Full final gate continues on head 0b0d9690a.

owner direction: no raw daemon-timeout traceback survives; j82-gate.log retains only ssh_askpass ETXTBSY, and j82-gate2.log has no Rust failures. Stop investigating the unreproduced daemon timeout. Recorded evidence: agent_daemon::real_binary_preserves_input_and_output_across_24_abrupt_stops completed 20 baseline + 20 follow-up runs with 0/40 failures on dev02, six Python CPU burners, all test/load processes pinned to CPUs 0–2, four named test suites swept concurrently. Follow-up daemon runs also overlapped the restart resource/full-suite loops and initial gate work. Per-run logs and durations are in j83/before/agent_daemon-* and j83/after/agent_daemon-*; no daemon deadline or implementation changed. The two reported daemon labels identify this single test. Full final gate continues on head 0b0d9690a.

kayg commented

2026-11-02 00:47:37 +00:00

Author

Owner

all J83 reproduction/repetition runs are finished. ordinary CPU-load sweeps: restart 0/20 before and 0/20 follow-up, agent_daemon 0/20 before and 0/20 after, askpass natural suite 0/20 before/after, status live-host 0/20 before/after. controlled cases: inactive-manager/absent user bus status 20/20 failures before -> 0/20 after; held-writer ETXTBSY regression 20/20 -> 0/20; default restart private 512 MiB scope 20/20 OOM -> 0/20 after stripping. Full restart suite: 20/20 passing before and after (same 18 tests). Gate remains active on final head. Precise affinity note: the sweep harness and burners were pinned to CPUs 0–2; systemd-created shim scopes retain the existing fixture CPU-weight propagation. Final gate runs with six burners over host CPUs 0–9 and CARGO_BUILD_JOBS=3.

kayg commented

2026-11-02 01:05:44 +00:00

Author

Owner

final-gate checkpoint on pushed head 0b0d9690a0: workspace test row PASS under six CPU burners (789 s), all named cases PASS inside that row; shim process-per-user, inode reuse, cargo-deny and architecture checks PASS. Static x86 build passed trust/linkage checks: 21,733,296 bytes, +289,472 against the gate retained baseline. arm64 static build active, then TS. Actual slice delta vs ffebd0561 will be measured after the gate. Reclaimed completed native test executables and inactive incremental cache from our target only; enough disk remains. Source/branch clean; no new commit needed. Small units correction: third commit message says 429 MiB; measured source image is 429,155,136 bytes (about 409 MiB), stripped 53,151,784 bytes. The exact byte counts in evidence are authoritative.

final-gate checkpoint on pushed head 0b0d9690a047d390dea378bc6b7a7ec9ffba0aa8: workspace test row PASS under six CPU burners (789 s), all named cases PASS inside that row; shim process-per-user, inode reuse, cargo-deny and architecture checks PASS. Static x86 build passed trust/linkage checks: 21,733,296 bytes, +289,472 against the gate retained baseline. arm64 static build active, then TS. Actual slice delta vs ffebd0561 will be measured after the gate. Reclaimed completed native test executables and inactive incremental cache from our target only; enough disk remains. Source/branch clean; no new commit needed. Small units correction: third commit message says 429 MiB; measured source image is 429,155,136 bytes (about 409 MiB), stripped 53,151,784 bytes. The exact byte counts in evidence are authoritative.

kayg commented

2026-11-02 01:11:00 +00:00

Author

Owner

J83 finish. Branch test-stability-load, head 0b0d9690a047d390dea378bc6b7a7ec9ffba0aa8, base ffebd0561a010690583481abc4d583ecaca4542e. Three atomic commits, pushed. No production code changed.

Status checks now use a cleared environment, private home/config/runtime/system-unit roots, and an immutable inactive-manager fixture. An unavailable real user manager means Unknown, so the old assertion depended on host state.
The askpass test retries only an ETXTBSY spawn error. A real-kernel regression retains a writer until the first exec poll, then closes it; an unrelated NotFound error remains unchanged. Authentication failures and running-helper outcomes never retry.
The default restart test reuses the existing stripped executable fixture in a child running the same exact test. Real default-path admission and the exact-one-open assertion remain. The measured source image was 429,155,136 bytes, stripped 53,151,784 bytes. No production deadline was extended.

Failures out of 20 under load:

Case	Before	After
default restart, ordinary CPU load	0/20	0/20
default restart, private 512 MiB cap plus CPU load	20/20 (OOM, 137)	0/20
agent_daemon / 24 abrupt stops (one test)	0/20	0/20
ssh_askpass, ordinary CPU load	0/20	0/20
ssh_askpass, controlled retained executable writer	20/20	0/20
status exit 5, ordinary live-host sweep	0/20	0/20
status exit 5, unavailable private user bus	20/20	0/20
full 18-test restart suite	0/20	0/20

Ordinary sweeps ran on dev02 with six CPU burners, the harness pinned to CPUs 0–2, and four suites swept concurrently. Existing managed-shim CPU-weight propagation remained. Restart memory scopes used MemoryMax=512M, MemorySwapMax=0, taskset 0–2 and nice 10, overlapping CPU load; the after loop also owned six burners. The first after run includes cold stripped-cache creation. Raw per-run logs, counts, commands and conditions are attached.

The historical restart timeout did not reproduce; the memory-pressure finding is an OOM reproduction, not proof of that timeout's cause. The historical daemon timeout did not reproduce in 40 loaded runs, and the daemon passed the final gate as well. The owner confirmed no raw daemon traceback survives and directed J83 to stop investigating it. No daemon change was made.

Final scripts/verify fast ran on this head through build-slot on dev02, CARGO_BUILD_JOBS=3, nice 10, six active CPU burners over CPUs 0–9. Complete output attached; summaries verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

 39 pass
 0 fail
 59970 expect() calls
Ran 39 tests across 10 files. [2.40s]

VERIFY_EXIT=0
RUN_EXIT=0

Static slice measurement against the actual base revision, using the same target and toolchain: x86_64 21,733,296 bytes, aarch64 19,857,608 bytes, +0 bytes each, byte-identical SHA-256. The gate's retained historical baselines report +289,472 and +228,368 bytes respectively; those are already present at the base. Native x86 runtime probes report the gate's expected SKIP on the arm64 host.

Cleanup complete: all runs finished, final load workers exited, /home/claude/worktrees/j83/target deleted, source clean, VM checkout restored to this head. Evidence retained outside the checkout at /home/claude/.cache/codex-scratch/j83 and on the control host at /home/kayg/.cache/codex-scratch/j83/evidence. No blocker or owner decision remains for this slice. Issue left open for review.

J83 finish. Branch `test-stability-load`, head `0b0d9690a047d390dea378bc6b7a7ec9ffba0aa8`, base `ffebd0561a010690583481abc4d583ecaca4542e`. Three atomic commits, pushed. No production code changed. - Status checks now use a cleared environment, private home/config/runtime/system-unit roots, and an immutable inactive-manager fixture. An unavailable real user manager means Unknown, so the old assertion depended on host state. - The askpass test retries only an ETXTBSY spawn error. A real-kernel regression retains a writer until the first exec poll, then closes it; an unrelated NotFound error remains unchanged. Authentication failures and running-helper outcomes never retry. - The default restart test reuses the existing stripped executable fixture in a child running the same exact test. Real default-path admission and the exact-one-open assertion remain. The measured source image was 429,155,136 bytes, stripped 53,151,784 bytes. No production deadline was extended. Failures out of 20 under load: | Case | Before | After | | --- | ---: | ---: | | default restart, ordinary CPU load | 0/20 | 0/20 | | default restart, private 512 MiB cap plus CPU load | 20/20 (OOM, 137) | 0/20 | | agent_daemon / 24 abrupt stops (one test) | 0/20 | 0/20 | | ssh_askpass, ordinary CPU load | 0/20 | 0/20 | | ssh_askpass, controlled retained executable writer | 20/20 | 0/20 | | status exit 5, ordinary live-host sweep | 0/20 | 0/20 | | status exit 5, unavailable private user bus | 20/20 | 0/20 | | full 18-test restart suite | 0/20 | 0/20 | Ordinary sweeps ran on dev02 with six CPU burners, the harness pinned to CPUs 0–2, and four suites swept concurrently. Existing managed-shim CPU-weight propagation remained. Restart memory scopes used MemoryMax=512M, MemorySwapMax=0, taskset 0–2 and nice 10, overlapping CPU load; the after loop also owned six burners. The first after run includes cold stripped-cache creation. Raw per-run logs, counts, commands and conditions are attached. The historical restart timeout did not reproduce; the memory-pressure finding is an OOM reproduction, not proof of that timeout's cause. The historical daemon timeout did not reproduce in 40 loaded runs, and the daemon passed the final gate as well. The owner confirmed no raw daemon traceback survives and directed J83 to stop investigating it. No daemon change was made. Final `scripts/verify fast` ran on this head through build-slot on dev02, CARGO_BUILD_JOBS=3, nice 10, six active CPU burners over CPUs 0–9. Complete output attached; summaries verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ``` ```text 39 pass 0 fail 59970 expect() calls Ran 39 tests across 10 files. [2.40s] VERIFY_EXIT=0 RUN_EXIT=0 ``` Static slice measurement against the actual base revision, using the same target and toolchain: x86_64 21,733,296 bytes, aarch64 19,857,608 bytes, **+0 bytes each, byte-identical SHA-256**. The gate's retained historical baselines report +289,472 and +228,368 bytes respectively; those are already present at the base. Native x86 runtime probes report the gate's expected SKIP on the arm64 host. Cleanup complete: all runs finished, final load workers exited, `/home/claude/worktrees/j83/target` deleted, source clean, VM checkout restored to this head. Evidence retained outside the checkout at `/home/claude/.cache/codex-scratch/j83` and on the control host at `/home/kayg/.cache/codex-scratch/j83/evidence`. No blocker or owner decision remains for this slice. Issue left open for review.

verify-fast-load.log

513 KiB

j83-evidence.tar.gz

127 KiB

kayg referenced this issue

2026-11-02 08:37:16 +00:00

[J96] Restore green Rust test row on dev #307

kayg referenced this issue

2026-11-02 11:54:29 +00:00

[J96] Restore green Rust test row on dev #307

kayg referenced this issue

2026-11-02 12:43:30 +00:00

[J96] Restore green Rust test row on dev #307

kayg commented

2026-11-04 17:04:43 +00:00

Author

Owner

J210 owner-authorized housekeeping: closing this completed scope because its work is merged on dev in 3f31e8cd3. Verified the merge commit is an ancestor of current origin/dev. The implementation reports and evidence remain in this issue.

J210 owner-authorized housekeeping: closing this completed scope because its work is merged on dev in [3f31e8cd3](https://git.kayg.org/kayg/infraternal/commit/3f31e8cd3b12ac3770db746e148ccccdd2a04175). Verified the merge commit is an ancestor of current origin/dev. The implementation reports and evidence remain in this issue.

kayg closed this issue

2026-11-04 17:04:44 +00:00

kayg referenced this issue

2026-11-04 17:07:06 +00:00

J210: overnight housekeeping, 2026-11-04 #350

Rows
Columns

J83: stabilize Rust tests under VM load #299