J376: diagnose remaining install-suite failures after #404

kayg commented

2026-11-08 16:50:06 +00:00

Owner

J376 starting on branch fix-install-suite-red, base b31c2da2dc (current origin/dev). Investigate whether remaining compute01 failures affect shipped Add host or fixture isolation. Plan: reproduce unmodified suite with its required helpers, compare isolated environment, add deterministic red coverage, fix only evidenced cause, then three consecutive compute01 suites, verify quick and verify fast. Estimate 90 minutes excluding build queues/cold builds. Production read-only; no merge/deploy or other jobs processes/scopes touched. Evidence under ~/j376-evidence on compute01. One build slot, nix develop, one target with worktree target path preserved (J155). Related #404; report there immediately if shipped code is defective.

J376 starting on branch fix-install-suite-red, base b31c2da2dcebd2efcb8ae371dba58073905d31e7 (current origin/dev). Investigate whether remaining compute01 failures affect shipped Add host or fixture isolation. Plan: reproduce unmodified suite with its required helpers, compare isolated environment, add deterministic red coverage, fix only evidenced cause, then three consecutive compute01 suites, verify quick and verify fast. Estimate 90 minutes excluding build queues/cold builds. Production read-only; no merge/deploy or other jobs processes/scopes touched. Evidence under ~/j376-evidence on compute01. One build slot, nix develop, one target with worktree target path preserved (J155). Related #404; report there immediately if shipped code is defective.

kayg commented

2026-11-08 16:52:19 +00:00

Author

Owner

Initial evidence: J372 fast log is pre-#404: it reports 107 active tests (current dev has the added concurrent-fixture regression) and install.rs:2398 is its Done assertion, whereas on b31c2da2d that source location is the initial stage call (the added test moved the recovery assertions). The standalone log instead fails at stage with Upload/127, consistent with missing examples/recovery-helper. #404 itself records this exact missing-helper prerequisite during J365 baseline setup. No new production defect established. Fresh unmodified b31c2da2d test build queued on compute01, artifact ~/j376-evidence/baseline.log; isolated fixture roots already unique. Further proof will distinguish these historical failures from current dev.

Initial evidence: J372 fast log is pre-#404: it reports 107 active tests (current dev has the added concurrent-fixture regression) and install.rs:2398 is its Done assertion, whereas on b31c2da2d that source location is the initial stage call (the added test moved the recovery assertions). The standalone log instead fails at stage with Upload/127, consistent with missing examples/recovery-helper. #404 itself records this exact missing-helper prerequisite during J365 baseline setup. No new production defect established. Fresh unmodified b31c2da2d test build queued on compute01, artifact ~/j376-evidence/baseline.log; isolated fixture roots already unique. Further proof will distinguish these historical failures from current dev.

kayg commented

2026-11-08 16:54:09 +00:00

Author

Owner

Historical-log attribution is retained at compute01 ~/j376-evidence/{j372-fast.log,j372-t-dev.log,pre-404-lines.txt,current-lines.txt}. The fast run merge base is 7c8bd2221 (printed in its header); #404 merged after that base. Its 9 exit-73 failures therefore do not prove a regression after 77b1255cb. The current standalone upload failure requires a separate clean-target reproduction. Moved focused reproduction to dev-02 after cancelling only my queued compute01 launcher PID 1038118; baseline-no-helper.log on dev-02. No code changes yet.

Historical-log attribution is retained at compute01 ~/j376-evidence/{j372-fast.log,j372-t-dev.log,pre-404-lines.txt,current-lines.txt}. The fast run merge base is 7c8bd2221 (printed in its header); #404 merged after that base. Its 9 exit-73 failures therefore do not prove a regression after 77b1255cb. The current standalone upload failure requires a separate clean-target reproduction. Moved focused reproduction to dev-02 after cancelling only my queued compute01 launcher PID 1038118; baseline-no-helper.log on dev-02. No code changes yet.

kayg commented

2026-11-08 16:56:57 +00:00

Author

Owner

Proposed narrow repair if the clean-target red confirms exit 127: keep the shipping runtime guard unchanged and have common::runtime_helper prepare its existing example once when a focused cargo test did not build it. Preserve the explicit INFRATERNAL_TEST_RUNTIME_HELPER override and profile-local target path; do not add retry, sleeps, serialization or shell substitutions. Existing recovery tests provide the red/green behavioral check. Missing helper is a test prerequisite, not host state. Compute01 now owns the only active J376 build slot; dev-02 queue was cancelled by its exact launcher PID 34677 before any build.

kayg commented

2026-11-08 16:58:38 +00:00

Author

Owner

Clean-target RED confirmed on unmodified current dev b31c2da2d, compute01: cargo test --locked -p infraternal-plugin-ssh --test install known_failed_setup_finishes_without_a_second_confirmation -- --exact fails at install.rs:2395 initial stage, Upload exit 127, 0 passed / 1 failed, BASELINE_EXIT=101. Artifact ~/j376-evidence/baseline-no-helper.log. target/debug/examples/recovery-helper is absent. This matches J372 standalone failure exactly, independent of the historical pre-#404 gate failures. Building only the missing existing helper next, then repeating unmodified test with an isolated HOME/runtime to prove attribution before repairing test support.

Clean-target RED confirmed on unmodified current dev b31c2da2d, compute01: cargo test --locked -p infraternal-plugin-ssh --test install known_failed_setup_finishes_without_a_second_confirmation -- --exact fails at install.rs:2395 initial stage, Upload exit 127, 0 passed / 1 failed, BASELINE_EXIT=101. Artifact ~/j376-evidence/baseline-no-helper.log. target/debug/examples/recovery-helper is absent. This matches J372 standalone failure exactly, independent of the historical pre-#404 gate failures. Building only the missing existing helper next, then repeating unmodified test with an isolated HOME/runtime to prove attribution before repairing test support.

kayg commented

2026-11-08 17:01:08 +00:00

Author

Owner

Control proof: on unchanged b31c2da2d, building only recovery-helper makes known_failed_setup_finishes_without_a_second_confirmation pass with freshly-created HOME and XDG_RUNTIME_DIR (ISOLATED_EXIT=0). compute01 ~/j376-evidence/baseline-with-helper.log and baseline-helper-build.log (HELPER_BUILD_EXIT=0). Full unchanged suite passed all nine historical failures but found a distinct current failure: checked_staging_survives_a_buffering_payload_reader_and_reuses_the_cache, rollback at install.rs:3022, SetupRunning/73. 107 passed / 1 failed / 2 ignored, PROOF_EXIT=101. Tracing this before claiming isolation or changing product locking. Proposed helper preparation repair remains local and has not changed this baseline run.

Control proof: on unchanged b31c2da2d, building only recovery-helper makes known_failed_setup_finishes_without_a_second_confirmation pass with freshly-created HOME and XDG_RUNTIME_DIR (ISOLATED_EXIT=0). compute01 ~/j376-evidence/baseline-with-helper.log and baseline-helper-build.log (HELPER_BUILD_EXIT=0). Full unchanged suite passed all nine historical failures but found a distinct current failure: checked_staging_survives_a_buffering_payload_reader_and_reuses_the_cache, rollback at install.rs:3022, SetupRunning/73. 107 passed / 1 failed / 2 ignored, PROOF_EXIT=101. Tracing this before claiming isolation or changing product locking. Proposed helper preparation repair remains local and has not changed this baseline run.

kayg commented

2026-11-08 17:05:34 +00:00

Author

Owner

Diagnostic update: the extra checked-reader rollback failure did not reproduce in 20 focused or 10 full diagnostic suites (trace-sweep.log, trace-suite-summary.log). A subsequent attempt to add variables to already-quoted generated programs corrupted nested shell quoting (exit 2), so lean-trace-* runs are INVALID and excluded. Moving the tiny refusal diagnostics into the original shell templates before quoting. No product fix selected from the additional one-off exit-73 failure.

kayg referenced this issue

2026-11-08 17:07:08 +00:00

Flaky: real_binary_rootless_cleanup_survives_sender_disconnect_before_readiness (~1/15 on dev) #404

kayg commented

2026-11-08 17:09:08 +00:00

Author

Owner

Completed diagnostics: 20 focused checked-reader tests, 10 initial full suites, and 20 correctly-template-instrumented full suites pass; the latter each report 108 passed / 0 failed / 2 ignored. Artifacts trace-sweep.log, trace-suite-summary.log, template-trace-summary.log and individual suite logs under compute01 ~/j376-evidence. No repeat or identified cause for the single additional checked-reader rollback/73; retain as an unresolved observation, not evidence of a shipped regression. All temporary shell/test diagnostics are restored. Now testing the narrow helper-preparation repair after deliberately removing only J376 target/debug/examples/recovery-helper. Existing initial-stage recovery test is the unchanged red/green oracle.

kayg commented

2026-11-08 17:09:57 +00:00

Author

Owner

GREEN after the same helper executable was removed: the unchanged focused recovery test passes with the test-support repair (1 passed / 0 failed, GREEN_EXIT=0). Artifact compute01 ~/j376-evidence/green-no-helper.log. Only tests/common/mod.rs changes: missing default helper is built once with the invoking test profile using locked/offline Cargo; explicit override remains supported. No shipping code or assertions changed. Starting requested three consecutive full suites on compute01. One target only: ~/worktrees/j376/target (physical directory, so J155 external-target symlink case does not arise). Dev-02 never built a J376 target; its temporary worktree removed.

kayg commented

2026-11-08 17:11:24 +00:00

Author

Owner

Requested compute01 sweep passes without retries, fixed sleeps, diagnostic instrumentation or assertion changes. Verbatim ~/j376-evidence/install-three.log:\nINSTALL_1=0\ntest result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.67s\nINSTALL_2=0\ntest result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.60s\nINSTALL_3=0\ntest result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.28s\nTHREE_CONSECUTIVE=PASS\nSWEEP_EXIT=0\nIndividual full logs install-{1,2,3}.log. verify quick running on compute01 (shorter queue); evidence quick.log. No runtime product changes, so shipping binary delta from this repair is zero by construction.

kayg commented

2026-11-08 17:15:02 +00:00

Author

Owner

J376 checkpoint: pushed WIP head 3f0867e65 on fix-install-suite-red, preserving the repair while verify quick builds. Code is confined to SSH test support; final commit message will be finalized after gates. One compute01 target: ~/worktrees/j376/target; no other J376 builds or queues. No merge or deployment.

Correctly formatted verbatim three-suite output (the earlier comment accidentally retained escaped line breaks):

INSTALL_1=0

test result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.67s

INSTALL_2=0

test result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.60s

INSTALL_3=0

test result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.28s

THREE_CONSECUTIVE=PASS
SWEEP_EXIT=0

Evidence: compute01 ~/j376-evidence/install-{1,2,3}.log, install-three.log; local /home/kayg/.cache/j376-evidence. All three runs used the original assertions and default parallelism, without retries. verify quick is in progress at ~/j376-evidence/quick.log.

J376 checkpoint: pushed WIP head 3f0867e65 on fix-install-suite-red, preserving the repair while verify quick builds. Code is confined to SSH test support; final commit message will be finalized after gates. One compute01 target: ~/worktrees/j376/target; no other J376 builds or queues. No merge or deployment. Correctly formatted verbatim three-suite output (the earlier comment accidentally retained escaped line breaks): ```text INSTALL_1=0 test result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.67s INSTALL_2=0 test result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.60s INSTALL_3=0 test result: ok. 108 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.28s THREE_CONSECUTIVE=PASS SWEEP_EXIT=0 ``` Evidence: compute01 ~/j376-evidence/install-{1,2,3}.log, install-three.log; local /home/kayg/.cache/j376-evidence. All three runs used the original assertions and default parallelism, without retries. verify quick is in progress at ~/j376-evidence/quick.log.

kayg commented

2026-11-08 17:18:21 +00:00

Author

Owner

Integrated newly advanced origin/dev e22e1b76d (only an unrelated palette test changed); pushed WIP head edbe2e05de. verify quick re-run on that base passes, QUICK_EXIT=0, 124 passed / 0 failed / 6 ignored. Artifacts compute01 ~/j376-evidence/quick-final.log and quick.log. Requested three-suite code is identical after rebase. Final verify fast launched on compute01 through the same single build slot and target; full log fast.log, launcher PID fast.pid. No merge or deployment.

Integrated newly advanced origin/dev e22e1b76d (only an unrelated palette test changed); pushed WIP head edbe2e05defc86ecbcae31630940a0e65666c609. verify quick re-run on that base passes, QUICK_EXIT=0, 124 passed / 0 failed / 6 ignored. Artifacts compute01 ~/j376-evidence/quick-final.log and quick.log. Requested three-suite code is identical after rebase. Final verify fast launched on compute01 through the same single build slot and target; full log fast.log, launcher PID fast.pid. No merge or deployment.

kayg commented

2026-11-08 17:44:22 +00:00

Author

Owner

J376 gate checkpoint at unchanged pushed head edbe2e05de (base e22e1b76d). Both quick runs pass; three requested standalone install suites pass; the full fast gate install suite also passed 108/0/2 in 6.65s. CLI feature checks, workspace clippy, helper build and fixture-path checks passed. Workspace tests are still active: server unit suite currently 334/1018 completed with no failure reported. Artifact compute01 ~/j376-evidence/fast.log; exact launcher PID 2166194, one slot, own physical target ~/worktrees/j376/target. Continuing in the foreground; no new code, merge or deployment. The single unexplained baseline checked-reader rollback/73 remains recorded above and has not repeated.

J376 gate checkpoint at unchanged pushed head edbe2e05defc86ecbcae31630940a0e65666c609 (base e22e1b76d). Both quick runs pass; three requested standalone install suites pass; the full fast gate install suite also passed 108/0/2 in 6.65s. CLI feature checks, workspace clippy, helper build and fixture-path checks passed. Workspace tests are still active: server unit suite currently 334/1018 completed with no failure reported. Artifact compute01 ~/j376-evidence/fast.log; exact launcher PID 2166194, one slot, own physical target ~/worktrees/j376/target. Continuing in the foreground; no new code, merge or deployment. The single unexplained baseline checked-reader rollback/73 remains recorded above and has not repeated.

kayg commented

2026-11-08 18:19:28 +00:00

Author

Owner

J376 gate checkpoint: finalized the atomic commit message at 046c88931c. Its tree 073d1d3c8f is identical to the running gate head edbe2e05de (git diff --exit-code is clean). No code changed during verification.

Fast has passed workspace tests, the install suite (108/0/2), all five privileged SSH tests, dependency/architecture/documentation checks, and x86 static trust/key-memory probes. The aarch64 static build is still actively compiling; frontend checks follow. No gate failure reported yet. One slot and one physical target, /home/claude/worktrees/j376/target; launcher PID 2166194. Logs retained at /home/claude/j376-evidence/fast.log and being copied to /home/claude/.cache/codex-scratch/j376/evidence/. No merge or deployment. Continuing foreground polling until exit.

J376 gate checkpoint: finalized the atomic commit message at 046c88931c65dacaf01e538793ae30170c533d03. Its tree 073d1d3c8f7e68e6f7be9751dae257c3adb1fcba is identical to the running gate head edbe2e05defc86ecbcae31630940a0e65666c609 (git diff --exit-code is clean). No code changed during verification. Fast has passed workspace tests, the install suite (108/0/2), all five privileged SSH tests, dependency/architecture/documentation checks, and x86 static trust/key-memory probes. The aarch64 static build is still actively compiling; frontend checks follow. No gate failure reported yet. One slot and one physical target, /home/claude/worktrees/j376/target; launcher PID 2166194. Logs retained at /home/claude/j376-evidence/fast.log and being copied to /home/claude/.cache/codex-scratch/j376/evidence/. No merge or deployment. Continuing foreground polling until exit.

kayg commented

2026-11-08 18:23:52 +00:00

Author

Owner

J376 finished: branch fix-install-suite-red, head 046c88931c, based on e22e1b76d6. One atomic test-support change, pushed; no merge or deployment.

Confirmed root cause: focused cargo test --test install does not build examples, leaving recovery-helper absent. The fixture calls it and reports Upload/127. The unchanged baseline fails without it and passes after building only that example, including with fresh HOME/runtime directories. Test support now prepares the existing helper once when absent. No shipping code, assertions, sleeps, retries, or dependencies changed. This confirmed defect does not affect production Add host.

J372's nine lock failures belong to the pre-#404 log (old merge base, 107 active tests and old source-line mapping), rather than establishing a post-77b1255cb regression. One distinct baseline buffering-reader rollback SetupRunning/73 was observed after the helper existed; it remains unexplained and did not recur across the recorded diagnostics, validation suites, or fast. Its production impact is not established; this fix does not claim to resolve it.

Red/green: baseline-no-helper.log has Upload/127 and BASELINE_EXIT=101. Building only the helper makes unchanged baseline pass; green-no-helper.log has 1 passed / 0 failed and GREEN_EXIT=0 after deleting the owned helper. Three consecutive clean compute01 install suites passed 108/0/2 each. Quick passed (124/0/6, QUICK_EXIT=0), including after rebase.

All final fast rows pass; #421 did not reproduce. Fast ran at edbe2e05de; the final head only amends the commit message. Both have tree 073d1d3c8f, and git diff --exit-code is clean.

Verbatim fast summary and frontend result excerpts (complete output attached as fast.log):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

 655 pass
 0 fail
 24808 expect() calls
Ran 655 tests across 102 files. [57.80s]

 104 pass
 0 fail
 434 expect() calls
Ran 104 tests across 17 files. [300.00ms]

 17 pass
 0 fail
 1794 expect() calls
Ran 17 tests across 4 files. [618.00ms]

 50 pass
 0 fail
 77253 expect() calls
Ran 50 tests across 13 files. [15.41s]

PT_INTERP: absent; size: 28171984 bytes; baseline 22418256; delta +5753728 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 25305312 bytes; baseline 20453280; delta +4852032 bytes (report only, D243 amended 2026-11-01)

FAST_EXIT=0

The static deltas above compare the gate's historical baselines, not this test-only slice; no product source changed. Evidence is retained on compute01 at /home/claude/.cache/codex-scratch/j376/evidence/ (original /home/claude/j376-evidence/ paths also retained), and locally at /home/kayg/.cache/codex-scratch/j376/evidence/. The issue-attach helper is unavailable, so selected receipts are attached through the Forgejo API. J376's sole build target /home/claude/worktrees/j376/target has been removed after gate completion. No J376 build or queue remains. Issue left open for review.

J376 finished: branch fix-install-suite-red, head 046c88931c65dacaf01e538793ae30170c533d03, based on e22e1b76d60992a75c2c80ba9ca8887a258e7ca5. One atomic test-support change, pushed; no merge or deployment. Confirmed root cause: focused cargo test --test install does not build examples, leaving recovery-helper absent. The fixture calls it and reports Upload/127. The unchanged baseline fails without it and passes after building only that example, including with fresh HOME/runtime directories. Test support now prepares the existing helper once when absent. No shipping code, assertions, sleeps, retries, or dependencies changed. This confirmed defect does not affect production Add host. J372's nine lock failures belong to the pre-#404 log (old merge base, 107 active tests and old source-line mapping), rather than establishing a post-77b1255cb regression. One distinct baseline buffering-reader rollback SetupRunning/73 was observed after the helper existed; it remains unexplained and did not recur across the recorded diagnostics, validation suites, or fast. Its production impact is not established; this fix does not claim to resolve it. Red/green: baseline-no-helper.log has Upload/127 and BASELINE_EXIT=101. Building only the helper makes unchanged baseline pass; green-no-helper.log has 1 passed / 0 failed and GREEN_EXIT=0 after deleting the owned helper. Three consecutive clean compute01 install suites passed 108/0/2 each. Quick passed (124/0/6, QUICK_EXIT=0), including after rebase. All final fast rows pass; #421 did not reproduce. Fast ran at edbe2e05defc86ecbcae31630940a0e65666c609; the final head only amends the commit message. Both have tree 073d1d3c8f7e68e6f7be9751dae257c3adb1fcba, and git diff --exit-code is clean. Verbatim fast summary and frontend result excerpts (complete output attached as fast.log): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ 655 pass 0 fail 24808 expect() calls Ran 655 tests across 102 files. [57.80s] 104 pass 0 fail 434 expect() calls Ran 104 tests across 17 files. [300.00ms] 17 pass 0 fail 1794 expect() calls Ran 17 tests across 4 files. [618.00ms] 50 pass 0 fail 77253 expect() calls Ran 50 tests across 13 files. [15.41s] PT_INTERP: absent; size: 28171984 bytes; baseline 22418256; delta +5753728 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 25305312 bytes; baseline 20453280; delta +4852032 bytes (report only, D243 amended 2026-11-01) FAST_EXIT=0 ``` The static deltas above compare the gate's historical baselines, not this test-only slice; no product source changed. Evidence is retained on compute01 at /home/claude/.cache/codex-scratch/j376/evidence/ (original /home/claude/j376-evidence/ paths also retained), and locally at /home/kayg/.cache/codex-scratch/j376/evidence/. The issue-attach helper is unavailable, so selected receipts are attached through the Forgejo API. J376's sole build target /home/claude/worktrees/j376/target has been removed after gate completion. No J376 build or queue remains. Issue left open for review.

fast.log

923 KiB

fast-summary.txt

1.8 KiB

install-three.log

366 B

quick-final.log

22 KiB

baseline-no-helper.log

9.5 KiB

green-no-helper.log

1.8 KiB