J149: verify quick pre-check and shared sccache on build VMs

kayg commented

2026-11-03 16:38:42 +00:00

Owner

Job J149. Problem: scripts/verify fast takes 20-40 min and every job builds a 25-40 GB target from scratch; 6 build slots, 4-5 queued, many jobs re-run the gate only to find a fmt/clippy error.

Plan (shortest safe path, no framework), estimate ~3 h:

scripts/verify quick: fmt, clippy + unit tests of crates changed vs merge-base with origin/dev, tracer typecheck when TS changed. Never replaces fast. Document in docs/PROCESS.md; BRIEF-COMMON lines posted here for the orchestrator.
Opt-in shared sccache on the build VMs (local disk cache under ~claude/.cache/sccache, size cap), enabled by an env var set by build-slot / dev shell on the VMs only. CI and release unchanged. Prove identical SHA-256 of the static release binary with and without cache; measure cold vs second-job build, wall + CPU. Dev-shell-only flake change if needed. Nothing enters the shipped binary.

Branch: j149-verify-quick, base 7e1cb858f (origin/dev).

Job J149. Problem: `scripts/verify fast` takes 20-40 min and every job builds a 25-40 GB target from scratch; 6 build slots, 4-5 queued, many jobs re-run the gate only to find a fmt/clippy error. Plan (shortest safe path, no framework), estimate ~3 h: 1. `scripts/verify quick`: fmt, clippy + unit tests of crates changed vs merge-base with origin/dev, tracer typecheck when TS changed. Never replaces `fast`. Document in docs/PROCESS.md; BRIEF-COMMON lines posted here for the orchestrator. 2. Opt-in shared sccache on the build VMs (local disk cache under ~claude/.cache/sccache, size cap), enabled by an env var set by build-slot / dev shell on the VMs only. CI and release unchanged. Prove identical SHA-256 of the static release binary with and without cache; measure cold vs second-job build, wall + CPU. Dev-shell-only flake change if needed. Nothing enters the shipped binary. Branch: `j149-verify-quick`, base 7e1cb858f (origin/dev).

kayg commented

2026-11-03 17:15:52 +00:00

Author

Owner

Progress 1: scripts/verify quick (branch j149-verify-quick, commits pushed; base 7e1cb858f)

Run on dev-02 (aarch64, load ~15-40 from other jobs, warm shared target), through build-slot, nix develop -c scripts/verify quick:

scenario	result	wall
S1 only scripts/docs changed	fmt only, pass	2 s
S2 `infraternal-status` changed	fmt, clippy (workspace), unit tests, pass	143 s
S3 deliberate fmt error	stops at fmt, exit 1	1 s
S4 `apps/tracer` TS changed	fmt + tracer typecheck, pass	11 s
S5 `infraternal` (bin) + `infraternal-status` changed	pass	576 s (the `infraternal` crate's unit tests alone: 570 s, 118 s of it test run)

Finding (evidence): my first version ran cargo clippy -p <changed>. For infraternal-status it failed inside an UNTOUCHED crate:
crates/infraternal-shim/src/child.rs:114: unnecessary_wraps (-D clippy::pedantic). Cargo unifies features over the selected packages only, so a narrower set makes a feature-dependent lint fire that the gate (with workspace features) never reports. Fix: clippy always runs --workspace --all-targets --locked -D warnings (the gate's own flags, so the gate then reuses those artifacts in a shared target); only the unit tests (--lib --bins) are limited to changed packages. This is a deviation from "clippy on the changed crates", chosen to avoid false failures. Cold workspace clippy is the main cost of a first quick run.

Not covered by quick (it says so when it passes): dependents of a changed crate, integration tests, static builds, cargo-deny, repository checks. Documented in docs/PROCESS.md (section 2, Definition of Done item 2).

Infra finding (for the orchestrator): compute01's build-slot queue sat stuck for 33+ min with slots 3 and 4 free: the head ticket is bash /root/j145-run.sh (pid 535918, started 17:07 CET), a root process that loaded the OLD build-slot before the 17:27 fix, so it still reads /root/infraternal-ops/slots (default 2) and waits for slot 1 or 2 while everything queues behind it. A running bash reads its script incrementally, so editing build-slot while waiters exist is unsafe as well. It resolves itself when slot 1 or 2 frees; killing that waiter (as root) frees the queue now.

Next: sccache measurement (static release build, no cache vs cold cache vs warm cache from a second worktree; SHA-256 of the binary; wall + CPU).

**Progress 1: `scripts/verify quick` (branch `j149-verify-quick`, commits pushed; base 7e1cb858f)** Run on dev-02 (aarch64, load ~15-40 from other jobs, warm shared target), through build-slot, `nix develop -c scripts/verify quick`: | scenario | result | wall | |---|---|---| | S1 only scripts/docs changed | fmt only, pass | 2 s | | S2 `infraternal-status` changed | fmt, clippy (workspace), unit tests, pass | 143 s | | S3 deliberate fmt error | stops at fmt, exit 1 | 1 s | | S4 `apps/tracer` TS changed | fmt + tracer typecheck, pass | 11 s | | S5 `infraternal` (bin) + `infraternal-status` changed | pass | 576 s (the `infraternal` crate's unit tests alone: 570 s, 118 s of it test run) | **Finding (evidence):** my first version ran `cargo clippy -p <changed>`. For `infraternal-status` it failed inside an UNTOUCHED crate: `crates/infraternal-shim/src/child.rs:114: unnecessary_wraps` (`-D clippy::pedantic`). Cargo unifies features over the selected packages only, so a narrower set makes a feature-dependent lint fire that the gate (with workspace features) never reports. Fix: clippy always runs `--workspace --all-targets --locked -D warnings` (the gate's own flags, so the gate then reuses those artifacts in a shared target); only the unit tests (`--lib --bins`) are limited to changed packages. This is a deviation from "clippy on the changed crates", chosen to avoid false failures. Cold workspace clippy is the main cost of a first quick run. Not covered by quick (it says so when it passes): dependents of a changed crate, integration tests, static builds, cargo-deny, repository checks. Documented in docs/PROCESS.md (section 2, Definition of Done item 2). **Infra finding (for the orchestrator):** compute01's build-slot queue sat stuck for 33+ min with slots 3 and 4 free: the head ticket is `bash /root/j145-run.sh` (pid 535918, started 17:07 CET), a root process that loaded the OLD build-slot before the 17:27 fix, so it still reads `/root/infraternal-ops/slots` (default 2) and waits for slot 1 or 2 while everything queues behind it. A running bash reads its script incrementally, so editing build-slot while waiters exist is unsafe as well. It resolves itself when slot 1 or 2 frees; killing that waiter (as root) frees the queue now. Next: sccache measurement (static release build, no cache vs cold cache vs warm cache from a second worktree; SHA-256 of the binary; wall + CPU).

kayg commented

2026-11-03 18:56:49 +00:00

Author

Owner

Progress 2: sccache measurements (dev-02, aarch64, 10 cores, shared with ~9 other jobs; loadavg before each run in the receipts). Branch j149-verify-quick, head 056e80153 (2 commits on e81eb5bb4).

Setup: INFRATERNAL_SCCACHE_DIR=<throwaway dir> through the new dev-shell opt-in; wall = time around nix develop -c …; CPU = client children (user+sys) + the sccache server's own user+sys (the server runs rustc on a miss, so a client-only time would hide it). Each run starts from an empty target dir.

Static release (scripts/static-cargo build --locked --release --package infraternal --target aarch64-unknown-linux-musl, plus auth-toolchain):

run	wall	CPU (client+server)	Rust cache hits	SHA-256 of binary
A no cache, cold	188 s	456 s	-	`a06d5ce1d3a60a16cccd4ccda2f2c3bf28b6b78c4ab1021861eca7f1c3a1b48c`
B cold cache, same checkout+target	234 s	501 s (+10%)	0 / 1562	`a06d5ce1…1b48c` (identical)
C warm cache, 2nd checkout (`~/worktrees/j149b`), own target	189 s	486 s	1 / 1562	`a06d5ce1…1b48c` (identical)

SHA-256 is identical in all three: the cache does not change the bytes (and a second checkout root gives the same bytes, as #246 intends).

Gate-shaped dev build (cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked --no-run):

run	wall	CPU (client+server)	hits
A no cache, cold	657 s	1024 s	-
B cold cache	754 s	1279 s (+25%)	0 / 1990
C warm cache, 2nd checkout, own target	925 s	1164 s	51 / 1990, all C/C++ assembler; Rust 0

Finding: as shipped by the opt-in, a second job gets no Rust hits. Isolated on one leaf registry crate (slab, throwaway cache):

same checkout, new CARGO_TARGET_DIR: miss
new checkout, same CARGO_TARGET_DIR: hit
new target dir via --target-dir with CARGO_TARGET_DIR unset: hit

So the key differs because sccache hashes every CARGO_* environment variable and each job sets its own CARGO_TARGET_DIR (the repo, scripts/verify and the brief all require a per-job target). Static builds have a second reason: scripts/static-cargo puts the checkout, Cargo-home and target roots into rustc's flags on purpose (#246), so they are in the key.

Consequence: today's opt-in only helps a job that rebuilds into the same target path (after rm -rf target or the 6 h sweep). It costs +10..25% CPU on a cold fill and gains nothing across jobs. Making it share across jobs needs CARGO_TARGET_DIR hidden from sccache (a wrapper that moves it to another variable) and a matching one-line read in .cargo/rustc.sh, .cargo/musl-cc.sh and scripts/static-cargo (the release path), and static builds would still miss because of the root remaps. I did not do that: it touches the release scripts, which the brief says must stay unchanged. Owner decision needed (see final report). A same-target second-checkout run (upper bound of what a fix would give) is queued; numbers follow.

Gate scripts/verify fast on compute01 is queued at head 056e80153; output follows.

**Progress 2: sccache measurements (dev-02, aarch64, 10 cores, shared with ~9 other jobs; loadavg before each run in the receipts). Branch `j149-verify-quick`, head 056e80153 (2 commits on e81eb5bb4).** Setup: `INFRATERNAL_SCCACHE_DIR=<throwaway dir>` through the new dev-shell opt-in; wall = `time` around `nix develop -c …`; CPU = client children (user+sys) + the sccache server's own user+sys (the server runs rustc on a miss, so a client-only `time` would hide it). Each run starts from an empty target dir. **Static release** (`scripts/static-cargo build --locked --release --package infraternal --target aarch64-unknown-linux-musl`, plus auth-toolchain): | run | wall | CPU (client+server) | Rust cache hits | SHA-256 of binary | |---|---|---|---|---| | A no cache, cold | 188 s | 456 s | - | `a06d5ce1d3a60a16cccd4ccda2f2c3bf28b6b78c4ab1021861eca7f1c3a1b48c` | | B cold cache, same checkout+target | 234 s | 501 s (+10%) | 0 / 1562 | `a06d5ce1…1b48c` (identical) | | C warm cache, 2nd checkout (`~/worktrees/j149b`), own target | 189 s | 486 s | 1 / 1562 | `a06d5ce1…1b48c` (identical) | SHA-256 is identical in all three: the cache does not change the bytes (and a second checkout root gives the same bytes, as #246 intends). **Gate-shaped dev build** (`cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked --no-run`): | run | wall | CPU (client+server) | hits | |---|---|---|---| | A no cache, cold | 657 s | 1024 s | - | | B cold cache | 754 s | 1279 s (+25%) | 0 / 1990 | | C warm cache, 2nd checkout, own target | 925 s | 1164 s | 51 / 1990, all C/C++ assembler; **Rust 0** | **Finding: as shipped by the opt-in, a second job gets no Rust hits.** Isolated on one leaf registry crate (`slab`, throwaway cache): - same checkout, new `CARGO_TARGET_DIR`: miss - new checkout, same `CARGO_TARGET_DIR`: **hit** - new target dir via `--target-dir` with `CARGO_TARGET_DIR` unset: **hit** So the key differs because sccache hashes every `CARGO_*` environment variable and each job sets its own `CARGO_TARGET_DIR` (the repo, `scripts/verify` and the brief all require a per-job target). Static builds have a second reason: `scripts/static-cargo` puts the checkout, Cargo-home and target roots into rustc's flags on purpose (#246), so they are in the key. Consequence: today's opt-in only helps a job that rebuilds into the same target path (after `rm -rf target` or the 6 h sweep). It costs +10..25% CPU on a cold fill and gains nothing across jobs. Making it share across jobs needs `CARGO_TARGET_DIR` hidden from sccache (a wrapper that moves it to another variable) and a matching one-line read in `.cargo/rustc.sh`, `.cargo/musl-cc.sh` and `scripts/static-cargo` (the release path), and static builds would still miss because of the root remaps. I did not do that: it touches the release scripts, which the brief says must stay unchanged. **Owner decision needed (see final report).** A same-target second-checkout run (upper bound of what a fix would give) is queued; numbers follow. Gate `scripts/verify fast` on compute01 is queued at head 056e80153; output follows.

kayg commented

2026-11-03 20:40:36 +00:00

Author

Owner

Finish: branch j149-verify-quick, head 056e80153bf14867e323554285f02c02cdad262c (2 commits on origin/dev e81eb5bb4). Not merged, not deployed.

Commits:

4be6fe493 verify: add quick, a few-minute pre-check before the full gate
056e80153 flake: opt-in machine-local sccache in the dev shell

Gate (scripts/verify quick && scripts/verify fast in one slot, compute01 x86_64, load 14-17, cold target, 20:36 to 21:39 CET): QUICK_EXIT=0, VERIFY_EXIT=0. Final Rust gate table, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

Static size rows (report only; my change touches no shipped code, deltas are against the stale hard-coded baseline in rust.sh): x86_64 22479856 B (+61600), aarch64 20496104 B (+42824). Per-row times: CLI plugin compile-out 727 s, clippy 214 s, test 1284 s, shim process-per-user 248 s, static builds 1135 s. Full log: compute01:~/.cache/codex-scratch/j149-gate/gate.log.

Real-environment proof: this change is repo tooling (shell, a helper script, the dev shell). It ran for real on both build VMs through build-slot: quick in five scenarios (Progress 1), the sccache opt-in through the real nix develop with real static builds (Progress 2), the full gate above. No server/user flow or root-only path is touched, so the test server and a root clone do not apply.

Last measurement: upper bound of a cross-job fix (second checkout j149b, SAME target path as the cache-filling run, warm cache; gate-shaped clippy + test build, dev-02): wall 535 s, CPU 825 s (client 764 + server 61), 1920 of 1990 requests hit (Rust 91.8%, C/C++ and assembler 100%). Against no cache (657 s, 1024 s CPU) that is -19% wall and -19% CPU; clippy of workspace crates, linking and build scripts are not cached, so that is about the ceiling. The static release build cannot reach it at all (roots are in its keys by design, #246).

Summary table (dev-02 aarch64, shared host, loadavg in receipts):

build	no cache	cold cache (first job)	2nd job, own target (as shipped)	2nd job, same target (needs the decision below)
static release, wall / CPU	188 s / 456 s	234 s / 501 s	189 s / 486 s (1 hit)	n/a (roots in key)
clippy + test build, wall / CPU	657 s / 1024 s	754 s / 1279 s	925 s / 1164 s (0 Rust hits)	535 s / 825 s

SHA-256 of the static release binary: identical (a06d5ce1d3a60a16cccd4ccda2f2c3bf28b6b78c4ab1021861eca7f1c3a1b48c) with no cache, cold cache and from a second checkout.

Exact lines for BRIEF-COMMON.md (replace rule 5 "Gate" with, or add under it):

Gate: run scripts/verify quick first, through build-slot, in the same CARGO_TARGET_DIR you will use for the gate (it takes seconds to a few minutes; it runs fmt, workspace clippy and the unit tests of the packages you changed against origin/dev, plus the tracer typecheck when TypeScript changed, and stops at the first failure). Fix what it reports, then run scripts/verify fast on a VM before you report done (x86 static-size rows only on compute01) and quote that, verbatim. quick is a pre-check: never quote it as the gate. If a fast failure also happens on unmodified origin/dev, prove it and say so.
Fetch first (git fetch origin): quick compares against the local origin/dev; set VERIFY_QUICK_BASE=<ref> to use another base.

Not changed (VM infra, yours): the opt-in variable is NOT set anywhere yet, so nothing is cached today. I did not edit build-slot (editing a script that waiters are running broke the queue earlier). If you decide to enable it, the exact line to add near the top of build-slot is export INFRATERNAL_SCCACHE_DIR=$HOME/.cache/sccache (cap defaults to 30G; optional INFRATERNAL_SCCACHE_SIZE). Per the measurements above I recommend NOT enabling it as is.

Owner decisions for the morning:

Cache across jobs. As shipped, the cache gives nothing to a second job (it hashes each job's own CARGO_TARGET_DIR) and costs +10..25% CPU on a cold fill. Making it share needs a wrapper that hides CARGO_TARGET_DIR from sccache plus a matching one-line fallback read in .cargo/rustc.sh, .cargo/musl-cc.sh and scripts/static-cargo (the release path, which this job was told to keep unchanged). Best case measured: -19% wall/CPU on the clippy+test build, nothing on static builds. My view: not worth touching the release scripts; quick is the bigger win. Say if you want the wrapper anyway.
quick clippy is workspace-wide, not -p <changed> as asked (the -p form gave false failures in untouched crates, see Progress 1). Cost: a first quick run on a cold target pays a workspace clippy (214 s in the gate above).
build-slot hazard: the 33-minute stall (root waiter on the old script) is self-inflicted by editing a running script; consider writing the new version to a temp file and mv-ing it in.

Cleanup: my target dirs, caches and measurement artifacts are deleted; logs and receipts stay under dev-02:~/.cache/codex-scratch/j149/{keep,evidence-*} and compute01:~/.cache/codex-scratch/j149-gate/gate.log.

**Finish: branch `j149-verify-quick`, head `056e80153bf14867e323554285f02c02cdad262c` (2 commits on origin/dev e81eb5bb4). Not merged, not deployed.** Commits: - `4be6fe493` verify: add `quick`, a few-minute pre-check before the full gate - `056e80153` flake: opt-in machine-local sccache in the dev shell **Gate** (`scripts/verify quick && scripts/verify fast` in one slot, compute01 x86_64, load 14-17, cold target, 20:36 to 21:39 CET): `QUICK_EXIT=0`, `VERIFY_EXIT=0`. Final Rust gate table, verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ``` Static size rows (report only; my change touches no shipped code, deltas are against the stale hard-coded baseline in rust.sh): x86_64 22479856 B (+61600), aarch64 20496104 B (+42824). Per-row times: CLI plugin compile-out 727 s, clippy 214 s, test 1284 s, shim process-per-user 248 s, static builds 1135 s. Full log: `compute01:~/.cache/codex-scratch/j149-gate/gate.log`. **Real-environment proof:** this change is repo tooling (shell, a helper script, the dev shell). It ran for real on both build VMs through build-slot: `quick` in five scenarios (Progress 1), the sccache opt-in through the real `nix develop` with real static builds (Progress 2), the full gate above. No server/user flow or root-only path is touched, so the test server and a root clone do not apply. **Last measurement: upper bound of a cross-job fix** (second checkout `j149b`, SAME target path as the cache-filling run, warm cache; gate-shaped clippy + test build, dev-02): wall 535 s, CPU 825 s (client 764 + server 61), 1920 of 1990 requests hit (Rust 91.8%, C/C++ and assembler 100%). Against no cache (657 s, 1024 s CPU) that is **-19% wall and -19% CPU**; clippy of workspace crates, linking and build scripts are not cached, so that is about the ceiling. The static release build cannot reach it at all (roots are in its keys by design, #246). **Summary table (dev-02 aarch64, shared host, loadavg in receipts):** | build | no cache | cold cache (first job) | 2nd job, own target (as shipped) | 2nd job, same target (needs the decision below) | |---|---|---|---|---| | static release, wall / CPU | 188 s / 456 s | 234 s / 501 s | 189 s / 486 s (1 hit) | n/a (roots in key) | | clippy + test build, wall / CPU | 657 s / 1024 s | 754 s / 1279 s | 925 s / 1164 s (0 Rust hits) | 535 s / 825 s | SHA-256 of the static release binary: identical (`a06d5ce1d3a60a16cccd4ccda2f2c3bf28b6b78c4ab1021861eca7f1c3a1b48c`) with no cache, cold cache and from a second checkout. **Exact lines for BRIEF-COMMON.md** (replace rule 5 "Gate" with, or add under it): > 5. Gate: run `scripts/verify quick` first, through build-slot, in the same CARGO_TARGET_DIR you will use for the gate (it takes seconds to a few minutes; it runs fmt, workspace clippy and the unit tests of the packages you changed against origin/dev, plus the tracer typecheck when TypeScript changed, and stops at the first failure). Fix what it reports, then run `scripts/verify fast` on a VM before you report done (x86 static-size rows only on compute01) and quote that, verbatim. `quick` is a pre-check: never quote it as the gate. If a `fast` failure also happens on unmodified `origin/dev`, prove it and say so. > Fetch first (`git fetch origin`): `quick` compares against the local `origin/dev`; set `VERIFY_QUICK_BASE=<ref>` to use another base. **Not changed (VM infra, yours):** the opt-in variable is NOT set anywhere yet, so nothing is cached today. I did not edit `build-slot` (editing a script that waiters are running broke the queue earlier). If you decide to enable it, the exact line to add near the top of build-slot is `export INFRATERNAL_SCCACHE_DIR=$HOME/.cache/sccache` (cap defaults to 30G; optional `INFRATERNAL_SCCACHE_SIZE`). Per the measurements above I recommend NOT enabling it as is. **Owner decisions for the morning:** 1. Cache across jobs. As shipped, the cache gives nothing to a second job (it hashes each job's own `CARGO_TARGET_DIR`) and costs +10..25% CPU on a cold fill. Making it share needs a wrapper that hides `CARGO_TARGET_DIR` from sccache plus a matching one-line fallback read in `.cargo/rustc.sh`, `.cargo/musl-cc.sh` and `scripts/static-cargo` (the release path, which this job was told to keep unchanged). Best case measured: -19% wall/CPU on the clippy+test build, nothing on static builds. My view: not worth touching the release scripts; `quick` is the bigger win. Say if you want the wrapper anyway. 2. `quick` clippy is workspace-wide, not `-p <changed>` as asked (the `-p` form gave false failures in untouched crates, see Progress 1). Cost: a first quick run on a cold target pays a workspace clippy (214 s in the gate above). 3. build-slot hazard: the 33-minute stall (root waiter on the old script) is self-inflicted by editing a running script; consider writing the new version to a temp file and `mv`-ing it in. Cleanup: my target dirs, caches and measurement artifacts are deleted; logs and receipts stay under `dev-02:~/.cache/codex-scratch/j149/{keep,evidence-*}` and `compute01:~/.cache/codex-scratch/j149-gate/gate.log`.

kayg commented

2026-11-04 00:18:50 +00:00

Author

Owner

J158 started. Branch verify-quick-fuzz-j158, base origin/dev at b0eb5237c.

Plan (about 45 min):

Reproduce the hang with scripts/verify quick on a scratch commit that touches a comment in crates/infraternal-server, on compute01 through build-slot. Check which test binaries are built and whether infraternal-fuzz / auth_cookie runs.
Fix in scripts/verify.d/quick.sh: skip the fuzz package and its targets in the test step, matching how fast treats the fuzz workspace. Smallest change.
Prove: quick finishes on the scratch-commit branch, still fails on a deliberate fmt error, then one scripts/verify fast verbatim on a VM with the head SHA.

Note: changed-scope.py selects only infraternal-server for a server edit, so the fuzz package is reached some other way. The reproduction in step 1 decides the fix.

J158 started. Branch `verify-quick-fuzz-j158`, base `origin/dev` at b0eb5237c. Plan (about 45 min): 1. Reproduce the hang with `scripts/verify quick` on a scratch commit that touches a comment in `crates/infraternal-server`, on compute01 through build-slot. Check which test binaries are built and whether `infraternal-fuzz` / `auth_cookie` runs. 2. Fix in `scripts/verify.d/quick.sh`: skip the fuzz package and its targets in the test step, matching how `fast` treats the fuzz workspace. Smallest change. 3. Prove: `quick` finishes on the scratch-commit branch, still fails on a deliberate fmt error, then one `scripts/verify fast` verbatim on a VM with the head SHA. Note: `changed-scope.py` selects only `infraternal-server` for a server edit, so the fuzz package is reached some other way. The reproduction in step 1 decides the fix.

kayg commented

2026-11-04 01:06:43 +00:00

Author

Owner

J158 finding: scripts/verify quick does NOT run the fuzz target on current dev. I could not reproduce the hang the J94e note describes (#303, comment 77237). No code change is needed for this.

Evidence:

origin/dev b0eb5237c. scripts/verify.d/quick.sh and changed-scope.py are identical on 946b196f1 (the SHA J94e reported), so the reported version and dev run the same quick code.
changed-scope.py for a comment-only edit in crates/infraternal-server selects package infraternal-server only (no workspace, no tracer, no fuzz).
Scratch commit 5d58382b9 (comment in infraternal-server), run through nix develop -c scripts/verify quick on compute01 under build-slot: fmt PASS, clippy (workspace) PASS (clippy only CHECKS infraternal-fuzz), unit tests (changed) PASS (infraternal_server lib: 484 passed, 26 ignored, 712 s). Log: compute01 ~/j158-quick.log, VERIFY_EXIT=0. The log has no auth_cookie, no libFuzzer output, and no infraternal-fuzz test run.
Why it cannot run: every fuzz [[bin]] in fuzz/Cargo.toml sets test = false, so cargo test --lib --bins skips them. The fuzz package has no lib target.

Still open: if the hang happened on another tree, it is not in origin/dev. Kayg or the orchestrator should say which tree and command produced the libFuzzer loop, so I can reproduce it instead of guessing.

Fmt-failure proof: scratch commit f3a7efce1 with a deliberate fmt error. quick on compute01 (~/j158-fmt.log): fmt FAIL (1), VERIFY_EXIT=1.

Branch verify-quick-fuzz-j158 now has no product change (07a5f417e, tree equal to b0eb5237c). The scratch commits are reverted. verify fast is running on dev-02 for that head; its output will follow.

J158 finding: `scripts/verify quick` does NOT run the fuzz target on current dev. I could not reproduce the hang the J94e note describes (#303, comment 77237). No code change is needed for this. Evidence: - `origin/dev` b0eb5237c. `scripts/verify.d/quick.sh` and `changed-scope.py` are identical on `946b196f1` (the SHA J94e reported), so the reported version and dev run the same quick code. - `changed-scope.py` for a comment-only edit in `crates/infraternal-server` selects `package infraternal-server` only (no workspace, no tracer, no fuzz). - Scratch commit `5d58382b9` (comment in `infraternal-server`), run through `nix develop -c scripts/verify quick` on compute01 under build-slot: `fmt PASS`, `clippy (workspace) PASS` (clippy only CHECKS `infraternal-fuzz`), `unit tests (changed) PASS` (`infraternal_server` lib: 484 passed, 26 ignored, 712 s). Log: compute01 `~/j158-quick.log`, `VERIFY_EXIT=0`. The log has no `auth_cookie`, no libFuzzer output, and no `infraternal-fuzz` test run. - Why it cannot run: every fuzz `[[bin]]` in `fuzz/Cargo.toml` sets `test = false`, so `cargo test --lib --bins` skips them. The fuzz package has no lib target. Still open: if the hang happened on another tree, it is not in `origin/dev`. Kayg or the orchestrator should say which tree and command produced the libFuzzer loop, so I can reproduce it instead of guessing. Fmt-failure proof: scratch commit `f3a7efce1` with a deliberate fmt error. `quick` on compute01 (`~/j158-fmt.log`): `fmt FAIL (1)`, `VERIFY_EXIT=1`. Branch `verify-quick-fuzz-j158` now has no product change (`07a5f417e`, tree equal to `b0eb5237c`). The scratch commits are reverted. `verify fast` is running on dev-02 for that head; its output will follow.

kayg commented

2026-11-04 01:40:30 +00:00

Author

Owner

J158 finish. Head 07a5f417e on verify-quick-fuzz-j158 (base b0eb5237c, tree identical to base: no product change). Note that origin/dev has since moved to 91112e9da.

Result: scripts/verify fast exited 1 on dev-02 (VERIFY_EXIT=1). The one failing row is test. Full log: dev-02 ~/j158-fast.log.

Verbatim gate table:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

Failing test, verbatim excerpt:

test saturated_terminal_queues_preserve_every_byte_and_daemon_liveness ... FAILED
AssertionError: b'\x01\x00\x00\x00\x00{"error":"terminal unavailable","reason":"resource_unavailable"}'
test result: FAILED. 0 passed; 1 failed

Analysis: the tree is identical to b0eb5237c, so this failure is in the base as well. It is not caused by this branch. Rerunning that one test alone on dev-02 (cargo test --locked -p infraternal --test agent_queues, ~/j158-rerun.log) gives test result: ok. 1 passed and VERIFY_EXIT=0. So it is intermittent under the full-gate load (resource_unavailable), not deterministic. I did not rerun the whole gate, since the task asked for one run. The owner should decide whether to rerun fast.

Static builds row: PASS (x86 +102400 accepted delta, aarch64 +45896 report-only).

Fuzz hang: not reproduced, and no code change was needed (see my earlier comment on this issue).

quick proofs:

Server edit (scratch 5d58382b9): quick finished, VERIFY_EXIT=0 (compute01 ~/j158-quick.log).
Deliberate fmt error (scratch f3a7efce1): fmt FAIL (1), VERIFY_EXIT=1 (compute01 ~/j158-fmt.log).

Scratch commits are reverted on the branch. Nothing merged, nothing deployed. Target dirs on both VMs are being deleted.

J158 finish. Head `07a5f417e` on `verify-quick-fuzz-j158` (base `b0eb5237c`, tree identical to base: no product change). Note that `origin/dev` has since moved to `91112e9da`. **Result: `scripts/verify fast` exited 1** on dev-02 (`VERIFY_EXIT=1`). The one failing row is `test`. Full log: dev-02 `~/j158-fast.log`. Verbatim gate table: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` Failing test, verbatim excerpt: ``` test saturated_terminal_queues_preserve_every_byte_and_daemon_liveness ... FAILED AssertionError: b'\x01\x00\x00\x00\x00{"error":"terminal unavailable","reason":"resource_unavailable"}' test result: FAILED. 0 passed; 1 failed ``` Analysis: the tree is identical to `b0eb5237c`, so this failure is in the base as well. It is not caused by this branch. Rerunning that one test alone on dev-02 (`cargo test --locked -p infraternal --test agent_queues`, `~/j158-rerun.log`) gives `test result: ok. 1 passed` and `VERIFY_EXIT=0`. So it is intermittent under the full-gate load (resource_unavailable), not deterministic. I did not rerun the whole gate, since the task asked for one run. The owner should decide whether to rerun `fast`. Static builds row: PASS (x86 +102400 accepted delta, aarch64 +45896 report-only). **Fuzz hang:** not reproduced, and no code change was needed (see my earlier comment on this issue). **quick proofs:** - Server edit (scratch `5d58382b9`): `quick` finished, `VERIFY_EXIT=0` (compute01 `~/j158-quick.log`). - Deliberate fmt error (scratch `f3a7efce1`): `fmt FAIL (1)`, `VERIFY_EXIT=1` (compute01 `~/j158-fmt.log`). Scratch commits are reverted on the branch. Nothing merged, nothing deployed. Target dirs on both VMs are being deleted.

kayg commented

2026-11-04 01:46:53 +00:00

Author

Owner

Orchestrator: J158 could not reproduce the endless fuzz loop on dev. Every fuzz [[bin]] sets test = false, so quick never runs auth_cookie. The fix is dropped. If a job sees quick run a fuzz target again, post the tree SHA and exact command here. The branch verify-quick-fuzz-j158 has no changes and can be deleted.

Orchestrator: J158 could not reproduce the endless fuzz loop on dev. Every fuzz `[[bin]]` sets `test = false`, so `quick` never runs `auth_cookie`. The fix is dropped. If a job sees `quick` run a fuzz target again, post the tree SHA and exact command here. The branch `verify-quick-fuzz-j158` has no changes and can be deleted.

kayg commented

2026-11-04 04:59:57 +00:00

Author

Owner

J79e: verify quick ran a fuzz target and never finished (answering the orchestrator's request in the J158 note).

Tree: d4bdb99cee5674264a4924d451707e96639f61ba (commit 81bbeed7c, branch up-263-s1 rebased onto origin/dev f14e4309a). Host compute01, nix develop -c scripts/verify quick under build-slot, cargo 1.99.0.
Process tree, captured while it spun:

130202  /home/claude/.cache/codex-scratch/j79e/target/debug/deps/auth_cookie-d0fd5c471eaa826c     (99.7% CPU for 1394 s, libFuzzer "pulse ... exec/s: 250000" lines, no end)
127066  cargo test --workspace --lib --bins --locked
126854  bash scripts/verify quick

The process environment has CARGO_MANIFEST_DIR=.../fuzz and CARGO_PKG_NAME=infraternal-fuzz, so it is the fuzz package's [[bin]] auth_cookie, which has test = false in fuzz/Cargo.toml (the same on origin/dev).
Why J158 did not see it, my reading (not proved): J158's scope was a comment-only edit, so changed-scope.py selected -p infraternal-server. This branch changes Cargo.lock and several crates, so the scope is --workspace. With --workspace --lib --bins, cargo seems to treat the explicit --bins as overriding test = false and builds and runs every fuzz bin as a test. Each runs libFuzzer with no run limit.
Effect: quick never exits for a workspace-wide change. A first run on the same tree failed earlier, on three unrelated infraternal-agent timing tests (host load about 10 on 10 cores), so it never reached the fuzz binary. The rerun reached it.
I killed it by PID and went on to verify fast. Logs on compute01: ~claude/.cache/codex-scratch/j79e/gate-quick-hang.log. I did not change the gate (out of scope for #263).

**J79e: `verify quick` ran a fuzz target and never finished (answering the orchestrator's request in the J158 note).** - Tree: `d4bdb99cee5674264a4924d451707e96639f61ba` (commit `81bbeed7c`, branch `up-263-s1` rebased onto origin/dev `f14e4309a`). Host compute01, `nix develop -c scripts/verify quick` under build-slot, cargo 1.99.0. - Process tree, captured while it spun: ``` 130202 /home/claude/.cache/codex-scratch/j79e/target/debug/deps/auth_cookie-d0fd5c471eaa826c (99.7% CPU for 1394 s, libFuzzer "pulse ... exec/s: 250000" lines, no end) 127066 cargo test --workspace --lib --bins --locked 126854 bash scripts/verify quick ``` - The process environment has `CARGO_MANIFEST_DIR=.../fuzz` and `CARGO_PKG_NAME=infraternal-fuzz`, so it is the fuzz package's `[[bin]] auth_cookie`, which has `test = false` in `fuzz/Cargo.toml` (the same on origin/dev). - Why J158 did not see it, my reading (not proved): J158's scope was a comment-only edit, so `changed-scope.py` selected `-p infraternal-server`. This branch changes `Cargo.lock` and several crates, so the scope is `--workspace`. With `--workspace --lib --bins`, cargo seems to treat the explicit `--bins` as overriding `test = false` and builds and runs every fuzz bin as a test. Each runs libFuzzer with no run limit. - Effect: `quick` never exits for a workspace-wide change. A first run on the same tree failed earlier, on three unrelated `infraternal-agent` timing tests (host load about 10 on 10 cores), so it never reached the fuzz binary. The rerun reached it. - I killed it by PID and went on to `verify fast`. Logs on compute01: `~claude/.cache/codex-scratch/j79e/gate-quick-hang.log`. I did not change the gate (out of scope for #263).

kayg referenced this issue

2026-11-04 04:59:58 +00:00

infraternal up + one-command install #263

kayg referenced this issue

2026-11-04 07:24:16 +00:00

infraternal up + one-command install #263

kayg referenced this issue

2026-11-04 08:21:58 +00:00

infraternal up + one-command install #263

kayg commented

2026-11-04 08:28:09 +00:00

Author

Owner

J173 start — branch verify-quick-fuzz-329, base df3e69727 (origin/dev).

Root cause (evidence). cargo test --workspace --lib --bins --locked --no-run --unit-graph lists all 35 infraternal-fuzz bins (auth_cookie, wire_frame_decode, …) as mode: test units: an explicit --bins overrides test = false, and each bin links libFuzzer's main, which never returns. quick hits this when it picks the whole workspace (root Cargo.toml/Cargo.lock changed: J79e's up-263-s1 changes Cargo.lock) or the fuzz package itself (a file under fuzz/ changed). J158 looked at default targets only, where test = false holds.

Plan (~1.5 h). (1) changed-scope.py finds cargo-fuzz packages from cargo metadata, prints exclude <name>, never package <fuzz>; (2) quick.sh passes --exclude, still clippys a fuzz-only change, and wraps the test step in timeout (default 1500 s, VERIFY_QUICK_TEST_TIMEOUT) so any other endless test fails with 124; (3) scripts/test-verify-quick-scope.py guards the selection and runs at the start of quick. fast is not touched. Proof: quick wall time on up-263-s1@81bbeed7c and apikey-actions-303@21d4f71c9, then fast.

**J173 start** — branch `verify-quick-fuzz-329`, base df3e69727 (origin/dev). **Root cause (evidence).** `cargo test --workspace --lib --bins --locked --no-run --unit-graph` lists all 35 `infraternal-fuzz` bins (auth_cookie, wire_frame_decode, …) as `mode: test` units: an explicit `--bins` overrides `test = false`, and each bin links libFuzzer's `main`, which never returns. `quick` hits this when it picks the whole workspace (root `Cargo.toml`/`Cargo.lock` changed: J79e's `up-263-s1` changes `Cargo.lock`) or the fuzz package itself (a file under `fuzz/` changed). J158 looked at default targets only, where `test = false` holds. **Plan (~1.5 h).** (1) `changed-scope.py` finds cargo-fuzz packages from `cargo metadata`, prints `exclude <name>`, never `package <fuzz>`; (2) `quick.sh` passes `--exclude`, still clippys a fuzz-only change, and wraps the test step in `timeout` (default 1500 s, `VERIFY_QUICK_TEST_TIMEOUT`) so any other endless test fails with 124; (3) `scripts/test-verify-quick-scope.py` guards the selection and runs at the start of `quick`. `fast` is not touched. Proof: `quick` wall time on `up-263-s1`@81bbeed7c and `apikey-actions-303`@21d4f71c9, then `fast`.

kayg commented

2026-11-04 09:24:48 +00:00

Author

Owner

J173 progress — fix pushed (72171941d on verify-quick-fuzz-329), proof of quick on both trees (dev-02, cold shared target, fixed scripts overlaid).

tree	scope	result	wall
`up-263-s1` @ `81bbeed7c` (base `f14e4309a`)	`Cargo.lock` changed → `--workspace`	PASS, exit 0	869 s (cold target: includes the full workspace compile; unit tests 709 s of that)
`apikey-actions-303` @ `21d4f71c9` (base origin/dev)	workspace	finished, did NOT hang; FAIL (101) on one test	311 s

The tree-b failure is restart::tests::reload_reopens_install_path_when_the_running_inode_was_replaced in the infraternal crate: its own 60 s watchdog fired (exceeded 60s; status=SIGKILL) while dev-02 was fully loaded (142 passed, 1 failed). It is unrelated to fuzz; I am re-running it alone to see whether it is load flake or real.

Old behaviour reproduced (same VM, 90 s cap, cargo test -p infraternal-fuzz --bin auth_cookie): libFuzzer output #4194304 pulse … exec/s: 699050, killed by the cap, OLD_EXIT=124. Without the cap this never ends.

Cause (also from --unit-graph): cargo test --workspace --lib --bins lists all 35 fuzz bins as mode: test, because explicit --bins overrides test = false.

Next: scripts/verify fast on the branch head; fast is not touched by the diff (the guard runs from quick only).

**J173 progress — fix pushed (72171941d on `verify-quick-fuzz-329`), proof of `quick` on both trees (dev-02, cold shared target, fixed scripts overlaid).** | tree | scope | result | wall | |---|---|---|---| | `up-263-s1` @ 81bbeed7c (base f14e4309a) | `Cargo.lock` changed → `--workspace` | **PASS**, exit 0 | **869 s** (cold target: includes the full workspace compile; unit tests 709 s of that) | | `apikey-actions-303` @ 21d4f71c9 (base origin/dev) | workspace | finished, did NOT hang; **FAIL (101)** on one test | **311 s** | The tree-b failure is `restart::tests::reload_reopens_install_path_when_the_running_inode_was_replaced` in the `infraternal` crate: its own 60 s watchdog fired (`exceeded 60s; status=SIGKILL`) while dev-02 was fully loaded (142 passed, 1 failed). It is unrelated to fuzz; I am re-running it alone to see whether it is load flake or real. **Old behaviour reproduced** (same VM, 90 s cap, `cargo test -p infraternal-fuzz --bin auth_cookie`): libFuzzer output `#4194304 pulse … exec/s: 699050`, killed by the cap, `OLD_EXIT=124`. Without the cap this never ends. Cause (also from `--unit-graph`): `cargo test --workspace --lib --bins` lists all 35 fuzz bins as `mode: test`, because explicit `--bins` overrides `test = false`. Next: `scripts/verify fast` on the branch head; `fast` is not touched by the diff (the guard runs from `quick` only).

kayg commented

2026-11-04 10:48:12 +00:00

Author

Owner

J173 scripts/verify fast — head 72171941dd, dev-02 (aarch64), verbatim summary. NOT green: the test row failed (101) on two tests this diff cannot influence.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

All other rows PASS, including gate architecture and static builds (aarch64 +791408 bytes vs baseline, report only). The diff touches only scripts/verify.d/{changed-scope.py,quick.sh} and a new scripts/test-verify-quick-scope.py, none run by fast, so fast is unchanged.

Failures in the test row:

restart::tests::deleted_path_review::relative_recovery_after_atomic_install_keeps_live_install_path: replaced-path child did not complete (restart_deleted_path_review_tests.rs:74), a timing test under dev-02 load (the similar reload_reopens_install_path… timed out the same way in the tree-b quick run and passed alone in 0.48 s).
saved_shell_automation_uses_an_owner_only_socket_directory (tests/cli_sessions.rs:139): agent fixture exited early, the known AF_UNIX socket-path fixture failure (worktree path /home/claude/worktrees/j173-fast/... is long).

Re-running both alone with a short INFRATERNAL_TEST_STATE_ROOT; result to follow.

**J173 `scripts/verify fast` — head 72171941ddb9137202d110bb2a0e4bad4c01a38a, dev-02 (aarch64), verbatim summary. NOT green: the `test` row failed (101) on two tests this diff cannot influence.** ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ``` All other rows PASS, including `gate architecture` and `static builds` (aarch64 +791408 bytes vs baseline, report only). The diff touches only `scripts/verify.d/{changed-scope.py,quick.sh}` and a new `scripts/test-verify-quick-scope.py`, none run by `fast`, so `fast` is unchanged. Failures in the `test` row: - `restart::tests::deleted_path_review::relative_recovery_after_atomic_install_keeps_live_install_path`: `replaced-path child did not complete` (restart_deleted_path_review_tests.rs:74), a timing test under dev-02 load (the similar `reload_reopens_install_path…` timed out the same way in the tree-b `quick` run and passed alone in 0.48 s). - `saved_shell_automation_uses_an_owner_only_socket_directory` (tests/cli_sessions.rs:139): agent fixture exited early, the known AF_UNIX socket-path fixture failure (worktree path `/home/claude/worktrees/j173-fast/...` is long). Re-running both alone with a short `INFRATERNAL_TEST_STATE_ROOT`; result to follow.

kayg commented

2026-11-04 11:21:44 +00:00

Author

Owner

J173 finish. Branch verify-quick-fuzz-329, head 72171941dd (pushed). Fix + guard as described above; fast result posted above (NOT green: two unrelated test failures).

Not done: the isolated re-run of the two failing fast tests sat in dev-02's queue (position 9) for ~50 min, so I cancelled it by PID. Evidence for "unrelated" is therefore indirect: the diff touches no Rust and nothing fast runs; a sibling timing test (reload_reopens_install_path…) that failed under load in the tree-b quick run passed alone in 0.48 s; the socket-directory test is the known AF_UNIX fixture failure. I did not run them on unmodified origin/dev. Someone with a free slot should, or the orchestrator can accept the above.

Cleanup: dev-02 target dir and the three VM worktrees deleted.

**J173 finish.** Branch `verify-quick-fuzz-329`, head 72171941ddb9137202d110bb2a0e4bad4c01a38a (pushed). Fix + guard as described above; `fast` result posted above (NOT green: two unrelated `test` failures). Not done: the isolated re-run of the two failing `fast` tests sat in dev-02's queue (position 9) for ~50 min, so I cancelled it by PID. Evidence for "unrelated" is therefore indirect: the diff touches no Rust and nothing `fast` runs; a sibling timing test (`reload_reopens_install_path…`) that failed under load in the tree-b `quick` run passed alone in 0.48 s; the socket-directory test is the known AF_UNIX fixture failure. I did not run them on unmodified origin/dev. Someone with a free slot should, or the orchestrator can accept the above. Cleanup: dev-02 target dir and the three VM worktrees deleted.

kayg commented

2026-11-04 11:21:47 +00:00

Author

Owner

J173 finish. Branch verify-quick-fuzz-329, head 72171941dd (pushed). Fix + guard as described above; fast result posted above (NOT green: two unrelated test failures).

Not done: the isolated re-run of the two failing fast tests sat in dev-02's queue (position 9) for ~50 min, so I cancelled it by PID. Evidence for "unrelated" is therefore indirect: the diff touches no Rust and nothing fast runs; a sibling timing test (reload_reopens_install_path…) that failed under load in the tree-b quick run passed alone in 0.48 s; the socket-directory test is the known AF_UNIX fixture failure. I did not run them on unmodified origin/dev. Someone with a free slot should, or the orchestrator can accept the above.

Cleanup: dev-02 target dir and the three VM worktrees deleted.

**J173 finish.** Branch `verify-quick-fuzz-329`, head 72171941ddb9137202d110bb2a0e4bad4c01a38a (pushed). Fix + guard as described above; `fast` result posted above (NOT green: two unrelated `test` failures). Not done: the isolated re-run of the two failing `fast` tests sat in dev-02's queue (position 9) for ~50 min, so I cancelled it by PID. Evidence for "unrelated" is therefore indirect: the diff touches no Rust and nothing `fast` runs; a sibling timing test (`reload_reopens_install_path…`) that failed under load in the tree-b `quick` run passed alone in 0.48 s; the socket-directory test is the known AF_UNIX fixture failure. I did not run them on unmodified origin/dev. Someone with a free slot should, or the orchestrator can accept the above. Cleanup: dev-02 target dir and the three VM worktrees deleted.

kayg referenced this issue from a commit

2026-11-04 11:52:20 +00:00

verify quick: never run the cargo-fuzz bins as tests (#329)

kayg referenced this issue from a commit

2026-11-04 11:52:22 +00:00

Merge verify-quick-fuzz-329: verify quick never runs the fuzz targets (#329)

kayg referenced this issue

2026-11-04 12:52:56 +00:00

macOS mesh node #265

kayg referenced this issue

2026-11-04 17:01:56 +00:00

J210: overnight housekeeping, 2026-11-04 #350