Cheaper gate iterations, same final strictness

kayg commented

2026-10-13 07:02:14 +00:00

Owner

Owner, 2026-10-13: make each job iteration cheaper without making the gate any less strict.

Today scripts/verify fast builds both static release binaries (x86_64 and aarch64, LTO) on every run, jobs run it many times per round, and the size step fails on stale caps on every branch, so it carries no signal.

Plan:

Time today's gate per step on the build host (cold and warm sccache), and report.
Split: an iteration mode (fmt, VT fmt, clippy, tests, cargo-deny, plugin compile-out — unchanged) and the final mode that also does both static release builds, linkage checks, the D244 key-heap check and the size check. scripts/verify fast stays the full final gate (same name, same strictness); the iteration mode gets a new name and the job rules say: iterate with it, run the full gate before reporting done.
Size check compares against the ratcheted cap from #198 (no raising; D243), printing bytes and delta.
Report the per-step times before/after and the CPU saved per iteration.

Owner, 2026-10-13: make each job iteration cheaper without making the gate any less strict. Today `scripts/verify fast` builds both static release binaries (x86_64 and aarch64, LTO) on every run, jobs run it many times per round, and the size step fails on stale caps on every branch, so it carries no signal. Plan: 1. Time today's gate per step on the build host (cold and warm sccache), and report. 2. Split: an iteration mode (fmt, VT fmt, clippy, tests, cargo-deny, plugin compile-out — unchanged) and the final mode that also does both static release builds, linkage checks, the D244 key-heap check and the size check. `scripts/verify fast` stays the full final gate (same name, same strictness); the iteration mode gets a new name and the job rules say: iterate with it, run the full gate before reporting done. 3. Size check compares against the ratcheted cap from #198 (no raising; D243), printing bytes and delta. 4. Report the per-step times before/after and the CPU saved per iteration.

fast-cold.log

292 KiB

iterate-warm.log

292 KiB

fast-final.log

266 KiB

kayg commented

2026-10-13 12:18:44 +00:00

Author

Owner

Start: branch gate-iterate, base dcf9dbb9. Adding scripts/verify iterate; timing fast cold/warm vs iterate on the build host.

Start: branch gate-iterate, base dcf9dbb9. Adding scripts/verify iterate; timing fast cold/warm vs iterate on the build host.

kayg commented

2026-10-13 12:18:57 +00:00

Author

Owner

Start: branch gate-iterate, base dcf9dbb9. Adding scripts/verify iterate; timing fast cold/warm vs iterate on the build host.

Start: branch gate-iterate, base dcf9dbb9. Adding scripts/verify iterate; timing fast cold/warm vs iterate on the build host.

kayg commented

2026-10-13 16:49:47 +00:00

Author

Owner

Resume: branch gate-iterate, existing base dcf9dbb, head dee1aa5f. Read #204's two comments; no parent is named. The branch is missing the #198 ratchet now on origin/dev, so I’m integrating that base before final measurements and review.

Resume: branch gate-iterate, existing base dcf9dbb, head dee1aa5f. Read #204's two comments; no parent is named. The branch is missing the #198 ratchet now on origin/dev, so I’m integrating that base before final measurements and review.

kayg commented

2026-10-13 16:51:39 +00:00

Author

Owner

Finding/repair: the original base dcf9dbb left scripts/verify.d/rust.sh on obsolete #57 size caps (16,142,672 / 13,555,672 B). Rebased the two #204 commits onto origin/dev 1f637cbb, which includes #198's exact D243 caps (22,204,304 / 19,638,744 B). Also removed the accidental 1,555-byte scripts/__pycache__/evidence_path.cpython-314.pyc from the docs commit. Current head 2039c275; worktree clean. Measuring cold/warm fast and iterate next.

Finding/repair: the original base dcf9dbb left `scripts/verify.d/rust.sh` on obsolete #57 size caps (16,142,672 / 13,555,672 B). Rebased the two #204 commits onto origin/dev 1f637cbb, which includes #198's exact D243 caps (22,204,304 / 19,638,744 B). Also removed the accidental 1,555-byte `scripts/__pycache__/evidence_path.cpython-314.pyc` from the docs commit. Current head 2039c275; worktree clean. Measuring cold/warm fast and iterate next.

kayg commented

2026-10-13 16:54:46 +00:00

Author

Owner

Finding during timing run: infraternal-dev load average sampled mid cold fast run was 30.90 / 21.75 / 11.08 (1/5/15 min). The run is scoped with CPUWeight=10000; I’ll record per-run load and qualify timings against this host load.

Finding during timing run: infraternal-dev load average sampled mid cold `fast` run was 30.90 / 21.75 / 11.08 (1/5/15 min). The run is scoped with CPUWeight=10000; I’ll record per-run load and qualify timings against this host load.

kayg commented

2026-10-13 17:00:34 +00:00

Author

Owner

Timing update: host load rose to 46.50 / 34.90 / 20.05 (1/5/15 min) while the isolated cold fast run was still in clippy. A full cold/warm matrix would add unnecessary load; I’ll complete the required cold fast, then compare a warmed iterate run and report the limited sample/load context.

Timing update: host load rose to 46.50 / 34.90 / 20.05 (1/5/15 min) while the isolated cold `fast` run was still in clippy. A full cold/warm matrix would add unnecessary load; I’ll complete the required cold `fast`, then compare a warmed `iterate` run and report the limited sample/load context.

kayg commented

2026-10-13 17:20:52 +00:00

Author

Owner

Finding in cold fast: crates/infraternal-server/src/auth/ca_unlock_audit_tests.rs:97, test unlock_audit_records_principal_time_outcome_without_material, failed with captured audit events left: 0, right: 4. The server library reports 418 passed / 1 failed / 46 ignored; Rust test-step timer is 959 s. This is not one of the allowed host-load timeout exceptions. Auth sources are unchanged; full gate continues through cargo-deny/static checks, then I’ll isolate the failure.

Finding in cold `fast`: `crates/infraternal-server/src/auth/ca_unlock_audit_tests.rs:97`, test `unlock_audit_records_principal_time_outcome_without_material`, failed with captured audit events `left: 0`, `right: 4`. The server library reports 418 passed / 1 failed / 46 ignored; Rust test-step timer is 959 s. This is not one of the allowed host-load timeout exceptions. Auth sources are unchanged; full gate continues through cargo-deny/static checks, then I’ll isolate the failure.

kayg commented

2026-10-13 17:27:39 +00:00

Author

Owner

Timing environment update: infraternal-dev peaked at a sampled load average of 86.65 / 77.95 / 58.70 (1/5/15 min) while the cold fast run built x86_64 static. Its scope had accumulated 2,507 CPU seconds. No additional cold matrix runs will be started; reported comparisons will be qualified as contention-affected.

Timing environment update: infraternal-dev peaked at a sampled load average of 86.65 / 77.95 / 58.70 (1/5/15 min) while the cold `fast` run built x86_64 static. Its scope had accumulated 2,507 CPU seconds. No additional cold matrix runs will be started; reported comparisons will be qualified as contention-affected.

kayg commented

2026-10-13 17:33:51 +00:00

Author

Owner

Finding: the fresh CARGO_TARGET_DIR run did not get a cold sccache. sccache --show-stats reports the shared location /home/kayg/.cache/sccache-infraternal, 22.28% overall hit rate, 23,949 Rust hits and 169,061 Rust misses. The default sccache server was reused despite a task SCCACHE_DIR; I will not stop or reset the shared server. Label current measurement fresh-target/shared-cache, not cold.

Finding: the fresh CARGO_TARGET_DIR run did not get a cold sccache. `sccache --show-stats` reports the shared location `/home/kayg/.cache/sccache-infraternal`, 22.28% overall hit rate, 23,949 Rust hits and 169,061 Rust misses. The default sccache server was reused despite a task SCCACHE_DIR; I will not stop or reset the shared server. Label current measurement fresh-target/shared-cache, not cold.

kayg commented

2026-10-13 17:40:26 +00:00

Author

Owner

Finding: the cold fast x86_64 static artifact is 22,236,272 B against #198's exact 22,204,304 B cap (+31,968 B). Production-trust, static linkage and D244 key-memory checks passed. The cap is unchanged; ARM is still building while I compare the recipe/toolchain with #198's evidence.

Finding: the cold `fast` x86_64 static artifact is 22,236,272 B against #198's exact 22,204,304 B cap (+31,968 B). Production-trust, static linkage and D244 key-memory checks passed. The cap is unchanged; ARM is still building while I compare the recipe/toolchain with #198's evidence.

kayg commented

2026-10-13 17:57:40 +00:00

Author

Owner

Cold-target scripts/verify fast completed on branch head 2039c275, but is red. The target was fresh while sccache was shared (22.28% hit rate), so this is not a cold-sccache sample. Wall 3,728.411 s; process CPU 3,868.610 s (user 3,477.447, sys 391.163); sampled load peaked at 86.65 / 77.95 / 58.70. Rust step wall timers: fmt 4 s, VT native fmt 0 s, CLI compile-out 245 s, clippy 487 s, tests 959 s, cargo-deny 5 s, static builds 2,016 s.

The static gate remains exact: x86_64 22,236,272 B (+31,968 over 22,204,304); aarch64 19,666,552 B (+27,808 over 19,638,744). Production trust and static linkage passed; x86_64 D244 key memory passed; ARM runtime check is unavailable on this x86 host. No cap was raised. Static boundary fixture also correctly refused the over-cap x86 artifact.

Rust test row: 418 passed / 1 failed / 46 ignored. Failure remains unlock_audit_records_principal_time_outcome_without_material at ca_unlock_audit_tests.rs:97 (0 captured audit events, expected 4). The Rust failure prevented TypeScript checks from running in this fast invocation. Full raw log: fast-cold.log.

Verbatim Rust gate summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Cold-target `scripts/verify fast` completed on branch head 2039c275, but is red. The target was fresh while sccache was shared (22.28% hit rate), so this is not a cold-sccache sample. Wall 3,728.411 s; process CPU 3,868.610 s (user 3,477.447, sys 391.163); sampled load peaked at 86.65 / 77.95 / 58.70. Rust step wall timers: fmt 4 s, VT native fmt 0 s, CLI compile-out 245 s, clippy 487 s, tests 959 s, cargo-deny 5 s, static builds 2,016 s. The static gate remains exact: x86_64 22,236,272 B (+31,968 over 22,204,304); aarch64 19,666,552 B (+27,808 over 19,638,744). Production trust and static linkage passed; x86_64 D244 key memory passed; ARM runtime check is unavailable on this x86 host. No cap was raised. Static boundary fixture also correctly refused the over-cap x86 artifact. Rust test row: 418 passed / 1 failed / 46 ignored. Failure remains `unlock_audit_records_principal_time_outcome_without_material` at `ca_unlock_audit_tests.rs:97` (0 captured audit events, expected 4). The Rust failure prevented TypeScript checks from running in this `fast` invocation. Full raw log: [fast-cold.log](https://git.kayg.org/attachments/b9a7ef37-3955-4c08-a318-3341916f7233). Verbatim Rust gate summary: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg commented

2026-10-13 17:57:52 +00:00

Author

Owner

Source-context finding for the D243 failure: current origin/dev includes #173 mesh-cutoff-fast after size-ratchet 88e8a0da. The first-parent source diff adds 957 lines and removes 57 across 24 Rust files, including mesh access-update and dataplane code; the #204 diff itself changes only gate/docs files. This points to post-#198 feature growth already present in the base, not a binary change from iterate mode. The line diff is context, not a byte attribution; caps remain unchanged.

Source-context finding for the D243 failure: current origin/dev includes #173 mesh-cutoff-fast after size-ratchet 88e8a0da. The first-parent source diff adds 957 lines and removes 57 across 24 Rust files, including mesh access-update and dataplane code; the #204 diff itself changes only gate/docs files. This points to post-#198 feature growth already present in the base, not a binary change from iterate mode. The line diff is context, not a byte attribution; caps remain unchanged.

kayg commented

2026-10-13 18:04:14 +00:00

Author

Owner

Audit-test isolation: direct execution of the exact workspace test binary from the failed gate passed with one test thread: 1 passed; 0 failed; 0 ignored; 464 filtered out; finished in 2.07s. The earlier package-only Cargo invocation matched 0 tests because it selected a different feature graph; it is discarded. The original full-workspace failure remains recorded and is not waived; I’ll run iterate with the established two-thread gate setting.

Audit-test isolation: direct execution of the exact workspace test binary from the failed gate passed with one test thread: `1 passed; 0 failed; 0 ignored; 464 filtered out; finished in 2.07s`. The earlier package-only Cargo invocation matched 0 tests because it selected a different feature graph; it is discarded. The original full-workspace failure remains recorded and is not waived; I’ll run `iterate` with the established two-thread gate setting.

kayg commented

2026-10-13 18:04:52 +00:00

Author

Owner

Step complete: 814f182f prints signed size delta in rust_check_static_binary, preserving #198 caps and every pass/fail branch. bash -n scripts/verify scripts/verify.d/rust.sh passed; signed-format probe printed delta: -31968 bytes. Pushed to gate-iterate. Next: run scripts/verify iterate with RUST_TEST_THREADS=2.

Step complete: 814f182f prints signed size delta in `rust_check_static_binary`, preserving #198 caps and every pass/fail branch. `bash -n scripts/verify scripts/verify.d/rust.sh` passed; signed-format probe printed `delta: -31968 bytes`. Pushed to `gate-iterate`. Next: run `scripts/verify iterate` with RUST_TEST_THREADS=2.

kayg commented

2026-10-13 18:15:08 +00:00

Author

Owner

Step complete: scripts/verify iterate passed at 814f182f (Rust: 419 passed, 0 failed, 46 ignored; TS protocol: 37 passed, 0 failed). Receipt: iterate-warm.log.

Gate step	Before: `fast` (s)	After: `iterate` (s)
fmt	4	2
VT native fmt	0	0
CLI plugin compile-out	245	1
clippy	487	1
Rust tests	959	514
cargo-deny	5	2
static release builds, linkage, key heap, size	2,016	skipped
whole command wall	3,728.411	576.290
whole command CPU (user + sys)	3,868.610	418.520

These are elapsed gate-run timings, not controlled performance benchmarks: the earlier fast used a fresh target with shared sccache, while iterate reused the target and shared sccache; host load also varied. The final strict fast run at the current head is still required.

Step complete: `scripts/verify iterate` passed at `814f182f` (Rust: 419 passed, 0 failed, 46 ignored; TS protocol: 37 passed, 0 failed). Receipt: [iterate-warm.log](https://git.kayg.org/attachments/8fecd873-e026-463f-8207-f39b27911cd0). | Gate step | Before: `fast` (s) | After: `iterate` (s) | |---|---:|---:| | fmt | 4 | 2 | | VT native fmt | 0 | 0 | | CLI plugin compile-out | 245 | 1 | | clippy | 487 | 1 | | Rust tests | 959 | 514 | | cargo-deny | 5 | 2 | | static release builds, linkage, key heap, size | 2,016 | skipped | | whole command wall | 3,728.411 | 576.290 | | whole command CPU (user + sys) | 3,868.610 | 418.520 | These are elapsed gate-run timings, not controlled performance benchmarks: the earlier `fast` used a fresh target with shared sccache, while `iterate` reused the target and shared sccache; host load also varied. The final strict `fast` run at the current head is still required.

kayg commented

2026-10-13 18:24:27 +00:00

Author

Owner

Finding during final fast: the shared build host is contended. uptime at 20:24:22 CEST reported load averages 23.04 / 21.99 / 24.34; ps found 4 concurrent scripts/verify fast|iterate shells. This makes the run unsuitable as a controlled timing comparison. The gate is still running for correctness.

Finding during final `fast`: the shared build host is contended. `uptime` at 20:24:22 CEST reported load averages 23.04 / 21.99 / 24.34; `ps` found 4 concurrent `scripts/verify fast|iterate` shells. This makes the run unsuitable as a controlled timing comparison. The gate is still running for correctness.

kayg commented

2026-10-13 18:28:55 +00:00

Author

Owner

Finish: gate-iterate head 814f182fa7365c25c3212bacf053aed66efdc675 is pushed to origin/gate-iterate. The implementation and docs are in commits 984a3c18, 2039c275, 814f182f. scripts/verify iterate passed at this head: Rust 419 passed / 0 failed / 46 ignored; TypeScript protocol 37 passed / 0 failed. Receipt: iterate-warm.log.

Per-step gate elapsed times (seconds):

Step	Before: first `fast`	After: `iterate`	Final `fast`
fmt	4	2	3
VT native fmt	0	0	0
CLI plugin compile-out	245	1	0
clippy	487	1	1
tests	959	514	597
cargo-deny	5	2	2
static builds, linkage, key heap and size	2,016	skipped	2
whole command wall	3,728.411	576.290	609.412
whole command CPU (user + sys)	3,868.610	418.520	433.408

Receipts: first fast, iterate, final fast. These are raw gate elapsed times, not a controlled performance comparison: the first fast used a fresh target; iterate and final fast reused it and shared sccache. The shared host was contended (load average 23.04 / 21.99 / 24.34 with four concurrent verify shells observed). No perf-VM timing was collected; scripts/perf-vm is absent and the perf VM did not have rustup, cargo or rustc installed.

Final scripts/verify fast ran at the branch head and exited 1 only in static builds: the test suite and preceding Rust gates passed. D243 size checks remain strict and reported x86_64 22,236,272 / 22,204,304 bytes (+31,968), aarch64 19,666,552 / 19,638,744 bytes (+27,808). The x86_64 D244 key-memory check passed; the aarch64 key-memory check was not run on this x86_64 host. These same size overages appeared in the earlier full fast run; this change touches only gate scripts and docs and does not alter product binaries.

The final Rust gate output, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
METRIC gate=fast-final cache=warm-target/shared-sccache wall_s=609.412 cpu_user_s=353.993 cpu_sys_s=79.415

Finish: `gate-iterate` head `814f182fa7365c25c3212bacf053aed66efdc675` is pushed to `origin/gate-iterate`. The implementation and docs are in commits `984a3c18`, `2039c275`, `814f182f`. `scripts/verify iterate` passed at this head: Rust 419 passed / 0 failed / 46 ignored; TypeScript protocol 37 passed / 0 failed. Receipt: [iterate-warm.log](https://git.kayg.org/attachments/8fecd873-e026-463f-8207-f39b27911cd0). Per-step gate elapsed times (seconds): | Step | Before: first `fast` | After: `iterate` | Final `fast` | |---|---:|---:|---:| | fmt | 4 | 2 | 3 | | VT native fmt | 0 | 0 | 0 | | CLI plugin compile-out | 245 | 1 | 0 | | clippy | 487 | 1 | 1 | | tests | 959 | 514 | 597 | | cargo-deny | 5 | 2 | 2 | | static builds, linkage, key heap and size | 2,016 | skipped | 2 | | whole command wall | 3,728.411 | 576.290 | 609.412 | | whole command CPU (user + sys) | 3,868.610 | 418.520 | 433.408 | Receipts: [first fast](https://git.kayg.org/attachments/b9a7ef37-3955-4c08-a318-3341916f7233), [iterate](https://git.kayg.org/attachments/8fecd873-e026-463f-8207-f39b27911cd0), [final fast](https://git.kayg.org/attachments/a4b5bfed-e0b9-4ad6-a4a9-e7f35e95787d). These are raw gate elapsed times, not a controlled performance comparison: the first fast used a fresh target; iterate and final fast reused it and shared sccache. The shared host was contended (load average 23.04 / 21.99 / 24.34 with four concurrent verify shells observed). No perf-VM timing was collected; `scripts/perf-vm` is absent and the perf VM did not have `rustup`, `cargo` or `rustc` installed. Final `scripts/verify fast` ran at the branch head and exited 1 only in `static builds`: the test suite and preceding Rust gates passed. D243 size checks remain strict and reported x86_64 22,236,272 / 22,204,304 bytes (+31,968), aarch64 19,666,552 / 19,638,744 bytes (+27,808). The x86_64 D244 key-memory check passed; the aarch64 key-memory check was not run on this x86_64 host. These same size overages appeared in the earlier full fast run; this change touches only gate scripts and docs and does not alter product binaries. The final Rust gate output, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ METRIC gate=fast-final cache=warm-target/shared-sccache wall_s=609.412 cpu_user_s=353.993 cpu_sys_s=79.415 ```

kayg referenced this issue from a commit

2026-10-14 17:42:58 +00:00

Add scripts/verify iterate: the gate minus the static release builds (#204)

kayg referenced this issue from a commit

2026-10-14 17:42:58 +00:00

Tell jobs to iterate with verify iterate and finish with verify fast (#204)

kayg referenced this issue from a commit

2026-10-14 17:42:58 +00:00

Show signed byte delta for D243 static caps

kayg referenced this issue from a commit

2026-10-14 17:42:58 +00:00

Merge gate-iterate: cheaper iterations, same final gate (#204)

~~kayg referenced this issue 2026-10-28 17:53:56 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg commented

2026-11-08 19:04:29 +00:00

Author

Owner

Closed by the orchestrator during the 2026-11-09 night housekeeping (owner order: "close stale prs/issues"). The work has landed on dev, and the latest note here is a finish or merge note (J387 inventory on #436). Reopen if anything is still missing.

kayg closed this issue

2026-11-08 19:04:30 +00:00

Rows
Columns

Cheaper gate iterations, same final strictness #204