Build speed and binary size: sccache, linker, profiles, size budget

kayg commented

2026-10-07 07:42:20 +00:00

Owner

Owner (2026-10-07): "keep our binary small; if builds take too long, set up a cache and other things to make Rust builds go brrr." There is no more RAM or compute until the Mac Studio arrives; the box has 8 cores and ~189 GB free disk.

Done (orchestrator, machine level): sccache and mold installed in the nix profile. Every infraternal Codex job now runs with RUSTC_WRAPPER=sccache (cache ~/.cache/sccache-infraternal, 40 GB cap), set in the job launcher only, so calternal builds on the shared box are untouched.

To do (repo level, job queued for the next free slot):

measure clean and incremental build times and the binary size before and after;
release profile tuned for size and speed (lto, codegen-units, panic strategy per existing decisions, strip, opt-level per crate where it matters);
mold or lld for dev and test builds (not for static release unless verified);
trim dependencies and features (cargo-bloat / cargo tree -d) to find the size drivers;
a binary-size budget in scripts/verify that fails on growth beyond an allowance;
document it in PROCESS.md.

Owner (2026-10-07): "keep our binary small; if builds take too long, set up a cache and other things to make Rust builds go brrr." There is no more RAM or compute until the Mac Studio arrives; the box has 8 cores and ~189 GB free disk. Done (orchestrator, machine level): sccache and mold installed in the nix profile. Every infraternal Codex job now runs with `RUSTC_WRAPPER=sccache` (cache `~/.cache/sccache-infraternal`, 40 GB cap), set in the job launcher only, so calternal builds on the shared box are untouched. To do (repo level, job queued for the next free slot): - measure clean and incremental build times and the binary size before and after; - release profile tuned for size and speed (lto, codegen-units, panic strategy per existing decisions, strip, opt-level per crate where it matters); - mold or lld for dev and test builds (not for static release unless verified); - trim dependencies and features (cargo-bloat / `cargo tree -d`) to find the size drivers; - a binary-size budget in `scripts/verify` that fails on growth beyond an allowance; - document it in PROCESS.md.

kayg referenced this issue

2026-10-07 07:42:20 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-07 11:14:18 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-07 11:14:37 +00:00

Author

Owner

Starting measurements on branch build-speed, base ac03b1913b. Issue has no existing comments or parent reference in its body.

Starting measurements on branch build-speed, base ac03b1913bfb6dd9b978477fc23ee5c88e276f67. Issue has no existing comments or parent reference in its body.

kayg commented

2026-10-07 11:15:15 +00:00

Author

Owner

Measurement environment: Debian 13 KVM guest infraternal-dev, Intel i9-9900K, 8 CPUs, 31 GiB RAM; initial 1/5/15-minute load 7.10/5.60/7.06. RUSTC_WRAPPER=sccache v0.18.0 is set; cache is 5 GiB of 40 GiB with 12.57% prior hit rate. mold and cargo-bloat are installed. Current release profile already has LTO, one codegen unit, strip, opt-level 3 and panic=unwind; docs/RUST.md and Cargo.toml cite tokio task isolation as the reason to keep unwind (D5/D7).

kayg commented

2026-10-07 11:31:49 +00:00

Author

Owner

Baseline timing started: clean cargo build --locked run 1/3 completed in 249.028 s inside systemd user scope at CPUWeight=10000. Load was 18.71/16.79/12.16 at start and 14.87/18.02/13.92 at finish on the 8-core i9-9900K; runs 2/3 and incremental runs continue.

kayg commented

2026-10-07 11:32:32 +00:00

Author

Owner

Completed baseline clean cargo build timings (seconds, runs 1/2/3): 249.028 / 151.268 / 139.476, median 151.268. Incremental touch medians: server 2.353 s (2.353 / 2.607 / 2.346), protocol 2.375 s (2.370 / 2.387 / 2.375). All Cargo invocations were under CPUWeight=10000; the 8-core host load varied from 14.17 to 22.58 at run boundaries. SHA remains ac03b1913b.

Completed baseline clean cargo build timings (seconds, runs 1/2/3): 249.028 / 151.268 / 139.476, median 151.268. Incremental touch medians: server 2.353 s (2.353 / 2.607 / 2.346), protocol 2.375 s (2.370 / 2.387 / 2.375). All Cargo invocations were under CPUWeight=10000; the 8-core host load varied from 14.17 to 22.58 at run boundaries. SHA remains ac03b1913bfb6dd9b978477fc23ee5c88e276f67.

kayg commented

2026-10-07 11:42:01 +00:00

Author

Owner

Completed baseline cargo test --locked --no-run timings (seconds, runs 1/2/3): 153.214 / 173.071 / 190.448, median 173.071. Incremental touch medians: server 4.202 s (4.141 / 4.532 / 4.202), protocol 4.687 s (5.139 / 3.944 / 4.687). All Cargo invocations used CPUWeight=10000; host load at run boundaries ranged from 14.87 to 32.85. SHA remains ac03b1913b.

Completed baseline cargo test --locked --no-run timings (seconds, runs 1/2/3): 153.214 / 173.071 / 190.448, median 173.071. Incremental touch medians: server 4.202 s (4.141 / 4.532 / 4.202), protocol 4.687 s (5.139 / 3.944 / 4.687). All Cargo invocations used CPUWeight=10000; host load at run boundaries ranged from 14.87 to 32.85. SHA remains ac03b1913bfb6dd9b978477fc23ee5c88e276f67.

kayg commented

2026-10-07 11:43:07 +00:00

Author

Owner

Static-build setup finding: the first direct x86_64 musl Cargo run stopped after 31.753 s because .cargo/musl-cc.sh requires the checksum-pinned Zig binary at CARGO_TARGET_DIR/auth-tools/zig/zig; PATH Zig alone is not used. The static gate prepares it with scripts/auth-toolchain. I am preparing the same toolchain under the approved scratch target and will repeat the timings.

kayg commented

2026-10-07 12:17:57 +00:00

Author

Owner

Completed baseline x86_64 musl release timings (seconds, runs 1/2/3): clean 485.626 / 316.849 / 293.421, median 316.849; touch-server 68.863 / 82.421 / 72.731, median 72.731; touch-protocol 81.163 / 77.135 / 78.817, median 78.817. Binary is a stripped static PIE, 16,142,672 bytes. Cargo builds ran at CPUWeight=10000; host load at run boundaries ranged from 11.98 to 26.80. SHA remains ac03b1913b. Bloat report is still building.

Completed baseline x86_64 musl release timings (seconds, runs 1/2/3): clean 485.626 / 316.849 / 293.421, median 316.849; touch-server 68.863 / 82.421 / 72.731, median 72.731; touch-protocol 81.163 / 77.135 / 78.817, median 78.817. Binary is a stripped static PIE, 16,142,672 bytes. Cargo builds ran at CPUWeight=10000; host load at run boundaries ranged from 11.98 to 26.80. SHA remains ac03b1913bfb6dd9b978477fc23ee5c88e276f67. Bloat report is still building.

kayg commented

2026-10-07 12:22:26 +00:00

Author

Owner

Baseline cargo-bloat --release --crates, x86_64 musl: unstripped analysis binary was 49.0 MiB with 10.8 MiB .text. Top entries: openssl_sys 1.9 MiB, std 1.0 MiB, h2 503.6 KiB, infraternal_server 501.9 KiB, tokio 323.0 KiB, clap_builder 240.2 KiB, infraternal_shim 208.2 KiB, axum 186.9 KiB, hyper 159.7 KiB, webauthn_rs_core 155.6 KiB, serde_json 154.3 KiB, regex_syntax 146.6 KiB, sqlx_core 141.2 KiB, sqlx_sqlite 138.5 KiB, regex_automata 121.0 KiB. The shipped stripped binary measured 16,142,672 bytes. Full report is in the run output; SHA remains ac03b1913b.

Baseline cargo-bloat --release --crates, x86_64 musl: unstripped analysis binary was 49.0 MiB with 10.8 MiB .text. Top entries: openssl_sys 1.9 MiB, std 1.0 MiB, h2 503.6 KiB, infraternal_server 501.9 KiB, tokio 323.0 KiB, clap_builder 240.2 KiB, infraternal_shim 208.2 KiB, axum 186.9 KiB, hyper 159.7 KiB, webauthn_rs_core 155.6 KiB, serde_json 154.3 KiB, regex_syntax 146.6 KiB, sqlx_core 141.2 KiB, sqlx_sqlite 138.5 KiB, regex_automata 121.0 KiB. The shipped stripped binary measured 16,142,672 bytes. Full report is in the run output; SHA remains ac03b1913bfb6dd9b978477fc23ee5c88e276f67.

kayg commented

2026-10-07 12:28:19 +00:00

Author

Owner

Finding during aarch64 baseline run 1: build output recorded four sccache warnings that its server shut down unexpectedly and rustc fell back to local compilation. The run is continuing; I will include cache stats and timing when it completes.

kayg commented

2026-10-07 12:46:30 +00:00

Author

Owner

Completed baseline aarch64 musl release timings (seconds, runs 1/2/3): clean 461.495 / 282.762 / 238.580, median 282.762; touch-server 65.646 / 67.610 / 66.060, median 66.060; touch-protocol 61.317 / 64.547 / 58.927, median 61.317. Binary is stripped static, 13,555,672 bytes. Final sccache stats: 80.62% overall hit rate, 9 GiB of 40 GiB; four server-shutdown fallback warnings occurred only in aarch64 clean run 1. cargo tree -d has 28 duplicate sections; only four crate names resolve multiple versions (base64, getrandom, hashbrown, syn), all through transitive dependency requirements, so no safe version change fits this task. SHA remains ac03b1913b.

Completed baseline aarch64 musl release timings (seconds, runs 1/2/3): clean 461.495 / 282.762 / 238.580, median 282.762; touch-server 65.646 / 67.610 / 66.060, median 66.060; touch-protocol 61.317 / 64.547 / 58.927, median 61.317. Binary is stripped static, 13,555,672 bytes. Final sccache stats: 80.62% overall hit rate, 9 GiB of 40 GiB; four server-shutdown fallback warnings occurred only in aarch64 clean run 1. cargo tree -d has 28 duplicate sections; only four crate names resolve multiple versions (base64, getrandom, hashbrown, syn), all through transitive dependency requirements, so no safe version change fits this task. SHA remains ac03b1913bfb6dd9b978477fc23ee5c88e276f67.

kayg commented

2026-10-07 12:48:21 +00:00

Author

Owner

Blocked on the required runtime benchmark runs: SSH to perf-test-69-63 fails with Host key verification failed, and ssh-keygen reports no trusted known_hosts entry. I will not bypass host-key verification. Needed from kayg: confirm the expected perf VM host key fingerprint or repair the trusted known_hosts entry, then I can run the existing benches under scripts/perf-run. Compilation measurements remain valid; local latency smoke and code review can continue.

kayg commented

2026-10-07 13:08:33 +00:00

Author

Owner

Pre-change scripts/verify fast completed under CPUWeight=10000. Latency smoke output: METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=294.96 p99_ms=295.46 result=PASS. Static sizes: x86_64 16142672 bytes; aarch64 13555672 bytes. Gate summary: fmt PASS; VT native fmt PASS; clippy PASS; test PASS; cargo-deny PASS; static builds PASS. Full gate log is at $HOME/.cache/codex-scratch/build-speed/logs/verify-before-fast.log. SHA remains ac03b1913b.

Pre-change scripts/verify fast completed under CPUWeight=10000. Latency smoke output: METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=294.96 p99_ms=295.46 result=PASS. Static sizes: x86_64 16142672 bytes; aarch64 13555672 bytes. Gate summary: fmt PASS; VT native fmt PASS; clippy PASS; test PASS; cargo-deny PASS; static builds PASS. Full gate log is at $HOME/.cache/codex-scratch/build-speed/logs/verify-before-fast.log. SHA remains ac03b1913bfb6dd9b978477fc23ee5c88e276f67.

kayg commented

2026-10-07 13:38:46 +00:00

Author

Owner

Post-mold build batch completed. cargo build --locked: clean median 242.983 s (250.571 / 158.894 / 242.983), server-touch 4.471 s (4.853 / 3.719 / 4.471), protocol-touch 5.586 s (5.624 / 5.300 / 5.586). Baselines were 151.268 s, 2.353 s, and 2.375 s. cargo test --locked --no-run: clean median 259.736 s (229.374 / 297.709 / 259.736), server-touch 10.361 s (10.361 / 10.675 / 9.183), protocol-touch 10.750 s (11.703 / 10.750 / 9.026); baselines were 173.071 s, 4.202 s, and 4.687 s. Post runs had one-minute load up to 37.27 versus baseline up to 32.85, so this does not establish a linker regression or speedup. I am verifying the selected linker path before deciding the commit.

kayg commented

2026-10-07 13:46:22 +00:00

Author

Owner

Cargo --verbose confirmed native x86_64 GNU rustc invocations use .cargo/mold-cc.sh, and the package build completed successfully in 4m25s under the high-load guest. The wrapper invokes cc with -fuse-ld=mold when mold is installed and falls back to the system linker when absent; static musl targets retain .cargo/rust-lld.sh.

kayg commented

2026-10-07 14:05:35 +00:00

Author

Owner

Native linker step committed as ea48642. Cargo verbose compilation confirmed .cargo/mold-cc.sh is selected for native x86_64 GNU; the musl builds keep rust-lld. Static sizes stayed at x86_64 16,142,672 bytes and aarch64 13,555,672 bytes. Before/after cargo build medians (seconds): clean 151.268/242.983, server touch 2.353/4.471, protocol touch 2.375/5.586. cargo test --no-run: clean 173.071/259.736, server touch 4.202/10.361, protocol touch 4.687/10.750. After runs had load up to 37 versus 33 before, so these do not establish a mold speedup. Latency smoke before/after p50/p99 was 294.96/295.46 ms and 297.77/323.81 ms, both PASS; pinned runtime benches remain blocked on the unverified perf VM host key.

Native linker step committed as ea48642. Cargo verbose compilation confirmed .cargo/mold-cc.sh is selected for native x86_64 GNU; the musl builds keep rust-lld. Static sizes stayed at x86_64 16,142,672 bytes and aarch64 13,555,672 bytes. Before/after cargo build medians (seconds): clean 151.268/242.983, server touch 2.353/4.471, protocol touch 2.375/5.586. cargo test --no-run: clean 173.071/259.736, server touch 4.202/10.361, protocol touch 4.687/10.750. After runs had load up to 37 versus 33 before, so these do not establish a mold speedup. Latency smoke before/after p50/p99 was 294.96/295.46 ms and 297.77/323.81 ms, both PASS; pinned runtime benches remain blocked on the unverified perf VM host key. scripts/verify fast Rust gate output: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=297.77 p99_ms=323.81 result=PASS

kayg commented

2026-10-07 14:06:53 +00:00

Author

Owner

Size gate implementation is in progress. The budget helper now accepts the measured x86_64 and aarch64 static baselines with a 262,144-byte margin. A synthetic x86_64 static PIE at 16,404,817 bytes was rejected against the 16,404,816-byte maximum; both real baseline binaries remain available under the scratch target for the full gate.

kayg commented

2026-10-07 14:27:24 +00:00

Author

Owner

Finished implementation at head 2de8da2aedc72aa96c565bf0aaf42dc9d9e09c97. The static size guard rejects an x86_64 artifact at 16,404,817 bytes and accepts both measured release artifacts; docs/PROCESS.md records the limits and build paths. No dependency versions, feature sets, or release profile settings changed: the four duplicate crate families are transitive, and runtime tuning awaits the pinned perf VM host key.

Final scripts/verify fast output, verbatim:
METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=287.68 p99_ms=307.67 result=PASS
PT_INTERP: absent; size: 16142672 bytes (baseline 16142672 + allowance 262144 = max 16404816)
PT_INTERP: absent; size: 13555672 bytes (baseline 13555672 + allowance 262144 = max 13817816)
+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| fmt | PASS |
| VT native fmt | PASS |
| clippy | PASS |
| test | PASS |
| cargo-deny | PASS |
| static builds | PASS |
+---------------------------+--------------+
stream differential: 134 streams, 11439 schedules, zero divergences
7 pass
0 fail
34390 expect() calls

Finished implementation at head 2de8da2aedc72aa96c565bf0aaf42dc9d9e09c97. The static size guard rejects an x86_64 artifact at 16,404,817 bytes and accepts both measured release artifacts; docs/PROCESS.md records the limits and build paths. No dependency versions, feature sets, or release profile settings changed: the four duplicate crate families are transitive, and runtime tuning awaits the pinned perf VM host key. Final scripts/verify fast output, verbatim: METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=287.68 p99_ms=307.67 result=PASS PT_INTERP: absent; size: 16142672 bytes (baseline 16142672 + allowance 262144 = max 16404816) PT_INTERP: absent; size: 13555672 bytes (baseline 13555672 + allowance 262144 = max 13817816) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ stream differential: 134 streams, 11439 schedules, zero divergences 7 pass 0 fail 34390 expect() calls

kayg commented

2026-10-07 14:37:12 +00:00

Author

Owner

Orchestrator measurement (2026-10-07 ~16:50): incremental cargo build -p infraternal after touching infraternal-server/src/lib.rs, alternating mold (ea48642) vs cc, 5 runs each under CPUWeight=10000, load 26-34: mold 27.97/30.93/29.99/27.49/29.03 s (median 29.03); cc 34.83/26.15/29.35/33.22/28.04 s (median 29.35). No measurable gain: compile time dominates link time on this host and load. ea48642 (mold) is not merged. The size-budget commit 2de8da2 is taken. The bigger lever for build time is fewer concurrent jobs and sccache hit rate; a later measurement on a quiet host may revisit mold.

Orchestrator measurement (2026-10-07 ~16:50): incremental `cargo build -p infraternal` after touching infraternal-server/src/lib.rs, alternating mold (ea48642) vs cc, 5 runs each under CPUWeight=10000, load 26-34: mold 27.97/30.93/29.99/27.49/29.03 s (median 29.03); cc 34.83/26.15/29.35/33.22/28.04 s (median 29.35). No measurable gain: compile time dominates link time on this host and load. **ea48642 (mold) is not merged.** The size-budget commit 2de8da2 is taken. The bigger lever for build time is fewer concurrent jobs and sccache hit rate; a later measurement on a quiet host may revisit mold.

kayg referenced this issue

2026-10-07 14:48:03 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-07 14:48:03 +00:00

Bug: stale setup-token.sock remains after owner enrollment #51

kayg referenced this issue

2026-10-07 14:48:04 +00:00

Bug: themes don't apply to the terminal in production #75

kayg referenced this issue

2026-10-07 21:35:02 +00:00

SSH bootstrap: add a real host end to end (demo, D138-D141) #87

kayg referenced this issue from a commit

2026-10-10 09:20:35 +00:00

build: enforce static binary size budgets (#57)

kayg referenced this issue

2026-10-11 20:11:27 +00:00

Add-host bootstrap fails on a NetBird-SSH-only host; failure step not logged #169

kayg referenced this issue

2026-10-11 20:20:12 +00:00

Add-host bootstrap fails on a NetBird-SSH-only host; failure step not logged #169

kayg referenced this issue from a commit

2026-10-11 20:33:11 +00:00

fix(verify): measure static allowance against shipped production

kayg referenced this issue

2026-10-11 21:18:46 +00:00

Release pipeline: signed builds on builds.infraternal.com, verified payloads, add-host by architecture (D220, D224, D225) #157

kayg referenced this issue from a commit

2026-10-11 21:22:07 +00:00

chore(verify): use measured static size baselines

kayg referenced this issue

2026-10-12 05:20:58 +00:00

Size reduction push (D243): static binary only goes down #175

kayg referenced this issue

2026-10-12 05:44:39 +00:00

Size reduction push (D243): static binary only goes down #175

kayg referenced this issue

2026-10-12 06:07:17 +00:00

Size reduction push (D243): static binary only goes down #175

kayg referenced this issue

2026-10-12 09:56:50 +00:00

Size reduction push (D243): static binary only goes down #175

kayg referenced this issue

2026-10-13 16:51:39 +00:00

Cheaper gate iterations, same final strictness #204

kayg referenced this issue from a commit

2026-10-15 14:31:54 +00:00

docs(process): require measured downward static ratchets (#198)

kayg commented

2026-10-28 17:52:30 +00:00

Author

Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Size-budget commit 2de8da2 taken and measured-ratchet follow-ups landed (4b2cd2ad4, e273fd309 in dev); mold measured as no gain and deliberately not merged, sccache set up. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Size-budget commit 2de8da2 taken and measured-ratchet follow-ups landed (4b2cd2ad4, e273fd309 in dev); mold measured as no gain and deliberately not merged, sccache set up. Reopen if anything is still missing.

kayg closed this issue

2026-10-28 17:52:30 +00:00

~~kayg referenced this issue 2026-10-28 17:53:54 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

Rows
Columns

Build speed and binary size: sccache, linker, profiles, size budget #57