BUILD: measure incremental, shared sccache and a fast linker for job builds

kayg commented

2026-10-03 11:26:29 +00:00

Owner

Owner question (2026-10-03)

"Are we using this to speed up our builds?" (LWN article on Rust's query-based incremental compilation.)

Facts today

Codex jobs run with CARGO_INCREMENTAL=0 (launcher header) to save disk, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_BUILD_JOBS=3, one target dir per job under /mnt/hdd/targets/jobs/ (HDD). So every job compiles all ~600 dependencies from scratch, and every edit-rebuild cycle inside a job redoes whole crates. 20+ jobs run at once on a 12-core / 94 GB VM. Server links are slow (minutes).
sccache is installed (/home/kayg/.nix-profile/bin/sccache) but unused. No mold/wild/clang on the host.

On the build VM, quiet as possible (note load), measure for calternal-server + one plugin crate:
A. baseline (today's settings): cold build, then 3 edit→rebuild cycles (touch a leaf plugin file, a mid-level crate, calternal-server main);
B. CARGO_INCREMENTAL=1 (dev profile): same cycles; record target dir size;
C. shared sccache (RUSTC_WRAPPER, one cache dir on the NVMe /home or SSD, size cap) across two separate target dirs: second job's cold build time = the cross-job win;
D. a faster linker (mold or wild via nix, -C link-arg=-fuse-ld=... or linker in a job-local config): link time of calternal-server;
E. combinations B+C+D, and C+D with incremental off.
Also measure peak RSS and disk per job (the VM OOM'd today at ~25 jobs; disk is the other constraint).
Decision rule (post it on this issue first): pick the combination with the lowest total wall time for "cold build + 3 rebuilds" whose per-job disk ≤ 1.5x baseline and peak RSS ≤ 1.2x baseline; shared-cache wins are counted per additional concurrent job.

Output

A table and a concrete patch to the job launcher/preamble (env vars, cache dir, linker config), written to a file in the issue, NOT applied: the orchestrator applies it. Note AGPL-compatibility of any tool added (mold MIT, wild MIT/Apache, sccache Apache-2.0).

## Owner question (2026-10-03) "Are we using this to speed up our builds?" (LWN article on Rust's query-based incremental compilation.) ## Facts today - Codex jobs run with `CARGO_INCREMENTAL=0` (launcher header) to save disk, `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_BUILD_JOBS=3`, one target dir per job under /mnt/hdd/targets/jobs/<job> (HDD). So every job compiles all ~600 dependencies from scratch, and every edit-rebuild cycle inside a job redoes whole crates. 20+ jobs run at once on a 12-core / 94 GB VM. Server links are slow (minutes). - `sccache` is installed (/home/kayg/.nix-profile/bin/sccache) but unused. No mold/wild/clang on the host. ## Task: measure, then recommend (decision rule written before the runs) On the build VM, quiet as possible (note load), measure for `calternal-server` + one plugin crate: A. baseline (today's settings): cold build, then 3 edit→rebuild cycles (touch a leaf plugin file, a mid-level crate, calternal-server main); B. `CARGO_INCREMENTAL=1` (dev profile): same cycles; record target dir size; C. shared `sccache` (RUSTC_WRAPPER, one cache dir on the NVMe /home or SSD, size cap) across two separate target dirs: second job's cold build time = the cross-job win; D. a faster linker (mold or wild via nix, `-C link-arg=-fuse-ld=...` or `linker` in a job-local config): link time of calternal-server; E. combinations B+C+D, and C+D with incremental off. Also measure peak RSS and disk per job (the VM OOM'd today at ~25 jobs; disk is the other constraint). Decision rule (post it on this issue first): pick the combination with the lowest total wall time for "cold build + 3 rebuilds" whose per-job disk ≤ 1.5x baseline and peak RSS ≤ 1.2x baseline; shared-cache wins are counted per additional concurrent job. ## Output A table and a concrete patch to the job launcher/preamble (env vars, cache dir, linker config), written to a file in the issue, NOT applied: the orchestrator applies it. Note AGPL-compatibility of any tool added (mold MIT, wild MIT/Apache, sccache Apache-2.0).

kayg commented

2026-10-03 11:33:10 +00:00

Author

Owner

Decision rule (recorded before any benchmark runs): for each treatment, score one job as the wall time for its cold build plus three source-touch rebuilds. A treatment is eligible only if its target-directory size is at most 1.5× baseline and its peak aggregate process-tree RSS is at most 1.2× baseline. Choose the eligible treatment with the lowest score; if none is eligible, recommend today's settings. For shared sccache, separately compare a second fresh target directory's cold build with the uncached baseline cold build, and count that saved wall time once per additional concurrent job. Report the two-target cache result separately so it does not hide a per-job limit failure.

Protocol: use the same checkout and build command for calternal-server plus calternal-plugin-files; touch one file in that plugin, one direct dependency crate, then calternal-server's main source. Run one interleaved A–E pass, keep CARGO_BUILD_JOBS=3 and debug info at line-tables-only, record wall time, peak aggregate RSS, target bytes, and host load for each build. Keep all Cargo outputs under this job's assigned target root; do not edit the launcher.

Starting state: branch job/buildspeed-1007, base SHA f06679b11c. At 2026-10-03 13:28 UTC, host load average was 34.33 / 30.31 / 32.18 on 12 CPUs; 75 GiB memory was available. I will label measurements with the observed load.

Decision rule (recorded before any benchmark runs): for each treatment, score one job as the wall time for its cold build plus three source-touch rebuilds. A treatment is eligible only if its target-directory size is at most 1.5× baseline and its peak aggregate process-tree RSS is at most 1.2× baseline. Choose the eligible treatment with the lowest score; if none is eligible, recommend today's settings. For shared sccache, separately compare a second fresh target directory's cold build with the uncached baseline cold build, and count that saved wall time once per additional concurrent job. Report the two-target cache result separately so it does not hide a per-job limit failure. Protocol: use the same checkout and build command for calternal-server plus calternal-plugin-files; touch one file in that plugin, one direct dependency crate, then calternal-server's main source. Run one interleaved A–E pass, keep CARGO_BUILD_JOBS=3 and debug info at line-tables-only, record wall time, peak aggregate RSS, target bytes, and host load for each build. Keep all Cargo outputs under this job's assigned target root; do not edit the launcher. Starting state: branch job/buildspeed-1007, base SHA f06679b11cde29cc0b7120fdab5f721389caf695. At 2026-10-03 13:28 UTC, host load average was 34.33 / 30.31 / 32.18 on 12 CPUs; 75 GiB memory was available. I will label measurements with the observed load.

kayg commented

2026-10-03 11:59:57 +00:00

Author

Owner

Measurement setup finding: the initial comment mislabeled the host's 13:28 reading as UTC. It came from uptime local time; host date reports Europe/Berlin (+02:00). Current sample: 2026-10-03 13:56:28 +02:00, load 39.62 / 40.04 / 37.09.

The live command environment also differs from the issue facts: RUSTC_WRAPPER=/home/kayg/.nix-profile/bin/sccache, SCCACHE_DIR=/srv/calternal/cache/sccache, and SCCACHE_CACHE_SIZE=40G are already set. sccache --show-stats reports 7,797 requests, 5,537 hits, and a 40 GiB max. /srv and /home share /dev/sdb1 (reported non-rotational), so the cache is on the same SSD-backed filesystem. A separate server attempt returned Server startup failed: Address in use; I will not stop, clear, or resize the active shared cache.

For A and B I will explicitly unset RUSTC_WRAPPER to measure the stated launcher baseline (incremental off/on). C and the combined treatments will use the existing shared cache. The cache is already warm, so these are cold Cargo target dirs with a warm shared cache; I will label that distinction.

Measurement setup finding: the initial comment mislabeled the host's 13:28 reading as UTC. It came from `uptime` local time; host `date` reports Europe/Berlin (+02:00). Current sample: 2026-10-03 13:56:28 +02:00, load 39.62 / 40.04 / 37.09. The live command environment also differs from the issue facts: `RUSTC_WRAPPER=/home/kayg/.nix-profile/bin/sccache`, `SCCACHE_DIR=/srv/calternal/cache/sccache`, and `SCCACHE_CACHE_SIZE=40G` are already set. `sccache --show-stats` reports 7,797 requests, 5,537 hits, and a 40 GiB max. `/srv` and `/home` share `/dev/sdb1` (reported non-rotational), so the cache is on the same SSD-backed filesystem. A separate server attempt returned `Server startup failed: Address in use`; I will not stop, clear, or resize the active shared cache. For A and B I will explicitly unset `RUSTC_WRAPPER` to measure the stated launcher baseline (incremental off/on). C and the combined treatments will use the existing shared cache. The cache is already warm, so these are cold Cargo target dirs with a warm shared cache; I will label that distinction.

kayg referenced this issue

2026-10-03 12:16:12 +00:00

CANVAS core: Excalidraw canvas as a Note, live co-editing, one event path, chrome, element links, export (§60) #976

kayg commented

2026-10-03 12:28:56 +00:00

Author

Owner

Live measurement finding (D, mold, cold build): the server binary reached the link stage. The process chain was cc → collect2 → ld.mold; at 14:27:10 +02:00, the final ld.mold process had been active 1:11, was in Linux state D, and held 805,252 KiB RSS. The build target had reached 3.6 GiB. At that sample, load was 56.40 / 43.35 / 39.75. The job target is on /mnt/hdd (/dev/sda, rotational flag 1). I will report the completed build wall time and peak process RSS after this candidate finishes.

Live measurement finding (D, mold, cold build): the server binary reached the link stage. The process chain was `cc → collect2 → ld.mold`; at 14:27:10 +02:00, the final `ld.mold` process had been active 1:11, was in Linux state `D`, and held 805,252 KiB RSS. The build target had reached 3.6 GiB. At that sample, load was 56.40 / 43.35 / 39.75. The job target is on `/mnt/hdd` (`/dev/sda`, rotational flag 1). I will report the completed build wall time and peak process RSS after this candidate finishes.

kayg commented

2026-10-03 12:30:18 +00:00

Author

Owner

Completed measurement D (mold only, incremental off, sccache off), cold Cargo target: wall 1347.834 s; peak aggregate RSS for the Cargo process group 3,312,328 KiB; target directory 4,195,454,976 bytes. Load went from 42.25 / 41.31 / 38.53 to 47.71 / 47.40 / 42.07 (1 / 5 / 15 minute averages). The server build succeeded. The earlier ld.mold process was observed in state D for at least 1:11; the completed build wall includes compile and all links.

Completed measurement D (mold only, incremental off, sccache off), cold Cargo target: wall `1347.834 s`; peak aggregate RSS for the Cargo process group `3,312,328 KiB`; target directory `4,195,454,976 bytes`. Load went from `42.25 / 41.31 / 38.53` to `47.71 / 47.40 / 42.07` (1 / 5 / 15 minute averages). The server build succeeded. The earlier `ld.mold` process was observed in state `D` for at least 1:11; the completed build wall includes compile and all links.

kayg referenced this issue

2026-10-03 12:49:08 +00:00

CANVAS core: Excalidraw canvas as a Note, live co-editing, one event path, chrome, element links, export (§60) #976

kayg commented

2026-10-03 13:52:26 +00:00

Author

Owner

Completed C (shared sccache, incremental off, default linker), cold Cargo target with the host cache already warm:

wall: 4818.230 s (Cargo reported 80m 06s)
peak per-job RSS (Cargo plus compiler/linker processes tagged to this target): 2,518,156 KiB
target directory: 4,071,424,000 bytes
load before: 47.71 / 47.40 / 42.07; after: 27.27 / 31.01 / 34.61 (1 / 5 / 15 minute averages)

The build succeeded. The default server link driver was active for at least 4:14 during this build. D (mold only) took 1347.834 s; the conditions and load differ, so this is a signal to compare against A and the combined runs, not an isolated sccache/linker effect. C2 follows immediately to measure the second fresh target against this now-warmed cache.

Completed C (shared sccache, incremental off, default linker), cold Cargo target with the host cache already warm: - wall: `4818.230 s` (Cargo reported `80m 06s`) - peak per-job RSS (Cargo plus compiler/linker processes tagged to this target): `2,518,156 KiB` - target directory: `4,071,424,000 bytes` - load before: `47.71 / 47.40 / 42.07`; after: `27.27 / 31.01 / 34.61` (1 / 5 / 15 minute averages) The build succeeded. The default server link driver was active for at least `4:14` during this build. D (mold only) took `1347.834 s`; the conditions and load differ, so this is a signal to compare against A and the combined runs, not an isolated sccache/linker effect. C2 follows immediately to measure the second fresh target against this now-warmed cache.

kayg commented

2026-10-03 14:19:00 +00:00

Author

Owner

C2 finding — the immediately following fresh target directory, sharing C's sccache cache, completed successfully in 1,651.245 s (27:31.245), peak measured job RSS 2,201,048 KiB, target directory 4,071,399,424 bytes. Load average (1/5/15m) was 27.27 / 31.01 / 34.61 at start and 29.82 / 29.74 / 29.85 at end. Against C's cold result (4,818.230 s, 2,518,156 KiB, 4,071,424,000 bytes), this second target saved 3,166.985 s (52:46.985, 65.7%) and used essentially the same disk. This is a measured cross-target saving; the cache is shared with other host jobs and was not reset, so it is not an isolated attribution to this job.

kayg commented

2026-10-03 14:35:38 +00:00

Author

Owner

A / E1 finding:

A baseline cold build succeeded in 927.065 s (15:27.065), peak measured job RSS 2,730,780 KiB, target directory 4,071,415,808 bytes. Load average (1/5/15m) was 29.82 / 29.74 / 29.85 at start and 26.47 / 25.93 / 27.85 at end.

E1 (incremental + sccache + mold) cannot run with the available sccache path: it exited in 1.094 s before compiling, with sccache: incremental compilation is prohibited: Unset CARGO_INCREMENTAL to continue. The current upstream Rust docs also state that rustc incremental compilation needs to be disabled for sccache: https://github.com/mozilla/sccache/blob/main/docs/Rust.md . I am excluding E1 from timings rather than treating this setup error as a build measurement. B and E0 remain measurable.

A / E1 finding: A baseline cold build succeeded in 927.065 s (15:27.065), peak measured job RSS 2,730,780 KiB, target directory 4,071,415,808 bytes. Load average (1/5/15m) was 29.82 / 29.74 / 29.85 at start and 26.47 / 25.93 / 27.85 at end. E1 (incremental + sccache + mold) cannot run with the available sccache path: it exited in 1.094 s before compiling, with `sccache: incremental compilation is prohibited: Unset CARGO_INCREMENTAL to continue.` The current upstream Rust docs also state that rustc incremental compilation needs to be disabled for sccache: https://github.com/mozilla/sccache/blob/main/docs/Rust.md . I am excluding E1 from timings rather than treating this setup error as a build measurement. B and E0 remain measurable.

kayg commented

2026-10-03 14:41:44 +00:00

Author

Owner

Threshold finding from the completed cold builds: D (mold alone) used 4,195,454,976 target bytes versus A's 4,071,415,808 (1.0305x, within the 1.5x disk cap), but its 3,312,328 KiB peak RSS was 1.2130x A's 2,730,780 KiB, 35,392 KiB above the 1.2x cap. So D alone cannot qualify under the posted decision rule, even before its rebuild timings. D ran at higher host load than A (42.25/41.31/38.53 at start versus 29.82/29.74/29.85); the strict RSS result is reported without load normalization. E0 may still qualify because shared-cache hits reduce concurrent compilation memory.

kayg commented

2026-10-03 14:56:21 +00:00

Author

Owner

B cold finding: the incremental-on target build succeeded in 1,148.804 s (19:08.804), peak measured job RSS 2,539,492 KiB, target directory 5,725,495,296 bytes. Load average (1/5/15m) was 26.35 / 25.89 / 27.56 at start and 34.48 / 30.19 / 29.27 at end. Against A cold, B took 221.739 s longer, used 0.930x the peak RSS, and occupied 1.406x the target disk. The cold directory is under the 1.5x limit, but only 381,628,416 bytes of headroom remain for incremental rebuild growth; I will check the size after each source touch.

kayg commented

2026-10-03 15:23:31 +00:00

Author

Owner

E0 cold finding (sccache + mold, incremental off): success in 1,480.337 s (24:40.337), peak measured job RSS 2,901,660 KiB, target directory 4,195,454,976 bytes. Load average (1/5/15m) was 34.48 / 30.19 / 27.56 at start and 18.23 / 22.61 / 25.14 at end. Against A cold, E0 took 553.272 s longer; its cold peak RSS is 1.063x A and disk is 1.0305x A, both inside the posted caps. The rebuild timings and final target size are still required to score E0.

kayg commented

2026-10-03 15:24:48 +00:00

Author

Owner

B plugin-rebuild finding: the first edit→rebuild (touching crates/plugins/files/src/lookup.rs) took 192.554 s, peaked at 2,757,724 KiB RSS, and left the target directory at 6,320,021,504 bytes. A's 1.5x disk limit is 6,107,123,712 bytes, so B is already 212,897,792 bytes over the cap after one of three rebuilds (1.552x A). Its RSS is still below the 1.2x limit. I will keep the remaining B runs in the interleaved schedule for their wall-time record, but B cannot qualify under the posted rule.

B plugin-rebuild finding: the first edit→rebuild (touching `crates/plugins/files/src/lookup.rs`) took 192.554 s, peaked at 2,757,724 KiB RSS, and left the target directory at 6,320,021,504 bytes. A's 1.5x disk limit is 6,107,123,712 bytes, so B is already 212,897,792 bytes over the cap after one of three rebuilds (1.552x A). Its RSS is still below the 1.2x limit. I will keep the remaining B runs in the interleaved schedule for their wall-time record, but B cannot qualify under the posted rule.

kayg commented

2026-10-03 15:28:39 +00:00

Author

Owner

E0 plugin-rebuild finding: the first edit cycle completed in 226.258 s, peak measured job RSS 3,103,308 KiB, target directory 4,195,446,784 bytes. The directory stayed effectively flat and remains within A's 1.5x cap. RSS is 1.136x A, still below the 1.2x cap by 173,628 KiB; later cycles could still disqualify it. The build and link finished successfully.

kayg commented

2026-10-03 15:35:03 +00:00

Author

Owner

A plugin-rebuild finding: touching crates/plugins/files/src/lookup.rs completed in 402.350 s, peak measured job RSS 2,605,972 KiB, target directory 4,071,415,808 bytes (unchanged at reported precision). Load average (1/5/15m) was as recorded in A-rebuild-plugin.log's result row.

The default server link's collect2 process ended at 17:33:34.160+02:00. A process sample at 17:31:56 showed that same PID at 00:23 elapsed, implying about 121 seconds for the complete driver interval (roughly ±1 s from the sampled start). The dedicated observer attached late and saw only 39.217 seconds, so 121 seconds is an estimate, not a full observer interval.

A plugin-rebuild finding: touching `crates/plugins/files/src/lookup.rs` completed in 402.350 s, peak measured job RSS 2,605,972 KiB, target directory 4,071,415,808 bytes (unchanged at reported precision). Load average (1/5/15m) was as recorded in `A-rebuild-plugin.log`'s result row. The default server link's `collect2` process ended at 17:33:34.160+02:00. A process sample at 17:31:56 showed that same PID at 00:23 elapsed, implying about 121 seconds for the complete driver interval (roughly ±1 s from the sampled start). The dedicated observer attached late and saw only 39.217 seconds, so 121 seconds is an estimate, not a full observer interval.

kayg commented

2026-10-03 15:35:23 +00:00

Author

Owner

Correction to my previous comment: the load samples are in the measurement result row, not the Cargo log. For A's plugin rebuild, load average (1/5/15m) was 21.62 / 18.98 / 22.11 at start and 25.68 / 23.46 / 23.05 at end.

kayg commented

2026-10-03 15:39:00 +00:00

Author

Owner

D plugin-rebuild finding: the first edit cycle completed in 202.199 s, peak measured job RSS 3,062,864 KiB, target directory 4,195,450,880 bytes. Load average (1/5/15m) was 25.68 / 23.46 / 23.05 at start and 14.41 / 18.31 / 21.07 at end. The server collect2/mold driver interval was 34.157 s, observed from process appearance to exit (±1 s sampling), versus A's estimated ~121 s default driver. D still cannot qualify because its earlier cold peak RSS exceeded the posted cap.

D plugin-rebuild finding: the first edit cycle completed in 202.199 s, peak measured job RSS 3,062,864 KiB, target directory 4,195,450,880 bytes. Load average (1/5/15m) was 25.68 / 23.46 / 23.05 at start and 14.41 / 18.31 / 21.07 at end. The server `collect2`/mold driver interval was 34.157 s, observed from process appearance to exit (±1 s sampling), versus A's estimated ~121 s default driver. D still cannot qualify because its earlier cold peak RSS exceeded the posted cap.

kayg commented

2026-10-03 15:42:44 +00:00

Author

Owner

C plugin-rebuild finding: the first edit cycle completed in 221.040 s, peak measured job RSS 2,929,072 KiB, target directory 4,071,424,000 bytes. Load average (1/5/15m) was 14.41 / 18.31 / 21.07 at start and 20.02 / 18.27 / 20.33 at end. The server collect2 driver interval was 64.327 s. The target size remained effectively flat.

C plugin-rebuild finding: the first edit cycle completed in 221.040 s, peak measured job RSS 2,929,072 KiB, target directory 4,071,424,000 bytes. Load average (1/5/15m) was 14.41 / 18.31 / 21.07 at start and 20.02 / 18.27 / 20.33 at end. The server `collect2` driver interval was 64.327 s. The target size remained effectively flat.

kayg commented

2026-10-03 15:44:22 +00:00

Author

Owner

C middle-rebuild finding: touching crates/calternal-plugin/src/lib.rs completed in 152.587 s, peak measured job RSS 2,954,940 KiB, target directory 4,071,424,000 bytes. Load average (1/5/15m) was 20.02 / 18.27 / 20.33 at start and 18.64 / 18.37 / 20.06 at end. The target size remained effectively flat.

C middle-rebuild finding: touching `crates/calternal-plugin/src/lib.rs` completed in 152.587 s, peak measured job RSS 2,954,940 KiB, target directory 4,071,424,000 bytes. Load average (1/5/15m) was 20.02 / 18.27 / 20.33 at start and 18.64 / 18.37 / 20.06 at end. The target size remained effectively flat.

kayg commented

2026-10-03 15:49:00 +00:00

Author

Owner

D middle-rebuild finding: touching crates/calternal-plugin/src/lib.rs completed in 253.135 s, peak measured job RSS 3,269,224 KiB, target directory 4,195,446,784 bytes. Load average (1/5/15m) was 18.64 / 18.37 / 20.06 at start and 23.04 / 18.95 / 19.69 at end. The collect2/mold server link interval was 45.011 s. This cycle's RSS is 7,712 KiB below the 1.2x threshold, but D's cold build already exceeded that threshold.

D middle-rebuild finding: touching `crates/calternal-plugin/src/lib.rs` completed in 253.135 s, peak measured job RSS 3,269,224 KiB, target directory 4,195,446,784 bytes. Load average (1/5/15m) was 18.64 / 18.37 / 20.06 at start and 23.04 / 18.95 / 19.69 at end. The `collect2`/mold server link interval was 45.011 s. This cycle's RSS is 7,712 KiB below the 1.2x threshold, but D's cold build already exceeded that threshold.

kayg commented

2026-10-03 15:49:12 +00:00

Author

Owner

Link-time update: the process observer recorded C's calternal-server collect2/system-linker interval at 64.327 s in the plugin rebuild and 29.438 s in the middle rebuild. A's default-linker plugin rebuild remained an estimate (~121 s) because the observer attached after that process had started. These are edit-cycle links under the current mixed host load, not isolated linker-only microbenchmarks.

Link-time update: the process observer recorded C's `calternal-server` `collect2`/system-linker interval at 64.327 s in the plugin rebuild and 29.438 s in the middle rebuild. A's default-linker plugin rebuild remained an estimate (~121 s) because the observer attached after that process had started. These are edit-cycle links under the current mixed host load, not isolated linker-only microbenchmarks.

kayg commented

2026-10-03 15:53:21 +00:00

Author

Owner

B middle-rebuild finding: touching crates/calternal-plugin/src/lib.rs completed in 262.318 s, peak measured job RSS 2,395,820 KiB, target directory 6,641,217,536 bytes (1.631x A). This is 534,093,824 bytes above A's 1.5x disk cap. The server collect2/system-linker interval was 69.418 s. Load average (1/5/15m) was 23.04 / 18.95 / 19.72 at start and 43.26 / 30.59 / 24.19 at end; the host became heavily loaded during this run.

B middle-rebuild finding: touching `crates/calternal-plugin/src/lib.rs` completed in 262.318 s, peak measured job RSS 2,395,820 KiB, target directory 6,641,217,536 bytes (1.631x A). This is 534,093,824 bytes above A's 1.5x disk cap. The server `collect2`/system-linker interval was 69.418 s. Load average (1/5/15m) was 23.04 / 18.95 / 19.72 at start and 43.26 / 30.59 / 24.19 at end; the host became heavily loaded during this run.

kayg referenced this issue

2026-10-03 15:59:17 +00:00

CANVAS core: Excalidraw canvas as a Note, live co-editing, one event path, chrome, element links, export (§60) #976

kayg commented

2026-10-03 16:02:27 +00:00

Author

Owner

A middle-rebuild finding: touching crates/calternal-plugin/src/lib.rs completed in 522.619 s, peak measured job RSS 2,351,928 KiB, target directory 4,071,415,808 bytes. The server collect2/system-linker interval was 70.926 s. Load average (1/5/15m) was 42.80 / 30.07 / 23.96 at start and 34.04 / 36.74 / 30.71 at end. A's cold + plugin + middle wall total is 1,852.034 s; the server-source rebuild is still unmeasured.

A middle-rebuild finding: touching `crates/calternal-plugin/src/lib.rs` completed in 522.619 s, peak measured job RSS 2,351,928 KiB, target directory 4,071,415,808 bytes. The server `collect2`/system-linker interval was 70.926 s. Load average (1/5/15m) was 42.80 / 30.07 / 23.96 at start and 34.04 / 36.74 / 30.71 at end. A's cold + plugin + middle wall total is 1,852.034 s; the server-source rebuild is still unmeasured.

kayg commented

2026-10-03 16:15:17 +00:00

Author

Owner

Final report — #1007 measurement job

Decision rule was posted before the first run: choose the lowest wall-time candidate for a cold build plus three rebuilds, only if target disk is at most 1.5× A and peak RSS is at most 1.2× A. The matrix ran past the four-hour time box before the server-source rebuild round. I stopped it; no complete cold-plus-three total exists, so I cannot select a winner under that rule. Keep the current launcher baseline for now. The patch proposal is at the end and is not applied.

Method and host

Each measured command was cargo build --locked --timings -p calternal-server -p calternal-plugin-files, with the assigned CARGO_TARGET_DIR preserved, CARGO_PROFILE_DEV_DEBUG=line-tables-only, and CARGO_BUILD_JOBS=3. The rebuild touches were crates/plugins/files/src/lookup.rs, crates/calternal-plugin/src/lib.rs, then crates/calternal-server/src/main.rs. They changed mtimes only, as specified by the probe.

Each treatment used its own target directory under the assigned target root. SCCACHE_DIR=/srv/calternal/cache/sccache was on the SSD and had a 40G cap; the cache was shared with other host jobs and was not reset. Build targets were on /mnt/hdd. RSS was sampled once per second; target bytes are du sizes after each completed command. Recorded load samples across the matrix ranged 10.81–47.71 (1m), 17.57–47.40 (5m), and 19.72–42.07 (15m). This was not a quiet-host run.

Measurements

Times are seconds; RSS is KiB; disk is bytes. “Partial total” is cold + leaf touch + middle touch. The required server-source touch is missing for every candidate.

Candidate	Cold: time / RSS / disk	Leaf rebuild: time / disk	Middle rebuild: time / disk	Partial total	Status
A baseline	927.065 / 2,730,780 / 4,071,415,808	402.350 / 4,071,415,808	522.619 / 4,071,415,808	1,852.034	Within caps so far
B incremental	1,148.804 / 2,539,492 / 5,725,495,296	192.554 / 6,320,021,504	262.318 / 6,641,217,536	1,603.676	Fails disk cap after first rebuild
C sccache	4,818.230 / 2,518,156 / 4,071,424,000	221.040 / 4,071,424,000	152.587 / 4,071,424,000	5,191.857	Within caps so far
D mold	1,347.834 / 3,312,328 / 4,195,454,976	202.199 / 4,195,450,880	253.135 / 4,195,446,784	1,803.168	Fails RSS cap on cold build
E0 sccache + mold, incremental off	1,480.337 / 2,901,660 / 4,195,454,976	226.258 / 4,195,446,784	289.469 / 4,195,446,784	1,996.064	Within caps so far

A’s limits are 6,107,123,712 bytes of target disk and 3,276,936 KiB RSS. B exceeded the disk limit by 212,897,792 bytes after its leaf rebuild, and by 534,093,824 bytes after the middle rebuild. D’s cold RSS was 35,392 KiB over the limit. E0’s largest completed RSS was 3,103,308 KiB (1.136× A), and its maximum completed target size was 4,195,454,976 bytes (1.031× A).

C2 was the second fresh target immediately after C, using the same shared cache: 1,651.245 s, 2,201,048 KiB RSS, and 4,071,399,424 target bytes. It saved 3,166.985 s (52:46.985, 65.7%) against C’s first target. This is the measured cross-target cache delta; C2 is not included in a candidate total, and its time was still longer than A’s cold build.

E1 (incremental + sccache + mold) exited before compiling in 1.094 s with status 101: sccache: incremental compilation is prohibited: Unset CARGO_INCREMENTAL to continue. Upstream Rust guidance still requires incremental compilation to be disabled for sccache. The current upstream sccache release is 0.18.0; the host used 0.17.0. Upstream mold 2.42.1 was used. The official sccache Rust docs confirm the incremental restriction: https://github.com/mozilla/sccache/blob/main/docs/Rust.md . Release pages: https://github.com/mozilla/sccache/releases/tag/v0.18.0 and https://github.com/rui314/mold/releases/tag/v2.42.1 . No tool was added to the repository. sccache is Apache-2.0 and mold is MIT.

Server link observations

These are process intervals during the rebuilds, not isolated linker microbenchmarks. The observer sampled once per second.

Candidate / touch	`collect2` interval
A / leaf	about 121 s (estimated from a process sample; observer attached late)
A / middle	70.926 s
B / middle	69.418 s
C / leaf	64.327 s
C / middle	29.438 s
D / leaf (mold)	34.157 s
D / middle (mold)	45.011 s
E0 / middle (mold)	50.857 s

A C server-source run began after the time box expired and was terminated after about 49 seconds; it produced no result row and is not counted. No candidate completed the server-source rebuild. Therefore all partial totals above are insufficient for the posted decision rule.

Recommendation and launcher patch proposal

Keep the existing baseline until one complete eligible cold-plus-three series is measured. B and D are already disqualified by their measured disk and RSS caps. C and E0 remain within caps so far, but their full totals are unknown; E0 is 144.030 s slower than A through the two completed edits. Do not enable incremental+sccache together.

Proposed preamble (no launcher change applied):

# Retain the measured baseline until a complete eligible series is available.
export CARGO_INCREMENTAL=0
unset RUSTC_WRAPPER RUSTC_WORKSPACE_WRAPPER CARGO_BUILD_RUSTC_WRAPPER
unset RUSTFLAGS CARGO_ENCODED_RUSTFLAGS
# Do not add SCCACHE_DIR/cache settings or mold to PATH yet.

Completion

No tracked files changed and no commit was made. HEAD remains f06679b11cde29cc0b7120fdab5f721389caf695. Rust/web gates were not run because this was measurement-only and changed no source. cargo clean ran with empty output; I verified the assigned target root has no debug, slot, or timing-report directories. Generated web output and package-install output were removed. The remaining matrix work is the server-source rebuild for A, B, C, D, and E0; then recompute each full total and select by the posted rule.

## Final report — #1007 measurement job Decision rule was posted before the first run: choose the lowest wall-time candidate for a cold build plus three rebuilds, only if target disk is at most 1.5× A and peak RSS is at most 1.2× A. The matrix ran past the four-hour time box before the server-source rebuild round. I stopped it; no complete cold-plus-three total exists, so I cannot select a winner under that rule. Keep the current launcher baseline for now. The patch proposal is at the end and is not applied. ### Method and host Each measured command was `cargo build --locked --timings -p calternal-server -p calternal-plugin-files`, with the assigned `CARGO_TARGET_DIR` preserved, `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, and `CARGO_BUILD_JOBS=3`. The rebuild touches were `crates/plugins/files/src/lookup.rs`, `crates/calternal-plugin/src/lib.rs`, then `crates/calternal-server/src/main.rs`. They changed mtimes only, as specified by the probe. Each treatment used its own target directory under the assigned target root. `SCCACHE_DIR=/srv/calternal/cache/sccache` was on the SSD and had a 40G cap; the cache was shared with other host jobs and was not reset. Build targets were on `/mnt/hdd`. RSS was sampled once per second; target bytes are `du` sizes after each completed command. Recorded load samples across the matrix ranged 10.81–47.71 (1m), 17.57–47.40 (5m), and 19.72–42.07 (15m). This was not a quiet-host run. ### Measurements Times are seconds; RSS is KiB; disk is bytes. “Partial total” is cold + leaf touch + middle touch. The required server-source touch is missing for every candidate. | Candidate | Cold: time / RSS / disk | Leaf rebuild: time / disk | Middle rebuild: time / disk | Partial total | Status | |---|---:|---:|---:|---:|---| | A baseline | 927.065 / 2,730,780 / 4,071,415,808 | 402.350 / 4,071,415,808 | 522.619 / 4,071,415,808 | 1,852.034 | Within caps so far | | B incremental | 1,148.804 / 2,539,492 / 5,725,495,296 | 192.554 / 6,320,021,504 | 262.318 / 6,641,217,536 | 1,603.676 | Fails disk cap after first rebuild | | C sccache | 4,818.230 / 2,518,156 / 4,071,424,000 | 221.040 / 4,071,424,000 | 152.587 / 4,071,424,000 | 5,191.857 | Within caps so far | | D mold | 1,347.834 / 3,312,328 / 4,195,454,976 | 202.199 / 4,195,450,880 | 253.135 / 4,195,446,784 | 1,803.168 | Fails RSS cap on cold build | | E0 sccache + mold, incremental off | 1,480.337 / 2,901,660 / 4,195,454,976 | 226.258 / 4,195,446,784 | 289.469 / 4,195,446,784 | 1,996.064 | Within caps so far | A’s limits are 6,107,123,712 bytes of target disk and 3,276,936 KiB RSS. B exceeded the disk limit by 212,897,792 bytes after its leaf rebuild, and by 534,093,824 bytes after the middle rebuild. D’s cold RSS was 35,392 KiB over the limit. E0’s largest completed RSS was 3,103,308 KiB (1.136× A), and its maximum completed target size was 4,195,454,976 bytes (1.031× A). C2 was the second fresh target immediately after C, using the same shared cache: 1,651.245 s, 2,201,048 KiB RSS, and 4,071,399,424 target bytes. It saved 3,166.985 s (52:46.985, 65.7%) against C’s first target. This is the measured cross-target cache delta; C2 is not included in a candidate total, and its time was still longer than A’s cold build. E1 (incremental + sccache + mold) exited before compiling in 1.094 s with status 101: `sccache: incremental compilation is prohibited: Unset CARGO_INCREMENTAL to continue.` Upstream Rust guidance still requires incremental compilation to be disabled for sccache. The current upstream sccache release is 0.18.0; the host used 0.17.0. Upstream mold 2.42.1 was used. The official sccache Rust docs confirm the incremental restriction: https://github.com/mozilla/sccache/blob/main/docs/Rust.md . Release pages: https://github.com/mozilla/sccache/releases/tag/v0.18.0 and https://github.com/rui314/mold/releases/tag/v2.42.1 . No tool was added to the repository. sccache is Apache-2.0 and mold is MIT. ### Server link observations These are process intervals during the rebuilds, not isolated linker microbenchmarks. The observer sampled once per second. | Candidate / touch | `collect2` interval | |---|---:| | A / leaf | about 121 s (estimated from a process sample; observer attached late) | | A / middle | 70.926 s | | B / middle | 69.418 s | | C / leaf | 64.327 s | | C / middle | 29.438 s | | D / leaf (mold) | 34.157 s | | D / middle (mold) | 45.011 s | | E0 / middle (mold) | 50.857 s | A C server-source run began after the time box expired and was terminated after about 49 seconds; it produced no result row and is not counted. No candidate completed the server-source rebuild. Therefore all partial totals above are insufficient for the posted decision rule. ### Recommendation and launcher patch proposal Keep the existing baseline until one complete eligible cold-plus-three series is measured. B and D are already disqualified by their measured disk and RSS caps. C and E0 remain within caps so far, but their full totals are unknown; E0 is 144.030 s slower than A through the two completed edits. Do not enable incremental+sccache together. Proposed preamble (no launcher change applied): ```sh # Retain the measured baseline until a complete eligible series is available. export CARGO_INCREMENTAL=0 unset RUSTC_WRAPPER RUSTC_WORKSPACE_WRAPPER CARGO_BUILD_RUSTC_WRAPPER unset RUSTFLAGS CARGO_ENCODED_RUSTFLAGS # Do not add SCCACHE_DIR/cache settings or mold to PATH yet. ``` ### Completion No tracked files changed and no commit was made. HEAD remains `f06679b11cde29cc0b7120fdab5f721389caf695`. Rust/web gates were not run because this was measurement-only and changed no source. `cargo clean` ran with empty output; I verified the assigned target root has no `debug`, slot, or timing-report directories. Generated web output and package-install output were removed. The remaining matrix work is the server-source rebuild for A, B, C, D, and E0; then recompute each full total and select by the posted rule.

Rows
Columns

BUILD: measure incremental, shared sccache and a fast linker for job builds #1007

Owner question (2026-10-03)

Facts today

Task: measure, then recommend (decision rule written before the runs)

Output

Final report — #1007 measurement job

Method and host

Measurements

Server link observations

Recommendation and launcher patch proposal

Completion