Sync: each download rescans the whole local folder (perf, and one unrelated change fails the pass)

kayg commented

2026-09-25 15:32:16 +00:00

Owner

Found by the sync-load job (#88). SyncEngine::download rescans the entire local folder to verify the one file it just downloaded. The cost grows with downloads × folder size, and any unrelated local change during the pass fails the whole pass. Likely a large part of #46 (random sync p95 ~1.5 s).

Fix: verify only the downloaded path (stat plus hash or size/mtime of that one file, using the new dev+ino+birth-time identity); an unrelated change is picked up by the next scan, not treated as a failure of this pass. Measure before and after with the existing sync benchmark or add one (p50/p95 for N random changes in a folder of 10k files); report the numbers in #46.

Found by the sync-load job (#88). SyncEngine::download rescans the entire local folder to verify the one file it just downloaded. The cost grows with downloads × folder size, and any unrelated local change during the pass fails the whole pass. Likely a large part of #46 (random sync p95 ~1.5 s). Fix: verify only the downloaded path (stat plus hash or size/mtime of that one file, using the new dev+ino+birth-time identity); an unrelated change is picked up by the next scan, not treated as a failure of this pass. Measure before and after with the existing sync benchmark or add one (p50/p95 for N random changes in a folder of 10k files); report the numbers in #46.

kayg commented

2026-09-25 20:59:49 +00:00

Author

Owner

Fixed on branch job/sync-download (based on job/sync-load), head 700fb05. Not merged.

Fix (9739dda): SyncEngine::download no longer rescans the folder. The install already hashes the downloaded bytes before the atomic temp-file-then-rename. It now keeps the temp file open through the rename, then compares the target's dev+ino with it. An open inode cannot be reused, so an equal identity proves that the target holds the verified bytes, and the entry uses the dev+ino+birth-time id. A replaced target fails only this path (LocalDestinationChanged, short retry), and the next pass keeps the local revision as a conflict copy. The target check just before the rename is unchanged. An unrelated local change is left for the next scan. 0373c44 also stops a file that changes during the pass's own scan from failing the pass.

Tests

crates/calternal-sync/download_race_campaign.py (new; a proxy holds the download response):
- PASS unrelated edit during download; committed in 0.137s; other edits sync while a file keeps changing (the base daemon fails this check: the rescan hit the changing file)
- PASS same-file edit during download; kept as a conflict copy
#75 stress: PASS 200 daemon-restart collisions; lost_updates=0; seed=17
PASS reused inode identity: type flip and refused rename converge
PASS pending uploads followed case-only rename chain and folder move
Unit test tracked_download_returns_the_installed_file_identity.

Numbers (10k files, 30 random remote changes, release, interleaved on a loaded host): p50 6.9 s / 8.0 s -> 0.95 s / 1.63 s, p95 12.9 s / 14.1 s -> 2.5 s / 4.1 s. The details and the other hot spots are in #46.

Gates

cargo fmt --check: exit 0, no output.
cargo clippy --workspace --all-targets -- -D warnings: Finished `dev` profile [unoptimized + debuginfo] target(s) in 13.77s
cargo test --workspace: exit 0; 60 test-result lines, 1009 passed, 0 failed, 9 ignored. (One earlier run had a load flake in calternal-collab owner_editor_viewer_and_live_revoke; it passed 3/3 alone and in the full rerun.)
bash tests/adversarial/run.sh: ==== FINDINGS 0, ==== ROUND 2 FINDINGS 0, restart probe: 0 findings. (The run before it reported one finding, calendar Event from Log :: SLOW 5.7s status 201, when the host was loaded: the task baseline was p50 3.3 s against 0.8 s in the clean run. Calendar is not touched here.)

Fixed on branch `job/sync-download` (based on `job/sync-load`), head `700fb05`. Not merged. **Fix** (`9739dda`): `SyncEngine::download` no longer rescans the folder. The install already hashes the downloaded bytes before the atomic temp-file-then-rename. It now keeps the temp file open through the rename, then compares the target's dev+ino with it. An open inode cannot be reused, so an equal identity proves that the target holds the verified bytes, and the entry uses the dev+ino+birth-time id. A replaced target fails only this path (`LocalDestinationChanged`, short retry), and the next pass keeps the local revision as a conflict copy. The target check just before the rename is unchanged. An unrelated local change is left for the next scan. `0373c44` also stops a file that changes during the pass's own scan from failing the pass. **Tests** - `crates/calternal-sync/download_race_campaign.py` (new; a proxy holds the download response): - `PASS unrelated edit during download; committed in 0.137s; other edits sync while a file keeps changing` (the base daemon fails this check: the rescan hit the changing file) - `PASS same-file edit during download; kept as a conflict copy` - #75 stress: `PASS 200 daemon-restart collisions; lost_updates=0; seed=17` - `PASS reused inode identity: type flip and refused rename converge` - `PASS pending uploads followed case-only rename chain and folder move` - Unit test `tracked_download_returns_the_installed_file_identity`. **Numbers** (10k files, 30 random remote changes, release, interleaved on a loaded host): p50 6.9 s / 8.0 s -> 0.95 s / 1.63 s, p95 12.9 s / 14.1 s -> 2.5 s / 4.1 s. The details and the other hot spots are in #46. **Gates** - `cargo fmt --check`: exit 0, no output. - `cargo clippy --workspace --all-targets -- -D warnings`: ``Finished `dev` profile [unoptimized + debuginfo] target(s) in 13.77s`` - `cargo test --workspace`: exit 0; 60 test-result lines, 1009 passed, 0 failed, 9 ignored. (One earlier run had a load flake in `calternal-collab` `owner_editor_viewer_and_live_revoke`; it passed 3/3 alone and in the full rerun.) - `bash tests/adversarial/run.sh`: `==== FINDINGS 0`, `==== ROUND 2 FINDINGS 0`, `restart probe: 0 findings`. (The run before it reported one finding, `calendar Event from Log :: SLOW 5.7s status 201`, when the host was loaded: the task baseline was p50 3.3 s against 0.8 s in the clean run. Calendar is not touched here.)

kayg referenced this issue

2026-09-25 20:59:50 +00:00

Sync: slow tail — p95 edit-to-visible 5.3 s in the random workload campaign #46

kayg referenced this issue

2026-09-25 21:00:49 +00:00

Sync: three campaigns broken on base (local_failure, mass_deletion start without a token; feed_rescan checks before commit) #121

kayg referenced this issue

2026-09-25 21:00:50 +00:00

Server: uploads slow from ~10/s to 0.8/s as a Home grows toward 7,000 files; search indexer burns 30+ CPU-min on 10k small files #122

kayg referenced this issue

2026-09-25 21:00:51 +00:00

Sync: use the change feed to track the remote side instead of a full remote listing every pass #123