Sync: intermittent lost update when a remote edit and a local edit meet across a daemon restart

kayg commented

2026-09-24 21:07:19 +00:00

Owner

Sync collision (merge-blocker class, owner rule): an intermittent lost update when the daemon restarts.

tests/adversarial/attack2.py, section "sync concurrent edit", failed once on main at 3edef03 (it passed on the previous run with the same sync code; the machine was heavily loaded, load average ~30 on 8 cores):

calternald syncs SyncRemote/ ↔ a local folder; both.txt = base is synced.
The daemon is stopped with SIGINT.
Remote: both.txt is replaced with remote edit (tus upload, upload-conflict-policy: replace).
Local: both.txt is written as local edit.
The daemon is restarted.

Expected: a conflict. The remote keeps both.txt = remote edit, and the local edit is uploaded as both (conflict, <device>, <time>).txt (engine Action::Conflict, crates/calternal-sync/src/engine.rs). Observed after 30 s: the server had only both.txt = local edit, with no conflict copy. The local edit overwrote the remote edit (the old bytes remain only in server file versions).

Relevant code: classify (baseline vs local vs remote), Action::Upload passing remote_entry.hash as expected_remote_hash, transfer.rs::upload_file (pending-upload journal, stat_file shortcut, create_upload(destination, size, expected_remote_hash)), and the server tus create path that checks the captured item ID and If-Match hash (crates/plugins/files/src/uploads.rs). Suspects:

On restart, the local watcher event is classified against a stale remote snapshot (the journal baseline) instead of a fresh listing, so the result is Upload with expected_remote_hash = baseline.
The server does not enforce the If-Match hash on some path (resumed pending upload, create-if-absent after recovery, or when the destination exists).
A pending-upload record from an earlier detection is reused with a stale expectation.

Do:

Write a deterministic regression test in crates/calternal-sync (or calternal-cli e2e) that reproduces the sequence above against a real test server, plus a stress loop (200 iterations, randomized delays, concurrent CPU load) that fails on any lost update. Run it on unmodified main first and report how often it fails.
Fix the root cause so that a local change is uploaded only with an If-Match on the hash the engine last saw and verified fresh from the server, the server rejects mismatches with 412 on every upload path, and a 412 turns into Action::Conflict. Never overwrite.
Re-run the stress loop (0 failures in 200) and tests/adversarial/run.sh (0 findings). Quote the numbers.

**Sync collision (merge-blocker class, owner rule): an intermittent lost update when the daemon restarts.** `tests/adversarial/attack2.py`, section "sync concurrent edit", failed once on main at `3edef03` (it passed on the previous run with the same sync code; the machine was heavily loaded, load average ~30 on 8 cores): 1. `calternald` syncs `SyncRemote/` ↔ a local folder; `both.txt` = `base` is synced. 2. The daemon is stopped with SIGINT. 3. Remote: `both.txt` is replaced with `remote edit` (tus upload, `upload-conflict-policy: replace`). 4. Local: `both.txt` is written as `local edit`. 5. The daemon is restarted. Expected: a conflict. The remote keeps `both.txt = remote edit`, and the local edit is uploaded as `both (conflict, <device>, <time>).txt` (engine `Action::Conflict`, `crates/calternal-sync/src/engine.rs`). Observed after 30 s: the server had only `both.txt = local edit`, with no conflict copy. The local edit **overwrote** the remote edit (the old bytes remain only in server file versions). Relevant code: `classify` (baseline vs local vs remote), `Action::Upload` passing `remote_entry.hash` as `expected_remote_hash`, `transfer.rs::upload_file` (pending-upload journal, `stat_file` shortcut, `create_upload(destination, size, expected_remote_hash)`), and the server tus create path that checks the captured item ID and If-Match hash (`crates/plugins/files/src/uploads.rs`). Suspects: - On restart, the local watcher event is classified against a stale remote snapshot (the journal baseline) instead of a fresh listing, so the result is `Upload` with `expected_remote_hash = baseline`. - The server does not enforce the If-Match hash on some path (resumed pending upload, create-if-absent after recovery, or when the destination exists). - A pending-upload record from an earlier detection is reused with a stale expectation. Do: 1. Write a deterministic regression test in `crates/calternal-sync` (or `calternal-cli` e2e) that reproduces the sequence above against a real test server, plus a stress loop (200 iterations, randomized delays, concurrent CPU load) that fails on any lost update. Run it on unmodified main first and report how often it fails. 2. Fix the root cause so that a local change is uploaded only with an If-Match on the hash the engine last saw **and verified fresh from the server**, the server rejects mismatches with 412 on every upload path, and a 412 turns into `Action::Conflict`. Never overwrite. 3. Re-run the stress loop (0 failures in 200) and `tests/adversarial/run.sh` (0 findings). Quote the numbers.

kayg commented

2026-09-24 21:08:29 +00:00

Author

Owner

Starting investigation on job/sync-race, base e25dbfacba1caf3df092f3a7f50164693ea3712d (main). Worktree is clean. I will first run the existing sync race probe on this unchanged base and record its loss rate before editing.

Starting investigation on `job/sync-race`, base `e25dbfacba1caf3df092f3a7f50164693ea3712d` (main). Worktree is clean. I will first run the existing sync race probe on this unchanged base and record its loss rate before editing.

kayg commented

2026-09-24 22:05:28 +00:00

Author

Owner

Unmodified-base reproduction on e25dbfacba1caf3df092f3a7f50164693ea3712d: ROUND2_SECTIONS=sync bash tests/adversarial/run.sh completed with exit 0 and ==== ROUND 2 FINDINGS 0. The original restart-collision scenario lost 0 updates in 1 attempt on this run. The host load at start was 24.43 on the shared 8-core machine. This is intermittent, so I am adding the required 200-round real-server stress case before production edits.

Unmodified-base reproduction on `e25dbfacba1caf3df092f3a7f50164693ea3712d`: `ROUND2_SECTIONS=sync bash tests/adversarial/run.sh` completed with exit 0 and `==== ROUND 2 FINDINGS 0`. The original restart-collision scenario lost 0 updates in 1 attempt on this run. The host load at start was 24.43 on the shared 8-core machine. This is intermittent, so I am adding the required 200-round real-server stress case before production edits.

kayg commented

2026-09-24 22:16:45 +00:00

Author

Owner

Pre-fix stress on the unchanged e25dbfacba1caf3df092f3a7f50164693ea3712d sync/server code completed: PASS 200 daemon-restart collisions; lost_updates=0; seed=75 (exit 0). The loop used randomized phase delays and one concurrent CPU worker; host load before it started was 16.26/14.83/12.08. The earlier single adversarial run was also 0/1. The random sequence did not hit the reported intermittent window, so I am adding a barrier-controlled stale-snapshot case before production changes.

Pre-fix stress on the unchanged `e25dbfacba1caf3df092f3a7f50164693ea3712d` sync/server code completed: `PASS 200 daemon-restart collisions; lost_updates=0; seed=75` (exit 0). The loop used randomized phase delays and one concurrent CPU worker; host load before it started was 16.26/14.83/12.08. The earlier single adversarial run was also 0/1. The random sequence did not hit the reported intermittent window, so I am adding a barrier-controlled stale-snapshot case before production changes.

kayg commented

2026-09-24 22:33:49 +00:00

Author

Owner

Finding on base e25dbfacba: the deterministic real-server barrier held a daemon upload with If-Match equal to the verified base hash, replaced the remote file, then released the stale upload. The server returned 412. On unchanged production code, the daemon issued another tree listing before uploading the conflict copy; the new regression assertion failed at that ordering. The original adversarial sync sequence and the randomized restart loop both remained 0 lost updates (1 original sequence, 200 stress rounds), so the barrier now covers the rare request window directly. After the fix, both the stale-listing barrier and the post-stat 412 barrier pass.

Finding on base e25dbfacba1caf3df092f3a7f50164693ea3712d: the deterministic real-server barrier held a daemon upload with If-Match equal to the verified base hash, replaced the remote file, then released the stale upload. The server returned 412. On unchanged production code, the daemon issued another tree listing before uploading the conflict copy; the new regression assertion failed at that ordering. The original adversarial sync sequence and the randomized restart loop both remained 0 lost updates (1 original sequence, 200 stress rounds), so the barrier now covers the rare request window directly. After the fix, both the stale-listing barrier and the post-stat 412 barrier pass.

kayg commented

2026-09-24 23:52:48 +00:00

Author

Owner

Completed on branch job/sync-race.

Head: ca99b198bd562ea6393c59d57261e1f5b1caea8a

The sync engine now re-stats a remote path immediately before upload and compares its stable item ID and content hash with the listing snapshot. If the remote revision changed, it preserves the local edit as a conflict and fetches the observed remote revision before another tree scan. Upload precondition failures are handled in that same pass. Tus completion also rejects a create-if-absent session if another upload created the destination after the session began.

The initial adversarial barrier on unmodified main confirmed the stale upload was rejected with HTTP 412, followed by another tree scan before the conflict upload. The unmodified random stress run saw 0 lost updates in 200. No permanent byte loss was observed in that baseline run. On the fix, both deterministic barriers passed, and the restart-collision stress passed 200/200 with 0 lost updates.

Final verification:

cargo fmt --check
(exit 0; no output)

cargo clippy --workspace --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 35s

cargo test --workspace
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.64s
test result: ok. 46 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.79s

PASS stale listing race; fresh stat preserved both revisions without a stale upload
PASS precondition race; stale upload rejected with HTTP 412; conflict resolved before another tree scan
restart collisions 200/200
PASS 200 daemon-restart collisions; lost_updates=0; seed=75

The final adversarial command was OPENSSL_NO_VENDOR=1 CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 ROUND2_SECTIONS=sync bash tests/adversarial/run.sh. Its round 1 reported 29 latency findings under shared-host load (task and journal storm requests took about 5–9 seconds); the server stayed alive and the requests returned expected statuses. The requested round 2 sync section completed with:

server alive at end: True

==== ROUND 2 FINDINGS 0

The full script exited 1 because of the round 1 latency findings.

Decisions where the design doc was silent: treat any changed remote item ID or hash as a revision conflict, even if bytes match; treat a remote 404 as a concurrent delete and retain local bytes in a conflict copy; and handle HTTP 412 or create-if-absent 409 before another full tree scan.

Commits:

f0de306 fix(sync): preserve remote edits across upload races
ca99b19 fix(sync): keep changed remote identities in conflicts

cargo clean output: Removed 12533 files, 7.6GiB total.

Completed on branch `job/sync-race`. Head: `ca99b198bd562ea6393c59d57261e1f5b1caea8a` The sync engine now re-stats a remote path immediately before upload and compares its stable item ID and content hash with the listing snapshot. If the remote revision changed, it preserves the local edit as a conflict and fetches the observed remote revision before another tree scan. Upload precondition failures are handled in that same pass. Tus completion also rejects a create-if-absent session if another upload created the destination after the session began. The initial adversarial barrier on unmodified main confirmed the stale upload was rejected with HTTP 412, followed by another tree scan before the conflict upload. The unmodified random stress run saw 0 lost updates in 200. No permanent byte loss was observed in that baseline run. On the fix, both deterministic barriers passed, and the restart-collision stress passed 200/200 with 0 lost updates. Final verification: ``` cargo fmt --check (exit 0; no output) ``` ``` cargo clippy --workspace --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 35s ``` ``` cargo test --workspace test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.64s test result: ok. 46 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.79s ``` ``` PASS stale listing race; fresh stat preserved both revisions without a stale upload PASS precondition race; stale upload rejected with HTTP 412; conflict resolved before another tree scan restart collisions 200/200 PASS 200 daemon-restart collisions; lost_updates=0; seed=75 ``` The final adversarial command was `OPENSSL_NO_VENDOR=1 CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 ROUND2_SECTIONS=sync bash tests/adversarial/run.sh`. Its round 1 reported 29 latency findings under shared-host load (task and journal storm requests took about 5–9 seconds); the server stayed alive and the requests returned expected statuses. The requested round 2 sync section completed with: ``` server alive at end: True ==== ROUND 2 FINDINGS 0 ``` The full script exited 1 because of the round 1 latency findings. Decisions where the design doc was silent: treat any changed remote item ID or hash as a revision conflict, even if bytes match; treat a remote 404 as a concurrent delete and retain local bytes in a conflict copy; and handle HTTP 412 or create-if-absent 409 before another full tree scan. Commits: - `f0de306 fix(sync): preserve remote edits across upload races` - `ca99b19 fix(sync): keep changed remote identities in conflicts` `cargo clean` output: `Removed 12533 files, 7.6GiB total`.

kayg referenced this issue from a commit

2026-09-25 00:24:44 +00:00

Merge job/sync-race: re-check remote revisions before upload; stale preconditions become conflicts (#75)

kayg referenced this issue

2026-09-25 02:59:55 +00:00

Sync: slow tail — p95 edit-to-visible 5.3 s in the random workload campaign #46

kayg referenced this issue

2026-09-25 05:10:51 +00:00

Sync: slow tail — p95 edit-to-visible 5.3 s in the random workload campaign #46

kayg referenced this issue

2026-09-25 13:33:23 +00:00

Sync: adversarial round 2 reported a lost concurrent edit and file/folder flips under heavy load #88

kayg referenced this issue

2026-09-25 15:31:38 +00:00

Sync: adversarial round 2 reported a lost concurrent edit and file/folder flips under heavy load #88

kayg referenced this issue

2026-09-25 20:59:49 +00:00

Sync: each download rescans the whole local folder (perf, and one unrelated change fails the pass) #97

kayg commented

2026-10-01 05:08:59 +00:00

Author

Owner

Completed on dev in 6c943d6f16 (Merge job/sync-race: re-check remote revisions before upload; stale preconditions become conflicts (#75)).

Completed on dev in 6c943d6f16df6b75e9a79b68423a0e0d8c67f640 (Merge job/sync-race: re-check remote revisions before upload; stale preconditions become conflicts (#75)).

kayg closed this issue

2026-10-01 05:08:59 +00:00

Rows
Columns

Sync: intermittent lost update when a remote edit and a local edit meet across a daemon restart #75