Sync: use the change feed to track the remote side instead of a full remote listing every pass

kayg commented

2026-09-25 21:00:51 +00:00

Owner

From #97: the sync client reads the change feed but still lists the whole remote tree every pass (now 8 folders in parallel). Tracking the remote side from the feed (with a periodic or on-gap full listing as the safety net) is the next big latency win for #46 (random p95 still ~1.55 s under load vs the 1 s target). Needs a design decision on gap detection and recovery; keep every data-safety property and the torn-listing guard (453747a).

From #97: the sync client reads the change feed but still lists the whole remote tree every pass (now 8 folders in parallel). Tracking the remote side from the feed (with a periodic or on-gap full listing as the safety net) is the next big latency win for #46 (random p95 still ~1.55 s under load vs the 1 s target). Needs a design decision on gap detection and recovery; keep every data-safety property and the torn-listing guard (453747a).

kayg commented

2026-09-26 12:05:09 +00:00

Author

Owner

Started #123 on branch job/change-feed, based on dev at 44b15bec4f.

Started #123 on branch job/change-feed, based on dev at 44b15bec4f4e78e788c85b7ec60d54892a293b84.

kayg commented

2026-09-26 12:06:20 +00:00

Author

Owner

Finding: reads from the journal cursor through the high-water cursor but discards every page, then lists the full selected tree on every pass (crates/calternal-sync/src/engine.rs:125-180, 503-516). Recovery decision: use HTTP 410 as the cursor-gap signal; numeric gaps alone are valid because the feed sequence is shared across users (files feed contract, crates/plugins/files/src/lib.rs:2457). Keep a full baseline on first sync, on 410, and every 24 hours; apply feed deltas between those scans. I will retain the listing high-water retry and duplicate-item guard.

kayg commented

2026-09-26 12:06:36 +00:00

Author

Owner

Finding: reconcile_once_confirmed reads /changes from the journal cursor through the high-water cursor but discards every page, then remote_tree lists the full selected tree on every pass (crates/calternal-sync/src/engine.rs, lines 125–180 and 503–516). Recovery decision: use HTTP 410 as the cursor-gap signal; numeric gaps alone are valid because the feed sequence is shared across users (crates/plugins/files/src/lib.rs, line 2457). Keep a full baseline on first sync, on 410, and every 24 hours; apply feed deltas between those scans. Retain the listing high-water retry and duplicate-item guard.

Finding: `reconcile_once_confirmed` reads `/changes` from the journal cursor through the high-water cursor but discards every page, then `remote_tree` lists the full selected tree on every pass (`crates/calternal-sync/src/engine.rs`, lines 125–180 and 503–516). Recovery decision: use HTTP 410 as the cursor-gap signal; numeric gaps alone are valid because the feed sequence is shared across users (`crates/plugins/files/src/lib.rs`, line 2457). Keep a full baseline on first sync, on 410, and every 24 hours; apply feed deltas between those scans. Retain the listing high-water retry and duplicate-item guard.

kayg commented

2026-09-26 12:36:33 +00:00

Author

Owner

Additional recovery case: if the journal cursor is greater than /changes/head (for example, the server index was restored from an older backup), do a full listing and reset the cursor to that head in the same verified journal transaction. Only a completed full inventory can safely move the cursor backward; incremental commits remain monotonic.

Additional recovery case: if the journal cursor is greater than `/changes/head` (for example, the server index was restored from an older backup), do a full listing and reset the cursor to that head in the same verified journal transaction. Only a completed full inventory can safely move the cursor backward; incremental commits remain monotonic.

kayg commented

2026-09-26 12:41:55 +00:00

Author

Owner

Finding: entries is the last common sync baseline, not a complete remote inventory. A pass can defer an unstable or locally unwritable path yet still commit the high-water cursor. If that cursor were then used to rebuild remote state from entries, the consumed feed event for the deferred path would be lost on the next pass. The fix is a separate durable remote snapshot paired atomically with its feed cursor; the common baseline remains unchanged until the path verifies.

Finding: `entries` is the last common sync baseline, not a complete remote inventory. A pass can defer an unstable or locally unwritable path yet still commit the high-water cursor. If that cursor were then used to rebuild remote state from `entries`, the consumed feed event for the deferred path would be lost on the next pass. The fix is a separate durable remote snapshot paired atomically with its feed cursor; the common baseline remains unchanged until the path verifies.

kayg commented

2026-09-26 15:24:13 +00:00

Author

Owner

Finding after the required dev merge: workspace Clippy failed with E0308 in crates/plugins/files/src/index.rs:620. The #122 unchanged-row fast path returns Ok(()), but record_once returns Result<bool, Failure>, and its documented false result means the file changed while hashing. I will make this unchanged row return Ok(true) and add a focused regression test before rerunning the affected crate gates.

Finding after the required `dev` merge: workspace Clippy failed with E0308 in `crates/plugins/files/src/index.rs:620`. The #122 unchanged-row fast path returns `Ok(())`, but `record_once` returns `Result<bool, Failure>`, and its documented `false` result means the file changed while hashing. I will make this unchanged row return `Ok(true)` and add a focused regression test before rerunning the affected crate gates.

kayg referenced this issue

2026-09-26 15:56:35 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-26 16:06:55 +00:00

Author

Owner

The merged files-plugin compile failure is fixed in commit 1185f28d. The unchanged-row path now returns Ok(true), matching record_once semantics; the regression test index_record_of_an_unchanged_file_is_idempotent passed (1 passed; 0 failed), and cargo clippy -p calternal-plugin-files --all-targets -- -D warnings finished successfully.

The merged files-plugin compile failure is fixed in commit `1185f28d`. The unchanged-row path now returns `Ok(true)`, matching `record_once` semantics; the regression test `index_record_of_an_unchanged_file_is_idempotent` passed (`1 passed; 0 failed`), and `cargo clippy -p calternal-plugin-files --all-targets -- -D warnings` finished successfully.

kayg commented

2026-09-26 17:19:04 +00:00

Author

Owner

Real-server campaign finding: feed_delta timed out after the Files API returned HTTP 200 for trashing Sync/moved.txt; the daemon left local moved.txt in place after 20 seconds. The separate feed_rescan, precondition_race (both revisions preserved), and local_failure campaigns passed. I am inspecting the persisted cursor/snapshot and returned feed entry to locate why this delete was not applied.

Real-server campaign finding: `feed_delta` timed out after the Files API returned HTTP 200 for trashing `Sync/moved.txt`; the daemon left local `moved.txt` in place after 20 seconds. The separate `feed_rescan`, `precondition_race` (both revisions preserved), and `local_failure` campaigns passed. I am inspecting the persisted cursor/snapshot and returned feed entry to locate why this delete was not applied.

kayg commented

2026-09-26 17:24:39 +00:00

Author

Owner

Root cause found: the feed correctly removed moved.txt from the durable remote snapshot and advanced its cursor to 15, but the planner counted the single TrashLocal action against a one-file baseline. deletion_guard(1, 1) returned MassDeletion, leaving the local file and verified baseline in place. I changed the guard to keep full-scan remote losses and local-to-remote deletions guarded while allowing item-scoped remote feed deletions to move into local trash. The sync crate passed (42 library tests, 2 daemon tests) and its Clippy check passed. I am rebuilding the daemon and rerunning the real-server campaign.

Root cause found: the feed correctly removed `moved.txt` from the durable remote snapshot and advanced its cursor to 15, but the planner counted the single `TrashLocal` action against a one-file baseline. `deletion_guard(1, 1)` returned `MassDeletion`, leaving the local file and verified baseline in place. I changed the guard to keep full-scan remote losses and local-to-remote deletions guarded while allowing item-scoped remote feed deletions to move into local trash. The sync crate passed (42 library tests, 2 daemon tests) and its Clippy check passed. I am rebuilding the daemon and rerunning the real-server campaign.

kayg commented

2026-09-26 17:41:42 +00:00

Author

Owner

The real-server feed campaign exposed a sync collision: the feed returned an item-specific trash event and the daemon advanced its durable cursor, but the one-file baseline triggered the broad mass-deletion guard. The local file and verified baseline stayed unchanged. I changed the guard so an incremental remote TrashLocal is applied, while full-scan removals and TrashRemote actions remain guarded. After rebuilding calternald, feed_delta, feed_rescan, precondition_race, and local_failure all passed.

The real-server feed campaign exposed a sync collision: the feed returned an item-specific `trash` event and the daemon advanced its durable cursor, but the one-file baseline triggered the broad mass-deletion guard. The local file and verified baseline stayed unchanged. I changed the guard so an incremental remote `TrashLocal` is applied, while full-scan removals and `TrashRemote` actions remain guarded. After rebuilding `calternald`, `feed_delta`, `feed_rescan`, `precondition_race`, and `local_failure` all passed.

kayg commented

2026-09-26 18:42:45 +00:00

Author

Owner

Resuming after the planned VM restart on branch job/change-feed. Current base dev is 2527f0d07e111a8717e986f57e1ca3c51242c486; the branch is clean at 8ea9a0f4ff2ff7d40775f17634cd450df62d9179. I am merging the advanced dev tip and will run the final gates before reporting the resulting head and exact gate output.

Resuming after the planned VM restart on branch `job/change-feed`. Current base `dev` is `2527f0d07e111a8717e986f57e1ca3c51242c486`; the branch is clean at `8ea9a0f4ff2ff7d40775f17634cd450df62d9179`. I am merging the advanced `dev` tip and will run the final gates before reporting the resulting head and exact gate output.

kayg commented

2026-09-26 19:07:13 +00:00

Author

Owner

Finding from the required post-merge local-server adversarial round: the public photo thumbnail checks failed because the test host lacks the sandbox runtime (calternal-media-sandbox, calternal-media-sandbox-dropcaps, and bwrap). The Audit photo had no thumbnail after 30 seconds; public 256/1024 thumbnail requests returned 404. MediaThumbnailer starts the missing wrapper, while tests/adversarial/run.sh starts the server directly without it. I filed the runner/runtime setup gap as #209. This run does not establish thumbnail behavior in the packaged runtime. SLOW-only latency reports are treated as shared-host load per the job rule.

Finding from the required post-merge local-server adversarial round: the public photo thumbnail checks failed because the test host lacks the sandbox runtime (`calternal-media-sandbox`, `calternal-media-sandbox-dropcaps`, and `bwrap`). The `Audit` photo had no thumbnail after 30 seconds; public 256/1024 thumbnail requests returned 404. `MediaThumbnailer` starts the missing wrapper, while `tests/adversarial/run.sh` starts the server directly without it. I filed the runner/runtime setup gap as #209. This run does not establish thumbnail behavior in the packaged runtime. SLOW-only latency reports are treated as shared-host load per the job rule.

kayg commented

2026-09-26 19:13:49 +00:00

Author

Owner

Finding: after merging dev, workspace Clippy failed with E0428 because both merge parents added a test named index_record_of_an_unchanged_file_is_idempotent in crates/plugins/files/src/lib.rs (at lines 2809 and 2881). The parent versions cover the same behavior with different assertions. I will keep the branch test that exercises repeated index::record calls and preserves the item ID, then rerun the Files crate checks before another workspace gate pass.

Finding: after merging `dev`, workspace Clippy failed with E0428 because both merge parents added a test named `index_record_of_an_unchanged_file_is_idempotent` in `crates/plugins/files/src/lib.rs` (at lines 2809 and 2881). The parent versions cover the same behavior with different assertions. I will keep the branch test that exercises repeated `index::record` calls and preserves the item ID, then rerun the Files crate checks before another workspace gate pass.

kayg commented

2026-09-26 19:41:07 +00:00

Author

Owner

Completed #123 on branch job/change-feed.

Head SHA: ed709db015f83ba723890cf23e42f281b4671774

Built

The sync client now keeps a durable remote snapshot paired atomically with its change-feed cursor. It applies remote create, move, and trash entries between verified full inventories. It performs a full remote inventory on first sync, every 24 hours, after HTTP 410, and when the saved cursor is ahead of the server head. A backward cursor reset is committed only with a completed full inventory. The existing listing high-water retry and duplicate-item guard remain in place. The common verified sync baseline does not advance for paths that did not verify.

An item-scoped remote trash event can apply its matching local trash action without tripping the broad mass-delete guard. Full-scan removals and remote trash actions retain the guard.

The current dev merge exposed the same unchanged-index test from both parents. Commit ed709db0 keeps the test that repeats index::record and checks item identity.

Files

crates/calternal-sync/src/engine.rs
crates/calternal-sync/src/journal.rs
crates/calternal-sync/src/plan.rs
crates/calternal-sync/src/remote.rs
crates/calternal-sync/src/bin/calternald.rs
crates/calternal-sync/feed_delta_campaign.py
crates/calternal-sync/integration_campaign.py
crates/calternal-sync/run_campaigns.py
crates/plugins/files/src/index.rs
crates/plugins/files/src/lib.rs
tests/adversarial/attack2.py

Gate output

cargo fmt --check: exit 0; stdout and stderr were empty.

cargo clippy --all-targets -- -D warnings: exit 0. Exact final line:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 08s

cargo test: exit 0; 67 result lines, 1193 passed, 0 failed, 12 ignored. Exact Files and sync crate lines:
- test result: ok. 99 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 27.18s
- test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.46s
bun run check: svelte-check found 0 errors and 0 warnings
bun run test: Test Files 55 passed (55); Tests 432 passed (432); Duration 36.09s (transform 71%, import 13%, environment 9%, tests 6%)
Focused sync campaigns:
- feed_delta: PASS feed deltas handled remote create, move, delete and a two-sided collision without another full listing
- feed_rescan: PASS feed 410 caused full remote rescan and advanced journal cursor
- all campaigns passed
tests/adversarial/run.sh exited 1. It reported SLOW-only latency warnings under shared-host load and non-SLOW public thumbnail 404s because the local host lacks calternal-media-sandbox, calternal-media-sandbox-dropcaps, and bwrap. I filed the local runner/runtime setup gap as #209. This round does not establish thumbnail behavior in the packaged runtime. The server stayed alive and the restart probe reported restart probe: 0 findings.

Decisions not covered by DESIGN.md

HTTP 410 is the cursor-gap signal. Numeric cursor gaps are normal because the feed sequence is shared across users.
Use a full inventory on first sync and every 24 hours, as well as after 410 or a cursor-ahead-of-head restore. Only a completed verified inventory may move the cursor backward.
Store the durable remote snapshot separately from the last common sync baseline, and update the remote snapshot and feed cursor in one journal transaction.
Treat a feed trash event as item-scoped for the local deletion guard; retain the broad guard for full-scan losses and TrashRemote actions.

Completed #123 on branch `job/change-feed`. Head SHA: `ed709db015f83ba723890cf23e42f281b4671774` ## Built The sync client now keeps a durable remote snapshot paired atomically with its change-feed cursor. It applies remote create, move, and trash entries between verified full inventories. It performs a full remote inventory on first sync, every 24 hours, after HTTP 410, and when the saved cursor is ahead of the server head. A backward cursor reset is committed only with a completed full inventory. The existing listing high-water retry and duplicate-item guard remain in place. The common verified sync baseline does not advance for paths that did not verify. An item-scoped remote trash event can apply its matching local trash action without tripping the broad mass-delete guard. Full-scan removals and remote trash actions retain the guard. The current `dev` merge exposed the same unchanged-index test from both parents. Commit `ed709db0` keeps the test that repeats `index::record` and checks item identity. ## Files - `crates/calternal-sync/src/engine.rs` - `crates/calternal-sync/src/journal.rs` - `crates/calternal-sync/src/plan.rs` - `crates/calternal-sync/src/remote.rs` - `crates/calternal-sync/src/bin/calternald.rs` - `crates/calternal-sync/feed_delta_campaign.py` - `crates/calternal-sync/integration_campaign.py` - `crates/calternal-sync/run_campaigns.py` - `crates/plugins/files/src/index.rs` - `crates/plugins/files/src/lib.rs` - `tests/adversarial/attack2.py` ## Gate output - `cargo fmt --check`: exit 0; stdout and stderr were empty. - `cargo clippy --all-targets -- -D warnings`: exit 0. Exact final line: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 08s ``` - `cargo test`: exit 0; 67 result lines, 1193 passed, 0 failed, 12 ignored. Exact Files and sync crate lines: - `test result: ok. 99 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 27.18s` - `test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.46s` - `bun run check`: `svelte-check found 0 errors and 0 warnings` - `bun run test`: `Test Files 55 passed (55)`; `Tests 432 passed (432)`; `Duration 36.09s (transform 71%, import 13%, environment 9%, tests 6%)` - Focused sync campaigns: - `feed_delta: PASS feed deltas handled remote create, move, delete and a two-sided collision without another full listing` - `feed_rescan: PASS feed 410 caused full remote rescan and advanced journal cursor` - `all campaigns passed` - `tests/adversarial/run.sh` exited 1. It reported SLOW-only latency warnings under shared-host load and non-SLOW public thumbnail 404s because the local host lacks `calternal-media-sandbox`, `calternal-media-sandbox-dropcaps`, and `bwrap`. I filed the local runner/runtime setup gap as #209. This round does not establish thumbnail behavior in the packaged runtime. The server stayed alive and the restart probe reported `restart probe: 0 findings`. ## Decisions not covered by DESIGN.md - HTTP 410 is the cursor-gap signal. Numeric cursor gaps are normal because the feed sequence is shared across users. - Use a full inventory on first sync and every 24 hours, as well as after 410 or a cursor-ahead-of-head restore. Only a completed verified inventory may move the cursor backward. - Store the durable remote snapshot separately from the last common sync baseline, and update the remote snapshot and feed cursor in one journal transaction. - Treat a feed trash event as item-scoped for the local deletion guard; retain the broad guard for full-scan losses and `TrashRemote` actions.

kayg commented

2026-09-26 19:44:02 +00:00

Author

Owner

Merged into dev at 8a2a9417. Deploy status on #203.

Merged into dev at 8a2a9417. Deploy status on #203.

kayg referenced this issue from a commit

2026-09-26 19:44:02 +00:00

Merge job/change-feed (#123)

kayg closed this issue

2026-09-26 19:44:03 +00:00

kayg referenced this issue