kayg commented

Owner

Sync collisions block merges (CLAUDE.md). Build a chaos harness in tests/ (not shipped): real server + 2-3 sync clients on temp dirs; random operations (create/edit/rename/move/delete files and folders, same file on two clients, case-only renames, Unicode NFC/NFD names, large files via tus) interleaved with faults: kill -9 client or server mid-upload/download, network cut via a local proxy that drops/stalls connections, clock skew, disk-full on the client (small tmpfs/loop or quota). After each run, converge and assert invariants: no data loss (every byte ever written by a client exists at its final path or as a conflict copy), no duplicates at two paths (#97 class), identical trees on all clients and server, index consistent with disk. Seeded and reproducible; run for a time budget. Every failure: minimal seed as a regression test, fix, commit atomically. Report seeds run and bugs fixed.

kayg commented

2026-09-26 09:41:29 +00:00

Author

Owner

Started #163 on job/sync-chaos at base 82d4c258bb. I am extending the existing real-server sync campaign fixtures with a seeded, time-budgeted chaos probe. It will cap clients at three, use small ordinary files, and reserve large data for tus fault cases. I will report concrete findings and final gate output here.

Started #163 on job/sync-chaos at base 82d4c258bbcc6f34e8805d0018004fa8e621b9a5. I am extending the existing real-server sync campaign fixtures with a seeded, time-budgeted chaos probe. It will cap clients at three, use small ordinary files, and reserve large data for tus fault cases. I will report concrete findings and final gate output here.

kayg commented

2026-09-26 09:55:38 +00:00

Author

Owner

Progress: the seeded probe is implemented in tests/adversarial/sync_chaos.py. It uses two real daemon installations, one real server, a proxy that can stall/drop a committed response, per-step byte/tree/Index checks, a 12 MiB tus interruption, and a private 4 MiB tmpfs for real client ENOSPC. Python syntax and a standalone proxy disconnect check pass. The first Rust binary build is still compiling dependencies on the shared host; no sync result or product finding yet.

kayg commented

2026-09-26 10:10:15 +00:00

Author

Owner

Progress: two harness commits are on the branch (e2b545c and 2bb4f12). I merged dev at 35d2cd5 before the final build and gates; the merge was clean. The build remains active on the shared host. A standalone test confirmed the proxy closes a response after the upstream has completed, and Python syntax passes. No live sync seed has run yet, so there is no product finding to claim.

Progress: two harness commits are on the branch (e2b545c and 2bb4f12). I merged dev at 35d2cd5 before the final build and gates; the merge was clean. The build remains active on the shared host. A standalone test confirmed the proxy closes a response after the upstream has completed, and Python syntax passes. No live sync seed has run yet, so there is no product finding to claim.

kayg commented

2026-09-26 10:24:43 +00:00

Author

Owner

Progress: the merged build was retried with RUSTC_WRAPPER= after the host-wide sccache failed while writing a dependency file under another worktree's removed target/tmp path. The uncached build is compiling normally. The harness commits are e2b545c, 2bb4f12 and 85b15c0. There is still no live sync result; I will report only seeded test output after the binaries finish.

Progress: the merged build was retried with `RUSTC_WRAPPER=` after the host-wide sccache failed while writing a dependency file under another worktree's removed target/tmp path. The uncached build is compiling normally. The harness commits are e2b545c, 2bb4f12 and 85b15c0. There is still no live sync result; I will report only seeded test output after the binaries finish.

kayg referenced this issue

2026-09-26 12:28:45 +00:00

Sync: NFC to NFD local rename leaves client tree divergent #169

kayg commented

2026-09-26 12:29:00 +00:00

Author

Owner

Finding from live seed 163: step 7 (NFC to NFD local rename) reproduced three times. The server and receiving installation retain café-5.txt; the originating installation retains decomposed cafe\u0301-5.txt after the convergence deadline. calternal-fs intentionally normalizes new server names to NFC. This is a permanent sync tree mismatch, not host load. Filed product fix as #169 because correcting client rename behavior changes calternal-sync outside this test job. The chaos probe keeps this failing case as a regression. I am continuing other operation and transfer cases on separate seeds.

Finding from live seed 163: step 7 (NFC to NFD local rename) reproduced three times. The server and receiving installation retain `café-5.txt`; the originating installation retains decomposed `cafe\u0301-5.txt` after the convergence deadline. `calternal-fs` intentionally normalizes new server names to NFC. This is a permanent sync tree mismatch, not host load. Filed product fix as #169 because correcting client rename behavior changes `calternal-sync` outside this test job. The chaos probe keeps this failing case as a regression. I am continuing other operation and transfer cases on separate seeds.

kayg referenced this issue

2026-09-26 12:33:58 +00:00

Sync: folder move leaves empty old directory on server and peer #170

kayg commented

2026-09-26 12:34:13 +00:00

Author

Owner

Further finding: seed 173 folder_move reproduced three times. Server and receiving installation retain the empty old folder-6 after the originating installation moves the folder to moved-6. All file bytes and both journals match; directory sets do not. Filed as #170. This is a convergence defect in product sync behavior; the exact directory assertion remains in the chaos probe. I am running seed 20 to exercise conflict and restore operations before the two known failing rename cases.

Further finding: seed 173 `folder_move` reproduced three times. Server and receiving installation retain the empty old `folder-6` after the originating installation moves the folder to `moved-6`. All file bytes and both journals match; directory sets do not. Filed as #170. This is a convergence defect in product sync behavior; the exact directory assertion remains in the chaos probe. I am running seed 20 to exercise conflict and restore operations before the two known failing rename cases.

kayg referenced this issue

2026-09-26 15:56:35 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 10:00:55 +00:00

Author

Owner

Resuming #163 on job/sync-chaos at checkpoint 369858e081. The work started from base 82d4c258bb; the checkpoint already contains a merge from dev. I read the earlier reports: seeds 163 and 173 reproduce NFC/NFD rename and empty-directory move convergence failures, filed as #169 and #170. I will continue with the remaining operations, merge current dev before final gates, and report the exact campaign and gate output.

Resuming #163 on job/sync-chaos at checkpoint 369858e0815efd0eda050355a55d723018e5881a. The work started from base 82d4c258bbcc6f34e8805d0018004fa8e621b9a5; the checkpoint already contains a merge from dev. I read the earlier reports: seeds 163 and 173 reproduce NFC/NFD rename and empty-directory move convergence failures, filed as #169 and #170. I will continue with the remaining operations, merge current dev before final gates, and report the exact campaign and gate output.

kayg commented

2026-09-27 10:20:59 +00:00

Author

Owner

Harness finding on seed 20: the first settle timed out with the API tree containing only lost-reply.bin, both client inventories also containing .calternal-sync-root-id, and both journals matching the Index. The local scanner explicitly treats that exact root marker as private client state, but the shared Python inventory did not filter it. This is a probe false failure, not a sync product defect. I am fixing the test inventory to omit the exact marker and valid atomic marker temps while retaining similarly named user files, then I will replay the same seed.

kayg referenced this issue

2026-09-27 10:23:51 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 10:24:34 +00:00

Author

Owner

Seed 163 now reproduces #169 on the merged branch after the inventory fix. At step 7 (NFC to NFD rename), the server and client 2 list NFC café-5.txt while client 1 lists decomposed café-5.txt; the other files agree and both journals match the Index. The convergence assertion fails at unicode nfd. This is the already-filed sync behavior issue, not a new probe mismatch.

kayg commented

2026-09-27 10:25:45 +00:00

Author

Owner

Seed 173 now reproduces #170 on the merged branch. After moving folder-6 to moved-6, file bytes and both journals agree, but the server and client 2 retain an empty folder-6 beside moved-6. The directory invariant fails at folder_move. This confirms the prior filed issue, not a separate load-only result.

kayg commented

2026-09-27 10:29:20 +00:00

Author

Owner

ENOSPC campaign finding: the client reports ENOSPC and the initial partial-file/journal assertions pass, but recovery times out after unmounting the 4 MiB tmpfs. At timeout, server and client 1 contain full-disk.bin and small-after-full.txt, client 2 is empty, and its journal differs. The probe mounts tmpfs over the whole sync root, then unmounts it; I suspect this removes the private root identity marker and trips the root-change guard. I am checking that fixture interaction before changing the campaign.

kayg commented

2026-09-27 10:30:17 +00:00

Author

Owner

The ENOSPC timeout is confirmed as a fixture error. In a private mount namespace, the same tmpfs remounted from 4 MiB to 16 MiB preserved the root marker bytes and the root directory's device/inode. Unmounting instead reveals the underlying directory without that marker, which correctly triggers the existing local_root_changed guard. I will keep the tmpfs mounted and enlarge it for recovery.

kayg referenced this issue

2026-09-27 11:13:19 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 12:04:09 +00:00

Author

Owner

sync-chaos final report

Branch: job/sync-chaos
Head: 3202b25ccd43a2067340974a50841174dd01189a
Final dev merge: 1701cef5bd8936076522eb18039c3c8caed66299

Built

Added a seeded adversarial harness in tests/adversarial/sync_chaos.py. It runs a real server and two sync clients, injects proxy disconnects/stalls and process kills, exercises file and folder races, checks client trees against server disk, Index, and journals, and covers a 12 MiB tus upload and client ENOSPC recovery.
Updated crates/calternal-sync/random_campaign.py to exclude only the reserved root identity marker and UUID-shaped sync download temp names from inventories. Added tests/adversarial/test_sync_inventory.py to verify these internal names are filtered and lookalike user files remain visible.
Kept the root identity stable in the disk-full fixture by remounting its tmpfs in place after ENOSPC.

Campaign results

PASS sync chaos seed=20 steps=8 seconds=120.0 large=False
PASS sync chaos seed=20 steps=3 seconds=1.0 large=True
PASS sync chaos disk-full seed=20 exact recovery after ENOSPC
Seed 163 reproduced an NFC/NFD rename convergence failure: the sender retained the decomposed name while the server and other client retained the NFC name. Filed as #169.
Seed 173 reproduced a folder-move convergence failure: the old empty source directory remained on the server and one client. Filed as #170.

Gates

cargo fmt --check: exit code 0; no output.

cargo clippy --all-targets -- -D warnings (exit code 0):

    Checking calternal-plugin-photos v0.0.1 (/home/kayg/Developer/calternal-wt/sync-chaos/crates/plugins/photos)
   Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/sync-chaos/crates/calternal-server)
    Checking calternal-plugin-calendar v0.0.1 (/home/kayg/Developer/calternal-wt/sync-chaos/crates/plugins/calendar)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.90s

cargo test (exit code 0):

test result: ok. 46 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

apps/web bun run check:

$ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/sync-chaos/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

apps/web bun run test:

 Test Files  78 passed (78)
      Tests  579 passed (579)
   Start at  13:59:40
   Duration  119.62s (transform 68%, import 12%, environment 12%, tests 7%, setup 2%)

cargo clean: Removed 16981 files, 14.1GiB total. Removed apps/web/.svelte-kit and apps/web/build after the gates.

Known gaps

The #169 and #170 convergence bugs remain open and are linked above. The seeded campaign did not find another blocking issue.

Decisions not specified in the design

The disk-full fixture remounts the same tmpfs with a larger size so the sync root keeps its filesystem identity marker while testing recovery.
Inventory filtering uses the existing shared private-file predicate and exact reserved-name patterns. Similar names without the required UUID shape remain ordinary user files.

# sync-chaos final report Branch: `job/sync-chaos` Head: `3202b25ccd43a2067340974a50841174dd01189a` Final `dev` merge: `1701cef5bd8936076522eb18039c3c8caed66299` ## Built - Added a seeded adversarial harness in `tests/adversarial/sync_chaos.py`. It runs a real server and two sync clients, injects proxy disconnects/stalls and process kills, exercises file and folder races, checks client trees against server disk, Index, and journals, and covers a 12 MiB tus upload and client ENOSPC recovery. - Updated `crates/calternal-sync/random_campaign.py` to exclude only the reserved root identity marker and UUID-shaped sync download temp names from inventories. Added `tests/adversarial/test_sync_inventory.py` to verify these internal names are filtered and lookalike user files remain visible. - Kept the root identity stable in the disk-full fixture by remounting its tmpfs in place after ENOSPC. ## Campaign results - `PASS sync chaos seed=20 steps=8 seconds=120.0 large=False` - `PASS sync chaos seed=20 steps=3 seconds=1.0 large=True` - `PASS sync chaos disk-full seed=20 exact recovery after ENOSPC` - Seed 163 reproduced an NFC/NFD rename convergence failure: the sender retained the decomposed name while the server and other client retained the NFC name. Filed as #169. - Seed 173 reproduced a folder-move convergence failure: the old empty source directory remained on the server and one client. Filed as #170. ## Gates `cargo fmt --check`: exit code 0; no output. `cargo clippy --all-targets -- -D warnings` (exit code 0): ```text Checking calternal-plugin-photos v0.0.1 (/home/kayg/Developer/calternal-wt/sync-chaos/crates/plugins/photos) Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/sync-chaos/crates/calternal-server) Checking calternal-plugin-calendar v0.0.1 (/home/kayg/Developer/calternal-wt/sync-chaos/crates/plugins/calendar) Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.90s ``` `cargo test` (exit code 0): ```text test result: ok. 46 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `apps/web bun run check`: ```text $ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/sync-chaos/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `apps/web bun run test`: ```text Test Files 78 passed (78) Tests 579 passed (579) Start at 13:59:40 Duration 119.62s (transform 68%, import 12%, environment 12%, tests 7%, setup 2%) ``` `cargo clean`: `Removed 16981 files, 14.1GiB total`. Removed `apps/web/.svelte-kit` and `apps/web/build` after the gates. ## Known gaps The #169 and #170 convergence bugs remain open and are linked above. The seeded campaign did not find another blocking issue. ## Decisions not specified in the design - The disk-full fixture remounts the same tmpfs with a larger size so the sync root keeps its filesystem identity marker while testing recovery. - Inventory filtering uses the existing shared private-file predicate and exact reserved-name patterns. Similar names without the required UUID shape remain ordinary user files.

kayg commented

2026-09-27 12:06:46 +00:00

Author

Owner

Merged in f1c07669. Convergence bugs found by seeds 163/173 tracked in #169/#170.

Merged in f1c07669. Convergence bugs found by seeds 163/173 tracked in #169/#170.

kayg referenced this issue from a commit

2026-09-27 12:06:46 +00:00

Merge job/sync-chaos (#163)

kayg closed this issue

2026-09-27 12:06:46 +00:00

kayg referenced this issue