Sync: folder move leaves empty old directory on server and peer

kayg commented

2026-09-26 12:33:58 +00:00

Owner

Seeded real-server sync chaos finds a persistent empty source directory after a folder move.

Reproduce from job/sync-chaos with built calternal-server and calternald binaries:

TMPDIR=$PWD/target/tmp python3 tests/adversarial/sync_chaos.py --seed 173 --seconds 180 --steps 7

Seed 173 has failed three times at folder_move after all file bytes and both journals converge. The originating installation creates folder-6/round-6.bin, waits for both installations and server, then renames folder-6 to moved-6. At the 35-second deadline, all three file trees contain moved-6/round-6.bin with identical bytes and no file under folder-6; the directories differ:

Server data directory: folder-6, moved-6.
Originating installation: moved-6.
Receiving installation: folder-6, moved-6.

The old empty folder persists in the server Index as well. The sync engine must remove the old directory on move and converge all three trees. Found in #163. The chaos probe keeps the exact directory assertion as a regression. Correcting this changes product sync behavior outside the test job.

Seeded real-server sync chaos finds a persistent empty source directory after a folder move. Reproduce from `job/sync-chaos` with built `calternal-server` and `calternald` binaries: ``` TMPDIR=$PWD/target/tmp python3 tests/adversarial/sync_chaos.py --seed 173 --seconds 180 --steps 7 ``` Seed 173 has failed three times at `folder_move` after all file bytes and both journals converge. The originating installation creates `folder-6/round-6.bin`, waits for both installations and server, then renames `folder-6` to `moved-6`. At the 35-second deadline, all three file trees contain `moved-6/round-6.bin` with identical bytes and no file under `folder-6`; the directories differ: - Server data directory: `folder-6`, `moved-6`. - Originating installation: `moved-6`. - Receiving installation: `folder-6`, `moved-6`. The old empty folder persists in the server Index as well. The sync engine must remove the old directory on move and converge all three trees. Found in #163. The chaos probe keeps the exact directory assertion as a regression. Correcting this changes product sync behavior outside the test job.

kayg referenced this issue

2026-09-26 12:34:13 +00:00

Sync chaos test: kill, cut and race the sync client against a real server #163

kayg referenced this issue

2026-09-27 10:00:55 +00:00

Sync chaos test: kill, cut and race the sync client against a real server #163

kayg referenced this issue

2026-09-27 10:25:45 +00:00

Sync chaos test: kill, cut and race the sync client against a real server #163

kayg referenced this issue

2026-09-27 12:04:09 +00:00

Sync chaos test: kill, cut and race the sync client against a real server #163

kayg commented

2026-09-27 12:08:34 +00:00

Author

Owner

Starting fix and reproduction on job/sync-converge, based on dev at f1c0766907d5d1a0a3a7ea0ae807e43dd2caf9f0.

Starting fix and reproduction on `job/sync-converge`, based on `dev` at `f1c0766907d5d1a0a3a7ea0ae807e43dd2caf9f0`.

kayg commented

2026-09-27 12:30:59 +00:00

Author

Owner

Reproduced before changes with the real-server chaos harness:

TMPDIR=$PWD/target/tmp python3 tests/adversarial/sync_chaos.py --seed 173 --seconds 180 --steps 7

It failed at folder_move. The file bytes, paths and both journals had converged, but the server data directory and receiving installation retained both folder-6 and moved-6; the originating installation had only moved-6. The harness also found the stale directory in the server Index.

Reproduced before changes with the real-server chaos harness: ``` TMPDIR=$PWD/target/tmp python3 tests/adversarial/sync_chaos.py --seed 173 --seconds 180 --steps 7 ``` It failed at `folder_move`. The file bytes, paths and both journals had converged, but the server data directory and receiving installation retained both `folder-6` and `moved-6`; the originating installation had only `moved-6`. The harness also found the stale directory in the server Index.

kayg commented

2026-09-27 12:33:20 +00:00

Author

Owner

Reproduced before changes with the real-server chaos harness:

TMPDIR=$PWD/target/tmp python3 tests/adversarial/sync_chaos.py --seed 173 --seconds 180 --steps 7

It failed at folder_move. Both daemon journals and file bytes matched, but the authoritative disk and installation 2 retained empty folder-6 beside moved-6. The Index directory rows also differed from installation 1.

Reproduced before changes with the real-server chaos harness: ``` TMPDIR=$PWD/target/tmp python3 tests/adversarial/sync_chaos.py --seed 173 --seconds 180 --steps 7 ``` It failed at `folder_move`. Both daemon journals and file bytes matched, but the authoritative disk and installation 2 retained empty `folder-6` beside `moved-6`. The Index directory rows also differed from installation 1.

kayg commented

2026-09-27 13:14:25 +00:00

Author

Owner

Root cause: rename following handled each child file separately. The Files API created the destination folder but did not move the folder inode, so an empty source folder remained on the server and receiving installations.

The sync engine now detects a folder move only when the tracked files still match by item ID, content hash and local file identity. It moves the folder through the existing Files API, mirrors it with a directory-handle-relative local rename, and rebases the baseline and inventories together. Selective pairs also check every descendant before moving a folder.

Verification so far: --regression folder_move passes against the real server; cargo test -p calternal-sync passes 50 library tests and 2 daemon tests; cargo clippy -p calternal-sync --all-targets -- -D warnings passes.

Root cause: rename following handled each child file separately. The Files API created the destination folder but did not move the folder inode, so an empty source folder remained on the server and receiving installations. The sync engine now detects a folder move only when the tracked files still match by item ID, content hash and local file identity. It moves the folder through the existing Files API, mirrors it with a directory-handle-relative local rename, and rebases the baseline and inventories together. Selective pairs also check every descendant before moving a folder. Verification so far: `--regression folder_move` passes against the real server; `cargo test -p calternal-sync` passes 50 library tests and 2 daemon tests; `cargo clippy -p calternal-sync --all-targets -- -D warnings` passes.

kayg commented

2026-09-27 14:37:37 +00:00

Author

Owner

Fixed Forgejo #170: moving a folder now removes the old directory on the server and peer Installations. The sync engine confirms the source subtree still matches the tracked tree, performs the server folder rename through the existing Files API, then safely renames the peer directory and rebases its path maps. This prevents an empty source directory from being recreated during reconciliation.

Branch: job/sync-converge
Head: 39a30cf88a2c35e9e137d69563448426f0cfdcb1
Latest dev merged: 74d6072643e59cf8668a0cbd9e02b3e36b79736e

Changes:

NFC normalization keeps a local NFD rename on the server's canonical NFC spelling. The local file is renamed through directory handles, and the journal and path map are updated to that spelling.
Folder moves are sent as one folder rename when the unchanged tracked subtree is fully inside the selected sync scope. Peer Installations rename the directory safely, and all path snapshots are rebased together. If the subtree is not a complete match, sync keeps the existing item-level behavior.
Added real-server deterministic regressions for both cases and aligned the random Unicode chaos assertion with the server's NFC spelling.

Validation:

Final-head deterministic real-server replays: PASS sync chaos regression=nfd_rename; PASS sync chaos regression=folder_move.
Random real-server chaos on bfb78b9a: PASS sync chaos matrix seeds=1..40,163,173 count=42 (10 randomized rounds per seed, plus built-in upload and interrupted-download cases). The later dev merge did not change sync reconciliation; both deterministic replays were repeated on final head.
cargo fmt --all -- --check: no output.
cargo clippy --all-targets -- -D warnings: Finished dev profile [unoptimized + debuginfo] target(s) in 12m 37s. Clippy emitted no diagnostics. The shell summary wrapper then printed zsh:1: read-only variable: status after Cargo exited.
cargo test: calternal-sync reported test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; finished in 1.43s; daemon tests reported test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s; command exit code 0.
bun run --cwd apps/web check: svelte-check found 0 errors and 0 warnings (exit 0).
bun run --cwd apps/web test: Test Files 80 passed (80) and Tests 585 passed (585) (exit 0).

Known gaps: Vitest printed jsdom notices (Could not parse CSS stylesheet and Not implemented: Window's scrollTo() method) but all tests passed. No sync gap remains known.

Decisions not stated in the design: the server's NFC spelling wins when a local rename changes only Unicode normalization; directory moves are treated as a unit only when every tracked descendant is unchanged and within the selected scope.

Fixed Forgejo #170: moving a folder now removes the old directory on the server and peer Installations. The sync engine confirms the source subtree still matches the tracked tree, performs the server folder rename through the existing Files API, then safely renames the peer directory and rebases its path maps. This prevents an empty source directory from being recreated during reconciliation. Branch: `job/sync-converge` Head: `39a30cf88a2c35e9e137d69563448426f0cfdcb1` Latest dev merged: `74d6072643e59cf8668a0cbd9e02b3e36b79736e` Changes: - NFC normalization keeps a local NFD rename on the server's canonical NFC spelling. The local file is renamed through directory handles, and the journal and path map are updated to that spelling. - Folder moves are sent as one folder rename when the unchanged tracked subtree is fully inside the selected sync scope. Peer Installations rename the directory safely, and all path snapshots are rebased together. If the subtree is not a complete match, sync keeps the existing item-level behavior. - Added real-server deterministic regressions for both cases and aligned the random Unicode chaos assertion with the server's NFC spelling. Validation: - Final-head deterministic real-server replays: `PASS sync chaos regression=nfd_rename`; `PASS sync chaos regression=folder_move`. - Random real-server chaos on `bfb78b9a`: `PASS sync chaos matrix seeds=1..40,163,173 count=42` (10 randomized rounds per seed, plus built-in upload and interrupted-download cases). The later `dev` merge did not change sync reconciliation; both deterministic replays were repeated on final head. - `cargo fmt --all -- --check`: no output. - `cargo clippy --all-targets -- -D warnings`: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 37s`. Clippy emitted no diagnostics. The shell summary wrapper then printed `zsh:1: read-only variable: status` after Cargo exited. - `cargo test`: `calternal-sync` reported `test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; finished in 1.43s`; daemon tests reported `test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s`; command exit code 0. - `bun run --cwd apps/web check`: `svelte-check found 0 errors and 0 warnings` (exit 0). - `bun run --cwd apps/web test`: `Test Files 80 passed (80)` and `Tests 585 passed (585)` (exit 0). Known gaps: Vitest printed jsdom notices (`Could not parse CSS stylesheet` and `Not implemented: Window's scrollTo() method`) but all tests passed. No sync gap remains known. Decisions not stated in the design: the server's NFC spelling wins when a local rename changes only Unicode normalization; directory moves are treated as a unit only when every tracked descendant is unchanged and within the selected scope.

kayg referenced this issue from a commit

2026-09-27 14:39:50 +00:00

Merge job/sync-converge (#169, #170)

kayg commented

2026-09-27 14:39:51 +00:00

Author

Owner

Merged in 235073a6.

Merged in 235073a6.

kayg closed this issue

2026-09-27 14:39:52 +00:00