kayg/calternal

Watch

Fork 0

PERF: Files startup reconciliation holds the shared mutation lock across a whole Home (#663) #750

New issue

Open

opened 2026-10-02 13:08:54 +00:00 by kayg · 3 comments

kayg commented

2026-10-02 13:08:54 +00:00

Owner

Context: disk IO and startup audit for #663 rule 8. Source is origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5; merge-round-7a 2f4482ded066d9c5d9c59130377907f7fd2916c9 retains the same behavior.

crates/plugins/files/src/index.rs:80-87 holds Root::lock_mutation for the complete recursive scan of one Home. Within it, reconcile_folder_locked:184 lists the directory and :265 prepares entries; :1077 hashes changed files. The queued implementation holds the same lock at :91. crates/calternal-fs/src/root.rs:117,263-264 stores this mutex in shared Root state. Upload completion (uploads.rs:1242) and normal mutations (lib.rs:2725,2758 and other routes) require it too.

Reasoned impact: while startup reconciliation scans a large Home or hashes newly imported media, interactive writes in ANY Home can wait for all remaining folders and changed bytes. The socket already being ready does not make those mutations usable. The dedicated startup thread's nice/idle IO priority cannot preempt an application lock. Pure Index reads have a separate WAL reader pool, but mutations still wait. No duration is measured here, and no crash is asserted.

Concrete fix: enumerate and hash outside the shared mutation lock in bounded pages. Acquire the lock only for fingerprint revalidation and one bounded publication batch. Release it before reading the next page. Reuse existing prepare/write_records and fingerprint logic. Preserve move/Trash recovery, upload-install exclusion, item identity, Sidecar pairing and the one successful completion notice. Do not simply remove the lock or disable reconciliation.

Regression tests: pause a reconcile hash after reading its file, then require an interactive write in a different folder and another User's Home to finish before releasing that hash. Race rename/trash/overwrite and verify that old hashes and item IDs cannot be published for replaced files. Verify interrupted pages resume or rescan safely. Measure a large cold Home plus concurrent small writes on the locked HDD emulator, separating lock wait from IO and SQL time.

Duplicate check: searched all open/closed issue titles and read #681 (foreground counts/badges), #476 (per-User DAV lock and fsync throughput), #470 (Photos completion order), and #549. This finding concerns the instance-wide background reconcile lock, not per-User DAV lock checks or foreground count queries. Keep #476's authorization guarantees unchanged.

No security blocker is asserted. Audit only; no product edits.

Context: disk IO and startup audit for #663 rule 8. Source is origin/dev `c4a61e8cf090170f35b1bed3350d9de20c83ecd5`; merge-round-7a `2f4482ded066d9c5d9c59130377907f7fd2916c9` retains the same behavior. `crates/plugins/files/src/index.rs:80-87` holds `Root::lock_mutation` for the complete recursive scan of one Home. Within it, `reconcile_folder_locked:184` lists the directory and `:265` prepares entries; `:1077` hashes changed files. The queued implementation holds the same lock at :91. `crates/calternal-fs/src/root.rs:117,263-264` stores this mutex in shared Root state. Upload completion (`uploads.rs:1242`) and normal mutations (`lib.rs:2725,2758` and other routes) require it too. Reasoned impact: while startup reconciliation scans a large Home or hashes newly imported media, interactive writes in ANY Home can wait for all remaining folders and changed bytes. The socket already being ready does not make those mutations usable. The dedicated startup thread's nice/idle IO priority cannot preempt an application lock. Pure Index reads have a separate WAL reader pool, but mutations still wait. No duration is measured here, and no crash is asserted. Concrete fix: enumerate and hash outside the shared mutation lock in bounded pages. Acquire the lock only for fingerprint revalidation and one bounded publication batch. Release it before reading the next page. Reuse existing prepare/write_records and fingerprint logic. Preserve move/Trash recovery, upload-install exclusion, item identity, Sidecar pairing and the one successful completion notice. Do not simply remove the lock or disable reconciliation. Regression tests: pause a reconcile hash after reading its file, then require an interactive write in a different folder and another User's Home to finish before releasing that hash. Race rename/trash/overwrite and verify that old hashes and item IDs cannot be published for replaced files. Verify interrupted pages resume or rescan safely. Measure a large cold Home plus concurrent small writes on the locked HDD emulator, separating lock wait from IO and SQL time. Duplicate check: searched all open/closed issue titles and read #681 (foreground counts/badges), #476 (per-User DAV lock and fsync throughput), #470 (Photos completion order), and #549. This finding concerns the instance-wide background reconcile lock, not per-User DAV lock checks or foreground count queries. Keep #476's authorization guarantees unchanged. No security blocker is asserted. Audit only; no product edits.

kayg referenced this issue

2026-10-02 13:17:40 +00:00

PERF: tab switch to fully painted on a hard disk with heavy data (Calendar much slower than Photos); stress, measure, fix #549

kayg referenced this issue

2026-10-02 13:18:27 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:25:22 +00:00

Author

Owner

Starting implementation for this issue on branch job/ioperf, based at 2f4482ded066d9c5d9c59130377907f7fd2916c9 (job/merge-round-7a). I have read the issue and the #663 audit evidence. I am inspecting the job/hddsql-549 changes before editing shared startup, filesystem, and database paths. I will add a regression test and a focused benchmark for each performance path, then report measured results and crate gates here.

Starting implementation for this issue on branch `job/ioperf`, based at `2f4482ded066d9c5d9c59130377907f7fd2916c9` (`job/merge-round-7a`). I have read the issue and the #663 audit evidence. I am inspecting the `job/hddsql-549` changes before editing shared startup, filesystem, and database paths. I will add a regression test and a focused benchmark for each performance path, then report measured results and crate gates here.

kayg commented

2026-10-02 13:34:15 +00:00

Author

Owner

Confirmed in crates/plugins/files/src/index.rs: reconcile_all holds the Root-wide mutation mutex across every directory traversal for each User, and each folder pass hashes changed files before releasing it. This serializes unrelated Home writes behind background reads. I am narrowing that scope and adding a paused-hash concurrency regression while preserving fingerprint checks and identity recovery.

Confirmed in `crates/plugins/files/src/index.rs`: `reconcile_all` holds the Root-wide mutation mutex across every directory traversal for each User, and each folder pass hashes changed files before releasing it. This serializes unrelated Home writes behind background reads. I am narrowing that scope and adding a paused-hash concurrency regression while preserving fingerprint checks and identity recovery.

kayg commented

2026-10-02 17:27:08 +00:00

Author

Owner

ioperf final report

Built changes for #748, #750, #764, #800 and #806. Head: d43e7829a3e0c8a483cc45bdbb965af5f5d1c713.

Changes

#748: validate pending migrations before taking a startup snapshot; unchanged restarts skip it.
#750: list and hash one folder outside the Root mutation lock, then validate generations and publish the prepared Index changes.
#764: idle workers subscribe to queue changes and wake for due-job and lease deadlines; the server fallback is 30 seconds.
#800: DAV stat/open reads use indexed hashes without holding the Root mutation lock and verify full file fingerprints.
#806: the server owns one bounded recursive Home watcher and forwards external hints through the shared Root change stream. Overflow triggers a coalesced rebuild, including open collaboration rooms.

Files

crates/calternal-db/src/migrations.rs, crates/calternal-db/src/worker.rs, crates/calternal-server/src/wire.rs, crates/plugins/files/src/dav.rs, crates/plugins/files/src/index.rs, crates/plugins/files/src/lib.rs, crates/calternal-fs/src/lib.rs, crates/calternal-fs/src/quota.rs, crates/calternal-fs/src/root.rs, crates/calternal-fs/tests/storage.rs, crates/calternal-search/src/indexer.rs, crates/calternal-collab/src/session.rs.

Gate output

cargo fmt --all --check: exit 0; no output.

cargo clippy --offline -p calternal-db --all-targets -- -D warnings:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 55s

RUST_TEST_THREADS=1 cargo test --offline -p calternal-db:

running 27 tests
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 68.51s

running 17 tests
test result: ok. 16 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 25.17s

running 1 test
test sqlite_pools_bound_readers_and_page_cache ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.80s

Doc-tests calternal_db
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Focused Files regressions:

running 4 tests
test dav::tests::open_read_does_not_wait_for_the_home_mutation_lock ... ok
test dav::tests::stat_does_not_wait_for_the_home_mutation_lock ... ok
test tests::reconcile_hash_does_not_hold_mutation_lock_across_homes ... ok
test tests::public_password_rejection_does_not_wait_for_data_mutation_lock ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 156 filtered out; finished in 4.16s

bun run check:

$ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
User browser caches use userStorage; only documented device/public-link exceptions remain.
Text sizes and UI shape values use shared role tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/ioperf/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

Known gaps

The full Files test command was interrupted with exit 130 when the shared target stayed blocked on I/O. The focused four tests passed.
Final clippy/test gates for Files, server, fs, search and collaboration were not completed. The #806 server/filesystem integration is therefore not compile-verified in this run.
bun run test, adversarial probing, and the requested per-feature performance profiles and measurements were not completed.
cargo clean and removal of the generated web build output remain undone.
No UI changed, so UX gaps and screenshots are N/A.

Decisions

The #764 cross-process fallback is 30 seconds; local queue hints and durable deadlines provide prompt wakeups.
Watcher overflow recovery waits for a 100 ms quiet window before repeating a full repair.
Folder reconciliation reuses the existing eight-attempt policy for changed snapshots.
External watcher events carry an explicit source on the existing Root change stream, so only external changes publish the Home feed.

No UX gaps were introduced. Commits are on job/ioperf; no push or deploy was made.

## ioperf final report Built changes for #748, #750, #764, #800 and #806. Head: `d43e7829a3e0c8a483cc45bdbb965af5f5d1c713`. ### Changes - #748: validate pending migrations before taking a startup snapshot; unchanged restarts skip it. - #750: list and hash one folder outside the Root mutation lock, then validate generations and publish the prepared Index changes. - #764: idle workers subscribe to queue changes and wake for due-job and lease deadlines; the server fallback is 30 seconds. - #800: DAV stat/open reads use indexed hashes without holding the Root mutation lock and verify full file fingerprints. - #806: the server owns one bounded recursive Home watcher and forwards external hints through the shared Root change stream. Overflow triggers a coalesced rebuild, including open collaboration rooms. ### Files `crates/calternal-db/src/migrations.rs`, `crates/calternal-db/src/worker.rs`, `crates/calternal-server/src/wire.rs`, `crates/plugins/files/src/dav.rs`, `crates/plugins/files/src/index.rs`, `crates/plugins/files/src/lib.rs`, `crates/calternal-fs/src/lib.rs`, `crates/calternal-fs/src/quota.rs`, `crates/calternal-fs/src/root.rs`, `crates/calternal-fs/tests/storage.rs`, `crates/calternal-search/src/indexer.rs`, `crates/calternal-collab/src/session.rs`. ### Gate output `cargo fmt --all --check`: exit 0; no output. `cargo clippy --offline -p calternal-db --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 55s ``` `RUST_TEST_THREADS=1 cargo test --offline -p calternal-db`: ``` running 27 tests test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 68.51s running 17 tests test result: ok. 16 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 25.17s running 1 test test sqlite_pools_bound_readers_and_page_cache ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.80s Doc-tests calternal_db running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Focused Files regressions: ``` running 4 tests test dav::tests::open_read_does_not_wait_for_the_home_mutation_lock ... ok test dav::tests::stat_does_not_wait_for_the_home_mutation_lock ... ok test tests::reconcile_hash_does_not_hold_mutation_lock_across_homes ... ok test tests::public_password_rejection_does_not_wait_for_data_mutation_lock ... ok test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 156 filtered out; finished in 4.16s ``` `bun run check`: ``` $ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json User browser caches use userStorage; only documented device/public-link exceptions remain. Text sizes and UI shape values use shared role tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/ioperf/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` ### Known gaps - The full Files test command was interrupted with exit 130 when the shared target stayed blocked on I/O. The focused four tests passed. - Final clippy/test gates for Files, server, fs, search and collaboration were not completed. The #806 server/filesystem integration is therefore not compile-verified in this run. - `bun run test`, adversarial probing, and the requested per-feature performance profiles and measurements were not completed. - `cargo clean` and removal of the generated web build output remain undone. - No UI changed, so UX gaps and screenshots are N/A. ### Decisions - The #764 cross-process fallback is 30 seconds; local queue hints and durable deadlines provide prompt wakeups. - Watcher overflow recovery waits for a 100 ms quiet window before repeating a full repair. - Folder reconciliation reuses the existing eight-attempt policy for changed snapshots. - External watcher events carry an explicit source on the existing Root change stream, so only external changes publish the Home feed. No UX gaps were introduced. Commits are on `job/ioperf`; no push or deploy was made.

kayg referenced this issue

2026-10-02 17:27:25 +00:00

PERF: skip full pre-migration Index snapshots on unchanged restarts (#663) #748

kayg referenced this issue

2026-10-02 17:27:25 +00:00

PERF: wake the job worker on queue changes instead of empty one-second writes (#663) #764

kayg referenced this issue

2026-10-02 17:27:26 +00:00

PERF: isolate WebDAV file reads from the global mutation queue (#663) #800

kayg referenced this issue

2026-10-02 17:27:27 +00:00

PERF: recursive Home watcher registration repeats full directory walks before readiness (#663) #806