PERF: wake the job worker on queue changes instead of empty one-second writes (#663)

kayg commented

2026-10-02 13:09:56 +00:00

Owner

Parent audit: #663. Baseline: c4a61e8cf0; unchanged in round 7a.

C2: Empty job queue causes a writer transaction every second

Status: source-confirmed performance finding. No crash or security exploit was reproduced.

Evidence: crates/calternal-server/src/wire.rs:4692 sets Worker poll to 1 s.
crates/calternal-db/src/worker.rs:300 attempts a lease for enabled kinds on
every empty pass; :333 sleeps and repeats. It does not subscribe to queue
changes, although Db::subscribe_job_changes exists. In jobs.rs,
lease_matching executes UPDATE ... RETURNING through the writer pool
(jobs.rs:645–673). Empty work still acquires the one writer connection.
This is one implicit statement transaction, not an explicit BEGIN block.

Impact estimate: 86,400 lease attempts per day per idle Instance. This is
baseline idle I/O and CPU, independent of User count. It does not imply
86,400 fsyncs: an unchanged transaction need not dirty a SQLite page.

Fix: wake on durable queue changes or the next due job/lease deadline. Keep
a longer fallback for out-of-process changes and recovery. Subscribe before
checking the queue so a wake cannot be lost.

Test: fake clock or query counter must show no one-second writer polling
on an empty queue. Immediate enqueue must wake promptly. Delayed jobs and
expired leases must still run at the correct deadline.

Duplicate search: all-state idle and unbounded; no matching empty-worker
issue. #367 is the broad performance review.

Parent audit: #663. Baseline: c4a61e8cf090170f35b1bed3350d9de20c83ecd5; unchanged in round 7a. C2: Empty job queue causes a writer transaction every second Status: source-confirmed performance finding. No crash or security exploit was reproduced. Evidence: crates/calternal-server/src/wire.rs:4692 sets Worker poll to 1 s. crates/calternal-db/src/worker.rs:300 attempts a lease for enabled kinds on every empty pass; :333 sleeps and repeats. It does not subscribe to queue changes, although Db::subscribe_job_changes exists. In jobs.rs, lease_matching executes UPDATE ... RETURNING through the writer pool (jobs.rs:645–673). Empty work still acquires the one writer connection. This is one implicit statement transaction, not an explicit BEGIN block. Impact estimate: 86,400 lease attempts per day per idle Instance. This is baseline idle I/O and CPU, independent of User count. It does not imply 86,400 fsyncs: an unchanged transaction need not dirty a SQLite page. Fix: wake on durable queue changes or the next due job/lease deadline. Keep a longer fallback for out-of-process changes and recovery. Subscribe before checking the queue so a wake cannot be lost. Test: fake clock or query counter must show no one-second writer polling on an empty queue. Immediate enqueue must wake promptly. Delayed jobs and expired leases must still run at the correct deadline. Duplicate search: all-state idle and unbounded; no matching empty-worker issue. #367 is the broad performance review.

kayg referenced this issue

2026-10-02 13:13:24 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:17:45 +00:00

Author

Owner

Additional core scheduler evidence: crates/calternal-server/src/wire.rs:5001–5050 rebuilds cron definitions and calls enqueue_due every second. Coordinate this with the Worker next-deadline wake change rather than leave a second one-second loop. Correction for exact mechanism: jobs.rs:645–673 runs UPDATE ... RETURNING directly on the writer pool; it is one implicit statement transaction, not an explicit BEGIN block. No claim of a dirty page or fsync on an empty pass.

kayg referenced this issue

2026-10-02 13:22:10 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:22:24 +00:00

PERF: publish Search generations without whole-manifest writer transactions (#663) #832

kayg commented

2026-10-02 13:25:24 +00:00

Author

Owner

Starting implementation for this issue on branch job/ioperf, based at 2f4482ded066d9c5d9c59130377907f7fd2916c9 (job/merge-round-7a). I have read the issue and the #663 audit evidence. I am inspecting the job/hddsql-549 changes before editing shared startup, filesystem, and database paths. I will add a regression test and a focused benchmark for each performance path, then report measured results and crate gates here.

Starting implementation for this issue on branch `job/ioperf`, based at `2f4482ded066d9c5d9c59130377907f7fd2916c9` (`job/merge-round-7a`). I have read the issue and the #663 audit evidence. I am inspecting the `job/hddsql-549` changes before editing shared startup, filesystem, and database paths. I will add a regression test and a focused benchmark for each performance path, then report measured results and crate gates here.

kayg commented

2026-10-02 13:34:16 +00:00

Author

Owner

Confirmed in crates/calternal-db/src/worker.rs: after an empty lease attempt, the worker sleeps for its one-second poll interval and repeats the writer-pool UPDATE ... RETURNING lease query. Db::subscribe_job_changes already exists and queue mutations publish hints. I am changing the idle wait to subscribe before checking, wake on durable queue changes or the next due deadline, and retain a longer recovery fallback.

Confirmed in `crates/calternal-db/src/worker.rs`: after an empty lease attempt, the worker sleeps for its one-second poll interval and repeats the writer-pool `UPDATE ... RETURNING` lease query. `Db::subscribe_job_changes` already exists and queue mutations publish hints. I am changing the idle wait to subscribe before checking, wake on durable queue changes or the next due deadline, and retain a longer recovery fallback.

kayg referenced this issue

2026-10-02 17:27:10 +00:00

PERF: Files startup reconciliation holds the shared mutation lock across a whole Home (#663) #750

kayg referenced this issue

2026-10-02 17:27:25 +00:00

PERF: skip full pre-migration Index snapshots on unchanged restarts (#663) #748

kayg commented

2026-10-02 17:27:25 +00:00

Author

Owner

ioperf final report

Built changes for #748, #750, #764, #800 and #806. Head: d43e7829a3e0c8a483cc45bdbb965af5f5d1c713.

Changes

#748: validate pending migrations before taking a startup snapshot; unchanged restarts skip it.
#750: list and hash one folder outside the Root mutation lock, then validate generations and publish the prepared Index changes.
#764: idle workers subscribe to queue changes and wake for due-job and lease deadlines; the server fallback is 30 seconds.
#800: DAV stat/open reads use indexed hashes without holding the Root mutation lock and verify full file fingerprints.
#806: the server owns one bounded recursive Home watcher and forwards external hints through the shared Root change stream. Overflow triggers a coalesced rebuild, including open collaboration rooms.

Files

crates/calternal-db/src/migrations.rs, crates/calternal-db/src/worker.rs, crates/calternal-server/src/wire.rs, crates/plugins/files/src/dav.rs, crates/plugins/files/src/index.rs, crates/plugins/files/src/lib.rs, crates/calternal-fs/src/lib.rs, crates/calternal-fs/src/quota.rs, crates/calternal-fs/src/root.rs, crates/calternal-fs/tests/storage.rs, crates/calternal-search/src/indexer.rs, crates/calternal-collab/src/session.rs.

Gate output

cargo fmt --all --check: exit 0; no output.

cargo clippy --offline -p calternal-db --all-targets -- -D warnings:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 55s

RUST_TEST_THREADS=1 cargo test --offline -p calternal-db:

running 27 tests
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 68.51s

running 17 tests
test result: ok. 16 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 25.17s

running 1 test
test sqlite_pools_bound_readers_and_page_cache ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.80s

Doc-tests calternal_db
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Focused Files regressions:

running 4 tests
test dav::tests::open_read_does_not_wait_for_the_home_mutation_lock ... ok
test dav::tests::stat_does_not_wait_for_the_home_mutation_lock ... ok
test tests::reconcile_hash_does_not_hold_mutation_lock_across_homes ... ok
test tests::public_password_rejection_does_not_wait_for_data_mutation_lock ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 156 filtered out; finished in 4.16s

bun run check:

$ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
User browser caches use userStorage; only documented device/public-link exceptions remain.
Text sizes and UI shape values use shared role tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/ioperf/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

Known gaps

The full Files test command was interrupted with exit 130 when the shared target stayed blocked on I/O. The focused four tests passed.
Final clippy/test gates for Files, server, fs, search and collaboration were not completed. The #806 server/filesystem integration is therefore not compile-verified in this run.
bun run test, adversarial probing, and the requested per-feature performance profiles and measurements were not completed.
cargo clean and removal of the generated web build output remain undone.
No UI changed, so UX gaps and screenshots are N/A.

Decisions

The #764 cross-process fallback is 30 seconds; local queue hints and durable deadlines provide prompt wakeups.
Watcher overflow recovery waits for a 100 ms quiet window before repeating a full repair.
Folder reconciliation reuses the existing eight-attempt policy for changed snapshots.
External watcher events carry an explicit source on the existing Root change stream, so only external changes publish the Home feed.

No UX gaps were introduced. Commits are on job/ioperf; no push or deploy was made.

## ioperf final report Built changes for #748, #750, #764, #800 and #806. Head: `d43e7829a3e0c8a483cc45bdbb965af5f5d1c713`. ### Changes - #748: validate pending migrations before taking a startup snapshot; unchanged restarts skip it. - #750: list and hash one folder outside the Root mutation lock, then validate generations and publish the prepared Index changes. - #764: idle workers subscribe to queue changes and wake for due-job and lease deadlines; the server fallback is 30 seconds. - #800: DAV stat/open reads use indexed hashes without holding the Root mutation lock and verify full file fingerprints. - #806: the server owns one bounded recursive Home watcher and forwards external hints through the shared Root change stream. Overflow triggers a coalesced rebuild, including open collaboration rooms. ### Files `crates/calternal-db/src/migrations.rs`, `crates/calternal-db/src/worker.rs`, `crates/calternal-server/src/wire.rs`, `crates/plugins/files/src/dav.rs`, `crates/plugins/files/src/index.rs`, `crates/plugins/files/src/lib.rs`, `crates/calternal-fs/src/lib.rs`, `crates/calternal-fs/src/quota.rs`, `crates/calternal-fs/src/root.rs`, `crates/calternal-fs/tests/storage.rs`, `crates/calternal-search/src/indexer.rs`, `crates/calternal-collab/src/session.rs`. ### Gate output `cargo fmt --all --check`: exit 0; no output. `cargo clippy --offline -p calternal-db --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 55s ``` `RUST_TEST_THREADS=1 cargo test --offline -p calternal-db`: ``` running 27 tests test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 68.51s running 17 tests test result: ok. 16 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 25.17s running 1 test test sqlite_pools_bound_readers_and_page_cache ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.80s Doc-tests calternal_db running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Focused Files regressions: ``` running 4 tests test dav::tests::open_read_does_not_wait_for_the_home_mutation_lock ... ok test dav::tests::stat_does_not_wait_for_the_home_mutation_lock ... ok test tests::reconcile_hash_does_not_hold_mutation_lock_across_homes ... ok test tests::public_password_rejection_does_not_wait_for_data_mutation_lock ... ok test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 156 filtered out; finished in 4.16s ``` `bun run check`: ``` $ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json User browser caches use userStorage; only documented device/public-link exceptions remain. Text sizes and UI shape values use shared role tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/ioperf/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` ### Known gaps - The full Files test command was interrupted with exit 130 when the shared target stayed blocked on I/O. The focused four tests passed. - Final clippy/test gates for Files, server, fs, search and collaboration were not completed. The #806 server/filesystem integration is therefore not compile-verified in this run. - `bun run test`, adversarial probing, and the requested per-feature performance profiles and measurements were not completed. - `cargo clean` and removal of the generated web build output remain undone. - No UI changed, so UX gaps and screenshots are N/A. ### Decisions - The #764 cross-process fallback is 30 seconds; local queue hints and durable deadlines provide prompt wakeups. - Watcher overflow recovery waits for a 100 ms quiet window before repeating a full repair. - Folder reconciliation reuses the existing eight-attempt policy for changed snapshots. - External watcher events carry an explicit source on the existing Root change stream, so only external changes publish the Home feed. No UX gaps were introduced. Commits are on `job/ioperf`; no push or deploy was made.

kayg referenced this issue

2026-10-02 17:27:26 +00:00

PERF: isolate WebDAV file reads from the global mutation queue (#663) #800

kayg referenced this issue

2026-10-02 17:27:27 +00:00

PERF: recursive Home watcher registration repeats full directory walks before readiness (#663) #806