[WIP] Cut idle scans and avoid unchanged index writes on ARM64

kayg commented

2026-10-27 21:21:33 +00:00

Owner

Idle quota accounting repeatedly walked every note, while task edits rewrote derived rows that had not changed. Native quota watching now updates usage when files change, retains a polling fallback, and bounds its reconciliation journal. Metadata updates preserve unchanged fields, task memberships and backlinks; a reader pool, borrowed property keys and smaller checkbox rows reduce repeated work. Fresh indexing batches plugin writes through the existing interface.

Invite validation performs the same covering lookup, constant-time secret comparison and legacy lookup for wrong, malformed, expired, revoked and used candidates within one read transaction. Tests cover lifecycle transitions and the validation operations. Primary FULL durability, receipt guarantees and file/directory sync barriers remain intact.

Supersedes draft #703 and remains stacked on #691 through issue/perf-move. References #686. Budgets and the performance harness are unchanged. Evidence belongs in issue comments; no measurements, logs or diagnostic probes are committed.

Performance acceptance remains blocked. At head 21180d0b2a82b8003cb1f1ee117a0a300ba5edaa, candidate warm task update p95 is 48.152, 43.162, 42.290 ms at 10k and 46.100, 43.995, 46.104 ms at 100k; all six exceed the unchanged 40 ms limit. First-touch at 10k is 24.177, 15.132, 22.687 ms and passes all three; at 100k it is 40.909, 39.967, 32.814 ms, with one miss. Cold growth is 6.079, 7.862, 8.566% at 10k and 7.966, 11.454, 22.262% at 100k. The 100k cold gate fails two of three repetitions and regresses in the third paired comparison. It is unresolved.

Candidate idle CPU is 0.000, 0.000, 0.000% across the three original 300-second windows. Long-note open p95 is 20.571, 20.747, 24.110 ms after preserving identical backlinks. Candidate invite max/min median ratios are 2.245, 2.119, 2.158, all below 2.5, but still above the near-1 goal. Application validation operations are equalized; the disk timing difference is not claimed eliminated.

Code checks passed: 1,185 workspace tests (20 ignored), clippy with warnings denied, fmt, both typechecks, 5,000 parser cases and 39 fsio tests. Web retains 14 existing warnings. Both before and after were rebuilt and alternated for three repetitions on the updated macOS 27.2 host after the required five-minute quiet period. All pre-reboot measurements are void. The draft remains blocked by task update and cold-index budgets. Full original outputs and noise tables are on issue #686.

Idle quota accounting repeatedly walked every note, while task edits rewrote derived rows that had not changed. Native quota watching now updates usage when files change, retains a polling fallback, and bounds its reconciliation journal. Metadata updates preserve unchanged fields, task memberships and backlinks; a reader pool, borrowed property keys and smaller checkbox rows reduce repeated work. Fresh indexing batches plugin writes through the existing interface. Invite validation performs the same covering lookup, constant-time secret comparison and legacy lookup for wrong, malformed, expired, revoked and used candidates within one read transaction. Tests cover lifecycle transitions and the validation operations. Primary FULL durability, receipt guarantees and file/directory sync barriers remain intact. Supersedes draft #703 and remains stacked on #691 through `issue/perf-move`. References #686. Budgets and the performance harness are unchanged. Evidence belongs in issue comments; no measurements, logs or diagnostic probes are committed. Performance acceptance remains blocked. At head `21180d0b2a82b8003cb1f1ee117a0a300ba5edaa`, candidate warm task update p95 is 48.152, 43.162, 42.290 ms at 10k and 46.100, 43.995, 46.104 ms at 100k; all six exceed the unchanged 40 ms limit. First-touch at 10k is 24.177, 15.132, 22.687 ms and passes all three; at 100k it is 40.909, 39.967, 32.814 ms, with one miss. Cold growth is 6.079, 7.862, 8.566% at 10k and 7.966, 11.454, 22.262% at 100k. The 100k cold gate fails two of three repetitions and regresses in the third paired comparison. It is unresolved. Candidate idle CPU is 0.000, 0.000, 0.000% across the three original 300-second windows. Long-note open p95 is 20.571, 20.747, 24.110 ms after preserving identical backlinks. Candidate invite max/min median ratios are 2.245, 2.119, 2.158, all below 2.5, but still above the near-1 goal. Application validation operations are equalized; the disk timing difference is not claimed eliminated. Code checks passed: 1,185 workspace tests (20 ignored), clippy with warnings denied, fmt, both typechecks, 5,000 parser cases and 39 fsio tests. Web retains 14 existing warnings. Both before and after were rebuilt and alternated for three repetitions on the updated macOS 27.2 host after the required five-minute quiet period. All pre-reboot measurements are void. The draft remains blocked by task update and cold-index budgets. Full original outputs and noise tables are on [issue #686](https://git.kayg.org/kayg/calternal/issues/686).

kayg added 19 commits

2026-10-27 21:21:33 +00:00

Reduce quota recount work and task write overhead b499f9a258

Reduce task index write amplification during cold fills a28185bfc9

Stream quota recounts and retain the bulk fill page cache d5aa104306

Avoid duplicate parsing and group later cold-fill writes bfc879f0e5

Commit a disk seed with its first property edit 1340a68106

Skip discarded checkbox fields during task index discovery b55bf2ad04

Reuse quota snapshot paths across drift recounts 473b6282b7

Read simple display properties without rebuilding a YAML tree bc354f92d6

Avoid unused property copies during task description aaf442509a

Keep closed task tags out of live indexes 3ac76b9e15