[WIP] Reduce ARM64 quota and task index overhead

kayg commented

2026-10-27 12:46:56 +00:00

Owner

At 100k notes, quota reconciliation repeatedly opened files and rebuilt path allocations, while task edits and background saves made redundant history commits. Reuse streamed quota snapshots, commit a disk seed with its first property edit, and coalesce background save bookkeeping. Durable acknowledgements still wait for the FULL history commit; file and directory syncs remain intact. The final trace records 2542 history WAL syncs for 2400 edits plus setup/shutdown, down from 5074.

Reduce cold task-index work through bounded overlapping reads and writes, fewer stored fields and closed-task rows, shared note fields, indexed link membership, and one final checkpoint for the fresh derived store. Skip unused body construction and repeated property parsing. The derived task store rebuilds at version 10. Invite failures now take one fixed-shape indexed lookup and one constant-time digest comparison as a consistency and defensive validation fix; auth schema 8 adds its covering index.

Validation passes: 434 notes tests, 36 task tests, 14 authentication security tests, 37 fsio tests, 41 additional authentication tests, relevant clippy checks, formatting and release builds. All original failing gates were measured three times before and after under perf-lock. Budgets and the harness are unchanged. Exact before/after/limit table, assessment and remaining failures.

RSS, invite ratios and API note-open pass all three final repetitions. Idle CPU remains about 0.17%; identical quota walks on disk and tmpfs identify the filesystem cost. Task warm-update p95 remains 43.14–46.80 ms at 100k; moving only application state to tmpfs passes every update assertion with the same physical note files and FULL history settings.

This is a draft because cold-index growth remains 15.62–18.12% against 10%. Whole-fixture tmpfs controls still show 13.66–14.89%, so the remaining index overhead is not fully explained by storage and needs more work. Both comparison hosts timed out on bounded SSH probes.

Stacked on #691 (issue/perf-move at 31c49e8b601ebe11469b9994e769b67184616061). References #686. Measured head: 5ed9dcc7e3604be3835fd87f7354d3e3f60610bb.

At 100k notes, quota reconciliation repeatedly opened files and rebuilt path allocations, while task edits and background saves made redundant history commits. Reuse streamed quota snapshots, commit a disk seed with its first property edit, and coalesce background save bookkeeping. Durable acknowledgements still wait for the FULL history commit; file and directory syncs remain intact. The final trace records 2542 history WAL syncs for 2400 edits plus setup/shutdown, down from 5074. Reduce cold task-index work through bounded overlapping reads and writes, fewer stored fields and closed-task rows, shared note fields, indexed link membership, and one final checkpoint for the fresh derived store. Skip unused body construction and repeated property parsing. The derived task store rebuilds at version 10. Invite failures now take one fixed-shape indexed lookup and one constant-time digest comparison as a consistency and defensive validation fix; auth schema 8 adds its covering index. Validation passes: 434 notes tests, 36 task tests, 14 authentication security tests, 37 fsio tests, 41 additional authentication tests, relevant clippy checks, formatting and release builds. All original failing gates were measured three times before and after under perf-lock. Budgets and the harness are unchanged. [Exact before/after/limit table](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-66156), [assessment and remaining failures](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-66160). RSS, invite ratios and API note-open pass all three final repetitions. Idle CPU remains about 0.17%; identical quota walks on disk and tmpfs identify the filesystem cost. Task warm-update p95 remains 43.14–46.80 ms at 100k; moving only application state to tmpfs passes every update assertion with the same physical note files and FULL history settings. This is a draft because cold-index growth remains 15.62–18.12% against 10%. Whole-fixture tmpfs controls still show 13.66–14.89%, so the remaining index overhead is not fully explained by storage and needs more work. Both comparison hosts timed out on bounded SSH probes. Stacked on #691 (`issue/perf-move` at `31c49e8b601ebe11469b9994e769b67184616061`). References #686. Measured head: `5ed9dcc7e3604be3835fd87f7354d3e3f60610bb`.

kayg added 16 commits

2026-10-27 12:46:56 +00:00

Reduce quota recount work and task write overhead b499f9a258

Reduce task index write amplification during cold fills a28185bfc9

Stream quota recounts and retain the bulk fill page cache d5aa104306

Avoid duplicate parsing and group later cold-fill writes bfc879f0e5

Commit a disk seed with its first property edit 1340a68106

Skip discarded checkbox fields during task index discovery b55bf2ad04

Reuse quota snapshot paths across drift recounts 473b6282b7

Read simple display properties without rebuilding a YAML tree bc354f92d6

Avoid unused property copies during task description aaf442509a

Keep closed task tags out of live indexes 3ac76b9e15