kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Keep CardDAV rebuilds and cold reads within their budgets #739

Closed
kayg wants to merge 1 commit from issue/carddav-read-perf-fix into issue/carddav-read
pull from: issue/carddav-read-perf-fix
merge into: kayg:issue/carddav-read
Conversation 2 Commits 1 Files changed 8 +289 -42
kayg commented 2026-10-28 18:15:12 +00:00
Owner
Copy link

CardDAV rebuilds and cold collection reads exceeded their budgets because the cache duplicated wire payloads and performed many SQLite page reads and writes. Store each payload once in revision history, keep compact live metadata for PROPFIND, and build replacement tables in memory before installing them with one atomic SQLite backup transaction. Multiget reuses a prepared lookup through a bounded 64 MiB mapping. The committed head lives beside the cursor, and an unchanged sync skips revision scanning.

This is a focused follow-up to #734 for #657, based on issue/carddav-read. The migration preserves wire rows, cursor, epoch and in-progress sync snapshots. WAL NORMAL, 4 KiB pages, default checkpoint policy and all timing limits remain unchanged. Rebuild staging temporarily uses memory proportional to the projected book. Format parsing and conversion stay in the contacts plugin.

Validation: fmt, all 20 CardDAV tests, workspace clippy with warnings denied, and workspace tests. Coverage includes old-cache migration with a paged snapshot and deletion, reads across WAL commits, metadata agreement with GET, and failed or busy rebuild rollback and retry.

Five cold and five warm measurements per operation on the assigned perf-test-02 VM, using the preceding job’s perf-run.sh method, Homes on /srv/homes and state on the OS SSD. Maximum across both conditions:

Operation, 1,000 cards Before max ms After max ms Limit
rebuild 85.749347 19.509446 <30 ms
sync 1.092146 1.134989 <2 ms
propfind 35.103631 9.052732 <30 ms
multiget 95.753909 14.469957 <30 ms

Every rebuild compared href, wire bytes, strong ETag and serialized property index against an independent incremental baseline, at both 1,000 and 10,000 cards. All comparisons passed. Raw tables, unsuccessful intermediate experiments and I/O profiles are posted on #657 and #734. Cold rebuild writes fell from 833 to 471 fragments; the final profile retains two durability syncs.

The workspace run passed 1,175 tests with 20 existing ignored tests. The first run hit an empty Bun bridge response while JavaScript dependencies were absent. After bun install --frozen-lockfile, both bridge tests and the full workspace retry passed without source changes. The first failed output is retained with the evidence.

Complete measurement and gate evidence

CardDAV rebuilds and cold collection reads exceeded their budgets because the cache duplicated wire payloads and performed many SQLite page reads and writes. Store each payload once in revision history, keep compact live metadata for PROPFIND, and build replacement tables in memory before installing them with one atomic SQLite backup transaction. Multiget reuses a prepared lookup through a bounded 64 MiB mapping. The committed head lives beside the cursor, and an unchanged sync skips revision scanning. This is a focused follow-up to #734 for #657, based on issue/carddav-read. The migration preserves wire rows, cursor, epoch and in-progress sync snapshots. WAL NORMAL, 4 KiB pages, default checkpoint policy and all timing limits remain unchanged. Rebuild staging temporarily uses memory proportional to the projected book. Format parsing and conversion stay in the contacts plugin. Validation: fmt, all 20 CardDAV tests, workspace clippy with warnings denied, and workspace tests. Coverage includes old-cache migration with a paged snapshot and deletion, reads across WAL commits, metadata agreement with GET, and failed or busy rebuild rollback and retry. Five cold and five warm measurements per operation on the assigned perf-test-02 VM, using the preceding job’s perf-run.sh method, Homes on /srv/homes and state on the OS SSD. Maximum across both conditions: | Operation, 1,000 cards | Before max ms | After max ms | Limit | |---|---:|---:|---:| | rebuild | 85.749347 | 19.509446 | <30 ms | | sync | 1.092146 | 1.134989 | <2 ms | | propfind | 35.103631 | 9.052732 | <30 ms | | multiget | 95.753909 | 14.469957 | <30 ms | Every rebuild compared href, wire bytes, strong ETag and serialized property index against an independent incremental baseline, at both 1,000 and 10,000 cards. All comparisons passed. Raw tables, unsuccessful intermediate experiments and I/O profiles are posted on #657 and #734. Cold rebuild writes fell from 833 to 471 fragments; the final profile retains two durability syncs. The workspace run passed 1,175 tests with 20 existing ignored tests. The first run hit an empty Bun bridge response while JavaScript dependencies were absent. After bun install --frozen-lockfile, both bridge tests and the full workspace retry passed without source changes. The first failed output is retained with the evidence. [Complete measurement and gate evidence](https://git.kayg.org/attachments/5b22b7a5-d8cc-456d-9f1d-330798bd0cb4)
Reduce CardDAV cache rebuild and cold read I/O
Some checks failed
CI / server (pull_request) Successful in 57s
Details
CI / rust (pull_request) Successful in 12m53s
Details
CI / web (pull_request) Has been cancelled
Details
a45e7482b5
kayg commented 2026-10-31 05:14:13 +00:00
Author
Owner
Copy link

Replaced by #777: same change rebased onto current dev (branch issue/carddav-read-perf-fix-r2), build and tests green on dev-02. This PR is left open and untouched; please close it when the replacement is accepted.

Replaced by #777: same change rebased onto current dev (branch `issue/carddav-read-perf-fix-r2`), build and tests green on dev-02. This PR is left open and untouched; please close it when the replacement is accepted.
kayg commented 2026-11-04 13:04:32 +00:00
Author
Owner
Copy link

Closing: superseded by #777 (named in the PR body or comments). Branch kept.

Closing: superseded by #777 (named in the PR body or comments). Branch kept.
kayg closed this pull request 2026-11-04 13:04:32 +00:00
Some checks failed
CI / server (pull_request) Successful in 57s
Details
CI / rust (pull_request) Successful in 12m53s
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!739
No description provided.