kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

WIP: Add CardDAV cache and read path (ADR 0064 pieces 3 and 4) #734

Closed
kayg wants to merge 1 commit from issue/carddav-read into dev
pull from: issue/carddav-read
merge into: kayg:dev
Conversation 15 Commits 1 Files changed 21 +2202
kayg commented 2026-10-28 17:26:59 +00:00
Owner
Copy link

Contact notes now have a disposable CardDAV cache and a read-only DAV adapter for ADR 0064 pieces 3 and 4. The cache stores vCard 3.0 bytes, strong ETags and property indexes in users//carddav.db. A journal subscriber projects creates, typing, imports, Trash, restore and undo, with a replay barrier before reads.

The adapter implements well-known discovery, the stable-user principal, one address book, PROPFIND, multiget, paged sync-collection and getctag. Initial sync pages use a fixed snapshot; delta pages include deletions and correct continuation tokens. VCard property selection stays in the existing contacts format crate, and reads use the cached field index.

The default server does not mount these routes. Constructing the router requires an authenticator; app passwords and server wiring remain piece 5. There are no anonymous credentials in product code. Writes, Lists and photos remain their later ADR pieces.

Validation:

  • Format check and workspace Clippy with warnings denied.
  • Workspace tests, plus 16 CardDAV tests and one wire-index test.
  • The journal tests passed 20 repeated runs after fixing a test race with the existing import-publication barrier. No fixed sleep was added.
  • RFC 6352 and RFC 6578 fixtures cover discovery, properties, full and partial multiget, paging, deletions, token isolation, XML escaping and malformed requests, GET, HEAD and conditional reads. Incremental and rebuilt cache rows compare byte for byte, including ETags and property indexes.

Required timing gates are pending. Both the perf VM (10.69.69.73) and Mac Studio (10.69.69.102) time out over SSH from this host. RFC fixtures are the approved fallback for unavailable macOS 27 traffic. No timing numbers are claimed. The perf example measures conversion plus a committed 1,000-card rebuild, a 1,000-card HTTP multiget, and an unchanged HTTP sync check. Run it only on 10by32-01-std under /usr/local/bin/perf-lock.sh, with CALTERNAL_PERF_VM=10by32-01-std, using cargo run --release -j3 -p calternal-carddav --example perf after sourcing /etc/calternal/env.sh.

Full gate output and the observed test-race failure are attached to issue #657. This PR stays a draft while the required VM gates are unavailable. No merge or deployment was performed.

Refs #657. ADR 0064.

Gate totals: 1171 passed, 20 ignored, 0 failed. Head: 82c5e64f51.

Complete gate output

Contact notes now have a disposable CardDAV cache and a read-only DAV adapter for ADR 0064 pieces 3 and 4. The cache stores vCard 3.0 bytes, strong ETags and property indexes in users/<stable-user-id>/carddav.db. A journal subscriber projects creates, typing, imports, Trash, restore and undo, with a replay barrier before reads. The adapter implements well-known discovery, the stable-user principal, one address book, PROPFIND, multiget, paged sync-collection and getctag. Initial sync pages use a fixed snapshot; delta pages include deletions and correct continuation tokens. VCard property selection stays in the existing contacts format crate, and reads use the cached field index. The default server does not mount these routes. Constructing the router requires an authenticator; app passwords and server wiring remain piece 5. There are no anonymous credentials in product code. Writes, Lists and photos remain their later ADR pieces. Validation: - Format check and workspace Clippy with warnings denied. - Workspace tests, plus 16 CardDAV tests and one wire-index test. - The journal tests passed 20 repeated runs after fixing a test race with the existing import-publication barrier. No fixed sleep was added. - RFC 6352 and RFC 6578 fixtures cover discovery, properties, full and partial multiget, paging, deletions, token isolation, XML escaping and malformed requests, GET, HEAD and conditional reads. Incremental and rebuilt cache rows compare byte for byte, including ETags and property indexes. Required timing gates are pending. Both the perf VM (10.69.69.73) and Mac Studio (10.69.69.102) time out over SSH from this host. RFC fixtures are the approved fallback for unavailable macOS 27 traffic. No timing numbers are claimed. The perf example measures conversion plus a committed 1,000-card rebuild, a 1,000-card HTTP multiget, and an unchanged HTTP sync check. Run it only on 10by32-01-std under /usr/local/bin/perf-lock.sh, with CALTERNAL_PERF_VM=10by32-01-std, using cargo run --release -j3 -p calternal-carddav --example perf after sourcing /etc/calternal/env.sh. Full gate output and the observed test-race failure are attached to issue #657. This PR stays a draft while the required VM gates are unavailable. No merge or deployment was performed. Refs #657. ADR 0064. Gate totals: 1171 passed, 20 ignored, 0 failed. Head: 82c5e64f512f026437159403c61cf3a71cb40f6a. [Complete gate output](https://git.kayg.org/attachments/a5612028-bd69-47b8-a431-8d073878ca98)
Add journal-backed CardDAV cache and read bridge
Some checks failed
CI / server (pull_request) Successful in 1m21s
Details
CI / rust (pull_request) Failing after 22m47s
Details
CI / web (pull_request) Failing after 2h6m19s
Details
82c5e64f51
kayg commented 2026-10-28 17:45:20 +00:00
Author
Owner
Copy link

Finished the perf-contacts measurement job for ADR 0064 pieces 3 and 4.

Measured head: 82c5e64f51, branch issue/carddav-read (detached checkout). PR base: dev at 3d0c6e33d5. Product code is unchanged and the worktree is clean. No commits, pushes, merges, deployments or issue closures.

Result: byte identity and the 2 ms sync check pass. The 30 ms rebuild gate fails cold and warm. The ADR 30 ms serving gate passes warm but fails cold for PROPFIND and multiget. These failures remain blockers; diagnostics do not replace or discard failed samples. No limits changed.

Environment and method: job-selected dedicated VM perf-test-02, KVM, 4 CPUs and 8 GiB cap; /srv/homes on rotational sdb/XFS and /home/agent/perf-state on nonrotational sda/ext4. Each operation ran serially through /usr/local/bin/perf-run.sh, which holds perf-lock.sh and the build lock, inside perf.slice (CPU quota 4, memory max 8 GiB); build-slot.sh was nested inside that held lock. Cold uses --cold before launching a fresh process and has no request warmup. Warm uses a fresh process with one identical untimed operation before its timed operation. Cache drop is the guest cache drop provided by perf-run.sh, not a claim that the hypervisor disk cache was cleared. Five independent measured invocations per operation per condition. p50/p95 use nearest rank; with five samples p95 is the maximum. Every sample must be strictly below its gate.

Raw baseline milliseconds, in execution order:

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 26.647671, 23.628417, 41.362540, 54.809073, 79.292679 41.362540 79.292679 FAIL (<30 ms, every sample)
rebuild 1000 warm 12.617011, 36.524844, 11.203179, 13.204586, 35.751018 13.204586 36.524844 FAIL (<30 ms, every sample)
rebuild 10000 cold 374.183759, 936.263512, 1039.411247, 518.447801, 1092.638194 936.263512 1092.638194 report only
rebuild 10000 warm 180.438818, 193.207032, 188.489252, 179.609285, 189.001645 188.489252 193.207032 report only
sync 1000 cold 1.022905, 1.256646, 1.153491, 1.140928, 1.139736 1.140928 1.256646 PASS (<2 ms, every sample)
sync 1000 warm 0.057649, 0.060474, 0.058660, 0.060333, 0.057789 0.058660 0.060474 PASS (<2 ms, every sample)
propfind 1000 cold 32.170438, 30.540329, 30.416135, 29.293872, 30.419771 30.419771 32.170438 FAIL (<30 ms, every sample)
propfind 1000 warm 2.788049, 2.669575, 2.697708, 2.636803, 2.654627 2.669575 2.788049 PASS (<30 ms, every sample)
multiget 1000 cold 38.570745, 37.299913, 39.914245, 38.646578, 41.307527 38.646578 41.307527 FAIL (<30 ms, every sample)
multiget 1000 warm 7.665099, 7.841641, 7.678524, 9.392742, 7.827424 7.827424 9.392742 PASS (<30 ms, every sample)

The 10,000-card rebuild is reported without inventing a new limit. The rebuild timer matches the original PR harness: source notes already in memory; note parsing, vCard 3.0 conversion, property indexing, hashing and the committed SQLite replacement are included. Home discovery/file reads and cache opening are outside that gate. Notes really reside under /srv/homes/perf/contacts-657; original cards and per-user carddav.db files reside under /home/agent/perf-state/perf-contacts-657. Every measured rebuild at both sizes compares href, wire bytes, strong ETag and serialized property index against a separate cache populated incrementally. All 20 comparisons passed. Cache generation and sync tokens rotate intentionally; this is row-byte identity, not physical SQLite file identity.

HTTP timing includes the router, unchanged-source replay barrier, XML parsing, cache reads and complete response body collection, before network time. As in the original gate, it uses the supplied test authenticator rather than the future piece 5 app-password path. PROPFIND uses Depth 1 and getetag/getcontenttype/getcontentlength; multiget requests getetag plus full address-data for all 1,000 contacts. All measured responses returned 207 with the expected counts: zero changed cards, 1,001 PROPFIND resources, or 1,000 multiget cards. PROPFIND bodies are 429,370 bytes; multiget bodies are 498,726 bytes. The source panics if reads attempt a source scan, conversion or change replay.

Where the failed gate spends time: baseline conversion/index creation was 6.024511–7.888589 ms for 1,000 cards. The slowest baseline rebuild spent 71.404090 ms in Cache::rebuild out of 79.292679 ms total. A separate syscall profile reproduced a failing cold rebuild at 69.806454 ms: conversion/indexing 6.676828 ms, SQLite 63.129626 ms, including 32.883864 ms in 414 pread64 calls, 0.977013 ms in 833 pwrite64 calls and 24.808069 ms in two fdatasync calls.

Five further diagnostic repetitions per failing operation/condition follow. These use an external LD_PRELOAD counter reset immediately before the operation and dumped immediately after its timer stops. Timings below are diagnostic and are not substituted into the baseline table.

Profile (1,000 cards) Run Total ms Conversion ms SQLite ms pread64 ms (calls) pwrite64 ms (calls) fdatasync ms (calls)
rebuild cold 1 52.500297 7.989440 44.510857 34.670432 (414) 1.049814 (833) 4.392199 (2)
rebuild cold 2 31.125230 7.504627 23.620603 10.011664 (414) 1.101278 (833) 8.429357 (2)
rebuild cold 3 25.818982 7.357229 18.461753 9.338100 (414) 0.928669 (833) 4.259848 (2)
rebuild cold 4 26.881792 7.889071 18.992721 9.303350 (414) 1.227415 (833) 4.298130 (2)
rebuild cold 5 48.853391 7.462197 41.391194 32.015158 (414) 1.054670 (833) 4.143679 (2)
rebuild warm 1 10.998292 6.357467 4.640825 0.000000 (0) 1.053657 (832) 0.000000 (0)
rebuild warm 2 10.139226 6.012618 4.126608 0.000000 (0) 0.931159 (832) 0.000000 (0)
rebuild warm 3 10.097828 6.046061 4.051767 0.000000 (0) 0.923993 (832) 0.000000 (0)
rebuild warm 4 10.232031 6.042194 4.189837 0.000000 (0) 0.939014 (832) 0.000000 (0)
rebuild warm 5 10.466982 6.017127 4.449855 0.000000 (0) 0.997863 (832) 0.000000 (0)
propfind cold 1 14.722024 — — 10.181432 (206) 0.000000 (0) 0.000000 (0)
propfind cold 2 11.746652 — — 7.085659 (206) 0.000000 (0) 0.000000 (0)
propfind cold 3 12.647718 — — 8.109969 (206) 0.000000 (0) 0.000000 (0)
propfind cold 4 12.977077 — — 8.063100 (206) 0.000000 (0) 0.000000 (0)
propfind cold 5 11.536005 — — 7.130840 (206) 0.000000 (0) 0.000000 (0)
multiget cold 1 22.268768 — — 11.183463 (206) 0.000000 (0) 0.000000 (0)
multiget cold 2 26.448797 — — 12.968421 (206) 0.000000 (0) 0.000000 (0)
multiget cold 3 21.552791 — — 10.770280 (206) 0.000000 (0) 0.000000 (0)
multiget cold 4 21.933296 — — 10.922584 (206) 0.000000 (0) 0.000000 (0)
multiget cold 5 26.928350 — — 14.277116 (206) 0.000000 (0) 0.000000 (0)

The later warm rebuild and cold HTTP diagnostics did not reproduce the initial outliers. Their syscall measurements show the variable read/durability cost, but do not establish the precise cause of every earlier outlier. Warm diagnostic rebuilds made no pread64 or fdatasync calls in their measured transactions; cold requests each made 206 pread64 calls. All other counted pread/pwrite/fsync variants were zero. perf.slice showed zero CPU throttling and zero memory limit/OOM events. No product change was made on this evidence alone.

Verbatim aggregate gate output (exit 1):

rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index)
rebuild_1000: FAIL max=79.292679 ms budget=<30 ms (5 cold + 5 warm)
sync_1000: PASS max=1.256646 ms budget=<2 ms (5 cold + 5 warm)
propfind_1000: FAIL max=32.170438 ms budget=<30 ms (5 cold + 5 warm)
multiget_1000: FAIL max=41.307527 ms budget=<30 ms (5 cold + 5 warm)

The optimized release build completed successfully with --locked -j3 under build-slot.sh. Evidence, temporary harness sources, exact invocation logs, raw JSON, machine details and all diagnostic runs are attached; nothing was committed to the repository.

Complete measurement evidence (ZIP SHA-256: 58fb3a077df8eae05abc5bf922091354dd1f01c231aa12fb070bec73e35ec59b).

Finished the perf-contacts measurement job for ADR 0064 pieces 3 and 4. Measured head: 82c5e64f512f026437159403c61cf3a71cb40f6a, branch issue/carddav-read (detached checkout). PR base: dev at 3d0c6e33d588335c745a660163bf8bf5cdc8412d. Product code is unchanged and the worktree is clean. No commits, pushes, merges, deployments or issue closures. Result: byte identity and the 2 ms sync check pass. The 30 ms rebuild gate fails cold and warm. The ADR 30 ms serving gate passes warm but fails cold for PROPFIND and multiget. These failures remain blockers; diagnostics do not replace or discard failed samples. No limits changed. Environment and method: job-selected dedicated VM perf-test-02, KVM, 4 CPUs and 8 GiB cap; /srv/homes on rotational sdb/XFS and /home/agent/perf-state on nonrotational sda/ext4. Each operation ran serially through /usr/local/bin/perf-run.sh, which holds perf-lock.sh and the build lock, inside perf.slice (CPU quota 4, memory max 8 GiB); build-slot.sh was nested inside that held lock. Cold uses --cold before launching a fresh process and has no request warmup. Warm uses a fresh process with one identical untimed operation before its timed operation. Cache drop is the guest cache drop provided by perf-run.sh, not a claim that the hypervisor disk cache was cleared. Five independent measured invocations per operation per condition. p50/p95 use nearest rank; with five samples p95 is the maximum. Every sample must be strictly below its gate. Raw baseline milliseconds, in execution order: | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 26.647671, 23.628417, 41.362540, 54.809073, 79.292679 | 41.362540 | 79.292679 | FAIL (<30 ms, every sample) | | rebuild | 1000 | warm | 12.617011, 36.524844, 11.203179, 13.204586, 35.751018 | 13.204586 | 36.524844 | FAIL (<30 ms, every sample) | | rebuild | 10000 | cold | 374.183759, 936.263512, 1039.411247, 518.447801, 1092.638194 | 936.263512 | 1092.638194 | report only | | rebuild | 10000 | warm | 180.438818, 193.207032, 188.489252, 179.609285, 189.001645 | 188.489252 | 193.207032 | report only | | sync | 1000 | cold | 1.022905, 1.256646, 1.153491, 1.140928, 1.139736 | 1.140928 | 1.256646 | PASS (<2 ms, every sample) | | sync | 1000 | warm | 0.057649, 0.060474, 0.058660, 0.060333, 0.057789 | 0.058660 | 0.060474 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 32.170438, 30.540329, 30.416135, 29.293872, 30.419771 | 30.419771 | 32.170438 | FAIL (<30 ms, every sample) | | propfind | 1000 | warm | 2.788049, 2.669575, 2.697708, 2.636803, 2.654627 | 2.669575 | 2.788049 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 38.570745, 37.299913, 39.914245, 38.646578, 41.307527 | 38.646578 | 41.307527 | FAIL (<30 ms, every sample) | | multiget | 1000 | warm | 7.665099, 7.841641, 7.678524, 9.392742, 7.827424 | 7.827424 | 9.392742 | PASS (<30 ms, every sample) | The 10,000-card rebuild is reported without inventing a new limit. The rebuild timer matches the original PR harness: source notes already in memory; note parsing, vCard 3.0 conversion, property indexing, hashing and the committed SQLite replacement are included. Home discovery/file reads and cache opening are outside that gate. Notes really reside under /srv/homes/perf/contacts-657; original cards and per-user carddav.db files reside under /home/agent/perf-state/perf-contacts-657. Every measured rebuild at both sizes compares href, wire bytes, strong ETag and serialized property index against a separate cache populated incrementally. All 20 comparisons passed. Cache generation and sync tokens rotate intentionally; this is row-byte identity, not physical SQLite file identity. HTTP timing includes the router, unchanged-source replay barrier, XML parsing, cache reads and complete response body collection, before network time. As in the original gate, it uses the supplied test authenticator rather than the future piece 5 app-password path. PROPFIND uses Depth 1 and getetag/getcontenttype/getcontentlength; multiget requests getetag plus full address-data for all 1,000 contacts. All measured responses returned 207 with the expected counts: zero changed cards, 1,001 PROPFIND resources, or 1,000 multiget cards. PROPFIND bodies are 429,370 bytes; multiget bodies are 498,726 bytes. The source panics if reads attempt a source scan, conversion or change replay. Where the failed gate spends time: baseline conversion/index creation was 6.024511–7.888589 ms for 1,000 cards. The slowest baseline rebuild spent 71.404090 ms in Cache::rebuild out of 79.292679 ms total. A separate syscall profile reproduced a failing cold rebuild at 69.806454 ms: conversion/indexing 6.676828 ms, SQLite 63.129626 ms, including 32.883864 ms in 414 pread64 calls, 0.977013 ms in 833 pwrite64 calls and 24.808069 ms in two fdatasync calls. Five further diagnostic repetitions per failing operation/condition follow. These use an external LD_PRELOAD counter reset immediately before the operation and dumped immediately after its timer stops. Timings below are diagnostic and are not substituted into the baseline table. | Profile (1,000 cards) | Run | Total ms | Conversion ms | SQLite ms | pread64 ms (calls) | pwrite64 ms (calls) | fdatasync ms (calls) | |---|---:|---:|---:|---:|---:|---:|---:| | rebuild cold | 1 | 52.500297 | 7.989440 | 44.510857 | 34.670432 (414) | 1.049814 (833) | 4.392199 (2) | | rebuild cold | 2 | 31.125230 | 7.504627 | 23.620603 | 10.011664 (414) | 1.101278 (833) | 8.429357 (2) | | rebuild cold | 3 | 25.818982 | 7.357229 | 18.461753 | 9.338100 (414) | 0.928669 (833) | 4.259848 (2) | | rebuild cold | 4 | 26.881792 | 7.889071 | 18.992721 | 9.303350 (414) | 1.227415 (833) | 4.298130 (2) | | rebuild cold | 5 | 48.853391 | 7.462197 | 41.391194 | 32.015158 (414) | 1.054670 (833) | 4.143679 (2) | | rebuild warm | 1 | 10.998292 | 6.357467 | 4.640825 | 0.000000 (0) | 1.053657 (832) | 0.000000 (0) | | rebuild warm | 2 | 10.139226 | 6.012618 | 4.126608 | 0.000000 (0) | 0.931159 (832) | 0.000000 (0) | | rebuild warm | 3 | 10.097828 | 6.046061 | 4.051767 | 0.000000 (0) | 0.923993 (832) | 0.000000 (0) | | rebuild warm | 4 | 10.232031 | 6.042194 | 4.189837 | 0.000000 (0) | 0.939014 (832) | 0.000000 (0) | | rebuild warm | 5 | 10.466982 | 6.017127 | 4.449855 | 0.000000 (0) | 0.997863 (832) | 0.000000 (0) | | propfind cold | 1 | 14.722024 | — | — | 10.181432 (206) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 2 | 11.746652 | — | — | 7.085659 (206) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 3 | 12.647718 | — | — | 8.109969 (206) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 4 | 12.977077 | — | — | 8.063100 (206) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 5 | 11.536005 | — | — | 7.130840 (206) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 1 | 22.268768 | — | — | 11.183463 (206) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 2 | 26.448797 | — | — | 12.968421 (206) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 3 | 21.552791 | — | — | 10.770280 (206) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 4 | 21.933296 | — | — | 10.922584 (206) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 5 | 26.928350 | — | — | 14.277116 (206) | 0.000000 (0) | 0.000000 (0) | The later warm rebuild and cold HTTP diagnostics did not reproduce the initial outliers. Their syscall measurements show the variable read/durability cost, but do not establish the precise cause of every earlier outlier. Warm diagnostic rebuilds made no pread64 or fdatasync calls in their measured transactions; cold requests each made 206 pread64 calls. All other counted pread/pwrite/fsync variants were zero. perf.slice showed zero CPU throttling and zero memory limit/OOM events. No product change was made on this evidence alone. Verbatim aggregate gate output (exit 1): ```text rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index) rebuild_1000: FAIL max=79.292679 ms budget=<30 ms (5 cold + 5 warm) sync_1000: PASS max=1.256646 ms budget=<2 ms (5 cold + 5 warm) propfind_1000: FAIL max=32.170438 ms budget=<30 ms (5 cold + 5 warm) multiget_1000: FAIL max=41.307527 ms budget=<30 ms (5 cold + 5 warm) ``` The optimized release build completed successfully with --locked -j3 under build-slot.sh. Evidence, temporary harness sources, exact invocation logs, raw JSON, machine details and all diagnostic runs are attached; nothing was committed to the repository. [Complete measurement evidence](https://git.kayg.org/attachments/e52a2034-f7da-4c84-adcf-a0e27b427933) (ZIP SHA-256: 58fb3a077df8eae05abc5bf922091354dd1f01c231aa12fb070bec73e35ec59b).
kayg commented 2026-10-28 17:46:58 +00:00
Author
Owner
Copy link

Starting perf-contacts-fix on branch issue/carddav-read-perf-fix, fetched base 82c5e64f51. Read the full prior tables and profile in comments 67096 and 67097. Reproducing 5 cold + 5 warm runs before changing code. The prior profile locates SQLite reads and durability waits; rebuild already uses one transaction and prepared inserts. Using assigned perf-test-02 as in the previous job (the task explicitly identifies this replacement VM). Hard rule requires a new branch and PR because updates to open PR branches are ignored; will link the follow-up from #734. No limits will change.

Starting perf-contacts-fix on branch issue/carddav-read-perf-fix, fetched base 82c5e64f512f026437159403c61cf3a71cb40f6a. Read the full prior tables and profile in comments 67096 and 67097. Reproducing 5 cold + 5 warm runs before changing code. The prior profile locates SQLite reads and durability waits; rebuild already uses one transaction and prepared inserts. Using assigned perf-test-02 as in the previous job (the task explicitly identifies this replacement VM). Hard rule requires a new branch and PR because updates to open PR branches are ignored; will link the follow-up from #734. No limits will change.
kayg commented 2026-10-28 17:48:53 +00:00
Author
Owner
Copy link

Profile review before edits: prior unmodified-head syscall profile uses counters scoped to the operation. Cold rebuild: 414 pread64, 833 pwrite64, 2 fdatasync; cold reads: 206 pread64, zero writes or syncs. Source review confirms rebuild has one transaction and two prepared insert statements. cards() selects complete precomputed rows ordered by href (SQLite uses the href index followed by table lookups); multiget prepares a new get statement for every requested href. No Home file reads or format conversions occur on the read path. SQLite bundled source defaults mmap_size to zero, so every cold page is fetched through xRead. Test a bounded 64 MiB mapping with SQLite-managed WAL coherence and cached get statements, retaining all existing durability settings.

Profile review before edits: prior unmodified-head syscall profile uses counters scoped to the operation. Cold rebuild: 414 pread64, 833 pwrite64, 2 fdatasync; cold reads: 206 pread64, zero writes or syncs. Source review confirms rebuild has one transaction and two prepared insert statements. cards() selects complete precomputed rows ordered by href (SQLite uses the href index followed by table lookups); multiget prepares a new get statement for every requested href. No Home file reads or format conversions occur on the read path. SQLite bundled source defaults mmap_size to zero, so every cold page is fetched through xRead. Test a bounded 64 MiB mapping with SQLite-managed WAL coherence and cached get statements, retaining all existing durability settings.
kayg commented 2026-10-28 17:49:21 +00:00
Author
Owner
Copy link

Fresh before measurements at 82c5e64f51 reproduced the failures. Same prior harness, fresh data namespace, Homes under /srv/homes/perf/contacts-fix-657 and state on the OS SSD. Each operation uses perf-run.sh and perf-lock.sh, 5 cold + 5 warm invocations; warm performs one untimed identical operation. Every sample is retained. All 20 rebuild row comparisons passed.

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 85.749347, 20.727567, 32.902707, 50.905726, 36.111467 36.111467 85.749347 FAIL (<30 ms, every sample)
rebuild 1000 warm 13.107354, 32.333165, 9.713585, 12.359236, 31.176197 13.107354 32.333165 FAIL (<30 ms, every sample)
rebuild 10000 cold 346.872530, 589.540054, 936.179287, 1088.261861, 780.247507 780.247507 1088.261861 report only
rebuild 10000 warm 200.452367, 204.157791, 198.634917, 240.282223, 199.049035 200.452367 240.282223 report only
sync 1000 cold 1.092146, 0.955207, 1.030419, 0.980315, 1.036341 1.030419 1.092146 PASS (<2 ms, every sample)
sync 1000 warm 0.057177, 0.056878, 0.058750, 0.057639, 0.056956 0.057177 0.058750 PASS (<2 ms, every sample)
propfind 1000 cold 35.103631, 29.413418, 28.018973, 30.116652, 32.720193 30.116652 35.103631 FAIL (<30 ms, every sample)
propfind 1000 warm 2.474969, 2.449030, 2.628417, 2.450051, 2.465260 2.465260 2.628417 PASS (<30 ms, every sample)
multiget 1000 cold 42.780871, 39.764743, 95.753909, 39.195092, 45.818720 42.780871 95.753909 FAIL (<30 ms, every sample)
multiget 1000 warm 7.560351, 7.578275, 8.878652, 7.581603, 7.533762 7.578275 8.878652 PASS (<30 ms, every sample)

Verbatim before gate output:

rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index)
rebuild_1000: FAIL max=85.749347 ms budget=<30 ms (5 cold + 5 warm)
sync_1000: PASS max=1.092146 ms budget=<2 ms (5 cold + 5 warm)
propfind_1000: FAIL max=35.103631 ms budget=<30 ms (5 cold + 5 warm)
multiget_1000: FAIL max=95.753909 ms budget=<30 ms (5 cold + 5 warm)

Testing bounded SQLite mapping and prepared lookup reuse first; no durability setting changed.

Fresh before measurements at 82c5e64f512f026437159403c61cf3a71cb40f6a reproduced the failures. Same prior harness, fresh data namespace, Homes under /srv/homes/perf/contacts-fix-657 and state on the OS SSD. Each operation uses perf-run.sh and perf-lock.sh, 5 cold + 5 warm invocations; warm performs one untimed identical operation. Every sample is retained. All 20 rebuild row comparisons passed. | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 85.749347, 20.727567, 32.902707, 50.905726, 36.111467 | 36.111467 | 85.749347 | FAIL (<30 ms, every sample) | | rebuild | 1000 | warm | 13.107354, 32.333165, 9.713585, 12.359236, 31.176197 | 13.107354 | 32.333165 | FAIL (<30 ms, every sample) | | rebuild | 10000 | cold | 346.872530, 589.540054, 936.179287, 1088.261861, 780.247507 | 780.247507 | 1088.261861 | report only | | rebuild | 10000 | warm | 200.452367, 204.157791, 198.634917, 240.282223, 199.049035 | 200.452367 | 240.282223 | report only | | sync | 1000 | cold | 1.092146, 0.955207, 1.030419, 0.980315, 1.036341 | 1.030419 | 1.092146 | PASS (<2 ms, every sample) | | sync | 1000 | warm | 0.057177, 0.056878, 0.058750, 0.057639, 0.056956 | 0.057177 | 0.058750 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 35.103631, 29.413418, 28.018973, 30.116652, 32.720193 | 30.116652 | 35.103631 | FAIL (<30 ms, every sample) | | propfind | 1000 | warm | 2.474969, 2.449030, 2.628417, 2.450051, 2.465260 | 2.465260 | 2.628417 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 42.780871, 39.764743, 95.753909, 39.195092, 45.818720 | 42.780871 | 95.753909 | FAIL (<30 ms, every sample) | | multiget | 1000 | warm | 7.560351, 7.578275, 8.878652, 7.581603, 7.533762 | 7.578275 | 8.878652 | PASS (<30 ms, every sample) | Verbatim before gate output: ```text rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index) rebuild_1000: FAIL max=85.749347 ms budget=<30 ms (5 cold + 5 warm) sync_1000: PASS max=1.092146 ms budget=<2 ms (5 cold + 5 warm) propfind_1000: FAIL max=35.103631 ms budget=<30 ms (5 cold + 5 warm) multiget_1000: FAIL max=95.753909 ms budget=<30 ms (5 cold + 5 warm) ``` Testing bounded SQLite mapping and prepared lookup reuse first; no durability setting changed.
kayg commented 2026-10-28 17:50:45 +00:00
Author
Owner
Copy link

First measured change: bounded 64 MiB SQLite mapping and cached get statement reuse. WAL NORMAL unchanged. Cold requests now pass; rebuild still fails. No samples discarded.

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 28.046666, 20.132738, 45.278755, 37.350990, 33.693276 33.693276 45.278755 FAIL (<30 ms, every sample)
rebuild 1000 warm 12.781129, 38.839342, 13.288083, 12.384884, 60.181647 13.288083 60.181647 FAIL (<30 ms, every sample)
rebuild 10000 cold 303.082769, 399.192309, 1102.662709, 854.578320, 774.898448 774.898448 1102.662709 report only
rebuild 10000 warm 192.999086, 182.014852, 185.003726, 186.205097, 181.595610 185.003726 192.999086 report only
sync 1000 cold 1.201602, 0.979523, 0.941442, 1.003269, 1.033235 1.003269 1.201602 PASS (<2 ms, every sample)
sync 1000 warm 0.059052, 0.058461, 0.058610, 0.068359, 0.058139 0.058610 0.068359 PASS (<2 ms, every sample)
propfind 1000 cold 12.749800, 12.670190, 12.652657, 12.543222, 13.332127 12.670190 13.332127 PASS (<30 ms, every sample)
propfind 1000 warm 2.465061, 2.441416, 2.449833, 2.407973, 2.542485 2.449833 2.542485 PASS (<30 ms, every sample)
multiget 1000 cold 14.025100, 14.916198, 14.304767, 16.050824, 16.159179 14.916198 16.159179 PASS (<30 ms, every sample)
multiget 1000 warm 4.752867, 4.767264, 5.028586, 4.812940, 4.757447 4.767264 5.028586 PASS (<30 ms, every sample)

Rebuild remains blocked by database replacement I/O; profiling that operation before the next change.

First measured change: bounded 64 MiB SQLite mapping and cached get statement reuse. WAL NORMAL unchanged. Cold requests now pass; rebuild still fails. No samples discarded. | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 28.046666, 20.132738, 45.278755, 37.350990, 33.693276 | 33.693276 | 45.278755 | FAIL (<30 ms, every sample) | | rebuild | 1000 | warm | 12.781129, 38.839342, 13.288083, 12.384884, 60.181647 | 13.288083 | 60.181647 | FAIL (<30 ms, every sample) | | rebuild | 10000 | cold | 303.082769, 399.192309, 1102.662709, 854.578320, 774.898448 | 774.898448 | 1102.662709 | report only | | rebuild | 10000 | warm | 192.999086, 182.014852, 185.003726, 186.205097, 181.595610 | 185.003726 | 192.999086 | report only | | sync | 1000 | cold | 1.201602, 0.979523, 0.941442, 1.003269, 1.033235 | 1.003269 | 1.201602 | PASS (<2 ms, every sample) | | sync | 1000 | warm | 0.059052, 0.058461, 0.058610, 0.068359, 0.058139 | 0.058610 | 0.068359 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 12.749800, 12.670190, 12.652657, 12.543222, 13.332127 | 12.670190 | 13.332127 | PASS (<30 ms, every sample) | | propfind | 1000 | warm | 2.465061, 2.441416, 2.449833, 2.407973, 2.542485 | 2.449833 | 2.542485 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 14.025100, 14.916198, 14.304767, 16.050824, 16.159179 | 14.916198 | 16.159179 | PASS (<30 ms, every sample) | | multiget | 1000 | warm | 4.752867, 4.767264, 5.028586, 4.812940, 4.757447 | 4.767264 | 5.028586 | PASS (<30 ms, every sample) | Rebuild remains blocked by database replacement I/O; profiling that operation before the next change.
kayg commented 2026-10-28 17:52:03 +00:00
Author
Owner
Copy link

Further diagnostic profile after mapping: rebuild uses zero explicit pread64 calls, about 834 pwrite64 calls, and cold commits make two fdatasync calls. This reproduces the I/O volume, although the failing warm outliers again did not reproduce during the separate profiles. Cold request profiles now make zero explicit page reads (mapped page faults are not counted as pread64). Testing 32 KiB SQLite pages to amortize replacement writes, with automatic checkpoints kept near the original 4 MiB (128 pages), WAL NORMAL unchanged. A one-time transactional VACUUM at open migrates old caches; migration tests cover rows, cursor and sync tokens.

Profile (1,000 cards) Run Total ms Conversion ms SQLite ms pread64 ms (calls) pwrite64 ms (calls) fdatasync ms (calls)
rebuild cold 1 26.524460 6.453659 20.070801 0.000000 (0) 0.968985 (833) 6.569256 (2)
rebuild cold 2 20.465285 6.415607 14.049678 0.000000 (0) 0.947598 (833) 7.151392 (2)
rebuild cold 3 19.683822 6.399205 13.284617 0.000000 (0) 1.266738 (835) 6.207435 (2)
rebuild cold 4 19.604253 6.363960 13.240293 0.000000 (0) 1.053995 (835) 6.469969 (2)
rebuild cold 5 19.465713 6.326469 13.139244 0.000000 (0) 0.958215 (835) 6.695943 (2)
rebuild warm 1 10.066729 5.966722 4.100007 0.000000 (0) 0.968197 (834) 0.000000 (0)
rebuild warm 2 10.647023 6.155437 4.491586 0.000000 (0) 0.970652 (834) 0.000000 (0)
rebuild warm 3 10.431416 6.041302 4.390114 0.000000 (0) 0.994552 (834) 0.000000 (0)
rebuild warm 4 10.448669 5.989064 4.459605 0.000000 (0) 1.009635 (834) 0.000000 (0)
rebuild warm 5 10.282285 6.042244 4.240041 0.000000 (0) 0.949630 (834) 0.000000 (0)
propfind cold 1 5.537905 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 2 5.784008 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 3 5.973074 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 4 5.287413 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 5 5.794347 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 1 8.222408 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 2 8.551016 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 3 26.958427 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 4 15.139249 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 5 23.869272 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
Further diagnostic profile after mapping: rebuild uses zero explicit pread64 calls, about 834 pwrite64 calls, and cold commits make two fdatasync calls. This reproduces the I/O volume, although the failing warm outliers again did not reproduce during the separate profiles. Cold request profiles now make zero explicit page reads (mapped page faults are not counted as pread64). Testing 32 KiB SQLite pages to amortize replacement writes, with automatic checkpoints kept near the original 4 MiB (128 pages), WAL NORMAL unchanged. A one-time transactional VACUUM at open migrates old caches; migration tests cover rows, cursor and sync tokens. | Profile (1,000 cards) | Run | Total ms | Conversion ms | SQLite ms | pread64 ms (calls) | pwrite64 ms (calls) | fdatasync ms (calls) | |---|---:|---:|---:|---:|---:|---:|---:| | rebuild cold | 1 | 26.524460 | 6.453659 | 20.070801 | 0.000000 (0) | 0.968985 (833) | 6.569256 (2) | | rebuild cold | 2 | 20.465285 | 6.415607 | 14.049678 | 0.000000 (0) | 0.947598 (833) | 7.151392 (2) | | rebuild cold | 3 | 19.683822 | 6.399205 | 13.284617 | 0.000000 (0) | 1.266738 (835) | 6.207435 (2) | | rebuild cold | 4 | 19.604253 | 6.363960 | 13.240293 | 0.000000 (0) | 1.053995 (835) | 6.469969 (2) | | rebuild cold | 5 | 19.465713 | 6.326469 | 13.139244 | 0.000000 (0) | 0.958215 (835) | 6.695943 (2) | | rebuild warm | 1 | 10.066729 | 5.966722 | 4.100007 | 0.000000 (0) | 0.968197 (834) | 0.000000 (0) | | rebuild warm | 2 | 10.647023 | 6.155437 | 4.491586 | 0.000000 (0) | 0.970652 (834) | 0.000000 (0) | | rebuild warm | 3 | 10.431416 | 6.041302 | 4.390114 | 0.000000 (0) | 0.994552 (834) | 0.000000 (0) | | rebuild warm | 4 | 10.448669 | 5.989064 | 4.459605 | 0.000000 (0) | 1.009635 (834) | 0.000000 (0) | | rebuild warm | 5 | 10.282285 | 6.042244 | 4.240041 | 0.000000 (0) | 0.949630 (834) | 0.000000 (0) | | propfind cold | 1 | 5.537905 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 2 | 5.784008 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 3 | 5.973074 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 4 | 5.287413 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 5 | 5.794347 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 1 | 8.222408 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 2 | 8.551016 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 3 | 26.958427 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 4 | 15.139249 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 5 | 23.869272 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) |
kayg commented 2026-10-28 17:54:05 +00:00
Author
Owner
Copy link

The 32 KiB page experiment still fails, so it will not be kept. Rebuild reaches 40.727440 ms and sync reaches 3.170842 ms. Byte comparisons pass.

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 22.296655, 15.383914, 29.750633, 21.350464, 15.309185 21.350464 29.750633 PASS (<30 ms, every sample)
rebuild 1000 warm 11.351230, 29.067068, 8.880170, 11.182126, 40.727440 11.351230 40.727440 FAIL (<30 ms, every sample)
rebuild 10000 cold 229.169765, 237.068818, 353.239104, 417.024882, 386.388047 353.239104 417.024882 report only
rebuild 10000 warm 171.982789, 166.019541, 166.490520, 164.890913, 159.917023 166.019541 171.982789 report only
sync 1000 cold 2.421651, 2.289686, 3.170842, 2.147163, 2.032149 2.289686 3.170842 FAIL (<2 ms, every sample)
sync 1000 warm 0.062927, 0.063467, 0.061104, 0.062777, 0.063167 0.062927 0.063467 PASS (<2 ms, every sample)
propfind 1000 cold 13.855749, 13.379216, 11.043623, 11.835341, 11.505270 11.835341 13.855749 PASS (<30 ms, every sample)
propfind 1000 warm 2.509464, 2.968305, 2.688327, 2.432282, 2.440557 2.509464 2.968305 PASS (<30 ms, every sample)
multiget 1000 cold 15.006425, 15.477167, 19.012795, 17.805845, 14.459380 15.477167 19.012795 PASS (<30 ms, every sample)
multiget 1000 warm 5.118765, 5.095991, 6.950681, 5.094078, 5.150384 5.118765 6.950681 PASS (<30 ms, every sample)

Returning to 4 KiB pages. The next change removes duplicate payload storage: the live-card table will point to its immutable revision instead of storing another vCard and property index. Both revision and live pointer still commit atomically. A validated unchanged sync token can return directly instead of issuing a revision query.

The 32 KiB page experiment still fails, so it will not be kept. Rebuild reaches 40.727440 ms and sync reaches 3.170842 ms. Byte comparisons pass. | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 22.296655, 15.383914, 29.750633, 21.350464, 15.309185 | 21.350464 | 29.750633 | PASS (<30 ms, every sample) | | rebuild | 1000 | warm | 11.351230, 29.067068, 8.880170, 11.182126, 40.727440 | 11.351230 | 40.727440 | FAIL (<30 ms, every sample) | | rebuild | 10000 | cold | 229.169765, 237.068818, 353.239104, 417.024882, 386.388047 | 353.239104 | 417.024882 | report only | | rebuild | 10000 | warm | 171.982789, 166.019541, 166.490520, 164.890913, 159.917023 | 166.019541 | 171.982789 | report only | | sync | 1000 | cold | 2.421651, 2.289686, 3.170842, 2.147163, 2.032149 | 2.289686 | 3.170842 | FAIL (<2 ms, every sample) | | sync | 1000 | warm | 0.062927, 0.063467, 0.061104, 0.062777, 0.063167 | 0.062927 | 0.063467 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 13.855749, 13.379216, 11.043623, 11.835341, 11.505270 | 11.835341 | 13.855749 | PASS (<30 ms, every sample) | | propfind | 1000 | warm | 2.509464, 2.968305, 2.688327, 2.432282, 2.440557 | 2.509464 | 2.968305 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 15.006425, 15.477167, 19.012795, 17.805845, 14.459380 | 15.477167 | 19.012795 | PASS (<30 ms, every sample) | | multiget | 1000 | warm | 5.118765, 5.095991, 6.950681, 5.094078, 5.150384 | 5.118765 | 6.950681 | PASS (<30 ms, every sample) | Returning to 4 KiB pages. The next change removes duplicate payload storage: the live-card table will point to its immutable revision instead of storing another vCard and property index. Both revision and live pointer still commit atomically. A validated unchanged sync token can return directly instead of issuing a revision query.
kayg commented 2026-10-28 17:56:48 +00:00
Author
Owner
Copy link

Removing duplicate payloads helps but still misses the gate: rebuild 41.987866 ms, sync 2.025800 ms, PROPFIND 46.479288 ms. Multiget passes. Retaining every sample below.

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 23.276558, 41.987866, 14.351330, 18.887857, 27.357500 23.276558 41.987866 FAIL (<30 ms, every sample)
rebuild 1000 warm 20.290907, 9.822008, 15.182347, 9.566050, 30.283031 15.182347 30.283031 FAIL (<30 ms, every sample)
rebuild 10000 cold 237.425242, 553.456780, 837.123497, 531.801417, 630.679669 553.456780 837.123497 report only
rebuild 10000 warm 156.663358, 155.379614, 156.409580, 194.527211, 159.380886 156.663358 194.527211 report only
sync 1000 cold 1.693621, 1.925703, 2.025800, 1.427647, 1.921756 1.921756 2.025800 FAIL (<2 ms, every sample)
sync 1000 warm 0.043971, 0.025297, 0.023394, 0.022791, 0.034704 0.025297 0.043971 PASS (<2 ms, every sample)
propfind 1000 cold 10.407578, 9.139238, 46.479288, 11.581724, 11.187159 11.187159 46.479288 FAIL (<30 ms, every sample)
propfind 1000 warm 2.742854, 3.916729, 2.787868, 2.634483, 2.720513 2.742854 3.916729 PASS (<30 ms, every sample)
multiget 1000 cold 15.085344, 12.948878, 13.390060, 17.789937, 14.384540 14.384540 17.789937 PASS (<30 ms, every sample)
multiget 1000 warm 5.954702, 5.479107, 7.119921, 5.237496, 5.995990 5.954702 7.119921 PASS (<30 ms, every sample)

Next: stage rebuild B-trees in memory and atomically install them through SQLite backup using the destination WAL NORMAL transaction, avoiding reads of old pages; keep a compact live metadata table for PROPFIND so it never reads bodies/indexes; store the head alongside the cursor in the same transaction so an unchanged sync reads only metadata. No checkpoint or durability setting is relaxed.

Removing duplicate payloads helps but still misses the gate: rebuild 41.987866 ms, sync 2.025800 ms, PROPFIND 46.479288 ms. Multiget passes. Retaining every sample below. | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 23.276558, 41.987866, 14.351330, 18.887857, 27.357500 | 23.276558 | 41.987866 | FAIL (<30 ms, every sample) | | rebuild | 1000 | warm | 20.290907, 9.822008, 15.182347, 9.566050, 30.283031 | 15.182347 | 30.283031 | FAIL (<30 ms, every sample) | | rebuild | 10000 | cold | 237.425242, 553.456780, 837.123497, 531.801417, 630.679669 | 553.456780 | 837.123497 | report only | | rebuild | 10000 | warm | 156.663358, 155.379614, 156.409580, 194.527211, 159.380886 | 156.663358 | 194.527211 | report only | | sync | 1000 | cold | 1.693621, 1.925703, 2.025800, 1.427647, 1.921756 | 1.921756 | 2.025800 | FAIL (<2 ms, every sample) | | sync | 1000 | warm | 0.043971, 0.025297, 0.023394, 0.022791, 0.034704 | 0.025297 | 0.043971 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 10.407578, 9.139238, 46.479288, 11.581724, 11.187159 | 11.187159 | 46.479288 | FAIL (<30 ms, every sample) | | propfind | 1000 | warm | 2.742854, 3.916729, 2.787868, 2.634483, 2.720513 | 2.742854 | 3.916729 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 15.085344, 12.948878, 13.390060, 17.789937, 14.384540 | 14.384540 | 17.789937 | PASS (<30 ms, every sample) | | multiget | 1000 | warm | 5.954702, 5.479107, 7.119921, 5.237496, 5.995990 | 5.954702 | 7.119921 | PASS (<30 ms, every sample) | Next: stage rebuild B-trees in memory and atomically install them through SQLite backup using the destination WAL NORMAL transaction, avoiding reads of old pages; keep a compact live metadata table for PROPFIND so it never reads bodies/indexes; store the head alongside the cursor in the same transaction so an unchanged sync reads only metadata. No checkpoint or durability setting is relaxed.
kayg commented 2026-10-28 17:59:23 +00:00
Author
Owner
Copy link

The final product change passes every measured gate. Fresh incremental baselines and all 20 measured rebuild comparisons pass. Also compared final rows at both sizes with the prior job’s independent incremental databases: identical href, vCard bytes, strong ETag and serialized property index. Homes remain on /srv/homes; state remains on the OS SSD. Same 5 cold + 5 warm method and strict limits. No durability setting or checkpoint threshold changed.

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 19.509446, 11.443830, 11.520274, 11.173687, 11.549077 11.520274 19.509446 PASS (<30 ms, every sample)
rebuild 1000 warm 17.104870, 9.957718, 10.268257, 9.354671, 9.214911 9.957718 17.104870 PASS (<30 ms, every sample)
rebuild 10000 cold 128.336459, 383.330295, 190.365571, 421.488210, 384.022982 383.330295 421.488210 report only
rebuild 10000 warm 101.281288, 98.627526, 98.511329, 106.660354, 98.094061 98.627526 106.660354 report only
sync 1000 cold 1.134989, 0.943442, 0.918003, 0.983787, 0.933343 0.943442 1.134989 PASS (<2 ms, every sample)
sync 1000 warm 0.028824, 0.020117, 0.020228, 0.020017, 0.020058 0.020117 0.028824 PASS (<2 ms, every sample)
propfind 1000 cold 6.448911, 7.153356, 6.651169, 9.052732, 7.916832 7.153356 9.052732 PASS (<30 ms, every sample)
propfind 1000 warm 2.290456, 3.135935, 2.419226, 2.240003, 3.127670 2.419226 3.135935 PASS (<30 ms, every sample)
multiget 1000 cold 14.420356, 14.469957, 11.998594, 13.532337, 12.303051 13.532337 14.469957 PASS (<30 ms, every sample)
multiget 1000 warm 5.151638, 5.157289, 5.135829, 7.577288, 5.058765 5.151638 7.577288 PASS (<30 ms, every sample)

Verbatim performance gate output:

rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index)
rebuild_1000: PASS max=19.509446 ms budget=<30 ms (5 cold + 5 warm)
sync_1000: PASS max=1.134989 ms budget=<2 ms (5 cold + 5 warm)
propfind_1000: PASS max=9.052732 ms budget=<30 ms (5 cold + 5 warm)
multiget_1000: PASS max=14.469957 ms budget=<30 ms (5 cold + 5 warm)

Running final I/O diagnostics, fmt, CardDAV tests, clippy and workspace tests before publishing the follow-up PR.

The final product change passes every measured gate. Fresh incremental baselines and all 20 measured rebuild comparisons pass. Also compared final rows at both sizes with the prior job’s independent incremental databases: identical href, vCard bytes, strong ETag and serialized property index. Homes remain on /srv/homes; state remains on the OS SSD. Same 5 cold + 5 warm method and strict limits. No durability setting or checkpoint threshold changed. | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 19.509446, 11.443830, 11.520274, 11.173687, 11.549077 | 11.520274 | 19.509446 | PASS (<30 ms, every sample) | | rebuild | 1000 | warm | 17.104870, 9.957718, 10.268257, 9.354671, 9.214911 | 9.957718 | 17.104870 | PASS (<30 ms, every sample) | | rebuild | 10000 | cold | 128.336459, 383.330295, 190.365571, 421.488210, 384.022982 | 383.330295 | 421.488210 | report only | | rebuild | 10000 | warm | 101.281288, 98.627526, 98.511329, 106.660354, 98.094061 | 98.627526 | 106.660354 | report only | | sync | 1000 | cold | 1.134989, 0.943442, 0.918003, 0.983787, 0.933343 | 0.943442 | 1.134989 | PASS (<2 ms, every sample) | | sync | 1000 | warm | 0.028824, 0.020117, 0.020228, 0.020017, 0.020058 | 0.020117 | 0.028824 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 6.448911, 7.153356, 6.651169, 9.052732, 7.916832 | 7.153356 | 9.052732 | PASS (<30 ms, every sample) | | propfind | 1000 | warm | 2.290456, 3.135935, 2.419226, 2.240003, 3.127670 | 2.419226 | 3.135935 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 14.420356, 14.469957, 11.998594, 13.532337, 12.303051 | 13.532337 | 14.469957 | PASS (<30 ms, every sample) | | multiget | 1000 | warm | 5.151638, 5.157289, 5.135829, 7.577288, 5.058765 | 5.151638 | 7.577288 | PASS (<30 ms, every sample) | Verbatim performance gate output: ```text rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index) rebuild_1000: PASS max=19.509446 ms budget=<30 ms (5 cold + 5 warm) sync_1000: PASS max=1.134989 ms budget=<2 ms (5 cold + 5 warm) propfind_1000: PASS max=9.052732 ms budget=<30 ms (5 cold + 5 warm) multiget_1000: PASS max=14.469957 ms budget=<30 ms (5 cold + 5 warm) ``` Running final I/O diagnostics, fmt, CardDAV tests, clippy and workspace tests before publishing the follow-up PR.
kayg commented 2026-10-28 18:00:49 +00:00
Author
Owner
Copy link

Final I/O evidence: cold rebuild explicit page reads fell from 414 to zero and write fragments from 833 to 471; warm writes fell from 832 to 470. Cold commits still make two fdatasync calls: durability was retained. Mapping uses page faults, so zero pread64 calls does not mean zero physical I/O. The compact PROPFIND table occupies 24 SQLite pages (98,304 bytes) for 1,000 cards; the former duplicated payload tables occupied about 400 pages together. Multiget reads precomputed wire bytes through the live revision pointer, with prepared statement reuse. No Home reads or card conversion occur in requests. Separate diagnostic profile follows; the successful uninstrumented gate samples remain the results.

Profile (1,000 cards) Run Total ms Conversion ms SQLite ms pread64 ms (calls) pwrite64 ms (calls) fdatasync ms (calls)
rebuild cold 1 17.971267 6.250318 11.720949 0.000000 (0) 0.703928 (471) 4.987497 (2)
rebuild cold 2 16.591900 6.601053 9.990847 0.000000 (0) 0.576148 (471) 4.695162 (2)
rebuild cold 3 16.532910 6.494754 10.038156 0.000000 (0) 0.572193 (471) 4.761937 (2)
rebuild cold 4 15.959139 6.424784 9.534355 0.000000 (0) 0.597624 (471) 4.374904 (2)
rebuild cold 5 16.351552 6.937562 9.413990 0.000000 (0) 0.563431 (471) 4.273384 (2)
rebuild warm 1 13.811375 8.562658 5.248717 0.000000 (0) 0.777561 (470) 0.000000 (0)
rebuild warm 2 10.050558 5.836575 4.213983 0.000000 (0) 0.562746 (470) 0.000000 (0)
rebuild warm 3 11.236986 6.517398 4.719588 0.000000 (0) 0.590762 (470) 0.000000 (0)
rebuild warm 4 9.811312 5.832508 3.978804 0.000000 (0) 0.539104 (470) 0.000000 (0)
rebuild warm 5 9.712899 5.838339 3.874560 0.000000 (0) 0.538233 (470) 0.000000 (0)
propfind cold 1 3.651995 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 2 4.284696 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 3 3.740500 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 4 3.596080 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
propfind cold 5 4.042342 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 1 6.965965 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 2 7.992695 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 3 7.023894 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 4 7.535149 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)
multiget cold 5 7.598759 — — 0.000000 (0) 0.000000 (0) 0.000000 (0)

Exact comparison and layout evidence:

1000 cards: identical to prior independent incremental baseline (href, wire bytes, ETag, serialized property index)
1000 cards: [('current_cards', 24, 98304), ('meta', 1, 4096), ('revisions', 201, 823296), ('revisions_href', 6, 24576), ('sqlite_autoindex_meta_1', 1, 4096), ('sqlite_schema', 1, 4096), ('sqlite_sequence', 1, 4096)]
10000 cards: identical to prior independent incremental baseline (href, wire bytes, ETag, serialized property index)
10000 cards: [('current_cards', 232, 950272), ('meta', 1, 4096), ('revisions', 2006, 8216576), ('revisions_href', 55, 225280), ('sqlite_autoindex_meta_1', 1, 4096), ('sqlite_schema', 1, 4096), ('sqlite_sequence', 1, 4096)]
Final I/O evidence: cold rebuild explicit page reads fell from 414 to zero and write fragments from 833 to 471; warm writes fell from 832 to 470. Cold commits still make two fdatasync calls: durability was retained. Mapping uses page faults, so zero pread64 calls does not mean zero physical I/O. The compact PROPFIND table occupies 24 SQLite pages (98,304 bytes) for 1,000 cards; the former duplicated payload tables occupied about 400 pages together. Multiget reads precomputed wire bytes through the live revision pointer, with prepared statement reuse. No Home reads or card conversion occur in requests. Separate diagnostic profile follows; the successful uninstrumented gate samples remain the results. | Profile (1,000 cards) | Run | Total ms | Conversion ms | SQLite ms | pread64 ms (calls) | pwrite64 ms (calls) | fdatasync ms (calls) | |---|---:|---:|---:|---:|---:|---:|---:| | rebuild cold | 1 | 17.971267 | 6.250318 | 11.720949 | 0.000000 (0) | 0.703928 (471) | 4.987497 (2) | | rebuild cold | 2 | 16.591900 | 6.601053 | 9.990847 | 0.000000 (0) | 0.576148 (471) | 4.695162 (2) | | rebuild cold | 3 | 16.532910 | 6.494754 | 10.038156 | 0.000000 (0) | 0.572193 (471) | 4.761937 (2) | | rebuild cold | 4 | 15.959139 | 6.424784 | 9.534355 | 0.000000 (0) | 0.597624 (471) | 4.374904 (2) | | rebuild cold | 5 | 16.351552 | 6.937562 | 9.413990 | 0.000000 (0) | 0.563431 (471) | 4.273384 (2) | | rebuild warm | 1 | 13.811375 | 8.562658 | 5.248717 | 0.000000 (0) | 0.777561 (470) | 0.000000 (0) | | rebuild warm | 2 | 10.050558 | 5.836575 | 4.213983 | 0.000000 (0) | 0.562746 (470) | 0.000000 (0) | | rebuild warm | 3 | 11.236986 | 6.517398 | 4.719588 | 0.000000 (0) | 0.590762 (470) | 0.000000 (0) | | rebuild warm | 4 | 9.811312 | 5.832508 | 3.978804 | 0.000000 (0) | 0.539104 (470) | 0.000000 (0) | | rebuild warm | 5 | 9.712899 | 5.838339 | 3.874560 | 0.000000 (0) | 0.538233 (470) | 0.000000 (0) | | propfind cold | 1 | 3.651995 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 2 | 4.284696 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 3 | 3.740500 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 4 | 3.596080 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | propfind cold | 5 | 4.042342 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 1 | 6.965965 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 2 | 7.992695 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 3 | 7.023894 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 4 | 7.535149 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | | multiget cold | 5 | 7.598759 | — | — | 0.000000 (0) | 0.000000 (0) | 0.000000 (0) | Exact comparison and layout evidence: ```text 1000 cards: identical to prior independent incremental baseline (href, wire bytes, ETag, serialized property index) 1000 cards: [('current_cards', 24, 98304), ('meta', 1, 4096), ('revisions', 201, 823296), ('revisions_href', 6, 24576), ('sqlite_autoindex_meta_1', 1, 4096), ('sqlite_schema', 1, 4096), ('sqlite_sequence', 1, 4096)] 10000 cards: identical to prior independent incremental baseline (href, wire bytes, ETag, serialized property index) 10000 cards: [('current_cards', 232, 950272), ('meta', 1, 4096), ('revisions', 2006, 8216576), ('revisions_href', 55, 225280), ('sqlite_autoindex_meta_1', 1, 4096), ('sqlite_schema', 1, 4096), ('sqlite_sequence', 1, 4096)] ```
kayg commented 2026-10-28 18:09:26 +00:00
Author
Owner
Copy link

The first workspace gate exited 101 in the unrelated notes test room::save::tests::review::history_durability::independent_652_real_syncclient_joining_snapshot_is_not_saved. Evidence: history_durability.rs:464:37, JSON parse error "EOF while parsing a value", after the Bun child returned no output. This worktree has no node_modules. The previous job log records the same SyncClient test requiring JavaScript dependencies. Fmt, all 20 CardDAV tests and workspace clippy with warnings denied passed. Installing dependencies from the frozen Bun lockfile, then rerunning the real SyncClient tests and the full workspace gate. The first failed log is retained; no test or limit is being weakened.

The first workspace gate exited 101 in the unrelated notes test room::save::tests::review::history_durability::independent_652_real_syncclient_joining_snapshot_is_not_saved. Evidence: history_durability.rs:464:37, JSON parse error "EOF while parsing a value", after the Bun child returned no output. This worktree has no node_modules. The previous job log records the same SyncClient test requiring JavaScript dependencies. Fmt, all 20 CardDAV tests and workspace clippy with warnings denied passed. Installing dependencies from the frozen Bun lockfile, then rerunning the real SyncClient tests and the full workspace gate. The first failed log is retained; no test or limit is being weakened.
kayg commented 2026-10-28 18:10:27 +00:00
Author
Owner
Copy link

Installed the frozen Bun dependencies (118 packages). The two real SyncClient tests now pass without changing product code or tests:

running 2 tests
test room::save::tests::review::history_durability::independent_652_real_syncclient_joining_snapshot_is_not_saved ... ok
test room::save::tests::review::history_durability::independent_652_real_syncclient_peers_and_restart_replay ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 429 filtered out; finished in 0.40s

The full workspace gate is running again. The first workspace failure is preserved in workspace-first.log and the gate history. Fmt, 20 CardDAV tests and workspace clippy still pass; source code is unchanged.

Installed the frozen Bun dependencies (118 packages). The two real SyncClient tests now pass without changing product code or tests: ```text running 2 tests test room::save::tests::review::history_durability::independent_652_real_syncclient_joining_snapshot_is_not_saved ... ok test room::save::tests::review::history_durability::independent_652_real_syncclient_peers_and_restart_replay ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 429 filtered out; finished in 0.40s ``` The full workspace gate is running again. The first workspace failure is preserved in workspace-first.log and the gate history. Fmt, 20 CardDAV tests and workspace clippy still pass; source code is unchanged.
kayg commented 2026-10-28 18:15:49 +00:00
Author
Owner
Copy link

Finished perf-contacts-fix. Head: a45e7482b5. Branch: issue/carddav-read-perf-fix.

Follow-up PR #739 targets issue/carddav-read, keeping its diff focused on the performance fix. The hard rule requires a new branch and PR because updates to an open PR branch are ignored. PR #734 and issue #657 remain open. No merge or deployment was performed.

All final gates pass. Payloads are stored once in revision history; live metadata holds the revision pointer, ETag and byte length. Rebuild constructs fresh tables in memory and installs them atomically through SQLite backup. PROPFIND reads compact metadata, multiget reuses prepared lookups through a bounded 64 MiB mapping, and unchanged sync checks read the committed head metadata. Legacy migration preserves rows, cursor, epoch and paged sync snapshots. WAL NORMAL, default 4 KiB pages, default automatic checkpoints and all limits remain unchanged. Staging temporarily uses memory proportional to the projected book.

Method matches the preceding job: assigned replacement VM perf-test-02, 4 CPU / 8 GiB parent perf.slice; Homes on rotational /srv/homes (/dev/sdb1, XFS), state on the OS SSD (/dev/sda1, ext4). Every operation is serial under perf-run.sh and perf-lock.sh. Five independent cold and five warm invocations per operation; cold drops guest caches, warm performs one identical untimed operation first. All samples are retained. Nearest-rank p50/p95 means p95 is the maximum with five samples; every measured sample must be strictly below its gate. The 10,000-card rebuild remains report only.

The rebuild timer includes in-memory note parsing, contact conversion, indexing, hashing, staging, SQLite replacement and commit. HTTP timing includes the router, unchanged-source barrier, XML parsing, cached reads and complete body collection with the original test authenticator. Source callbacks panic on an attempted Home scan, conversion or change replay during reads. Responses remain 207 with expected resource counts and unchanged PROPFIND/multiget body sizes.

Before, fetched base 82c5e64f51:

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 85.749347, 20.727567, 32.902707, 50.905726, 36.111467 36.111467 85.749347 FAIL (<30 ms, every sample)
rebuild 1000 warm 13.107354, 32.333165, 9.713585, 12.359236, 31.176197 13.107354 32.333165 FAIL (<30 ms, every sample)
rebuild 10000 cold 346.872530, 589.540054, 936.179287, 1088.261861, 780.247507 780.247507 1088.261861 report only
rebuild 10000 warm 200.452367, 204.157791, 198.634917, 240.282223, 199.049035 200.452367 240.282223 report only
sync 1000 cold 1.092146, 0.955207, 1.030419, 0.980315, 1.036341 1.030419 1.092146 PASS (<2 ms, every sample)
sync 1000 warm 0.057177, 0.056878, 0.058750, 0.057639, 0.056956 0.057177 0.058750 PASS (<2 ms, every sample)
propfind 1000 cold 35.103631, 29.413418, 28.018973, 30.116652, 32.720193 30.116652 35.103631 FAIL (<30 ms, every sample)
propfind 1000 warm 2.474969, 2.449030, 2.628417, 2.450051, 2.465260 2.465260 2.628417 PASS (<30 ms, every sample)
multiget 1000 cold 42.780871, 39.764743, 95.753909, 39.195092, 45.818720 42.780871 95.753909 FAIL (<30 ms, every sample)
multiget 1000 warm 7.560351, 7.578275, 8.878652, 7.581603, 7.533762 7.578275 8.878652 PASS (<30 ms, every sample)

After:

Operation Cards Cache Raw milliseconds (runs 1–5) p50 ms p95 ms Gate
rebuild 1000 cold 19.509446, 11.443830, 11.520274, 11.173687, 11.549077 11.520274 19.509446 PASS (<30 ms, every sample)
rebuild 1000 warm 17.104870, 9.957718, 10.268257, 9.354671, 9.214911 9.957718 17.104870 PASS (<30 ms, every sample)
rebuild 10000 cold 128.336459, 383.330295, 190.365571, 421.488210, 384.022982 383.330295 421.488210 report only
rebuild 10000 warm 101.281288, 98.627526, 98.511329, 106.660354, 98.094061 98.627526 106.660354 report only
sync 1000 cold 1.134989, 0.943442, 0.918003, 0.983787, 0.933343 0.943442 1.134989 PASS (<2 ms, every sample)
sync 1000 warm 0.028824, 0.020117, 0.020228, 0.020017, 0.020058 0.020117 0.028824 PASS (<2 ms, every sample)
propfind 1000 cold 6.448911, 7.153356, 6.651169, 9.052732, 7.916832 7.153356 9.052732 PASS (<30 ms, every sample)
propfind 1000 warm 2.290456, 3.135935, 2.419226, 2.240003, 3.127670 2.419226 3.135935 PASS (<30 ms, every sample)
multiget 1000 cold 14.420356, 14.469957, 11.998594, 13.532337, 12.303051 13.532337 14.469957 PASS (<30 ms, every sample)
multiget 1000 warm 5.151638, 5.157289, 5.135829, 7.577288, 5.058765 5.151638 7.577288 PASS (<30 ms, every sample)

Verbatim final performance gate output (exit 0):

rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index)
rebuild_1000: PASS max=19.509446 ms budget=<30 ms (5 cold + 5 warm)
sync_1000: PASS max=1.134989 ms budget=<2 ms (5 cold + 5 warm)
propfind_1000: PASS max=9.052732 ms budget=<30 ms (5 cold + 5 warm)
multiget_1000: PASS max=14.469957 ms budget=<30 ms (5 cold + 5 warm)

Verbatim code gate history and final test totals:

fmt: exit 0
carddav: exit 0
clippy: exit 0
workspace: exit 101
workspace retry after frozen Bun install: exit 0
carddav: 20 passed, 0 failed, 0 ignored
workspace: 1175 passed, 0 failed, 20 ignored

The first workspace run failed when a Bun bridge returned empty output while JavaScript dependencies were absent. Installing from the frozen Bun lockfile made both real SyncClient tests pass in 0.40 seconds, then the full workspace retry passed. No product or test source changed for that retry. The original failure and all retry logs are retained.

All 20 measured rebuilds at 1,000 and 10,000 cards compare every href, wire byte, strong ETag and serialized property index with independent incremental baselines. Final rows also match the previous job's independent incremental databases exactly. This is row-byte identity; generations intentionally rotate.

Final diagnostic profiles show cold rebuild explicit reads falling from 414 to zero and write fragments from 833 to 471. Cold commits still perform two fdatasync calls. Mapping uses page faults, so zero pread calls does not imply zero physical I/O. The PROPFIND metadata table occupies 24 SQLite pages for 1,000 cards. Profiles are separate diagnostics and do not replace any gate samples. All unsuccessful intermediate experiments are retained, including the removed larger-page experiment.

Complete raw measurements, profiles and gate logs (ZIP SHA-256: 1eb9a9198a227ca0bd58293e9feedf0421ea119af618f7adbae15703ae7bb801). Evidence stays outside the repository; the commit contains only CardDAV product code and tests.

Finished perf-contacts-fix. Head: a45e7482b5216a364a55a8a67c0bc3327d9f6663. Branch: issue/carddav-read-perf-fix. [Follow-up PR #739](https://git.kayg.org/kayg/calternal/pulls/739) targets issue/carddav-read, keeping its diff focused on the performance fix. The hard rule requires a new branch and PR because updates to an open PR branch are ignored. PR #734 and issue #657 remain open. No merge or deployment was performed. All final gates pass. Payloads are stored once in revision history; live metadata holds the revision pointer, ETag and byte length. Rebuild constructs fresh tables in memory and installs them atomically through SQLite backup. PROPFIND reads compact metadata, multiget reuses prepared lookups through a bounded 64 MiB mapping, and unchanged sync checks read the committed head metadata. Legacy migration preserves rows, cursor, epoch and paged sync snapshots. WAL NORMAL, default 4 KiB pages, default automatic checkpoints and all limits remain unchanged. Staging temporarily uses memory proportional to the projected book. Method matches the preceding job: assigned replacement VM perf-test-02, 4 CPU / 8 GiB parent perf.slice; Homes on rotational /srv/homes (/dev/sdb1, XFS), state on the OS SSD (/dev/sda1, ext4). Every operation is serial under perf-run.sh and perf-lock.sh. Five independent cold and five warm invocations per operation; cold drops guest caches, warm performs one identical untimed operation first. All samples are retained. Nearest-rank p50/p95 means p95 is the maximum with five samples; every measured sample must be strictly below its gate. The 10,000-card rebuild remains report only. The rebuild timer includes in-memory note parsing, contact conversion, indexing, hashing, staging, SQLite replacement and commit. HTTP timing includes the router, unchanged-source barrier, XML parsing, cached reads and complete body collection with the original test authenticator. Source callbacks panic on an attempted Home scan, conversion or change replay during reads. Responses remain 207 with expected resource counts and unchanged PROPFIND/multiget body sizes. Before, fetched base 82c5e64f512f026437159403c61cf3a71cb40f6a: | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 85.749347, 20.727567, 32.902707, 50.905726, 36.111467 | 36.111467 | 85.749347 | FAIL (<30 ms, every sample) | | rebuild | 1000 | warm | 13.107354, 32.333165, 9.713585, 12.359236, 31.176197 | 13.107354 | 32.333165 | FAIL (<30 ms, every sample) | | rebuild | 10000 | cold | 346.872530, 589.540054, 936.179287, 1088.261861, 780.247507 | 780.247507 | 1088.261861 | report only | | rebuild | 10000 | warm | 200.452367, 204.157791, 198.634917, 240.282223, 199.049035 | 200.452367 | 240.282223 | report only | | sync | 1000 | cold | 1.092146, 0.955207, 1.030419, 0.980315, 1.036341 | 1.030419 | 1.092146 | PASS (<2 ms, every sample) | | sync | 1000 | warm | 0.057177, 0.056878, 0.058750, 0.057639, 0.056956 | 0.057177 | 0.058750 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 35.103631, 29.413418, 28.018973, 30.116652, 32.720193 | 30.116652 | 35.103631 | FAIL (<30 ms, every sample) | | propfind | 1000 | warm | 2.474969, 2.449030, 2.628417, 2.450051, 2.465260 | 2.465260 | 2.628417 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 42.780871, 39.764743, 95.753909, 39.195092, 45.818720 | 42.780871 | 95.753909 | FAIL (<30 ms, every sample) | | multiget | 1000 | warm | 7.560351, 7.578275, 8.878652, 7.581603, 7.533762 | 7.578275 | 8.878652 | PASS (<30 ms, every sample) | After: | Operation | Cards | Cache | Raw milliseconds (runs 1–5) | p50 ms | p95 ms | Gate | |---|---:|---|---|---:|---:|---| | rebuild | 1000 | cold | 19.509446, 11.443830, 11.520274, 11.173687, 11.549077 | 11.520274 | 19.509446 | PASS (<30 ms, every sample) | | rebuild | 1000 | warm | 17.104870, 9.957718, 10.268257, 9.354671, 9.214911 | 9.957718 | 17.104870 | PASS (<30 ms, every sample) | | rebuild | 10000 | cold | 128.336459, 383.330295, 190.365571, 421.488210, 384.022982 | 383.330295 | 421.488210 | report only | | rebuild | 10000 | warm | 101.281288, 98.627526, 98.511329, 106.660354, 98.094061 | 98.627526 | 106.660354 | report only | | sync | 1000 | cold | 1.134989, 0.943442, 0.918003, 0.983787, 0.933343 | 0.943442 | 1.134989 | PASS (<2 ms, every sample) | | sync | 1000 | warm | 0.028824, 0.020117, 0.020228, 0.020017, 0.020058 | 0.020117 | 0.028824 | PASS (<2 ms, every sample) | | propfind | 1000 | cold | 6.448911, 7.153356, 6.651169, 9.052732, 7.916832 | 7.153356 | 9.052732 | PASS (<30 ms, every sample) | | propfind | 1000 | warm | 2.290456, 3.135935, 2.419226, 2.240003, 3.127670 | 2.419226 | 3.135935 | PASS (<30 ms, every sample) | | multiget | 1000 | cold | 14.420356, 14.469957, 11.998594, 13.532337, 12.303051 | 13.532337 | 14.469957 | PASS (<30 ms, every sample) | | multiget | 1000 | warm | 5.151638, 5.157289, 5.135829, 7.577288, 5.058765 | 5.151638 | 7.577288 | PASS (<30 ms, every sample) | Verbatim final performance gate output (exit 0): ```text rebuild byte identity: PASS (every cold and warm measured rebuild, 1,000 and 10,000 cards; href, wire bytes, ETag and serialized property index) rebuild_1000: PASS max=19.509446 ms budget=<30 ms (5 cold + 5 warm) sync_1000: PASS max=1.134989 ms budget=<2 ms (5 cold + 5 warm) propfind_1000: PASS max=9.052732 ms budget=<30 ms (5 cold + 5 warm) multiget_1000: PASS max=14.469957 ms budget=<30 ms (5 cold + 5 warm) ``` Verbatim code gate history and final test totals: ```text fmt: exit 0 carddav: exit 0 clippy: exit 0 workspace: exit 101 workspace retry after frozen Bun install: exit 0 carddav: 20 passed, 0 failed, 0 ignored workspace: 1175 passed, 0 failed, 20 ignored ``` The first workspace run failed when a Bun bridge returned empty output while JavaScript dependencies were absent. Installing from the frozen Bun lockfile made both real SyncClient tests pass in 0.40 seconds, then the full workspace retry passed. No product or test source changed for that retry. The original failure and all retry logs are retained. All 20 measured rebuilds at 1,000 and 10,000 cards compare every href, wire byte, strong ETag and serialized property index with independent incremental baselines. Final rows also match the previous job's independent incremental databases exactly. This is row-byte identity; generations intentionally rotate. Final diagnostic profiles show cold rebuild explicit reads falling from 414 to zero and write fragments from 833 to 471. Cold commits still perform two fdatasync calls. Mapping uses page faults, so zero pread calls does not imply zero physical I/O. The PROPFIND metadata table occupies 24 SQLite pages for 1,000 cards. Profiles are separate diagnostics and do not replace any gate samples. All unsuccessful intermediate experiments are retained, including the removed larger-page experiment. [Complete raw measurements, profiles and gate logs](https://git.kayg.org/attachments/5b22b7a5-d8cc-456d-9f1d-330798bd0cb4) (ZIP SHA-256: 1eb9a9198a227ca0bd58293e9feedf0421ea119af618f7adbae15703ae7bb801). Evidence stays outside the repository; the commit contains only CardDAV product code and tests.
kayg commented 2026-10-31 05:14:13 +00:00
Author
Owner
Copy link

Replaced by #776: same change rebased onto current dev (branch issue/carddav-read-r2), build and tests green on dev-02. This PR is left open and untouched; please close it when the replacement is accepted.

Replaced by #776: same change rebased onto current dev (branch `issue/carddav-read-r2`), build and tests green on dev-02. This PR is left open and untouched; please close it when the replacement is accepted.
kayg commented 2026-11-04 13:04:31 +00:00
Author
Owner
Copy link

Closing: superseded by #776 (named in the PR body or comments). Branch kept.

Closing: superseded by #776 (named in the PR body or comments). Branch kept.
kayg closed this pull request 2026-11-04 13:04:31 +00:00
Some checks failed
CI / server (pull_request) Successful in 1m21s
Details
CI / rust (pull_request) Failing after 22m47s
Details
CI / web (pull_request) Failing after 2h6m19s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!734
No description provided.