Tags read p95 rises during batched rewrite compared with dev #1499

Open
opened 2026-10-09 05:19:35 +00:00 by kayg · 1 comment
Owner

The #1186 tagperf4 A/B profile found higher GET /api/v1/tags p95 during the candidate rewrite. Both builds run on perf-test-69-63 under perf-run, with the same frozen e1672b919 production SPA, fresh 1,000-Note Homes, sixteen 1 Hz Tag readers, and one separate-User conditional Note-body writer (250 ms delay after each successful write).

Baseline is origin/dev cc1af3c12; candidate Rust source is a77e71920 (quota error fix plus merged dev). All requested code gates pass. Both servers have a 65,536 descriptor budget. Neither read nor write route returned an error in these pairs.

Pair Baseline Tag-read p95 ms Candidate Tag-read p95 ms Baseline completion s Candidate completion s Baseline load5 start/end Candidate load5 start/end
1 46.98 56.42 >900.00 25.70 1.95 / 6.22 6.38 / 6.29
2 47.58 84.20 869.06 25.83 4.69 / 5.79 5.65 / 5.59
3 49.50 113.17 >900.02 25.76 5.51 / 6.03 5.96 / 5.86

Median per-run p95: baseline 47.58 ms, candidate 84.20 ms (+76.97%). The route-p95 regression threshold in bench/compare.py is 10%. These are medians of per-run p95, not a pooled request percentile.

Interpretation limit: candidate request windows are about 26 seconds; baseline windows are about 15 minutes. Startup background work overlaps both, but the short candidate windows give startup work more weight. This observation does not isolate a steady-state route regression or its cause. It is still the requested during-rewrite measurement and must not be hidden. Candidate other-User write p95 improves in all three pairs, and every candidate UI clock meets 500 ms.

Raw profiles and the scoped baseline are in docs/perf/runs/tag-rewrite-1186-tagperf4.json and docs/perf/baseline.json; method and table are in docs/perf/2026-10-09-tagperf4-1186.md. The 5,000-Note pairs are running; their results will be added. Performance findings do not block a merge under the owner rule.

The #1186 tagperf4 A/B profile found higher GET /api/v1/tags p95 during the candidate rewrite. Both builds run on perf-test-69-63 under perf-run, with the same frozen e1672b919 production SPA, fresh 1,000-Note Homes, sixteen 1 Hz Tag readers, and one separate-User conditional Note-body writer (250 ms delay after each successful write). Baseline is origin/dev cc1af3c12; candidate Rust source is a77e71920 (quota error fix plus merged dev). All requested code gates pass. Both servers have a 65,536 descriptor budget. Neither read nor write route returned an error in these pairs. | Pair | Baseline Tag-read p95 ms | Candidate Tag-read p95 ms | Baseline completion s | Candidate completion s | Baseline load5 start/end | Candidate load5 start/end | | ---: | ---: | ---: | ---: | ---: | --- | --- | | 1 | 46.98 | 56.42 | >900.00 | 25.70 | 1.95 / 6.22 | 6.38 / 6.29 | | 2 | 47.58 | 84.20 | 869.06 | 25.83 | 4.69 / 5.79 | 5.65 / 5.59 | | 3 | 49.50 | 113.17 | >900.02 | 25.76 | 5.51 / 6.03 | 5.96 / 5.86 | Median per-run p95: baseline 47.58 ms, candidate 84.20 ms (+76.97%). The route-p95 regression threshold in bench/compare.py is 10%. These are medians of per-run p95, not a pooled request percentile. Interpretation limit: candidate request windows are about 26 seconds; baseline windows are about 15 minutes. Startup background work overlaps both, but the short candidate windows give startup work more weight. This observation does not isolate a steady-state route regression or its cause. It is still the requested during-rewrite measurement and must not be hidden. Candidate other-User write p95 improves in all three pairs, and every candidate UI clock meets 500 ms. Raw profiles and the scoped baseline are in docs/perf/runs/tag-rewrite-1186-tagperf4.json and docs/perf/baseline.json; method and table are in docs/perf/2026-10-09-tagperf4-1186.md. The 5,000-Note pairs are running; their results will be added. Performance findings do not block a merge under the owner rule.
Author
Owner

#1499 remains open. A new matched 1,000-Note pair against origin/dev 9dc65437e measured Tag-list p95 47.36 ms baseline versus 60.81 ms candidate (+28.4%; earlier candidate was about 84 ms). p50 was 22.12 versus 20.47 ms. Both had zero read/write errors. Candidate completed 1,000 Notes in 24.249 s; baseline was still running at 847/1,000 at its 900-second cap. load5 before/after: baseline 6.70/5.83, candidate 6.36/6.06. Candidate terminal ordinary-read p95 was 29.04 ms.

The Tag-list route uses the reader pool, with recursive prefix expansion and two GROUP BY stages per request. The acceptance writer lock is not on this read path. Bounded writer batches remove the long authority transaction but do not remove recursive read CPU work. The candidate also packs startup/read bursts into a much shorter measurement window; those bursts have more p95 weight than in a 900-second baseline window. CPU competition/window weighting is a supported explanation, not isolated attribution. Three 5,000-Note pairs were mixed: baseline p95 215.81/203.31/208.33 ms, candidate 196.00/234.42/197.99 ms.

All runs held perf-run's shared lock and used frozen production SPA e1672b919. Raw records, hashes, actual order, load and CPU/RSS are committed in docs/perf/runs/tag-rewrite-1186-tagfix5.json (a35ea071d). No issue is closed.

#1499 remains open. A new matched 1,000-Note pair against origin/dev 9dc65437e measured Tag-list p95 47.36 ms baseline versus 60.81 ms candidate (+28.4%; earlier candidate was about 84 ms). p50 was 22.12 versus 20.47 ms. Both had zero read/write errors. Candidate completed 1,000 Notes in 24.249 s; baseline was still running at 847/1,000 at its 900-second cap. load5 before/after: baseline 6.70/5.83, candidate 6.36/6.06. Candidate terminal ordinary-read p95 was 29.04 ms. The Tag-list route uses the reader pool, with recursive prefix expansion and two GROUP BY stages per request. The acceptance writer lock is not on this read path. Bounded writer batches remove the long authority transaction but do not remove recursive read CPU work. The candidate also packs startup/read bursts into a much shorter measurement window; those bursts have more p95 weight than in a 900-second baseline window. CPU competition/window weighting is a supported explanation, not isolated attribution. Three 5,000-Note pairs were mixed: baseline p95 215.81/203.31/208.33 ms, candidate 196.00/234.42/197.99 ms. All runs held perf-run's shared lock and used frozen production SPA e1672b919. Raw records, hashes, actual order, load and CPU/RSS are committed in docs/perf/runs/tag-rewrite-1186-tagfix5.json (a35ea071d). No issue is closed.
Commenting is not possible because the repository is archived.
No labels
No milestone
No project
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal-v0#1499
No description provided.