Perf: review the Notes bulk-create tail seen in #1393 #1426

Open
opened 2026-10-08 00:53:44 +00:00 by kayg · 2 comments
Owner

Five interleaved before/after samples in #1393 found a higher tail for 100 sequential Notes creates on the perf VM. This is a timing finding, not a failure or a data-safety problem. It does not block the merge.

Before release: ef0aa12b1. After release: c1300457f. Command: perf-run calternal bulkwrite-1393-average 30 -- bash /root/calternal-1393/target/tmp/1393-average.sh, using bench/bulk-write-1393.mjs with a fresh Instance for each sample, 100 creates, concurrency 1. Both variants ran on the VM root ext4 disk; HDD results are separate. All samples passed and no slow sample was removed.

  • Batch durable-response wall time p50: 5.974 s before, 6.059 s after (+1.4%).
  • Batch wall time p95: 6.025 s before, 7.199 s after (+19.5%, over the published 10% latency review threshold).
  • Median of each run's request p95: 77.0 ms before, 76.5 ms after.
  • The slow after sample had one 551.4 ms maximum request and a 125.6 ms request p95. Its start load was 0.99 / 1.45 / 1.94.
  • Median server CPU over writes and Search: 2,650 ms before, 2,500 ms after. Peak RSS medians: 159.9 / 159.8 MiB.

The signal is in batch-tail time, not the typical request. There is no matching bulk Notes profile in docs/perf/baseline.json. Cause is unproven; review this in the normal periodic performance round. Do not add more synchronous scanning or weaken durability to resolve it. #1393 has the complete paired table and raw sample evidence in its worktree artifacts.

Five interleaved before/after samples in #1393 found a higher tail for 100 sequential Notes creates on the perf VM. This is a timing finding, not a failure or a data-safety problem. It does not block the merge. Before release: `ef0aa12b1`. After release: `c1300457f`. Command: `perf-run calternal bulkwrite-1393-average 30 -- bash /root/calternal-1393/target/tmp/1393-average.sh`, using `bench/bulk-write-1393.mjs` with a fresh Instance for each sample, 100 creates, concurrency 1. Both variants ran on the VM root ext4 disk; HDD results are separate. All samples passed and no slow sample was removed. - Batch durable-response wall time p50: 5.974 s before, 6.059 s after (+1.4%). - Batch wall time p95: 6.025 s before, 7.199 s after (+19.5%, over the published 10% latency review threshold). - Median of each run's request p95: 77.0 ms before, 76.5 ms after. - The slow after sample had one 551.4 ms maximum request and a 125.6 ms request p95. Its start load was 0.99 / 1.45 / 1.94. - Median server CPU over writes and Search: 2,650 ms before, 2,500 ms after. Peak RSS medians: 159.9 / 159.8 MiB. The signal is in batch-tail time, not the typical request. There is no matching bulk Notes profile in `docs/perf/baseline.json`. Cause is unproven; review this in the normal periodic performance round. Do not add more synchronous scanning or weaken durability to resolve it. #1393 has the complete paired table and raw sample evidence in its worktree artifacts.
Author
Owner

Numerical clarification for the retained five-sample data: the Notes after median is 6060.706 ms (6.061 s), before 5974.260 ms (5.974 s). The p95 values and about 1.4% median change in the report are unchanged.

Numerical clarification for the retained five-sample data: the Notes after median is 6060.706 ms (6.061 s), before 5974.260 ms (5.974 s). The p95 values and about 1.4% median change in the report are unchanged.
Author
Owner

Metric scope: the 19.5% change is the p95 of five complete 100-write batch durations. bench/compare.py publishes a 10% threshold for route p95. The batch tail is above that reference level, but it is not an automatic route-p95 violation; median per-sample request p95 is stable (77.0 ms before, 76.5 ms after). This issue asks for review of the batch tail and does not establish a code cause.

Metric scope: the 19.5% change is the p95 of five complete 100-write batch durations. `bench/compare.py` publishes a 10% threshold for route p95. The batch tail is above that reference level, but it is not an automatic route-p95 violation; median per-sample request p95 is stable (77.0 ms before, 76.5 ms after). This issue asks for review of the batch tail and does not establish a code cause.
Commenting is not possible because the repository is archived.
No labels
No milestone
No project
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal-v0#1426
No description provided.