kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

WIP: Perf squeeze: reduce search text allocations and tag scanning #514

Closed
kayg wants to merge 0 commits from issue/perf-squeeze into dev
pull from: issue/perf-squeeze
merge into: kayg:dev
Conversation 1 Commits - Files changed -
kayg commented 2026-10-16 22:17:49 +00:00
Owner
Copy link

Search document construction copied every complete note and allocated new strings for lines whose text did not change. Tag scanning also parsed code, links and addresses on lines that could not contain a tag. Borrow unchanged search text until the document is joined, allocate only when removing markup, and skip tag parsing on lines without hashes after fence tracking.

Two commits, with regression tests for borrowed text, Unicode, links, replacement order and code fences. Durability, isolation and read concurrency are unchanged.

Five interleaved direct benchmarks used the same executable and fixture text, with identical output. Each measurement ran 30 passes over 1,535,671 bytes and 17,588 lines:

Work Before median ms After median ms Reduction
Clean and join search text 319.417 115.279 63.9%
Scan inline tags 348.028 142.184 59.1%

These measure the affected functions. Whole-server measurements and concurrent request tails follow below. CPU profiles of cold indexing, search catch-up and journal catch-up are summarized in issue #512.

Validation at be555c1dfc:

  • cargo fmt --all --check: exit 0.
  • cargo clippy --workspace --all-targets -- -D warnings: exit 0.
  • cargo test --workspace: exit 0, including isolation tests.

Full gate output is recorded on the issue. Related to #512.

The performance gate remains red on HDD. The tables retain every primary run and the observed maxima, including regressions. Budgets are unchanged. Whole-server timing varies substantially; the unchanged baseline validation run also became faster, so the direct benchmarks give the clearest evidence for each code change.

100k cold indexing Before median s After median s Before max s After max s
ssd 33.190 22.963 39.207 32.186
hdd 49.445 42.414 56.417 53.068

Some primary maxima worsened:

  • ssd-10000 sync.save_ack: 92.444 → 348.306 ms.
  • hdd-100000 sync.save_ack: 1218.771 → 1995.020 ms.
  • hdd-100000 shutdown: 3082.647 → 3331.564 ms.

Concurrent tails also regressed in this single probe per phase and disk. SSD durable-append p99 rose from 155.103 to 302.955 ms; HDD search p99 rose from 1113.187 to 1851.468 ms. The PR is work in progress because these results do not establish an overall tail-latency improvement. The CPU changes and their tests are ready for review; further tail analysis is needed before merging.

Twenty interleaved startup checks per binary gave RSS medians 13.672 → 13.723 MiB, a 0.051 MiB difference within the observed process spread. Startup p95 was 13.871 → 14.207 MiB and maximum 14.031 → 14.699 MiB.

HDD/100k RSS after use, primary idle sample and full validation repeat: before 37.938, 47.199 MiB. These samples quantify process-to-process variation in memory after use.
HDD/100k RSS after use, primary idle sample and full validation repeat: final 43.309, 47.211 MiB. These samples quantify process-to-process variation in memory after use.

Extra HDD/100k run Cold index s Save ack p95 ms Save ack max ms Idle RSS MiB Peak RSS MiB
before 43.431 66.096 211.471 47.199 95.090
final 39.059 62.093 382.697 47.211 92.934

The additional validation pair ran the entire cold and warm workload, including typing and 300 seconds idle. Its complete table is on issue #512. These two runs are separate from the primary five-run table.

Before/after server performance. Base 312bc892ae; head be555c1dfc. Same release toolchain, fixture seed and harness; dropped page cache before each start. Five interleaved runs per cell. Latency distributions use the median of per-run p50/p95, and max is the largest sample. Single-value metrics pool the runs. Idle measurements use one 300-second sample per cell after rooms settle. Both binaries were built with matching Cargo feature unification: cargo build --release --locked -j4 -p calternal -p calternal-perf -p calternal-fixtures. Each invocation resets from the pristine seed-1 fixture and includes one cold and one warm run; pass 5 also includes the stock 60-second typing load and extra quiet checks. Memory follows the harness MB label (MiB).

ssd-10000

Metric Unit Before p50 Before p95 Before max After p50 After p95 After max
startup.healthz ms 32.293 46.417 46.417 30.235 37.659 37.659
home.first_page ms 12.030 26.745 26.745 11.365 14.541 14.541
home.page ms 1.379 2.096 7.982 1.405 1.693 4.276
home.page_title ms 1.238 3.118 5.634 1.150 1.705 4.975
home.page_tag ms 9.242 12.737 182.386 8.669 11.246 87.584
home.page_next ms 1.417 2.107 4.798 1.281 1.773 3.552
home.page_big ms 28.815 34.508 50.323 28.390 29.420 34.461
search.query ms 3.554 7.285 65.865 3.678 10.086 75.070
search.first_query ms 27.849 43.668 43.668 27.526 79.223 79.223
search.catch_up s 4.012 6.016 6.027 4.026 4.026 6.034
open.note ms 1.254 2.620 8.883 1.073 2.803 5.073
open.api_get ms 7.650 9.381 14.894 7.199 8.571 14.859
sync.save_ack ms 5.985 8.562 92.444 5.903 10.210 348.306
storm.save_ack ms 10.403 23.176 113.846 8.138 22.508 141.278
journal.catch_up ms 62.046 94.674 94.674 61.780 65.217 65.217
cold.index s 2.056 2.267 2.267 1.498 1.677 1.677
cold.startup ms 97.708 134.832 134.832 70.602 79.012 79.012
open.long_note ms 9.570 12.641 12.641 9.955 12.699 12.699
journal.page ms 5.588 7.392 14.115 5.398 6.435 6.830
shutdown ms 66.128 67.453 67.453 45.918 67.229 67.229
storm.edits_per_s edits/s 2217.073 3214.040 3214.040 2501.178 3980.039 3980.039
storm.disk_written MB 50.520 54.227 54.227 50.901 52.302 52.302
trash.delete ms 13.784 23.300 78.310 12.287 20.364 73.600
trash.restore ms 16.036 26.539 174.210 15.090 27.179 60.917
trash.forget ms 7.604 12.991 47.241 7.399 11.601 29.835
mem.idle MB 13.199 13.414 13.414 13.230 14.031 14.031
mem.peak MB 38.957 40.543 40.543 37.152 40.902 40.902
cold.mem_peak MB 50.082 50.875 50.875 51.672 51.711 51.711
idle.rss MB 26.535 26.535 26.535 26.758 26.758 26.758
idle.cpu_pct % 0.000 0.000 0.000 0.000 0.000 0.000
idle.wakeups_per_s /s 0.000 0.000 0.000 0.000 0.000 0.000

ssd-100000

Metric Unit Before p50 Before p95 Before max After p50 After p95 After max
startup.healthz ms 47.894 89.280 89.280 37.325 49.970 49.970
home.first_page ms 55.071 99.624 99.624 18.709 41.162 41.162
home.page ms 2.361 3.694 8.512 2.088 2.767 5.372
home.page_title ms 2.588 7.041 11.183 1.893 4.152 55.433
home.page_tag ms 6.699 11.686 299.157 5.668 7.760 176.247
home.page_next ms 2.708 5.130 14.808 2.125 3.301 11.855
home.page_big ms 411.416 643.016 1049.351 133.137 220.357 914.898
search.query ms 5.773 13.903 760.582 4.587 9.729 801.232
search.first_query ms 61.681 77.668 77.668 41.368 72.185 72.185
search.catch_up s 92.739 98.791 104.979 68.355 68.355 92.909
open.note ms 2.274 5.296 34.235 1.368 4.234 12.606
open.api_get ms 12.275 16.739 32.028 9.572 19.706 340.469
sync.save_ack ms 9.602 35.688 209.367 8.629 14.916 115.846
storm.save_ack ms 17.587 44.747 937.979 13.730 31.191 736.756
journal.catch_up ms 1126.472 1309.691 1309.691 771.246 1541.604 1541.604
cold.index s 33.190 39.207 39.207 22.963 32.186 32.186
cold.startup ms 169.768 213.792 213.792 85.390 110.088 110.088
open.long_note ms 13.182 23.185 23.185 13.927 17.811 17.811
journal.page ms 9.220 14.758 50.788 6.558 10.514 93.871
shutdown ms 88.419 93.827 93.827 65.785 115.179 115.179
storm.edits_per_s edits/s 1281.929 2471.487 2471.487 1555.176 4283.820 4283.820
storm.disk_written MB 49.512 56.533 56.533 44.257 44.376 44.376
trash.delete ms 18.098 28.986 287.089 19.575 33.131 68.809
trash.restore ms 23.266 39.272 301.490 23.498 36.498 298.570
trash.forget ms 11.330 24.394 77.260 11.017 19.061 57.756
mem.idle MB 13.109 13.328 13.328 13.289 13.352 13.352
mem.peak MB 96.254 99.703 99.703 91.637 96.473 96.473
cold.mem_peak MB 119.871 121.469 121.469 117.426 121.172 121.172
idle.rss MB 41.789 41.789 41.789 39.508 39.508 39.508
idle.cpu_pct % 0.000 0.000 0.000 0.000 0.000 0.000
idle.wakeups_per_s /s 0.000 0.000 0.000 0.000 0.000 0.000

hdd-10000

Metric Unit Before p50 Before p95 Before max After p50 After p95 After max
startup.healthz ms 41.033 178.483 178.483 33.838 61.189 61.189
home.first_page ms 31.201 43.028 43.028 18.645 68.831 68.831
home.page ms 1.816 3.533 18.805 1.364 1.815 7.108
home.page_title ms 1.517 2.152 14.762 1.229 2.578 12.484
home.page_tag ms 11.141 16.243 542.228 8.503 17.351 243.223
home.page_next ms 1.913 3.674 23.067 1.496 1.759 24.673
home.page_big ms 33.585 46.160 131.010 33.118 36.892 114.975
search.query ms 3.738 8.538 88.265 4.871 8.764 90.615
search.first_query ms 184.362 199.424 199.424 153.797 266.908 266.908
search.catch_up s 10.035 10.035 12.078 8.024 8.034 10.047
open.note ms 1.351 3.433 25.293 1.310 4.802 32.175
open.api_get ms 35.976 60.643 96.257 37.224 60.436 236.259
sync.save_ack ms 15.036 56.526 1516.069 14.427 49.491 1321.199
storm.save_ack ms 47.024 134.363 741.346 42.665 114.206 496.167
journal.catch_up ms 66.749 194.828 194.828 66.972 135.687 135.687
cold.index s 4.100 5.092 5.092 3.860 4.078 4.078
cold.startup ms 432.376 690.480 690.480 466.030 623.636 623.636
open.long_note ms 18.735 36.723 36.723 17.182 51.035 51.035
journal.page ms 5.897 7.721 31.740 6.140 7.196 16.690
shutdown ms 89.005 228.973 228.973 193.058 263.189 263.189
storm.edits_per_s edits/s 497.648 633.855 633.855 585.738 611.050 611.050
storm.disk_written MB 81.326 91.574 91.574 71.856 80.327 80.327
trash.delete ms 52.399 115.189 262.883 56.079 91.103 245.028
trash.restore ms 72.064 179.774 494.562 73.554 151.461 466.337
trash.forget ms 36.029 85.912 318.037 34.219 61.051 204.856
mem.idle MB 13.242 13.336 13.336 13.199 13.332 13.332
mem.peak MB 39.695 40.633 40.633 39.121 41.074 41.074
cold.mem_peak MB 50.340 53.176 53.176 50.715 51.609 51.609
idle.rss MB 26.441 26.441 26.441 26.727 26.727 26.727
idle.cpu_pct % 0.000 0.000 0.000 0.000 0.000 0.000
idle.wakeups_per_s /s 0.000 0.000 0.000 0.000 0.000 0.000

hdd-100000

Metric Unit Before p50 Before p95 Before max After p50 After p95 After max
startup.healthz ms 104.669 157.592 157.592 66.592 85.778 85.778
home.first_page ms 120.894 136.916 136.916 67.718 86.076 86.076
home.page ms 2.490 10.606 19.555 1.619 2.174 5.770
home.page_title ms 2.074 5.776 26.489 1.327 1.868 12.102
home.page_tag ms 6.993 14.752 252.036 4.923 6.160 183.359
home.page_next ms 2.784 5.371 18.184 1.604 2.346 12.684
home.page_big ms 453.527 668.061 1358.472 193.059 283.536 1016.940
search.query ms 5.676 12.601 730.219 4.606 12.972 707.446
search.first_query ms 338.579 434.254 434.254 402.598 406.983 406.983
search.catch_up s 132.778 142.799 144.976 118.419 118.419 136.895
open.note ms 1.308 6.811 151.554 1.117 2.667 10.359
open.api_get ms 41.418 114.401 349.620 40.827 89.825 298.186
sync.save_ack ms 14.212 82.811 1218.771 14.824 58.937 1995.020
storm.save_ack ms 44.949 129.829 418.613 38.313 101.175 381.548
journal.catch_up ms 901.677 1127.429 1127.429 639.896 1217.029 1217.029
cold.index s 49.445 56.417 56.417 42.414 53.068 53.068
cold.startup ms 862.585 2513.378 2513.378 664.830 1053.712 1053.712
open.long_note ms 11.796 26.785 26.785 9.764 44.867 44.867
journal.page ms 7.448 13.184 32.491 5.727 8.130 29.115
shutdown ms 2414.701 3082.647 3082.647 2179.835 3331.564 3331.564
storm.edits_per_s edits/s 545.469 622.575 622.575 661.722 705.388 705.388
storm.disk_written MB 65.839 67.592 67.592 57.516 63.365 63.365
trash.delete ms 75.191 128.685 239.133 74.444 149.597 403.720
trash.restore ms 87.383 171.128 307.504 90.418 164.083 337.787
trash.forget ms 49.332 101.264 148.389 44.311 101.103 174.414
mem.idle MB 13.289 13.316 13.316 13.223 13.438 13.438
mem.peak MB 93.508 95.859 95.859 91.848 94.980 94.980
cold.mem_peak MB 131.770 136.805 136.805 129.691 136.598 136.598
idle.rss MB 37.938 37.938 37.938 43.309 43.309 43.309
idle.cpu_pct % 0.000 0.000 0.000 0.000 0.000 0.000
idle.wakeups_per_s /s 0.003 0.003 0.003 0.000 0.000 0.000

20 concurrent clients on an indexed 100k-note Home, after note and search indexes settle. No think time. Each client runs 50 mixed rounds with page, note, search and durable append operations. Each operation has 1,000 samples per phase and disk; all 1,000 appended markers were verified exactly once. Latencies are milliseconds.

Disk Operation Before p50 Before p95 Before p99 Before max After p50 After p95 After p99 After max
ssd notes.list 23.954 62.956 81.148 106.318 12.379 30.373 44.321 73.946
ssd notes.get 10.323 36.480 74.479 140.765 5.324 18.200 47.674 104.773
ssd search.query 76.503 184.521 245.662 286.207 56.592 121.839 171.461 275.802
ssd notes.append 55.499 112.510 155.103 236.001 31.102 202.308 302.955 363.992
hdd notes.list 7.750 17.816 22.687 25.815 12.669 29.402 36.163 50.615
hdd notes.get 3.937 13.103 566.419 698.729 6.451 20.525 100.471 581.054
hdd search.query 32.810 322.049 1113.187 1707.499 54.615 244.054 1851.468 1972.262
hdd notes.append 81.628 499.968 872.065 879.829 91.105 325.264 593.469 646.600

ssd post-load RSS / peak RSS (MiB): before 102.371 / 102.371; after 91.383 / 91.383.

hdd post-load RSS / peak RSS (MiB): before 97.363 / 97.363; after 103.289 / 103.289.

This mixed load is closed loop: slow writes reduce subsequent read traffic. Compare each operation within a disk rather than ranking disks from read latency alone.

Aggregate gated breaches, using the same percentile method as the harness:

before aggregated gated breaches (same percentile method as the harness):
ssd-10000: none
ssd-100000: none
hdd-10000: cold.startup: p95 690.480 > 500; p50 run spread 76.1%; open.api_get: p95 60.643 > 50; p50 run spread 25.1%; sync.save_ack: p95 56.526 > 40; p50 run spread 58.9%
hdd-100000: cold.startup: p95 2513.378 > 500; p50 run spread 220.6%; home.first_page: p95 136.916 > 100; p50 run spread 64.4%; open.api_get: p95 114.401 > 50; p50 run spread 40.5%; shutdown: p95 3082.647 > 1000; p50 run spread 58.7%; sync.save_ack: p95 82.811 > 40; p50 run spread 27.9%
final aggregated gated breaches (same percentile method as the harness):
ssd-10000: none
ssd-100000: none
hdd-10000: cold.startup: p95 623.636 > 500; p50 run spread 65.6%; open.api_get: p95 60.436 > 50; p50 run spread 10.4%; sync.save_ack: p95 49.491 > 40; p50 run spread 17.7%
hdd-100000: cold.startup: p95 1053.712 > 500; p50 run spread 120.2%; open.api_get: p95 89.825 > 50; p50 run spread 16.7%; shutdown: p95 3331.564 > 1000; p50 run spread 144.9%; sync.save_ack: p95 58.937 > 40; p50 run spread 22.0%
Search document construction copied every complete note and allocated new strings for lines whose text did not change. Tag scanning also parsed code, links and addresses on lines that could not contain a tag. Borrow unchanged search text until the document is joined, allocate only when removing markup, and skip tag parsing on lines without hashes after fence tracking. Two commits, with regression tests for borrowed text, Unicode, links, replacement order and code fences. Durability, isolation and read concurrency are unchanged. Five interleaved direct benchmarks used the same executable and fixture text, with identical output. Each measurement ran 30 passes over 1,535,671 bytes and 17,588 lines: | Work | Before median ms | After median ms | Reduction | |---|---:|---:|---:| | Clean and join search text | 319.417 | 115.279 | 63.9% | | Scan inline tags | 348.028 | 142.184 | 59.1% | These measure the affected functions. Whole-server measurements and concurrent request tails follow below. CPU profiles of cold indexing, search catch-up and journal catch-up are summarized in [issue #512](https://git.kayg.org/kayg/calternal/issues/512). Validation at be555c1dfc1f1da75ac8bb9c74b3cadf3fcf793b: - `cargo fmt --all --check`: exit 0. - `cargo clippy --workspace --all-targets -- -D warnings`: exit 0. - `cargo test --workspace`: exit 0, including isolation tests. Full gate output is recorded on the issue. Related to #512. The performance gate remains red on HDD. The tables retain every primary run and the observed maxima, including regressions. Budgets are unchanged. Whole-server timing varies substantially; the unchanged baseline validation run also became faster, so the direct benchmarks give the clearest evidence for each code change. | 100k cold indexing | Before median s | After median s | Before max s | After max s | |---|---:|---:|---:|---:| | ssd | 33.190 | 22.963 | 39.207 | 32.186 | | hdd | 49.445 | 42.414 | 56.417 | 53.068 | Some primary maxima worsened: - ssd-10000 sync.save_ack: 92.444 → 348.306 ms. - hdd-100000 sync.save_ack: 1218.771 → 1995.020 ms. - hdd-100000 shutdown: 3082.647 → 3331.564 ms. Concurrent tails also regressed in this single probe per phase and disk. SSD durable-append p99 rose from 155.103 to 302.955 ms; HDD search p99 rose from 1113.187 to 1851.468 ms. The PR is work in progress because these results do not establish an overall tail-latency improvement. The CPU changes and their tests are ready for review; further tail analysis is needed before merging. Twenty interleaved startup checks per binary gave RSS medians 13.672 → 13.723 MiB, a 0.051 MiB difference within the observed process spread. Startup p95 was 13.871 → 14.207 MiB and maximum 14.031 → 14.699 MiB. HDD/100k RSS after use, primary idle sample and full validation repeat: before 37.938, 47.199 MiB. These samples quantify process-to-process variation in memory after use. HDD/100k RSS after use, primary idle sample and full validation repeat: final 43.309, 47.211 MiB. These samples quantify process-to-process variation in memory after use. | Extra HDD/100k run | Cold index s | Save ack p95 ms | Save ack max ms | Idle RSS MiB | Peak RSS MiB | |---|---:|---:|---:|---:|---:| | before | 43.431 | 66.096 | 211.471 | 47.199 | 95.090 | | final | 39.059 | 62.093 | 382.697 | 47.211 | 92.934 | The additional validation pair ran the entire cold and warm workload, including typing and 300 seconds idle. Its complete table is on [issue #512](https://git.kayg.org/kayg/calternal/issues/512). These two runs are separate from the primary five-run table. Before/after server performance. Base 312bc892ae17f513f141cd8d09abb62ed864c6d5; head be555c1dfc1f1da75ac8bb9c74b3cadf3fcf793b. Same release toolchain, fixture seed and harness; dropped page cache before each start. Five interleaved runs per cell. Latency distributions use the median of per-run p50/p95, and max is the largest sample. Single-value metrics pool the runs. Idle measurements use one 300-second sample per cell after rooms settle. Both binaries were built with matching Cargo feature unification: `cargo build --release --locked -j4 -p calternal -p calternal-perf -p calternal-fixtures`. Each invocation resets from the pristine seed-1 fixture and includes one cold and one warm run; pass 5 also includes the stock 60-second typing load and extra quiet checks. Memory follows the harness MB label (MiB). ssd-10000 | Metric | Unit | Before p50 | Before p95 | Before max | After p50 | After p95 | After max | |---|---|---:|---:|---:|---:|---:|---:| | startup.healthz | ms | 32.293 | 46.417 | 46.417 | 30.235 | 37.659 | 37.659 | | home.first_page | ms | 12.030 | 26.745 | 26.745 | 11.365 | 14.541 | 14.541 | | home.page | ms | 1.379 | 2.096 | 7.982 | 1.405 | 1.693 | 4.276 | | home.page_title | ms | 1.238 | 3.118 | 5.634 | 1.150 | 1.705 | 4.975 | | home.page_tag | ms | 9.242 | 12.737 | 182.386 | 8.669 | 11.246 | 87.584 | | home.page_next | ms | 1.417 | 2.107 | 4.798 | 1.281 | 1.773 | 3.552 | | home.page_big | ms | 28.815 | 34.508 | 50.323 | 28.390 | 29.420 | 34.461 | | search.query | ms | 3.554 | 7.285 | 65.865 | 3.678 | 10.086 | 75.070 | | search.first_query | ms | 27.849 | 43.668 | 43.668 | 27.526 | 79.223 | 79.223 | | search.catch_up | s | 4.012 | 6.016 | 6.027 | 4.026 | 4.026 | 6.034 | | open.note | ms | 1.254 | 2.620 | 8.883 | 1.073 | 2.803 | 5.073 | | open.api_get | ms | 7.650 | 9.381 | 14.894 | 7.199 | 8.571 | 14.859 | | sync.save_ack | ms | 5.985 | 8.562 | 92.444 | 5.903 | 10.210 | 348.306 | | storm.save_ack | ms | 10.403 | 23.176 | 113.846 | 8.138 | 22.508 | 141.278 | | journal.catch_up | ms | 62.046 | 94.674 | 94.674 | 61.780 | 65.217 | 65.217 | | cold.index | s | 2.056 | 2.267 | 2.267 | 1.498 | 1.677 | 1.677 | | cold.startup | ms | 97.708 | 134.832 | 134.832 | 70.602 | 79.012 | 79.012 | | open.long_note | ms | 9.570 | 12.641 | 12.641 | 9.955 | 12.699 | 12.699 | | journal.page | ms | 5.588 | 7.392 | 14.115 | 5.398 | 6.435 | 6.830 | | shutdown | ms | 66.128 | 67.453 | 67.453 | 45.918 | 67.229 | 67.229 | | storm.edits_per_s | edits/s | 2217.073 | 3214.040 | 3214.040 | 2501.178 | 3980.039 | 3980.039 | | storm.disk_written | MB | 50.520 | 54.227 | 54.227 | 50.901 | 52.302 | 52.302 | | trash.delete | ms | 13.784 | 23.300 | 78.310 | 12.287 | 20.364 | 73.600 | | trash.restore | ms | 16.036 | 26.539 | 174.210 | 15.090 | 27.179 | 60.917 | | trash.forget | ms | 7.604 | 12.991 | 47.241 | 7.399 | 11.601 | 29.835 | | mem.idle | MB | 13.199 | 13.414 | 13.414 | 13.230 | 14.031 | 14.031 | | mem.peak | MB | 38.957 | 40.543 | 40.543 | 37.152 | 40.902 | 40.902 | | cold.mem_peak | MB | 50.082 | 50.875 | 50.875 | 51.672 | 51.711 | 51.711 | | idle.rss | MB | 26.535 | 26.535 | 26.535 | 26.758 | 26.758 | 26.758 | | idle.cpu_pct | % | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | | idle.wakeups_per_s | /s | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | ssd-100000 | Metric | Unit | Before p50 | Before p95 | Before max | After p50 | After p95 | After max | |---|---|---:|---:|---:|---:|---:|---:| | startup.healthz | ms | 47.894 | 89.280 | 89.280 | 37.325 | 49.970 | 49.970 | | home.first_page | ms | 55.071 | 99.624 | 99.624 | 18.709 | 41.162 | 41.162 | | home.page | ms | 2.361 | 3.694 | 8.512 | 2.088 | 2.767 | 5.372 | | home.page_title | ms | 2.588 | 7.041 | 11.183 | 1.893 | 4.152 | 55.433 | | home.page_tag | ms | 6.699 | 11.686 | 299.157 | 5.668 | 7.760 | 176.247 | | home.page_next | ms | 2.708 | 5.130 | 14.808 | 2.125 | 3.301 | 11.855 | | home.page_big | ms | 411.416 | 643.016 | 1049.351 | 133.137 | 220.357 | 914.898 | | search.query | ms | 5.773 | 13.903 | 760.582 | 4.587 | 9.729 | 801.232 | | search.first_query | ms | 61.681 | 77.668 | 77.668 | 41.368 | 72.185 | 72.185 | | search.catch_up | s | 92.739 | 98.791 | 104.979 | 68.355 | 68.355 | 92.909 | | open.note | ms | 2.274 | 5.296 | 34.235 | 1.368 | 4.234 | 12.606 | | open.api_get | ms | 12.275 | 16.739 | 32.028 | 9.572 | 19.706 | 340.469 | | sync.save_ack | ms | 9.602 | 35.688 | 209.367 | 8.629 | 14.916 | 115.846 | | storm.save_ack | ms | 17.587 | 44.747 | 937.979 | 13.730 | 31.191 | 736.756 | | journal.catch_up | ms | 1126.472 | 1309.691 | 1309.691 | 771.246 | 1541.604 | 1541.604 | | cold.index | s | 33.190 | 39.207 | 39.207 | 22.963 | 32.186 | 32.186 | | cold.startup | ms | 169.768 | 213.792 | 213.792 | 85.390 | 110.088 | 110.088 | | open.long_note | ms | 13.182 | 23.185 | 23.185 | 13.927 | 17.811 | 17.811 | | journal.page | ms | 9.220 | 14.758 | 50.788 | 6.558 | 10.514 | 93.871 | | shutdown | ms | 88.419 | 93.827 | 93.827 | 65.785 | 115.179 | 115.179 | | storm.edits_per_s | edits/s | 1281.929 | 2471.487 | 2471.487 | 1555.176 | 4283.820 | 4283.820 | | storm.disk_written | MB | 49.512 | 56.533 | 56.533 | 44.257 | 44.376 | 44.376 | | trash.delete | ms | 18.098 | 28.986 | 287.089 | 19.575 | 33.131 | 68.809 | | trash.restore | ms | 23.266 | 39.272 | 301.490 | 23.498 | 36.498 | 298.570 | | trash.forget | ms | 11.330 | 24.394 | 77.260 | 11.017 | 19.061 | 57.756 | | mem.idle | MB | 13.109 | 13.328 | 13.328 | 13.289 | 13.352 | 13.352 | | mem.peak | MB | 96.254 | 99.703 | 99.703 | 91.637 | 96.473 | 96.473 | | cold.mem_peak | MB | 119.871 | 121.469 | 121.469 | 117.426 | 121.172 | 121.172 | | idle.rss | MB | 41.789 | 41.789 | 41.789 | 39.508 | 39.508 | 39.508 | | idle.cpu_pct | % | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | | idle.wakeups_per_s | /s | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | hdd-10000 | Metric | Unit | Before p50 | Before p95 | Before max | After p50 | After p95 | After max | |---|---|---:|---:|---:|---:|---:|---:| | startup.healthz | ms | 41.033 | 178.483 | 178.483 | 33.838 | 61.189 | 61.189 | | home.first_page | ms | 31.201 | 43.028 | 43.028 | 18.645 | 68.831 | 68.831 | | home.page | ms | 1.816 | 3.533 | 18.805 | 1.364 | 1.815 | 7.108 | | home.page_title | ms | 1.517 | 2.152 | 14.762 | 1.229 | 2.578 | 12.484 | | home.page_tag | ms | 11.141 | 16.243 | 542.228 | 8.503 | 17.351 | 243.223 | | home.page_next | ms | 1.913 | 3.674 | 23.067 | 1.496 | 1.759 | 24.673 | | home.page_big | ms | 33.585 | 46.160 | 131.010 | 33.118 | 36.892 | 114.975 | | search.query | ms | 3.738 | 8.538 | 88.265 | 4.871 | 8.764 | 90.615 | | search.first_query | ms | 184.362 | 199.424 | 199.424 | 153.797 | 266.908 | 266.908 | | search.catch_up | s | 10.035 | 10.035 | 12.078 | 8.024 | 8.034 | 10.047 | | open.note | ms | 1.351 | 3.433 | 25.293 | 1.310 | 4.802 | 32.175 | | open.api_get | ms | 35.976 | 60.643 | 96.257 | 37.224 | 60.436 | 236.259 | | sync.save_ack | ms | 15.036 | 56.526 | 1516.069 | 14.427 | 49.491 | 1321.199 | | storm.save_ack | ms | 47.024 | 134.363 | 741.346 | 42.665 | 114.206 | 496.167 | | journal.catch_up | ms | 66.749 | 194.828 | 194.828 | 66.972 | 135.687 | 135.687 | | cold.index | s | 4.100 | 5.092 | 5.092 | 3.860 | 4.078 | 4.078 | | cold.startup | ms | 432.376 | 690.480 | 690.480 | 466.030 | 623.636 | 623.636 | | open.long_note | ms | 18.735 | 36.723 | 36.723 | 17.182 | 51.035 | 51.035 | | journal.page | ms | 5.897 | 7.721 | 31.740 | 6.140 | 7.196 | 16.690 | | shutdown | ms | 89.005 | 228.973 | 228.973 | 193.058 | 263.189 | 263.189 | | storm.edits_per_s | edits/s | 497.648 | 633.855 | 633.855 | 585.738 | 611.050 | 611.050 | | storm.disk_written | MB | 81.326 | 91.574 | 91.574 | 71.856 | 80.327 | 80.327 | | trash.delete | ms | 52.399 | 115.189 | 262.883 | 56.079 | 91.103 | 245.028 | | trash.restore | ms | 72.064 | 179.774 | 494.562 | 73.554 | 151.461 | 466.337 | | trash.forget | ms | 36.029 | 85.912 | 318.037 | 34.219 | 61.051 | 204.856 | | mem.idle | MB | 13.242 | 13.336 | 13.336 | 13.199 | 13.332 | 13.332 | | mem.peak | MB | 39.695 | 40.633 | 40.633 | 39.121 | 41.074 | 41.074 | | cold.mem_peak | MB | 50.340 | 53.176 | 53.176 | 50.715 | 51.609 | 51.609 | | idle.rss | MB | 26.441 | 26.441 | 26.441 | 26.727 | 26.727 | 26.727 | | idle.cpu_pct | % | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | | idle.wakeups_per_s | /s | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | hdd-100000 | Metric | Unit | Before p50 | Before p95 | Before max | After p50 | After p95 | After max | |---|---|---:|---:|---:|---:|---:|---:| | startup.healthz | ms | 104.669 | 157.592 | 157.592 | 66.592 | 85.778 | 85.778 | | home.first_page | ms | 120.894 | 136.916 | 136.916 | 67.718 | 86.076 | 86.076 | | home.page | ms | 2.490 | 10.606 | 19.555 | 1.619 | 2.174 | 5.770 | | home.page_title | ms | 2.074 | 5.776 | 26.489 | 1.327 | 1.868 | 12.102 | | home.page_tag | ms | 6.993 | 14.752 | 252.036 | 4.923 | 6.160 | 183.359 | | home.page_next | ms | 2.784 | 5.371 | 18.184 | 1.604 | 2.346 | 12.684 | | home.page_big | ms | 453.527 | 668.061 | 1358.472 | 193.059 | 283.536 | 1016.940 | | search.query | ms | 5.676 | 12.601 | 730.219 | 4.606 | 12.972 | 707.446 | | search.first_query | ms | 338.579 | 434.254 | 434.254 | 402.598 | 406.983 | 406.983 | | search.catch_up | s | 132.778 | 142.799 | 144.976 | 118.419 | 118.419 | 136.895 | | open.note | ms | 1.308 | 6.811 | 151.554 | 1.117 | 2.667 | 10.359 | | open.api_get | ms | 41.418 | 114.401 | 349.620 | 40.827 | 89.825 | 298.186 | | sync.save_ack | ms | 14.212 | 82.811 | 1218.771 | 14.824 | 58.937 | 1995.020 | | storm.save_ack | ms | 44.949 | 129.829 | 418.613 | 38.313 | 101.175 | 381.548 | | journal.catch_up | ms | 901.677 | 1127.429 | 1127.429 | 639.896 | 1217.029 | 1217.029 | | cold.index | s | 49.445 | 56.417 | 56.417 | 42.414 | 53.068 | 53.068 | | cold.startup | ms | 862.585 | 2513.378 | 2513.378 | 664.830 | 1053.712 | 1053.712 | | open.long_note | ms | 11.796 | 26.785 | 26.785 | 9.764 | 44.867 | 44.867 | | journal.page | ms | 7.448 | 13.184 | 32.491 | 5.727 | 8.130 | 29.115 | | shutdown | ms | 2414.701 | 3082.647 | 3082.647 | 2179.835 | 3331.564 | 3331.564 | | storm.edits_per_s | edits/s | 545.469 | 622.575 | 622.575 | 661.722 | 705.388 | 705.388 | | storm.disk_written | MB | 65.839 | 67.592 | 67.592 | 57.516 | 63.365 | 63.365 | | trash.delete | ms | 75.191 | 128.685 | 239.133 | 74.444 | 149.597 | 403.720 | | trash.restore | ms | 87.383 | 171.128 | 307.504 | 90.418 | 164.083 | 337.787 | | trash.forget | ms | 49.332 | 101.264 | 148.389 | 44.311 | 101.103 | 174.414 | | mem.idle | MB | 13.289 | 13.316 | 13.316 | 13.223 | 13.438 | 13.438 | | mem.peak | MB | 93.508 | 95.859 | 95.859 | 91.848 | 94.980 | 94.980 | | cold.mem_peak | MB | 131.770 | 136.805 | 136.805 | 129.691 | 136.598 | 136.598 | | idle.rss | MB | 37.938 | 37.938 | 37.938 | 43.309 | 43.309 | 43.309 | | idle.cpu_pct | % | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | | idle.wakeups_per_s | /s | 0.003 | 0.003 | 0.003 | 0.000 | 0.000 | 0.000 | 20 concurrent clients on an indexed 100k-note Home, after note and search indexes settle. No think time. Each client runs 50 mixed rounds with page, note, search and durable append operations. Each operation has 1,000 samples per phase and disk; all 1,000 appended markers were verified exactly once. Latencies are milliseconds. | Disk | Operation | Before p50 | Before p95 | Before p99 | Before max | After p50 | After p95 | After p99 | After max | |---|---|---:|---:|---:|---:|---:|---:|---:|---:| | ssd | notes.list | 23.954 | 62.956 | 81.148 | 106.318 | 12.379 | 30.373 | 44.321 | 73.946 | | ssd | notes.get | 10.323 | 36.480 | 74.479 | 140.765 | 5.324 | 18.200 | 47.674 | 104.773 | | ssd | search.query | 76.503 | 184.521 | 245.662 | 286.207 | 56.592 | 121.839 | 171.461 | 275.802 | | ssd | notes.append | 55.499 | 112.510 | 155.103 | 236.001 | 31.102 | 202.308 | 302.955 | 363.992 | | hdd | notes.list | 7.750 | 17.816 | 22.687 | 25.815 | 12.669 | 29.402 | 36.163 | 50.615 | | hdd | notes.get | 3.937 | 13.103 | 566.419 | 698.729 | 6.451 | 20.525 | 100.471 | 581.054 | | hdd | search.query | 32.810 | 322.049 | 1113.187 | 1707.499 | 54.615 | 244.054 | 1851.468 | 1972.262 | | hdd | notes.append | 81.628 | 499.968 | 872.065 | 879.829 | 91.105 | 325.264 | 593.469 | 646.600 | ssd post-load RSS / peak RSS (MiB): before 102.371 / 102.371; after 91.383 / 91.383. hdd post-load RSS / peak RSS (MiB): before 97.363 / 97.363; after 103.289 / 103.289. This mixed load is closed loop: slow writes reduce subsequent read traffic. Compare each operation within a disk rather than ranking disks from read latency alone. Aggregate gated breaches, using the same percentile method as the harness: ```text before aggregated gated breaches (same percentile method as the harness): ssd-10000: none ssd-100000: none hdd-10000: cold.startup: p95 690.480 > 500; p50 run spread 76.1%; open.api_get: p95 60.643 > 50; p50 run spread 25.1%; sync.save_ack: p95 56.526 > 40; p50 run spread 58.9% hdd-100000: cold.startup: p95 2513.378 > 500; p50 run spread 220.6%; home.first_page: p95 136.916 > 100; p50 run spread 64.4%; open.api_get: p95 114.401 > 50; p50 run spread 40.5%; shutdown: p95 3082.647 > 1000; p50 run spread 58.7%; sync.save_ack: p95 82.811 > 40; p50 run spread 27.9% final aggregated gated breaches (same percentile method as the harness): ssd-10000: none ssd-100000: none hdd-10000: cold.startup: p95 623.636 > 500; p50 run spread 65.6%; open.api_get: p95 60.436 > 50; p50 run spread 10.4%; sync.save_ack: p95 49.491 > 40; p50 run spread 17.7% hdd-100000: cold.startup: p95 1053.712 > 500; p50 run spread 120.2%; open.api_get: p95 89.825 > 50; p50 run spread 16.7%; shutdown: p95 3331.564 > 1000; p50 run spread 144.9%; sync.save_ack: p95 58.937 > 40; p50 run spread 22.0% ```
perf(notes): skip tag parsing on lines without hashes
Some checks failed
CI / server (pull_request) Successful in 28s
Details
CI / rust (pull_request) Successful in 2m11s
Details
CI / web (pull_request) Failing after 50m46s
Details
be555c1dfc
kayg commented 2026-10-17 20:10:15 +00:00
Author
Owner
Copy link

Landed in train y, #565.

Landed in train y, #565.
kayg closed this pull request 2026-10-17 20:10:15 +00:00
Some checks failed
CI / server (pull_request) Successful in 28s
Details
CI / rust (pull_request) Successful in 2m11s
Details
CI / web (pull_request) Failing after 50m46s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!514
No description provided.