kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

Files slice 1: Home identity, joined index, and reconcile #1010

Open
kayg wants to merge 0 commits from feat/files-s1-identity into dev
pull from: feat/files-s1-identity
merge into: kayg:dev
Conversation 10 Commits - Files changed -
kayg commented 2026-11-07 18:24:22 +00:00
Owner
Copy link

Status (2026-11-10, head 5517ce595)

  • Design: one schema transaction; folder-stamp check at restart plus one full walk once the Home is quiet; Files walks only while quiet; Files-only side tables, so Notes rows keep #970's bytes. Details in the simpler design and round 4 / round 5 notes.
  • Gates: fmt, clippy -D warnings, workspace tests and the 1M listing gate pass.
  • Perf: #1010-share flat (ABAB ×5 with #970, head-copy and dev-copy arms). Head vs dev is red only where #970 alone is red, so this waits for #970's cold fix and is not READY.

Plan

Implement Files slice 1 as a backend Home identity layer over the two real Home roots selected by amended ADR 0062. Each root keeps the same relative folders and its own files; the resolver joins their listings and refuses to serve or reconcile when either expected root is missing, mismatched, or unmounted. No overlay, symlink, hard-link, or content-addressed storage is involved.

Reuse the accepted #641 folder marker contract: .calternal-id carries a canonical UUIDv7 through folder renames and moves. Marker collisions are resolved deterministically and recorded; an ambiguous path keeps both real entries. Markdown IDs remain in calternal-id frontmatter. Other file IDs use an app-managed row plus a portable hidden marker, with no content-hash-only identity inference. Reuse the Notes event/index and watcher behavior through shared APIs where it fits; do not create a second copy of those implementations.

The per-user index is rebuildable from durable markers and stores both roots in one logical namespace. First-page listing uses a composite B-tree key and keyset cursor: O(log N + page size). Normal ID and parent lookup is O(log N); a single-row update is O(log N). Reconciliation is O(N) only in background batches with bounded working memory. No request sorts a directory or loads the whole Home.

Failure table

Failure Effect Detection Recovery Test
Either root is missing or has the wrong installation identity The joined Home is incomplete Validate both mounted roots and role/installation markers before serving Fail closed without creating a root or deleting index rows; remount and reconcile Hide each root in turn, then restore it and compare IDs
Watch queue overflows or a watch cannot be installed Some outside changes are not delivered Overflow/error event or watch-limit result Retire stale watch state and enqueue paged reconciliation Inject overflow and constrained watch installation; verify convergence
A folder or file marker is copied, malformed, or duplicated Two entries may claim one ID Compare marker ownership with indexed identity and path Preserve the established ID; assign and durably record a fresh ID for the copy; keep ambiguous bytes Copy, corrupt, and delete markers, including after restart
Startup or reconciliation stops mid-page Index may lag the real trees Persisted scan generation/cursor and root validation Restart the page walk idempotently; publish only committed rows Restart between pages, then compare the joined tree and IDs

Required gates

Gate Required evidence Status
Identity Rename, move, copy, restart, missing-root, and watcher-overflow tests preserve the specified IDs Pending
Listing scale First page on a 1M-entry metadata fixture; p50/p99 plus peak memory Pending
Notes hot paths ABAB ×5 at one perf epoch for save, startup, and index, with both perf locks and a dev-vs-dev control Pending
Build and checks Workspace cargo test, clippy, and web checks Pending

The account envelope stays 50 million files and 1 PiB logical size. The first-page target remains under 100 ms. Scale fixtures contain metadata only and sparse logical sizes. Results and benchmark artifacts will be posted in PR comments per ADR 0003.

<!-- files-1010-status --> ## Status (2026-11-10, head `5517ce595`) - **Design:** one schema transaction; folder-stamp check at restart plus one full walk once the Home is quiet; Files walks only while quiet; Files-only side tables, so Notes rows keep #970's bytes. Details in [the simpler design](https://git.kayg.org/kayg/calternal/pulls/1010#issuecomment-89027) and [round 4](https://git.kayg.org/kayg/calternal/pulls/1010#issuecomment-89287) / [round 5](https://git.kayg.org/kayg/calternal/pulls/1010#issuecomment-89423) notes. - **Gates:** fmt, clippy `-D warnings`, workspace tests and the 1M listing gate pass. - **Perf:** [#1010-share flat](https://git.kayg.org/kayg/calternal/pulls/1010#issuecomment-89523) (ABAB ×5 with #970, head-copy and dev-copy arms). Head vs dev is red only where #970 alone is red, so this waits for #970's cold fix and is not READY. <!-- files-1010-status --> ## Plan Implement Files slice 1 as a backend Home identity layer over the two real Home roots selected by amended ADR 0062. Each root keeps the same relative folders and its own files; the resolver joins their listings and refuses to serve or reconcile when either expected root is missing, mismatched, or unmounted. No overlay, symlink, hard-link, or content-addressed storage is involved. Reuse the accepted #641 folder marker contract: `.calternal-id` carries a canonical UUIDv7 through folder renames and moves. Marker collisions are resolved deterministically and recorded; an ambiguous path keeps both real entries. Markdown IDs remain in `calternal-id` frontmatter. Other file IDs use an app-managed row plus a portable hidden marker, with no content-hash-only identity inference. Reuse the Notes event/index and watcher behavior through shared APIs where it fits; do not create a second copy of those implementations. The per-user index is rebuildable from durable markers and stores both roots in one logical namespace. First-page listing uses a composite B-tree key and keyset cursor: O(log N + page size). Normal ID and parent lookup is O(log N); a single-row update is O(log N). Reconciliation is O(N) only in background batches with bounded working memory. No request sorts a directory or loads the whole Home. ## Failure table | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | Either root is missing or has the wrong installation identity | The joined Home is incomplete | Validate both mounted roots and role/installation markers before serving | Fail closed without creating a root or deleting index rows; remount and reconcile | Hide each root in turn, then restore it and compare IDs | | Watch queue overflows or a watch cannot be installed | Some outside changes are not delivered | Overflow/error event or watch-limit result | Retire stale watch state and enqueue paged reconciliation | Inject overflow and constrained watch installation; verify convergence | | A folder or file marker is copied, malformed, or duplicated | Two entries may claim one ID | Compare marker ownership with indexed identity and path | Preserve the established ID; assign and durably record a fresh ID for the copy; keep ambiguous bytes | Copy, corrupt, and delete markers, including after restart | | Startup or reconciliation stops mid-page | Index may lag the real trees | Persisted scan generation/cursor and root validation | Restart the page walk idempotently; publish only committed rows | Restart between pages, then compare the joined tree and IDs | ## Required gates | Gate | Required evidence | Status | |---|---|---| | Identity | Rename, move, copy, restart, missing-root, and watcher-overflow tests preserve the specified IDs | Pending | | Listing scale | First page on a 1M-entry metadata fixture; p50/p99 plus peak memory | Pending | | Notes hot paths | ABAB ×5 at one perf epoch for save, startup, and index, with both perf locks and a dev-vs-dev control | Pending | | Build and checks | Workspace `cargo test`, clippy, and web checks | Pending | The account envelope stays 50 million files and 1 PiB logical size. The first-page target remains under 100 ms. Scale fixtures contain metadata only and sparse logical sizes. Results and benchmark artifacts will be posted in PR comments per ADR 0003.
kayg commented 2026-11-07 20:21:42 +00:00
Author
Owner
Copy link

Namespace overlap and implementation path

Reviewing #970 showed that it already owns the shared Files current namespace and the three-root storage contract. The current #1010 draft added a separate HomeIndex under Notes metadata and modeled the SSD Home plus state as its two roots, so that draft duplicates the index and does not match ADR 0062.

I will make this branch build on #970's calternal-files::Namespace history and remove the separate HomeIndex. The Files resolver will validate SSD Home, HDD Home, and SSD app state, store its index under app state, and reconcile the two Home trees through the same current namespace. #1010 remains targeted at dev; until #970 lands, its commits are ancestors of this branch, so the shared implementation has one source in Git history and the dependent diff can shrink after #970 merges.

The current gate table remains pending until this combined implementation has exact-head results. I will keep the existing Notes watcher/index paths shared and report any touched Notes hot-path deltas with the required locked ABAB comparison.

## Namespace overlap and implementation path Reviewing #970 showed that it already owns the shared Files current namespace and the three-root storage contract. The current #1010 draft added a separate `HomeIndex` under Notes metadata and modeled the SSD Home plus state as its two roots, so that draft duplicates the index and does not match ADR 0062. I will make this branch build on #970's `calternal-files::Namespace` history and remove the separate `HomeIndex`. The Files resolver will validate SSD Home, HDD Home, and SSD app state, store its index under app state, and reconcile the two Home trees through the same current namespace. #1010 remains targeted at `dev`; until #970 lands, its commits are ancestors of this branch, so the shared implementation has one source in Git history and the dependent diff can shrink after #970 merges. The current gate table remains pending until this combined implementation has exact-head results. I will keep the existing Notes watcher/index paths shared and report any touched Notes hot-path deltas with the required locked ABAB comparison.
kayg commented 2026-11-09 06:09:27 +00:00
Author
Owner
Copy link

Takeover status (head 46faabb28, merged with dev d4b32af94; Rust identical to d00b48b6e)

Previous agent stopped on a usage cap. State found: branch had no conflicts with #970 beyond one in crates/perf/src/scenarios/cold.rs (kept both functions).

Fixed a real startup bug (commit afb3d29b6): WatchEvents::recv could report the queue closed while a rescan was still pending (the last sender is dropped right after it sends the startup rescan when no watcher is installed). The startup scan then never ran and scan.wait hung. This was the cause of shutdown_settlement_state_machine failing ~3 of 4 runs on this branch (0 of 3 on dev). Test rescan_sent_just_before_the_inbox_drops_is_delivered (2,000 iterations) added. After the fix: 6 of 6 runs pass.

New test #1015 waits on: legacy_attachment_ids_survive_every_retry_boundary (crates/notes/src/home_files/tests.rs): files written before identity existed get one ID each on first scan, kept after a repeated scan, a restart and a rename.

Gate Command (on calternal-dev-01-compute02, via ~/bin/slot) Result
fmt cargo fmt --all --check pass
clippy cargo clippy --workspace --all-targets --locked -j4 -- -D warnings pass
tests cargo test --workspace --locked -j4 --no-fail-fast all pass (131 test binaries ok)
settlement flake shutdown_settlement_state_machine x6 on fixed head 6/6 pass

Leftover predecessor unit files-s1-full-test-3-1af2 was stopped (no files-s1* units remain on the perf VMs). One load-only timeout in calternal-contacts-actions crash_after_trash_move_... did not reproduce in 5 isolated runs nor in the next full run (flaky rule: no block).

Perf: fresh release binaries (head d00b48b6e, dev d4b32af94, sha256 checked) staged on perf-01; queued behind Mail/p928/Photos on perf-01 with a blocking perf-lock, then perf-02. Plan: 1M listing gate, ABAB x5 (dev legacy layout vs head joined layout) with a dev-copy control each cycle, inside both locks. Results follow in a comment.

## Takeover status (head `46faabb28`, merged with dev `d4b32af94`; Rust identical to `d00b48b6e`) Previous agent stopped on a usage cap. State found: branch had no conflicts with #970 beyond one in `crates/perf/src/scenarios/cold.rs` (kept both functions). **Fixed a real startup bug** (commit `afb3d29b6`): `WatchEvents::recv` could report the queue closed while a rescan was still pending (the last sender is dropped right after it sends the startup rescan when no watcher is installed). The startup scan then never ran and `scan.wait` hung. This was the cause of `shutdown_settlement_state_machine` failing ~3 of 4 runs on this branch (0 of 3 on dev). Test `rescan_sent_just_before_the_inbox_drops_is_delivered` (2,000 iterations) added. After the fix: 6 of 6 runs pass. **New test #1015 waits on**: `legacy_attachment_ids_survive_every_retry_boundary` (`crates/notes/src/home_files/tests.rs`): files written before identity existed get one ID each on first scan, kept after a repeated scan, a restart and a rename. | Gate | Command (on `calternal-dev-01-compute02`, via `~/bin/slot`) | Result | |---|---|---| | fmt | `cargo fmt --all --check` | pass | | clippy | `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` | pass | | tests | `cargo test --workspace --locked -j4 --no-fail-fast` | all pass (131 test binaries ok) | | settlement flake | `shutdown_settlement_state_machine` x6 on fixed head | 6/6 pass | Leftover predecessor unit `files-s1-full-test-3-1af2` was stopped (no `files-s1*` units remain on the perf VMs). One load-only timeout in `calternal-contacts-actions` `crash_after_trash_move_...` did not reproduce in 5 isolated runs nor in the next full run (flaky rule: no block). Perf: fresh release binaries (head `d00b48b6e`, dev `d4b32af94`, sha256 checked) staged on perf-01; queued behind Mail/p928/Photos on perf-01 with a blocking `perf-lock`, then perf-02. Plan: 1M listing gate, ABAB x5 (dev legacy layout vs head joined layout) with a dev-copy control each cycle, inside both locks. Results follow in a comment.
kayg commented 2026-11-09 07:56:06 +00:00
Author
Owner
Copy link

Perf: ABAB x5 + control, inside both perf locks (perf-01 then perf-02), session 2026-11-09 06:54–07:55Z

Head d00b48b6e (Rust identical to PR head 46faabb28) vs dev d4b32af94, release builds, sha256 checked. Dev runs the legacy single-root layout, head runs the joined three-root layout (SSD Home, HDD Home, app state). Per cycle: A1 dev, B1 head, A2 dev, B2 head, then a dev-copy control (same dev binary, copied). 5 cycles, same boot (7a65f072), sizes 1k/10k/100k, --runs 1 --cold-runs 1 --hot-only --gate. Medians of per-arm p50/p95 (n = 10 dev, 10 head, 5 control). Locks released at 07:55Z.

Control (dev vs dev-copy) is within about 1% on the cold metrics, so the cold deltas below are not noise.

Scale gate (1M entries, files-listing-bench, page 50)

cold_first_us=104 p50_us=59 p99_us=99 active_map_bytes=53085308 peak_rss_kib=56460. The plan's gate is a first page under 100 ms: pass (about 0.1 ms).

Regressions vs dev (they block merge under the no-regression rule)

Metric 1k 10k 100k
cold.first_page p50 +30% (435 → 568 ms) +115% (333 → 714 ms) +168% (2.72 → 7.28 s)
cold.index +25% +23% +19% (22.9 → 27.1 s)
cold.notes_per_s -20% -19% -16%
cold.disk_written +3% +6% +17% (1.92 → 2.24 GB)
startup.healthz +43% +43% +25% (7.3 → 9.2 ms)
home.page_big p50 -1% +2% +48% (19.0 → 28.2 ms), p95 +57%
open.api_get p95 +11% +1% +345% (14.8 → 65.9 ms)
notes.append.first +6% +77% (104 → 183 ms; control also +150%, noisy) -79% (better)
notes.append p50 -2% -8% +9% (p95 +13%)
open.note -7% +4% +10% (p95 +18%)

cold.* runs once per arm, so these are medians of 10 arms; the control's median stays within about 1% of dev.

Improvements

home.first_page 1k -37%; notes.append.first 100k -79%; sync.save_ack p95 100k -41%; sync.round_trip 100k -19%; memory flat to -7%.

Flat (inside control noise)

home.page, home.page_next, home.page_tag, home.page_title, search.*, sync.durable_write, shutdown, open.api_get.warm, mem.*.

Likely cause (not verified)

The cold open now reads and writes the shared current namespace (joined-tree reconcile plus bounded 8,192-row commits) in addition to the Notes discovery dev does, so the first page waits behind extra index writes (+17% bytes written at 100k). startup.healthz +2 ms is the extra root validation and resolver open.

State

Not READY FOR MERGE under the no-perf-regression rule: the cold-path, home.page_big and open.api_get p95 deltas need a fix or kayg's explicit waiver. Everything else is green: fmt, clippy -D warnings, full workspace tests, scale gate, legacy-attachment ID test, settlement flake fixed.

Full table (all metrics) 
size metric unit | dev p50 | head p50 d% | ctl p50 d% | dev p95 | head p95 d% | n(dev/head/ctl)
1000 cold.disk_written MB | 18.813 | 19.419 +3.2% | 18.784 -0.2% | 18.813 | 19.419 +3.2% | 10 10 5
1000 cold.first_page ms | 435.473 | 567.589 +30.3% | 432.892 -0.6% | 435.473 | 567.589 +30.3% | 10 10 5
1000 cold.index s | 0.538 | 0.669 +24.5% | 0.535 -0.5% | 0.538 | 0.669 +24.5% | 10 10 5
1000 cold.mem_peak MB | 45.088 | 47.998 +6.5% | 46.273 +2.6% | 45.088 | 47.998 +6.5% | 10 10 5
1000 cold.notes_per_s notes/s | 1859.29 | 1493.762 -19.7% | 1868.279 +0.5% | 1859.29 | 1493.762 -19.7% | 10 10 5
1000 cold.startup ms | 20.349 | 22.157 +8.9% | 23.192 +14.0% | 20.349 | 22.157 +8.9% | 10 10 5
1000 home.first_page ms | 2.787 | 1.769 -36.5% | 2.126 -23.7% | 2.787 | 1.769 -36.5% | 10 10 5
1000 home.page ms | 0.678 | 0.727 +7.3% | 0.672 -0.9% | 1.222 | 1.408 +15.2% | 10 10 5
1000 home.page_big ms | 8.938 | 8.826 -1.2% | 8.897 -0.5% | 8.938 | 8.826 -1.2% | 10 10 5
1000 home.page_next ms | 0.714 | 0.719 +0.7% | 0.695 -2.7% | 0.803 | 0.792 -1.3% | 10 10 5
1000 home.page_tag ms | 1.247 | 1.272 +2.0% | 1.21 -3.0% | 1.446 | 1.494 +3.3% | 10 10 5
1000 home.page_title ms | 0.611 | 0.61 -0.2% | 0.57 -6.7% | 0.715 | 0.713 -0.3% | 10 10 5
1000 mem.after_first_page MB | 21.34 | 20.857 -2.3% | 21.668 +1.5% | 21.34 | 20.857 -2.3% | 10 10 5
1000 mem.after_opens MB | 34.705 | 35.404 +2.0% | 35.336 +1.8% | 34.705 | 35.404 +2.0% | 10 10 5
1000 mem.after_search MB | 47.57 | 48.371 +1.7% | 47.777 +0.4% | 47.57 | 48.371 +1.7% | 10 10 5
1000 mem.idle MB | 18.232 | 19.148 +5.0% | 18.77 +2.9% | 18.232 | 19.148 +5.0% | 10 10 5
1000 notes.append ms | 5.336 | 5.231 -2.0% | 5.277 -1.1% | 13.402 | 13.103 -2.2% | 10 10 5
1000 notes.append.first ms | 21.315 | 22.6 +6.0% | 24.109 +13.1% | 21.315 | 22.6 +6.0% | 10 10 5
1000 open.api_get ms | 7.488 | 7.515 +0.4% | 7.518 +0.4% | 13.772 | 15.316 +11.2% | 10 10 5
1000 open.api_get.warm ms | 0.162 | 0.164 +1.4% | 0.17 +5.3% | 0.209 | 0.207 -1.2% | 10 10 5
1000 open.long_note ms | 15.102 | 15.582 +3.2% | 15.884 +5.2% | 15.102 | 15.582 +3.2% | 10 10 5
1000 open.note ms | 0.701 | 0.651 -7.2% | 0.663 -5.5% | 2.266 | 2.121 -6.4% | 10 10 5
1000 search.catch_up s | 2.003 | 2.003 +0.0% | 2.003 +0.0% | 2.003 | 2.003 +0.0% | 10 10 5
1000 search.first_query ms | 27.772 | 31.144 +12.1% | 30.12 +8.5% | 27.772 | 31.144 +12.1% | 10 10 5
1000 search.query ms | 1.149 | 1.167 +1.5% | 1.161 +1.0% | 2.137 | 2.238 +4.7% | 10 10 5
1000 shutdown ms | 43.058 | 43.031 -0.1% | 42.175 -2.0% | 43.058 | 43.031 -0.1% | 10 10 5
1000 startup.healthz ms | 4.968 | 7.109 +43.1% | 4.899 -1.4% | 4.968 | 7.109 +43.1% | 10 10 5
1000 sync.durable_write ms | 24.98 | 24.951 -0.1% | 25.116 +0.5% | 33.142 | 31.43 -5.2% | 10 10 5
1000 sync.round_trip ms | 0.214 | 0.209 -2.6% | 0.208 -3.1% | 0.343 | 0.352 +2.6% | 10 10 5
1000 sync.save_ack ms | 3.696 | 3.717 +0.6% | 3.668 -0.7% | 5.504 | 5.71 +3.7% | 10 10 5
10000 cold.disk_written MB | 150.252 | 159.322 +6.0% | 150.569 +0.2% | 150.252 | 159.322 +6.0% | 10 10 5
10000 cold.first_page ms | 332.798 | 713.989 +114.5% | 325.348 -2.2% | 332.798 | 713.989 +114.5% | 10 10 5
10000 cold.index s | 1.697 | 2.086 +22.9% | 1.676 -1.2% | 1.697 | 2.086 +22.9% | 10 10 5
10000 cold.mem_peak MB | 143.535 | 145.127 +1.1% | 143.309 -0.2% | 143.535 | 145.127 +1.1% | 10 10 5
10000 cold.notes_per_s notes/s | 5894.999 | 4794.997 -18.7% | 5965.417 +1.2% | 5894.999 | 4794.997 -18.7% | 10 10 5
10000 cold.startup ms | 19.833 | 21.198 +6.9% | 18.787 -5.3% | 19.833 | 21.198 +6.9% | 10 10 5
10000 home.first_page ms | 2.36 | 2.351 -0.4% | 2.622 +11.1% | 2.36 | 2.351 -0.4% | 10 10 5
10000 home.page ms | 0.76 | 0.831 +9.3% | 0.816 +7.4% | 1.312 | 1.07 -18.5% | 10 10 5
10000 home.page_big ms | 16.733 | 17.087 +2.1% | 16.519 -1.3% | 17.715 | 19.529 +10.2% | 10 10 5
10000 home.page_next ms | 0.688 | 0.71 +3.2% | 0.7 +1.8% | 0.808 | 0.924 +14.4% | 10 10 5
10000 home.page_tag ms | 5.363 | 5.433 +1.3% | 5.382 +0.4% | 7.117 | 7.254 +1.9% | 10 10 5
10000 home.page_title ms | 0.644 | 0.69 +7.2% | 0.679 +5.6% | 0.773 | 1.272 +64.6% | 10 10 5
10000 mem.after_first_page MB | 21.277 | 21.504 +1.1% | 21.367 +0.4% | 21.277 | 21.504 +1.1% | 10 10 5
10000 mem.after_opens MB | 40.227 | 41.617 +3.5% | 40.938 +1.8% | 40.227 | 41.617 +3.5% | 10 10 5
10000 mem.after_search MB | 74.918 | 74.867 -0.1% | 75.184 +0.4% | 74.918 | 74.867 -0.1% | 10 10 5
10000 mem.idle MB | 18.775 | 19.283 +2.7% | 18.609 -0.9% | 18.775 | 19.283 +2.7% | 10 10 5
10000 notes.append ms | 5.492 | 5.062 -7.8% | 5.274 -4.0% | 14.713 | 14.445 -1.8% | 10 10 5
10000 notes.append.first ms | 103.797 | 183.348 +76.6% | 259.36 +149.9% | 103.797 | 183.348 +76.6% | 10 10 5
10000 open.api_get ms | 7.686 | 7.703 +0.2% | 7.613 -0.9% | 13.7 | 13.808 +0.8% | 10 10 5
10000 open.api_get.warm ms | 0.164 | 0.165 +0.2% | 0.159 -2.9% | 0.216 | 0.211 -2.0% | 10 10 5
10000 open.long_note ms | 15.951 | 15.942 -0.1% | 16.012 +0.4% | 15.951 | 15.942 -0.1% | 10 10 5
10000 open.note ms | 0.61 | 0.633 +3.7% | 0.594 -2.7% | 1.294 | 1.283 -0.9% | 10 10 5
10000 search.catch_up s | 4.012 | 4.011 -0.0% | 4.014 +0.1% | 4.012 | 4.011 -0.0% | 10 10 5
10000 search.first_query ms | 29.406 | 27.417 -6.8% | 27.438 -6.7% | 29.406 | 27.417 -6.8% | 10 10 5
10000 search.query ms | 1.603 | 1.509 -5.9% | 1.808 +12.7% | 6.255 | 7.799 +24.7% | 10 10 5
10000 shutdown ms | 43.114 | 43.135 +0.0% | 43.056 -0.1% | 43.114 | 43.135 +0.0% | 10 10 5
10000 startup.healthz ms | 4.931 | 7.039 +42.7% | 5.084 +3.1% | 4.931 | 7.039 +42.7% | 10 10 5
10000 sync.durable_write ms | 24.99 | 24.963 -0.1% | 24.941 -0.2% | 31.052 | 30.264 -2.5% | 10 10 5
10000 sync.round_trip ms | 0.207 | 0.214 +3.5% | 0.213 +3.2% | 0.349 | 0.348 -0.4% | 10 10 5
10000 sync.save_ack ms | 3.729 | 3.702 -0.7% | 3.713 -0.4% | 6.279 | 6.824 +8.7% | 10 10 5
100000 cold.disk_written MB | 1919.152 | 2237.45 +16.6% | 1919.189 +0.0% | 1919.152 | 2237.45 +16.6% | 10 10 5
100000 cold.first_page ms | 2720.993 | 7283.372 +167.7% | 2751.979 +1.1% | 2720.993 | 7283.372 +167.7% | 10 10 5
100000 cold.index s | 22.859 | 27.114 +18.6% | 22.966 +0.5% | 22.859 | 27.114 +18.6% | 10 10 5
100000 cold.mem_peak MB | 286.926 | 282.439 -1.6% | 289.355 +0.8% | 286.926 | 282.439 -1.6% | 10 10 5
100000 cold.notes_per_s notes/s | 4375.476 | 3688.112 -15.7% | 4354.334 -0.5% | 4375.476 | 3688.112 -15.7% | 10 10 5
100000 cold.startup ms | 20.303 | 21.906 +7.9% | 19.937 -1.8% | 20.303 | 21.906 +7.9% | 10 10 5
100000 home.first_page ms | 3.315 | 3.491 +5.3% | 3.224 -2.7% | 3.315 | 3.491 +5.3% | 10 10 5
100000 home.page ms | 0.961 | 0.892 -7.2% | 0.972 +1.2% | 1.828 | 1.207 -33.9% | 10 10 5
100000 home.page_big ms | 18.997 | 28.183 +48.4% | 18.969 -0.2% | 24.504 | 38.57 +57.4% | 10 10 5
100000 home.page_next ms | 0.764 | 0.841 +10.0% | 0.766 +0.2% | 1.314 | 1.034 -21.3% | 10 10 5
100000 home.page_tag ms | 52.733 | 51.903 -1.6% | 52.917 +0.3% | 59.835 | 55.278 -7.6% | 10 10 5
100000 home.page_title ms | 0.814 | 0.811 -0.4% | 0.903 +10.9% | 1.056 | 1.028 -2.6% | 10 10 5
100000 mem.after_first_page MB | 22.367 | 22.172 -0.9% | 22.715 +1.6% | 22.367 | 22.172 -0.9% | 10 10 5
100000 mem.after_opens MB | 98.402 | 96.197 -2.2% | 94.48 -4.0% | 98.402 | 96.197 -2.2% | 10 10 5
100000 mem.after_search MB | 101.928 | 94.588 -7.2% | 98.484 -3.4% | 101.928 | 94.588 -7.2% | 10 10 5
100000 mem.idle MB | 19.318 | 19.02 -1.5% | 19.855 +2.8% | 19.318 | 19.02 -1.5% | 10 10 5
100000 notes.append ms | 5.713 | 6.236 +9.2% | 6.045 +5.8% | 19.235 | 21.69 +12.8% | 10 10 5
100000 notes.append.first ms | 889.959 | 186.333 -79.1% | 449.07 -49.5% | 889.959 | 186.333 -79.1% | 10 10 5
100000 open.api_get ms | 8.059 | 8.204 +1.8% | 8.28 +2.7% | 14.807 | 65.915 +345.2% | 10 10 5
100000 open.api_get.warm ms | 0.166 | 0.172 +3.9% | 0.173 +4.5% | 0.21 | 0.217 +3.3% | 10 10 5
100000 open.long_note ms | 96.605 | 100.05 +3.6% | 96.335 -0.3% | 96.605 | 100.05 +3.6% | 10 10 5
100000 open.note ms | 6.221 | 6.822 +9.7% | 7.457 +19.9% | 20.602 | 24.318 +18.0% | 10 10 5
100000 search.catch_up s | 58.136 | 59.138 +1.7% | 58.142 +0.0% | 58.136 | 59.138 +1.7% | 10 10 5
100000 search.first_query ms | 29.39 | 32.264 +9.8% | 27.632 -6.0% | 29.39 | 32.264 +9.8% | 10 10 5
100000 search.query ms | 2.373 | 2.41 +1.6% | 2.284 -3.7% | 23.516 | 24.248 +3.1% | 10 10 5
100000 shutdown ms | 43.336 | 43.299 -0.1% | 43.401 +0.1% | 43.336 | 43.299 -0.1% | 10 10 5
100000 startup.healthz ms | 7.338 | 9.154 +24.7% | 7.272 -0.9% | 7.338 | 9.154 +24.7% | 10 10 5
100000 sync.durable_write ms | 29.645 | 27.86 -6.0% | 29.389 -0.9% | 44.856 | 46.599 +3.9% | 10 10 5
100000 sync.round_trip ms | 0.336 | 0.273 -18.7% | 0.327 -2.8% | 0.579 | 0.46 -20.5% | 10 10 5
100000 sync.save_ack ms | 4.502 | 3.982 -11.5% | 4.639 +3.0% | 67.179 | 39.298 -41.5% | 10 10 5
## Perf: ABAB x5 + control, inside both perf locks (perf-01 then perf-02), session 2026-11-09 06:54–07:55Z Head `d00b48b6e` (Rust identical to PR head `46faabb28`) vs dev `d4b32af94`, release builds, sha256 checked. Dev runs the legacy single-root layout, head runs the joined three-root layout (SSD Home, HDD Home, app state). Per cycle: A1 dev, B1 head, A2 dev, B2 head, then a dev-copy control (same dev binary, copied). 5 cycles, same boot (`7a65f072`), sizes 1k/10k/100k, `--runs 1 --cold-runs 1 --hot-only --gate`. Medians of per-arm p50/p95 (n = 10 dev, 10 head, 5 control). Locks released at 07:55Z. **Control (dev vs dev-copy) is within about 1% on the cold metrics**, so the cold deltas below are not noise. ### Scale gate (1M entries, `files-listing-bench`, page 50) `cold_first_us=104 p50_us=59 p99_us=99 active_map_bytes=53085308 peak_rss_kib=56460`. The plan's gate is a first page under 100 ms: pass (about 0.1 ms). ### Regressions vs dev (they block merge under the no-regression rule) | Metric | 1k | 10k | 100k | |---|---|---|---| | `cold.first_page` p50 | +30% (435 → 568 ms) | +115% (333 → 714 ms) | **+168% (2.72 → 7.28 s)** | | `cold.index` | +25% | +23% | +19% (22.9 → 27.1 s) | | `cold.notes_per_s` | -20% | -19% | -16% | | `cold.disk_written` | +3% | +6% | +17% (1.92 → 2.24 GB) | | `startup.healthz` | +43% | +43% | +25% (7.3 → 9.2 ms) | | `home.page_big` p50 | -1% | +2% | **+48% (19.0 → 28.2 ms)**, p95 +57% | | `open.api_get` p95 | +11% | +1% | **+345% (14.8 → 65.9 ms)** | | `notes.append.first` | +6% | +77% (104 → 183 ms; control also +150%, noisy) | -79% (better) | | `notes.append` p50 | -2% | -8% | +9% (p95 +13%) | | `open.note` | -7% | +4% | +10% (p95 +18%) | `cold.*` runs once per arm, so these are medians of 10 arms; the control's median stays within about 1% of dev. ### Improvements `home.first_page` 1k -37%; `notes.append.first` 100k -79%; `sync.save_ack` p95 100k -41%; `sync.round_trip` 100k -19%; memory flat to -7%. ### Flat (inside control noise) `home.page`, `home.page_next`, `home.page_tag`, `home.page_title`, `search.*`, `sync.durable_write`, `shutdown`, `open.api_get.warm`, `mem.*`. ### Likely cause (not verified) The cold open now reads and writes the shared current namespace (joined-tree reconcile plus bounded 8,192-row commits) in addition to the Notes discovery dev does, so the first page waits behind extra index writes (+17% bytes written at 100k). `startup.healthz` +2 ms is the extra root validation and resolver open. ### State Not READY FOR MERGE under the no-perf-regression rule: the cold-path, `home.page_big` and `open.api_get` p95 deltas need a fix or kayg's explicit waiver. Everything else is green: fmt, clippy `-D warnings`, full workspace tests, scale gate, legacy-attachment ID test, settlement flake fixed. <details><summary>Full table (all metrics)</summary> ``` size metric unit | dev p50 | head p50 d% | ctl p50 d% | dev p95 | head p95 d% | n(dev/head/ctl) 1000 cold.disk_written MB | 18.813 | 19.419 +3.2% | 18.784 -0.2% | 18.813 | 19.419 +3.2% | 10 10 5 1000 cold.first_page ms | 435.473 | 567.589 +30.3% | 432.892 -0.6% | 435.473 | 567.589 +30.3% | 10 10 5 1000 cold.index s | 0.538 | 0.669 +24.5% | 0.535 -0.5% | 0.538 | 0.669 +24.5% | 10 10 5 1000 cold.mem_peak MB | 45.088 | 47.998 +6.5% | 46.273 +2.6% | 45.088 | 47.998 +6.5% | 10 10 5 1000 cold.notes_per_s notes/s | 1859.29 | 1493.762 -19.7% | 1868.279 +0.5% | 1859.29 | 1493.762 -19.7% | 10 10 5 1000 cold.startup ms | 20.349 | 22.157 +8.9% | 23.192 +14.0% | 20.349 | 22.157 +8.9% | 10 10 5 1000 home.first_page ms | 2.787 | 1.769 -36.5% | 2.126 -23.7% | 2.787 | 1.769 -36.5% | 10 10 5 1000 home.page ms | 0.678 | 0.727 +7.3% | 0.672 -0.9% | 1.222 | 1.408 +15.2% | 10 10 5 1000 home.page_big ms | 8.938 | 8.826 -1.2% | 8.897 -0.5% | 8.938 | 8.826 -1.2% | 10 10 5 1000 home.page_next ms | 0.714 | 0.719 +0.7% | 0.695 -2.7% | 0.803 | 0.792 -1.3% | 10 10 5 1000 home.page_tag ms | 1.247 | 1.272 +2.0% | 1.21 -3.0% | 1.446 | 1.494 +3.3% | 10 10 5 1000 home.page_title ms | 0.611 | 0.61 -0.2% | 0.57 -6.7% | 0.715 | 0.713 -0.3% | 10 10 5 1000 mem.after_first_page MB | 21.34 | 20.857 -2.3% | 21.668 +1.5% | 21.34 | 20.857 -2.3% | 10 10 5 1000 mem.after_opens MB | 34.705 | 35.404 +2.0% | 35.336 +1.8% | 34.705 | 35.404 +2.0% | 10 10 5 1000 mem.after_search MB | 47.57 | 48.371 +1.7% | 47.777 +0.4% | 47.57 | 48.371 +1.7% | 10 10 5 1000 mem.idle MB | 18.232 | 19.148 +5.0% | 18.77 +2.9% | 18.232 | 19.148 +5.0% | 10 10 5 1000 notes.append ms | 5.336 | 5.231 -2.0% | 5.277 -1.1% | 13.402 | 13.103 -2.2% | 10 10 5 1000 notes.append.first ms | 21.315 | 22.6 +6.0% | 24.109 +13.1% | 21.315 | 22.6 +6.0% | 10 10 5 1000 open.api_get ms | 7.488 | 7.515 +0.4% | 7.518 +0.4% | 13.772 | 15.316 +11.2% | 10 10 5 1000 open.api_get.warm ms | 0.162 | 0.164 +1.4% | 0.17 +5.3% | 0.209 | 0.207 -1.2% | 10 10 5 1000 open.long_note ms | 15.102 | 15.582 +3.2% | 15.884 +5.2% | 15.102 | 15.582 +3.2% | 10 10 5 1000 open.note ms | 0.701 | 0.651 -7.2% | 0.663 -5.5% | 2.266 | 2.121 -6.4% | 10 10 5 1000 search.catch_up s | 2.003 | 2.003 +0.0% | 2.003 +0.0% | 2.003 | 2.003 +0.0% | 10 10 5 1000 search.first_query ms | 27.772 | 31.144 +12.1% | 30.12 +8.5% | 27.772 | 31.144 +12.1% | 10 10 5 1000 search.query ms | 1.149 | 1.167 +1.5% | 1.161 +1.0% | 2.137 | 2.238 +4.7% | 10 10 5 1000 shutdown ms | 43.058 | 43.031 -0.1% | 42.175 -2.0% | 43.058 | 43.031 -0.1% | 10 10 5 1000 startup.healthz ms | 4.968 | 7.109 +43.1% | 4.899 -1.4% | 4.968 | 7.109 +43.1% | 10 10 5 1000 sync.durable_write ms | 24.98 | 24.951 -0.1% | 25.116 +0.5% | 33.142 | 31.43 -5.2% | 10 10 5 1000 sync.round_trip ms | 0.214 | 0.209 -2.6% | 0.208 -3.1% | 0.343 | 0.352 +2.6% | 10 10 5 1000 sync.save_ack ms | 3.696 | 3.717 +0.6% | 3.668 -0.7% | 5.504 | 5.71 +3.7% | 10 10 5 10000 cold.disk_written MB | 150.252 | 159.322 +6.0% | 150.569 +0.2% | 150.252 | 159.322 +6.0% | 10 10 5 10000 cold.first_page ms | 332.798 | 713.989 +114.5% | 325.348 -2.2% | 332.798 | 713.989 +114.5% | 10 10 5 10000 cold.index s | 1.697 | 2.086 +22.9% | 1.676 -1.2% | 1.697 | 2.086 +22.9% | 10 10 5 10000 cold.mem_peak MB | 143.535 | 145.127 +1.1% | 143.309 -0.2% | 143.535 | 145.127 +1.1% | 10 10 5 10000 cold.notes_per_s notes/s | 5894.999 | 4794.997 -18.7% | 5965.417 +1.2% | 5894.999 | 4794.997 -18.7% | 10 10 5 10000 cold.startup ms | 19.833 | 21.198 +6.9% | 18.787 -5.3% | 19.833 | 21.198 +6.9% | 10 10 5 10000 home.first_page ms | 2.36 | 2.351 -0.4% | 2.622 +11.1% | 2.36 | 2.351 -0.4% | 10 10 5 10000 home.page ms | 0.76 | 0.831 +9.3% | 0.816 +7.4% | 1.312 | 1.07 -18.5% | 10 10 5 10000 home.page_big ms | 16.733 | 17.087 +2.1% | 16.519 -1.3% | 17.715 | 19.529 +10.2% | 10 10 5 10000 home.page_next ms | 0.688 | 0.71 +3.2% | 0.7 +1.8% | 0.808 | 0.924 +14.4% | 10 10 5 10000 home.page_tag ms | 5.363 | 5.433 +1.3% | 5.382 +0.4% | 7.117 | 7.254 +1.9% | 10 10 5 10000 home.page_title ms | 0.644 | 0.69 +7.2% | 0.679 +5.6% | 0.773 | 1.272 +64.6% | 10 10 5 10000 mem.after_first_page MB | 21.277 | 21.504 +1.1% | 21.367 +0.4% | 21.277 | 21.504 +1.1% | 10 10 5 10000 mem.after_opens MB | 40.227 | 41.617 +3.5% | 40.938 +1.8% | 40.227 | 41.617 +3.5% | 10 10 5 10000 mem.after_search MB | 74.918 | 74.867 -0.1% | 75.184 +0.4% | 74.918 | 74.867 -0.1% | 10 10 5 10000 mem.idle MB | 18.775 | 19.283 +2.7% | 18.609 -0.9% | 18.775 | 19.283 +2.7% | 10 10 5 10000 notes.append ms | 5.492 | 5.062 -7.8% | 5.274 -4.0% | 14.713 | 14.445 -1.8% | 10 10 5 10000 notes.append.first ms | 103.797 | 183.348 +76.6% | 259.36 +149.9% | 103.797 | 183.348 +76.6% | 10 10 5 10000 open.api_get ms | 7.686 | 7.703 +0.2% | 7.613 -0.9% | 13.7 | 13.808 +0.8% | 10 10 5 10000 open.api_get.warm ms | 0.164 | 0.165 +0.2% | 0.159 -2.9% | 0.216 | 0.211 -2.0% | 10 10 5 10000 open.long_note ms | 15.951 | 15.942 -0.1% | 16.012 +0.4% | 15.951 | 15.942 -0.1% | 10 10 5 10000 open.note ms | 0.61 | 0.633 +3.7% | 0.594 -2.7% | 1.294 | 1.283 -0.9% | 10 10 5 10000 search.catch_up s | 4.012 | 4.011 -0.0% | 4.014 +0.1% | 4.012 | 4.011 -0.0% | 10 10 5 10000 search.first_query ms | 29.406 | 27.417 -6.8% | 27.438 -6.7% | 29.406 | 27.417 -6.8% | 10 10 5 10000 search.query ms | 1.603 | 1.509 -5.9% | 1.808 +12.7% | 6.255 | 7.799 +24.7% | 10 10 5 10000 shutdown ms | 43.114 | 43.135 +0.0% | 43.056 -0.1% | 43.114 | 43.135 +0.0% | 10 10 5 10000 startup.healthz ms | 4.931 | 7.039 +42.7% | 5.084 +3.1% | 4.931 | 7.039 +42.7% | 10 10 5 10000 sync.durable_write ms | 24.99 | 24.963 -0.1% | 24.941 -0.2% | 31.052 | 30.264 -2.5% | 10 10 5 10000 sync.round_trip ms | 0.207 | 0.214 +3.5% | 0.213 +3.2% | 0.349 | 0.348 -0.4% | 10 10 5 10000 sync.save_ack ms | 3.729 | 3.702 -0.7% | 3.713 -0.4% | 6.279 | 6.824 +8.7% | 10 10 5 100000 cold.disk_written MB | 1919.152 | 2237.45 +16.6% | 1919.189 +0.0% | 1919.152 | 2237.45 +16.6% | 10 10 5 100000 cold.first_page ms | 2720.993 | 7283.372 +167.7% | 2751.979 +1.1% | 2720.993 | 7283.372 +167.7% | 10 10 5 100000 cold.index s | 22.859 | 27.114 +18.6% | 22.966 +0.5% | 22.859 | 27.114 +18.6% | 10 10 5 100000 cold.mem_peak MB | 286.926 | 282.439 -1.6% | 289.355 +0.8% | 286.926 | 282.439 -1.6% | 10 10 5 100000 cold.notes_per_s notes/s | 4375.476 | 3688.112 -15.7% | 4354.334 -0.5% | 4375.476 | 3688.112 -15.7% | 10 10 5 100000 cold.startup ms | 20.303 | 21.906 +7.9% | 19.937 -1.8% | 20.303 | 21.906 +7.9% | 10 10 5 100000 home.first_page ms | 3.315 | 3.491 +5.3% | 3.224 -2.7% | 3.315 | 3.491 +5.3% | 10 10 5 100000 home.page ms | 0.961 | 0.892 -7.2% | 0.972 +1.2% | 1.828 | 1.207 -33.9% | 10 10 5 100000 home.page_big ms | 18.997 | 28.183 +48.4% | 18.969 -0.2% | 24.504 | 38.57 +57.4% | 10 10 5 100000 home.page_next ms | 0.764 | 0.841 +10.0% | 0.766 +0.2% | 1.314 | 1.034 -21.3% | 10 10 5 100000 home.page_tag ms | 52.733 | 51.903 -1.6% | 52.917 +0.3% | 59.835 | 55.278 -7.6% | 10 10 5 100000 home.page_title ms | 0.814 | 0.811 -0.4% | 0.903 +10.9% | 1.056 | 1.028 -2.6% | 10 10 5 100000 mem.after_first_page MB | 22.367 | 22.172 -0.9% | 22.715 +1.6% | 22.367 | 22.172 -0.9% | 10 10 5 100000 mem.after_opens MB | 98.402 | 96.197 -2.2% | 94.48 -4.0% | 98.402 | 96.197 -2.2% | 10 10 5 100000 mem.after_search MB | 101.928 | 94.588 -7.2% | 98.484 -3.4% | 101.928 | 94.588 -7.2% | 10 10 5 100000 mem.idle MB | 19.318 | 19.02 -1.5% | 19.855 +2.8% | 19.318 | 19.02 -1.5% | 10 10 5 100000 notes.append ms | 5.713 | 6.236 +9.2% | 6.045 +5.8% | 19.235 | 21.69 +12.8% | 10 10 5 100000 notes.append.first ms | 889.959 | 186.333 -79.1% | 449.07 -49.5% | 889.959 | 186.333 -79.1% | 10 10 5 100000 open.api_get ms | 8.059 | 8.204 +1.8% | 8.28 +2.7% | 14.807 | 65.915 +345.2% | 10 10 5 100000 open.api_get.warm ms | 0.166 | 0.172 +3.9% | 0.173 +4.5% | 0.21 | 0.217 +3.3% | 10 10 5 100000 open.long_note ms | 96.605 | 100.05 +3.6% | 96.335 -0.3% | 96.605 | 100.05 +3.6% | 10 10 5 100000 open.note ms | 6.221 | 6.822 +9.7% | 7.457 +19.9% | 20.602 | 24.318 +18.0% | 10 10 5 100000 search.catch_up s | 58.136 | 59.138 +1.7% | 58.142 +0.0% | 58.136 | 59.138 +1.7% | 10 10 5 100000 search.first_query ms | 29.39 | 32.264 +9.8% | 27.632 -6.0% | 29.39 | 32.264 +9.8% | 10 10 5 100000 search.query ms | 2.373 | 2.41 +1.6% | 2.284 -3.7% | 23.516 | 24.248 +3.1% | 10 10 5 100000 shutdown ms | 43.336 | 43.299 -0.1% | 43.401 +0.1% | 43.336 | 43.299 -0.1% | 10 10 5 100000 startup.healthz ms | 7.338 | 9.154 +24.7% | 7.272 -0.9% | 7.338 | 9.154 +24.7% | 10 10 5 100000 sync.durable_write ms | 29.645 | 27.86 -6.0% | 29.389 -0.9% | 44.856 | 46.599 +3.9% | 10 10 5 100000 sync.round_trip ms | 0.336 | 0.273 -18.7% | 0.327 -2.8% | 0.579 | 0.46 -20.5% | 10 10 5 100000 sync.save_ack ms | 4.502 | 3.982 -11.5% | 4.639 +3.0% | 67.179 | 39.298 -41.5% | 10 10 5 ``` </details>
kayg commented 2026-11-09 09:55:49 +00:00
Author
Owner
Copy link

Perf fix design (round 1): why head was slower than dev, and the fix

Profiled on compute02 (directional; the perf VMs decide) with release builds of dev d4b32af94 (legacy layout), #970 head 381ce6750 (joined layout) and this PR's head 46faabb28 (joined layout), 10k cold runs, strace on the server.

Root cause per metric

Metric Cause found Evidence
cold.first_page, cold.index, cold.notes_per_s, cold.disk_written Every Notes row written by the cold scan now updates five B-trees in files/current.db instead of #970's two: current_files PK, current_files_by_path, the new full current_files_by_scan_generation, the new current_children table and its (parent_key, name_key, id) index. UUIDv7-keyed inserts into extra trees dirty many more pages per 8,192-row commit. In the first-page window head did 21,335 pwrite + 19,717 pread on current.db vs 4,500 + 3,169 for #970 (same 10k Home, same scan code). All other syscall counts (statx, openat, read) matched within 1%.
open.api_get p95, home.page_big (100k, hot restart) Home::reconcile_files takes self.index() once and holds the Home's index mutex for the whole Files walk, including every getdents/statx/marker read over both trees. Any request needing index() (note open, path_of, list validation) waits behind it. At 100k the walk is seconds long, so a few requests in each run land on it (p95, not p50). Code path (home_files.rs reconcile_files/reconcile_files_paths); matches the shape (p50 flat, p95 4×).
startup.healthz Not reproduced between #970 and this PR at 10k on compute02 (same syscall counts up to listen). Measured in the fixed head vs dev in the locked ABAB; if it stays above dev it gets its own round.

Fix

  1. Listing reads the path index, not a second child table. page_children range-scans current_files_by_path(path_key, id) from <parent>/; when it meets a descendant <parent>/<child>/… it seeks once past that child's whole subtree. current_children (and its backfill + namespace_schema) is dropped; a database from the earlier draft has them removed once on open.
  2. Scan cursor index covers Files-walk rows only. current_files_by_scan_generation becomes a partial index WHERE path <> 'Notes' AND path NOT GLOB 'Notes/*' — exactly the predicate of the stale-row query, so the walk still uses it, and Notes writes never touch it. The stale query reads scan_generation < ? (walk generations only grow), a true range.
  3. One transaction per walk page. Rows, inode evidence and the seen mark commit together (Namespace::commit_scan_page): one WAL sync per page instead of three, and a crash leaves the whole page or none. Unchanged inode evidence is not rewritten.
  4. The walk never holds the index lock across I/O. It borrows the index per lookup and per page commit (home_files/access.rs). Safe because the walk is the only writer of non-Notes rows and runs one at a time from the watcher loop, while Notes writes only touch Notes/*.md rows that the walk skips.

The first Notes page already comes from meta.db and never waits for identity assignment; the Files walk starts only after the Notes scan, so nothing here moves onto the first-page path.

Big-O, before → after (N = rows in the user's namespace, P = page size, W = rows in one walk page = 256)

Operation Before After
Notes row write (cold scan, save, rename) 5 B-tree updates, O(5 log N) 2 B-tree updates, O(2 log N) (same as #970)
Folder page (page_children) O(log N + P) O(P · log N) worst case (one extra seek per child folder with descendants), O(log N + P) for flat folders
Walk page commit 3 transactions / 3 syncs, O(W log N) 1 transaction / 1 sync, O(W log N)
Stale-row sweep full scan of a whole-table index, O(N) per call range read of a partial index, O(log N + stale)
Request latency during a walk blocked for the whole walk, O(total walk) blocked for at most one lookup or one page commit, O(W log N)

Identity guarantees are unchanged: IDs, markers, inode evidence and the collision journal are written by the same code in the same order; only lock scope and transaction grouping change. Gates to re-run: identity tests (incl. 2,000-iteration rescan, legacy_attachment_ids_survive_every_retry_boundary), workspace tests/clippy/fmt, the 1M listing gate, and the locked ABAB ×5 with a dev-copy control.

## Perf fix design (round 1): why head was slower than dev, and the fix Profiled on compute02 (directional; the perf VMs decide) with release builds of dev `d4b32af94` (legacy layout), #970 head `381ce6750` (joined layout) and this PR's head `46faabb28` (joined layout), 10k cold runs, strace on the server. ### Root cause per metric | Metric | Cause found | Evidence | |---|---|---| | `cold.first_page`, `cold.index`, `cold.notes_per_s`, `cold.disk_written` | Every Notes row written by the cold scan now updates **five** B-trees in `files/current.db` instead of #970's two: `current_files` PK, `current_files_by_path`, the new full `current_files_by_scan_generation`, the new `current_children` table and its `(parent_key, name_key, id)` index. UUIDv7-keyed inserts into extra trees dirty many more pages per 8,192-row commit. | In the first-page window head did 21,335 `pwrite` + 19,717 `pread` on `current.db` vs 4,500 + 3,169 for #970 (same 10k Home, same scan code). All other syscall counts (statx, openat, read) matched within 1%. | | `open.api_get` p95, `home.page_big` (100k, hot restart) | `Home::reconcile_files` takes `self.index()` once and holds the Home's index mutex for the **whole** Files walk, including every `getdents`/`statx`/marker read over both trees. Any request needing `index()` (note open, `path_of`, list validation) waits behind it. At 100k the walk is seconds long, so a few requests in each run land on it (p95, not p50). | Code path (`home_files.rs` `reconcile_files`/`reconcile_files_paths`); matches the shape (p50 flat, p95 4×). | | `startup.healthz` | Not reproduced between #970 and this PR at 10k on compute02 (same syscall counts up to listen). Measured in the fixed head vs dev in the locked ABAB; if it stays above dev it gets its own round. | | ### Fix 1. **Listing reads the path index, not a second child table.** `page_children` range-scans `current_files_by_path(path_key, id)` from `<parent>/`; when it meets a descendant `<parent>/<child>/…` it seeks once past that child's whole subtree. `current_children` (and its backfill + `namespace_schema`) is dropped; a database from the earlier draft has them removed once on open. 2. **Scan cursor index covers Files-walk rows only.** `current_files_by_scan_generation` becomes a partial index `WHERE path <> 'Notes' AND path NOT GLOB 'Notes/*'` — exactly the predicate of the stale-row query, so the walk still uses it, and Notes writes never touch it. The stale query reads `scan_generation < ?` (walk generations only grow), a true range. 3. **One transaction per walk page.** Rows, inode evidence and the seen mark commit together (`Namespace::commit_scan_page`): one WAL sync per page instead of three, and a crash leaves the whole page or none. Unchanged inode evidence is not rewritten. 4. **The walk never holds the index lock across I/O.** It borrows the index per lookup and per page commit (`home_files/access.rs`). Safe because the walk is the only writer of non-Notes rows and runs one at a time from the watcher loop, while Notes writes only touch `Notes/*.md` rows that the walk skips. The first Notes page already comes from `meta.db` and never waits for identity assignment; the Files walk starts only after the Notes scan, so nothing here moves onto the first-page path. ### Big-O, before → after (N = rows in the user's namespace, P = page size, W = rows in one walk page = 256) | Operation | Before | After | |---|---|---| | Notes row write (cold scan, save, rename) | 5 B-tree updates, O(5 log N) | 2 B-tree updates, O(2 log N) (same as #970) | | Folder page (`page_children`) | O(log N + P) | O(P · log N) worst case (one extra seek per child folder with descendants), O(log N + P) for flat folders | | Walk page commit | 3 transactions / 3 syncs, O(W log N) | 1 transaction / 1 sync, O(W log N) | | Stale-row sweep | full scan of a whole-table index, O(N) per call | range read of a partial index, O(log N + stale) | | Request latency during a walk | blocked for the whole walk, O(total walk) | blocked for at most one lookup or one page commit, O(W log N) | Identity guarantees are unchanged: IDs, markers, inode evidence and the collision journal are written by the same code in the same order; only lock scope and transaction grouping change. Gates to re-run: identity tests (incl. 2,000-iteration rescan, `legacy_attachment_ids_survive_every_retry_boundary`), workspace tests/clippy/fmt, the 1M listing gate, and the locked ABAB ×5 with a dev-copy control.
kayg commented 2026-11-09 16:59:19 +00:00
Author
Owner
Copy link

Perf round 3: ABAB ×5 with a #970 arm and a dev-copy control (both perf locks, FIFO turn)

Session 2026-11-09 15:56–16:56Z (21:26–22:26 IST), boot 7a65f072 (same epoch as the earlier sessions), perf-01 lock via perf-turn files-1010 then perf-02's lock, both released at the end. Per cycle: dev (legacy layout), head (joined layout), #970 381ce6750 (joined), dev, head, dev-copy control. Sizes 1k/10k/100k, --runs 1 --cold-runs 1 --hot-only --gate; medians of per-arm p50/p95, n = 10 dev, 10 head, 5 #970, 5 control. All 30 arms exited 0. sha256 checked at turn time.

  • head = PR head 0c99d74e9 (release built from 9520d6fed; the two differ only in a #[cfg(test)] module)
  • dev = d4b32af94 (the merge base)
  • Change from earlier sessions: head arms now run the server binary directly. The earlier sessions ran it through a bash wrapper, which forked bash before every server start. That wrapper was the constant +2 ms in startup.healthz: with it gone, healthz is flat at 1k and 100k.

1M listing gate

files_listing entries=1000000 page_size=50 cold_first_us=99 p50_us=58 p99_us=85 active_map_bytes=53085308 rss_after_open_kib=56548 peak_rss_kib=56548

First page under 100 ms: pass (0.099 ms).

Full table

size metric dev p50 head p50 Δ head #970 p50 Δ #970 ctl Δ dev p95 head p95 Δ head p95 Δ #970 p95 ctl Δ p95 n dev/head/970/ctl
1000 cold.disk_written (MB) 18.8 19.1 +1.7% 19 +0.9% +0.5% 18.8 19.1 +1.7% +0.9% +0.5% 10/10/5/5
1000 cold.first_page (ms) 406 491 +20.9% 441 +8.5% +0.5% 406 491 +20.9% +8.5% +0.5% 10/10/5/5
1000 cold.index (s) 0.509 0.593 +16.6% 0.543 +6.7% +0.3% 0.509 0.593 +16.6% +6.7% +0.3% 10/10/5/5
1000 cold.mem_peak (MB) 46 48.2 +4.8% 47.8 +4.0% +1.6% 46 48.2 +4.8% +4.0% +1.6% 10/10/5/5
1000 cold.notes_per_s (notes/s) 1.97e+03 1.69e+03 -14.2% 1.84e+03 -6.3% -0.3% 1.97e+03 1.69e+03 -14.2% -6.3% -0.3% 10/10/5/5
1000 cold.startup (ms) 19.8 19.8 +0.0% 19.8 -0.1% +0.6% 19.8 19.8 +0.0% -0.1% +0.6% 10/10/5/5
1000 home.first_page (ms) 1.75 2.33 +33.4% 2.66 +52.6% +4.5% 1.75 2.33 +33.4% +52.6% +4.5% 10/10/5/5
1000 home.page (ms) 0.68 0.713 +4.8% 0.72 +5.8% -5.6% 1.21 1.91 +57.1% +5.8% -1.8% 10/10/5/5
1000 home.page_big (ms) 8.66 8.88 +2.5% 8.86 +2.2% -1.7% 8.66 8.88 +2.5% +2.2% -1.7% 10/10/5/5
1000 home.page_next (ms) 0.726 0.736 +1.4% 0.727 +0.1% +1.3% 0.815 0.816 +0.1% +4.7% -2.4% 10/10/5/5
1000 home.page_tag (ms) 1.24 1.29 +3.7% 1.23 -1.1% -0.6% 1.45 1.4 -3.1% +2.4% -7.6% 10/10/5/5
1000 home.page_title (ms) 0.597 0.602 +0.8% 0.587 -1.8% -0.5% 0.707 0.707 -0.0% -7.8% -2.3% 10/10/5/5
1000 mem.after_first_page (MB) 20.9 20.8 -0.1% 21.3 +1.9% +2.9% 20.9 20.8 -0.1% +1.9% +2.9% 10/10/5/5
1000 mem.after_opens (MB) 34.6 35.4 +2.4% 34.6 +0.1% -0.1% 34.6 35.4 +2.4% +0.1% -0.1% 10/10/5/5
1000 mem.after_search (MB) 47.7 48.3 +1.3% 48 +0.7% +0.3% 47.7 48.3 +1.3% +0.7% +0.3% 10/10/5/5
1000 mem.idle (MB) 18.3 18.8 +2.4% 18.8 +2.3% +1.5% 18.3 18.8 +2.4% +2.3% +1.5% 10/10/5/5
1000 notes.append (ms) 4.84 4.78 -1.2% 4.86 +0.4% +0.0% 12 12.5 +4.5% +6.6% +2.2% 10/10/5/5
1000 notes.append.first (ms) 22.4 17 -24.0% 15.8 -29.4% -25.6% 22.4 17 -24.0% -29.4% -25.6% 10/10/5/5
1000 open.api_get (ms) 7.5 7.42 -1.1% 7.21 -3.9% -1.0% 11.8 11.3 -4.5% +7.9% +15.5% 10/10/5/5
1000 open.api_get.warm (ms) 0.163 0.164 +0.8% 0.166 +2.1% +0.1% 0.213 0.209 -1.9% -2.3% +0.6% 10/10/5/5
1000 open.long_note (ms) 15.1 15.2 +0.9% 14.8 -1.4% +3.3% 15.1 15.2 +0.9% -1.4% +3.3% 10/10/5/5
1000 open.note (ms) 0.66 0.656 -0.5% 0.674 +2.2% +4.4% 2.26 2.21 -2.2% -5.1% -3.2% 10/10/5/5
1000 search.catch_up (s) 2 2 +0.0% 2 +0.0% +0.0% 2 2 +0.0% +0.0% +0.0% 10/10/5/5
1000 search.first_query (ms) 28.1 26.7 -4.9% 26.4 -6.2% -3.3% 28.1 26.7 -4.9% -6.2% -3.3% 10/10/5/5
1000 search.query (ms) 1.25 1.23 -1.5% 1.23 -1.8% -9.5% 2.34 2.17 -7.2% -6.4% -10.2% 10/10/5/5
1000 shutdown (ms) 43.1 43.1 -0.1% 43.1 -0.0% -0.1% 43.1 43.1 -0.1% -0.0% -0.1% 10/10/5/5
1000 startup.healthz (ms) 4.94 4.96 +0.4% 4.84 -2.0% +3.8% 4.94 4.96 +0.4% -2.0% +3.8% 10/10/5/5
1000 sync.durable_write (ms) 24.9 24.7 -0.8% 24.8 -0.4% +0.5% 30 28.2 -5.9% -5.0% +13.0% 10/10/5/5
1000 sync.round_trip (ms) 0.201 0.206 +2.4% 0.198 -1.4% +6.9% 0.325 0.339 +4.5% -2.4% +6.9% 10/10/5/5
1000 sync.save_ack (ms) 3.66 3.65 -0.3% 3.74 +2.3% +0.5% 4.78 5.04 +5.4% +10.8% +5.0% 10/10/5/5
10000 cold.disk_written (MB) 150 152 +1.4% 152 +1.4% -0.1% 150 152 +1.4% +1.4% -0.1% 10/10/5/5
10000 cold.first_page (ms) 331 518 +56.3% 459 +38.6% -7.3% 331 518 +56.3% +38.6% -7.3% 10/10/5/5
10000 cold.index (s) 1.48 1.7 +14.5% 1.7 +14.5% -2.5% 1.48 1.7 +14.5% +14.5% -2.5% 10/10/5/5
10000 cold.mem_peak (MB) 143 145 +1.4% 145 +1.1% +0.3% 143 145 +1.4% +1.1% +0.3% 10/10/5/5
10000 cold.notes_per_s (notes/s) 6.75e+03 5.9e+03 -12.7% 5.9e+03 -12.7% +2.5% 6.75e+03 5.9e+03 -12.7% -12.7% +2.5% 10/10/5/5
10000 cold.startup (ms) 20 19.9 -0.6% 17.9 -10.4% -0.3% 20 19.9 -0.6% -10.4% -0.3% 10/10/5/5
10000 home.first_page (ms) 2.46 2.54 +3.0% 2.28 -7.4% -3.9% 2.46 2.54 +3.0% -7.4% -3.9% 10/10/5/5
10000 home.page (ms) 0.759 0.766 +1.0% 0.852 +12.2% +10.3% 1.79 1.27 -29.2% -43.9% +111.6% 10/10/5/5
10000 home.page_big (ms) 16.9 17 +0.7% 16.7 -0.9% -1.3% 18 18.1 +0.7% -0.6% -1.1% 10/10/5/5
10000 home.page_next (ms) 0.7 0.733 +4.7% 0.713 +1.9% -0.2% 0.821 0.897 +9.2% +8.4% +5.4% 10/10/5/5
10000 home.page_tag (ms) 5.32 5.57 +4.8% 5.55 +4.3% -0.7% 7.53 7.94 +5.5% -2.7% +1.5% 10/10/5/5
10000 home.page_title (ms) 0.634 0.724 +14.3% 0.746 +17.7% +3.6% 0.833 1.4 +68.6% +14.7% +7.0% 10/10/5/5
10000 mem.after_first_page (MB) 21.4 21.4 +0.4% 21.4 -0.0% -0.1% 21.4 21.4 +0.4% -0.0% -0.1% 10/10/5/5
10000 mem.after_opens (MB) 39.9 40.2 +0.7% 39.9 -0.0% +0.9% 39.9 40.2 +0.7% -0.0% +0.9% 10/10/5/5
10000 mem.after_search (MB) 75.3 75.6 +0.4% 74.7 -0.8% +0.2% 75.3 75.6 +0.4% -0.8% +0.2% 10/10/5/5
10000 mem.idle (MB) 18.5 19.3 +4.0% 18.9 +1.7% +0.1% 18.5 19.3 +4.0% +1.7% +0.1% 10/10/5/5
10000 notes.append (ms) 4.9 4.9 +0.0% 4.98 +1.6% +4.9% 12 11.8 -1.8% +3.5% +3.5% 10/10/5/5
10000 notes.append.first (ms) 55.8 52.1 -6.6% 48.5 -13.2% -8.7% 55.8 52.1 -6.6% -13.2% -8.7% 10/10/5/5
10000 open.api_get (ms) 7.45 7.34 -1.5% 7.39 -0.8% -2.9% 13.3 12.2 -8.4% +17.6% -4.5% 10/10/5/5
10000 open.api_get.warm (ms) 0.171 0.16 -6.2% 0.165 -3.5% -5.1% 0.215 0.206 -4.2% +0.7% -2.3% 10/10/5/5
10000 open.long_note (ms) 16.2 16.6 +2.0% 16.4 +0.8% +3.1% 16.2 16.6 +2.0% +0.8% +3.1% 10/10/5/5
10000 open.note (ms) 0.661 0.633 -4.3% 0.625 -5.5% -8.0% 1.37 1.3 -5.8% +3.9% -6.6% 10/10/5/5
10000 search.catch_up (s) 4.02 4.01 -0.0% 4.01 -0.1% -0.1% 4.02 4.01 -0.0% -0.1% -0.1% 10/10/5/5
10000 search.first_query (ms) 28.4 29.3 +3.2% 27.5 -2.9% -3.4% 28.4 29.3 +3.2% -2.9% -3.4% 10/10/5/5
10000 search.query (ms) 1.35 2.29 +69.1% 1.35 -0.1% +24.2% 10.6 11.5 +8.7% -34.0% -11.9% 10/10/5/5
10000 shutdown (ms) 43.1 43.2 +0.1% 43 -0.2% +0.0% 43.1 43.2 +0.1% -0.2% +0.0% 10/10/5/5
10000 startup.healthz (ms) 4.92 5.61 +14.0% 4.85 -1.4% -1.0% 4.92 5.61 +14.0% -1.4% -1.0% 10/10/5/5
10000 sync.durable_write (ms) 24.9 24.8 -0.2% 24.3 -2.1% -2.2% 28.7 29 +1.0% +2.3% +0.9% 10/10/5/5
10000 sync.round_trip (ms) 0.208 0.206 -0.9% 0.212 +2.0% -6.7% 0.345 0.344 -0.1% -2.2% -3.6% 10/10/5/5
10000 sync.save_ack (ms) 3.66 3.66 -0.1% 3.67 +0.2% -0.4% 5.45 5.27 -3.3% -2.4% -2.2% 10/10/5/5
100000 cold.disk_written (MB) 1.92e+03 2.06e+03 +7.5% 2.06e+03 +7.3% -0.0% 1.92e+03 2.06e+03 +7.5% +7.3% -0.0% 10/10/5/5
100000 cold.first_page (ms) 1.82e+03 4.15e+03 +128.1% 4.35e+03 +139.5% -2.1% 1.82e+03 4.15e+03 +128.1% +139.5% -2.1% 10/10/5/5
100000 cold.index (s) 15.6 18 +15.6% 17.9 +14.9% -0.0% 15.6 18 +15.6% +14.9% -0.0% 10/10/5/5
100000 cold.mem_peak (MB) 292 286 -2.2% 287 -1.7% +0.1% 292 286 -2.2% -1.7% +0.1% 10/10/5/5
100000 cold.notes_per_s (notes/s) 6.41e+03 5.55e+03 -13.5% 5.58e+03 -13.0% +0.0% 6.41e+03 5.55e+03 -13.5% -13.0% +0.0% 10/10/5/5
100000 cold.startup (ms) 20.8 19.9 -4.7% 19.8 -4.8% -3.8% 20.8 19.9 -4.7% -4.8% -3.8% 10/10/5/5
100000 home.first_page (ms) 3.21 3.19 -0.7% 3.1 -3.3% -4.9% 3.21 3.19 -0.7% -3.3% -4.9% 10/10/5/5
100000 home.page (ms) 0.952 0.957 +0.5% 0.95 -0.3% -7.9% 1.2 1.35 +12.5% +125.4% -0.8% 10/10/5/5
100000 home.page_big (ms) 18.9 28.1 +49.1% 29.5 +56.4% -0.5% 28.9 37.6 +30.0% +45.0% +0.3% 10/10/5/5
100000 home.page_next (ms) 0.725 0.807 +11.3% 0.769 +6.0% -2.3% 1.05 1.02 -2.5% -9.1% +42.1% 10/10/5/5
100000 home.page_tag (ms) 52.4 52.5 +0.3% 52.5 +0.1% +0.3% 57.3 56.1 -2.2% -4.8% +4.1% 10/10/5/5
100000 home.page_title (ms) 0.859 0.858 -0.2% 0.783 -8.9% -4.0% 1.08 1.06 -2.2% -15.3% -7.6% 10/10/5/5
100000 mem.after_first_page (MB) 22.6 22.1 -1.9% 22 -2.4% +0.2% 22.6 22.1 -1.9% -2.4% +0.2% 10/10/5/5
100000 mem.after_opens (MB) 95.7 95.6 -0.0% 97.4 +1.8% -0.0% 95.7 95.6 -0.0% +1.8% -0.0% 10/10/5/5
100000 mem.after_search (MB) 103 91.4 -11.1% 93.6 -9.0% -2.9% 103 91.4 -11.1% -9.0% -2.9% 10/10/5/5
100000 mem.idle (MB) 18.4 18.8 +2.0% 18.7 +1.9% +0.8% 18.4 18.8 +2.0% +1.9% +0.8% 10/10/5/5
100000 notes.append (ms) 6.56 6.58 +0.4% 6.55 -0.1% +7.7% 20.8 19 -8.2% -9.8% -14.2% 10/10/5/5
100000 notes.append.first (ms) 990 1.15e+03 +15.7% 1.26e+03 +27.0% -3.5% 990 1.15e+03 +15.7% +27.0% -3.5% 10/10/5/5
100000 open.api_get (ms) 7.71 8.79 +14.1% 7.55 -2.0% -1.1% 14 32.1 +128.4% +76.5% -8.5% 10/10/5/5
100000 open.api_get.warm (ms) 0.165 0.165 +0.2% 0.171 +3.7% +5.9% 0.212 0.21 -1.1% +1.1% +4.1% 10/10/5/5
100000 open.long_note (ms) 68.8 67.3 -2.3% 74.9 +8.8% +2.5% 68.8 67.3 -2.3% +8.8% +2.5% 10/10/5/5
100000 open.note (ms) 1.47 0.905 -38.5% 0.896 -39.1% +21.2% 4.85 4.1 -15.5% -13.2% +40.7% 10/10/5/5
100000 search.catch_up (s) 46.1 46.1 -0.0% 46.1 -0.1% -0.0% 46.1 46.1 -0.0% -0.1% -0.0% 10/10/5/5
100000 search.first_query (ms) 27.4 26.7 -2.3% 27.4 -0.0% +1.6% 27.4 26.7 -2.3% -0.0% +1.6% 10/10/5/5
100000 search.query (ms) 2.51 1.92 -23.4% 1.97 -21.5% -29.8% 25.4 20.2 -20.3% -19.3% -24.3% 10/10/5/5
100000 shutdown (ms) 43.1 43.1 -0.0% 43.1 -0.1% -0.2% 43.1 43.1 -0.0% -0.1% -0.2% 10/10/5/5
100000 startup.healthz (ms) 4.86 4.99 +2.7% 4.86 -0.1% +1.3% 4.86 4.99 +2.7% -0.1% +1.3% 10/10/5/5
100000 sync.durable_write (ms) 24.8 25 +0.7% 24.8 -0.2% -0.3% 32.5 31.8 -2.4% -3.2% +4.7% 10/10/5/5
100000 sync.round_trip (ms) 0.216 0.208 -3.4% 0.217 +0.6% -3.1% 0.337 0.344 +2.1% +3.3% +5.6% 10/10/5/5
100000 sync.save_ack (ms) 3.7 3.67 -1.0% 3.73 +0.8% -1.3% 5.29 4.46 -15.7% -7.1% -5.7% 10/10/5/5
## Perf round 3: ABAB ×5 with a #970 arm and a dev-copy control (both perf locks, FIFO turn) Session 2026-11-09 15:56–16:56Z (21:26–22:26 IST), boot `7a65f072` (same epoch as the earlier sessions), perf-01 lock via `perf-turn files-1010` then perf-02's lock, both released at the end. Per cycle: dev (legacy layout), head (joined layout), #970 `381ce6750` (joined), dev, head, dev-copy control. Sizes 1k/10k/100k, `--runs 1 --cold-runs 1 --hot-only --gate`; medians of per-arm p50/p95, n = 10 dev, 10 head, 5 #970, 5 control. All 30 arms exited 0. sha256 checked at turn time. - head = PR head `0c99d74e9` (release built from `9520d6fed`; the two differ only in a `#[cfg(test)]` module) - dev = `d4b32af94` (the merge base) - **Change from earlier sessions:** head arms now run the server binary directly. The earlier sessions ran it through a bash wrapper, which forked bash before every server start. That wrapper was the constant +2 ms in `startup.healthz`: with it gone, healthz is flat at 1k and 100k. ### 1M listing gate `files_listing entries=1000000 page_size=50 cold_first_us=99 p50_us=58 p99_us=85 active_map_bytes=53085308 rss_after_open_kib=56548 peak_rss_kib=56548` First page under 100 ms: **pass** (0.099 ms). ### Full table | size | metric | dev p50 | head p50 | Δ head | #970 p50 | Δ #970 | ctl Δ | dev p95 | head p95 | Δ head p95 | Δ #970 p95 | ctl Δ p95 | n dev/head/970/ctl | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 1000 | cold.disk_written (MB) | 18.8 | 19.1 | +1.7% | 19 | +0.9% | +0.5% | 18.8 | 19.1 | +1.7% | +0.9% | +0.5% | 10/10/5/5 | | 1000 | cold.first_page (ms) | 406 | 491 | +20.9% | 441 | +8.5% | +0.5% | 406 | 491 | +20.9% | +8.5% | +0.5% | 10/10/5/5 | | 1000 | cold.index (s) | 0.509 | 0.593 | +16.6% | 0.543 | +6.7% | +0.3% | 0.509 | 0.593 | +16.6% | +6.7% | +0.3% | 10/10/5/5 | | 1000 | cold.mem_peak (MB) | 46 | 48.2 | +4.8% | 47.8 | +4.0% | +1.6% | 46 | 48.2 | +4.8% | +4.0% | +1.6% | 10/10/5/5 | | 1000 | cold.notes_per_s (notes/s) | 1.97e+03 | 1.69e+03 | -14.2% | 1.84e+03 | -6.3% | -0.3% | 1.97e+03 | 1.69e+03 | -14.2% | -6.3% | -0.3% | 10/10/5/5 | | 1000 | cold.startup (ms) | 19.8 | 19.8 | +0.0% | 19.8 | -0.1% | +0.6% | 19.8 | 19.8 | +0.0% | -0.1% | +0.6% | 10/10/5/5 | | 1000 | home.first_page (ms) | 1.75 | 2.33 | +33.4% | 2.66 | +52.6% | +4.5% | 1.75 | 2.33 | +33.4% | +52.6% | +4.5% | 10/10/5/5 | | 1000 | home.page (ms) | 0.68 | 0.713 | +4.8% | 0.72 | +5.8% | -5.6% | 1.21 | 1.91 | +57.1% | +5.8% | -1.8% | 10/10/5/5 | | 1000 | home.page_big (ms) | 8.66 | 8.88 | +2.5% | 8.86 | +2.2% | -1.7% | 8.66 | 8.88 | +2.5% | +2.2% | -1.7% | 10/10/5/5 | | 1000 | home.page_next (ms) | 0.726 | 0.736 | +1.4% | 0.727 | +0.1% | +1.3% | 0.815 | 0.816 | +0.1% | +4.7% | -2.4% | 10/10/5/5 | | 1000 | home.page_tag (ms) | 1.24 | 1.29 | +3.7% | 1.23 | -1.1% | -0.6% | 1.45 | 1.4 | -3.1% | +2.4% | -7.6% | 10/10/5/5 | | 1000 | home.page_title (ms) | 0.597 | 0.602 | +0.8% | 0.587 | -1.8% | -0.5% | 0.707 | 0.707 | -0.0% | -7.8% | -2.3% | 10/10/5/5 | | 1000 | mem.after_first_page (MB) | 20.9 | 20.8 | -0.1% | 21.3 | +1.9% | +2.9% | 20.9 | 20.8 | -0.1% | +1.9% | +2.9% | 10/10/5/5 | | 1000 | mem.after_opens (MB) | 34.6 | 35.4 | +2.4% | 34.6 | +0.1% | -0.1% | 34.6 | 35.4 | +2.4% | +0.1% | -0.1% | 10/10/5/5 | | 1000 | mem.after_search (MB) | 47.7 | 48.3 | +1.3% | 48 | +0.7% | +0.3% | 47.7 | 48.3 | +1.3% | +0.7% | +0.3% | 10/10/5/5 | | 1000 | mem.idle (MB) | 18.3 | 18.8 | +2.4% | 18.8 | +2.3% | +1.5% | 18.3 | 18.8 | +2.4% | +2.3% | +1.5% | 10/10/5/5 | | 1000 | notes.append (ms) | 4.84 | 4.78 | -1.2% | 4.86 | +0.4% | +0.0% | 12 | 12.5 | +4.5% | +6.6% | +2.2% | 10/10/5/5 | | 1000 | notes.append.first (ms) | 22.4 | 17 | -24.0% | 15.8 | -29.4% | -25.6% | 22.4 | 17 | -24.0% | -29.4% | -25.6% | 10/10/5/5 | | 1000 | open.api_get (ms) | 7.5 | 7.42 | -1.1% | 7.21 | -3.9% | -1.0% | 11.8 | 11.3 | -4.5% | +7.9% | +15.5% | 10/10/5/5 | | 1000 | open.api_get.warm (ms) | 0.163 | 0.164 | +0.8% | 0.166 | +2.1% | +0.1% | 0.213 | 0.209 | -1.9% | -2.3% | +0.6% | 10/10/5/5 | | 1000 | open.long_note (ms) | 15.1 | 15.2 | +0.9% | 14.8 | -1.4% | +3.3% | 15.1 | 15.2 | +0.9% | -1.4% | +3.3% | 10/10/5/5 | | 1000 | open.note (ms) | 0.66 | 0.656 | -0.5% | 0.674 | +2.2% | +4.4% | 2.26 | 2.21 | -2.2% | -5.1% | -3.2% | 10/10/5/5 | | 1000 | search.catch_up (s) | 2 | 2 | +0.0% | 2 | +0.0% | +0.0% | 2 | 2 | +0.0% | +0.0% | +0.0% | 10/10/5/5 | | 1000 | search.first_query (ms) | 28.1 | 26.7 | -4.9% | 26.4 | -6.2% | -3.3% | 28.1 | 26.7 | -4.9% | -6.2% | -3.3% | 10/10/5/5 | | 1000 | search.query (ms) | 1.25 | 1.23 | -1.5% | 1.23 | -1.8% | -9.5% | 2.34 | 2.17 | -7.2% | -6.4% | -10.2% | 10/10/5/5 | | 1000 | shutdown (ms) | 43.1 | 43.1 | -0.1% | 43.1 | -0.0% | -0.1% | 43.1 | 43.1 | -0.1% | -0.0% | -0.1% | 10/10/5/5 | | 1000 | startup.healthz (ms) | 4.94 | 4.96 | +0.4% | 4.84 | -2.0% | +3.8% | 4.94 | 4.96 | +0.4% | -2.0% | +3.8% | 10/10/5/5 | | 1000 | sync.durable_write (ms) | 24.9 | 24.7 | -0.8% | 24.8 | -0.4% | +0.5% | 30 | 28.2 | -5.9% | -5.0% | +13.0% | 10/10/5/5 | | 1000 | sync.round_trip (ms) | 0.201 | 0.206 | +2.4% | 0.198 | -1.4% | +6.9% | 0.325 | 0.339 | +4.5% | -2.4% | +6.9% | 10/10/5/5 | | 1000 | sync.save_ack (ms) | 3.66 | 3.65 | -0.3% | 3.74 | +2.3% | +0.5% | 4.78 | 5.04 | +5.4% | +10.8% | +5.0% | 10/10/5/5 | | 10000 | cold.disk_written (MB) | 150 | 152 | +1.4% | 152 | +1.4% | -0.1% | 150 | 152 | +1.4% | +1.4% | -0.1% | 10/10/5/5 | | 10000 | cold.first_page (ms) | 331 | 518 | +56.3% | 459 | +38.6% | -7.3% | 331 | 518 | +56.3% | +38.6% | -7.3% | 10/10/5/5 | | 10000 | cold.index (s) | 1.48 | 1.7 | +14.5% | 1.7 | +14.5% | -2.5% | 1.48 | 1.7 | +14.5% | +14.5% | -2.5% | 10/10/5/5 | | 10000 | cold.mem_peak (MB) | 143 | 145 | +1.4% | 145 | +1.1% | +0.3% | 143 | 145 | +1.4% | +1.1% | +0.3% | 10/10/5/5 | | 10000 | cold.notes_per_s (notes/s) | 6.75e+03 | 5.9e+03 | -12.7% | 5.9e+03 | -12.7% | +2.5% | 6.75e+03 | 5.9e+03 | -12.7% | -12.7% | +2.5% | 10/10/5/5 | | 10000 | cold.startup (ms) | 20 | 19.9 | -0.6% | 17.9 | -10.4% | -0.3% | 20 | 19.9 | -0.6% | -10.4% | -0.3% | 10/10/5/5 | | 10000 | home.first_page (ms) | 2.46 | 2.54 | +3.0% | 2.28 | -7.4% | -3.9% | 2.46 | 2.54 | +3.0% | -7.4% | -3.9% | 10/10/5/5 | | 10000 | home.page (ms) | 0.759 | 0.766 | +1.0% | 0.852 | +12.2% | +10.3% | 1.79 | 1.27 | -29.2% | -43.9% | +111.6% | 10/10/5/5 | | 10000 | home.page_big (ms) | 16.9 | 17 | +0.7% | 16.7 | -0.9% | -1.3% | 18 | 18.1 | +0.7% | -0.6% | -1.1% | 10/10/5/5 | | 10000 | home.page_next (ms) | 0.7 | 0.733 | +4.7% | 0.713 | +1.9% | -0.2% | 0.821 | 0.897 | +9.2% | +8.4% | +5.4% | 10/10/5/5 | | 10000 | home.page_tag (ms) | 5.32 | 5.57 | +4.8% | 5.55 | +4.3% | -0.7% | 7.53 | 7.94 | +5.5% | -2.7% | +1.5% | 10/10/5/5 | | 10000 | home.page_title (ms) | 0.634 | 0.724 | +14.3% | 0.746 | +17.7% | +3.6% | 0.833 | 1.4 | +68.6% | +14.7% | +7.0% | 10/10/5/5 | | 10000 | mem.after_first_page (MB) | 21.4 | 21.4 | +0.4% | 21.4 | -0.0% | -0.1% | 21.4 | 21.4 | +0.4% | -0.0% | -0.1% | 10/10/5/5 | | 10000 | mem.after_opens (MB) | 39.9 | 40.2 | +0.7% | 39.9 | -0.0% | +0.9% | 39.9 | 40.2 | +0.7% | -0.0% | +0.9% | 10/10/5/5 | | 10000 | mem.after_search (MB) | 75.3 | 75.6 | +0.4% | 74.7 | -0.8% | +0.2% | 75.3 | 75.6 | +0.4% | -0.8% | +0.2% | 10/10/5/5 | | 10000 | mem.idle (MB) | 18.5 | 19.3 | +4.0% | 18.9 | +1.7% | +0.1% | 18.5 | 19.3 | +4.0% | +1.7% | +0.1% | 10/10/5/5 | | 10000 | notes.append (ms) | 4.9 | 4.9 | +0.0% | 4.98 | +1.6% | +4.9% | 12 | 11.8 | -1.8% | +3.5% | +3.5% | 10/10/5/5 | | 10000 | notes.append.first (ms) | 55.8 | 52.1 | -6.6% | 48.5 | -13.2% | -8.7% | 55.8 | 52.1 | -6.6% | -13.2% | -8.7% | 10/10/5/5 | | 10000 | open.api_get (ms) | 7.45 | 7.34 | -1.5% | 7.39 | -0.8% | -2.9% | 13.3 | 12.2 | -8.4% | +17.6% | -4.5% | 10/10/5/5 | | 10000 | open.api_get.warm (ms) | 0.171 | 0.16 | -6.2% | 0.165 | -3.5% | -5.1% | 0.215 | 0.206 | -4.2% | +0.7% | -2.3% | 10/10/5/5 | | 10000 | open.long_note (ms) | 16.2 | 16.6 | +2.0% | 16.4 | +0.8% | +3.1% | 16.2 | 16.6 | +2.0% | +0.8% | +3.1% | 10/10/5/5 | | 10000 | open.note (ms) | 0.661 | 0.633 | -4.3% | 0.625 | -5.5% | -8.0% | 1.37 | 1.3 | -5.8% | +3.9% | -6.6% | 10/10/5/5 | | 10000 | search.catch_up (s) | 4.02 | 4.01 | -0.0% | 4.01 | -0.1% | -0.1% | 4.02 | 4.01 | -0.0% | -0.1% | -0.1% | 10/10/5/5 | | 10000 | search.first_query (ms) | 28.4 | 29.3 | +3.2% | 27.5 | -2.9% | -3.4% | 28.4 | 29.3 | +3.2% | -2.9% | -3.4% | 10/10/5/5 | | 10000 | search.query (ms) | 1.35 | 2.29 | +69.1% | 1.35 | -0.1% | +24.2% | 10.6 | 11.5 | +8.7% | -34.0% | -11.9% | 10/10/5/5 | | 10000 | shutdown (ms) | 43.1 | 43.2 | +0.1% | 43 | -0.2% | +0.0% | 43.1 | 43.2 | +0.1% | -0.2% | +0.0% | 10/10/5/5 | | 10000 | startup.healthz (ms) | 4.92 | 5.61 | +14.0% | 4.85 | -1.4% | -1.0% | 4.92 | 5.61 | +14.0% | -1.4% | -1.0% | 10/10/5/5 | | 10000 | sync.durable_write (ms) | 24.9 | 24.8 | -0.2% | 24.3 | -2.1% | -2.2% | 28.7 | 29 | +1.0% | +2.3% | +0.9% | 10/10/5/5 | | 10000 | sync.round_trip (ms) | 0.208 | 0.206 | -0.9% | 0.212 | +2.0% | -6.7% | 0.345 | 0.344 | -0.1% | -2.2% | -3.6% | 10/10/5/5 | | 10000 | sync.save_ack (ms) | 3.66 | 3.66 | -0.1% | 3.67 | +0.2% | -0.4% | 5.45 | 5.27 | -3.3% | -2.4% | -2.2% | 10/10/5/5 | | 100000 | cold.disk_written (MB) | 1.92e+03 | 2.06e+03 | +7.5% | 2.06e+03 | +7.3% | -0.0% | 1.92e+03 | 2.06e+03 | +7.5% | +7.3% | -0.0% | 10/10/5/5 | | 100000 | cold.first_page (ms) | 1.82e+03 | 4.15e+03 | +128.1% | 4.35e+03 | +139.5% | -2.1% | 1.82e+03 | 4.15e+03 | +128.1% | +139.5% | -2.1% | 10/10/5/5 | | 100000 | cold.index (s) | 15.6 | 18 | +15.6% | 17.9 | +14.9% | -0.0% | 15.6 | 18 | +15.6% | +14.9% | -0.0% | 10/10/5/5 | | 100000 | cold.mem_peak (MB) | 292 | 286 | -2.2% | 287 | -1.7% | +0.1% | 292 | 286 | -2.2% | -1.7% | +0.1% | 10/10/5/5 | | 100000 | cold.notes_per_s (notes/s) | 6.41e+03 | 5.55e+03 | -13.5% | 5.58e+03 | -13.0% | +0.0% | 6.41e+03 | 5.55e+03 | -13.5% | -13.0% | +0.0% | 10/10/5/5 | | 100000 | cold.startup (ms) | 20.8 | 19.9 | -4.7% | 19.8 | -4.8% | -3.8% | 20.8 | 19.9 | -4.7% | -4.8% | -3.8% | 10/10/5/5 | | 100000 | home.first_page (ms) | 3.21 | 3.19 | -0.7% | 3.1 | -3.3% | -4.9% | 3.21 | 3.19 | -0.7% | -3.3% | -4.9% | 10/10/5/5 | | 100000 | home.page (ms) | 0.952 | 0.957 | +0.5% | 0.95 | -0.3% | -7.9% | 1.2 | 1.35 | +12.5% | +125.4% | -0.8% | 10/10/5/5 | | 100000 | home.page_big (ms) | 18.9 | 28.1 | +49.1% | 29.5 | +56.4% | -0.5% | 28.9 | 37.6 | +30.0% | +45.0% | +0.3% | 10/10/5/5 | | 100000 | home.page_next (ms) | 0.725 | 0.807 | +11.3% | 0.769 | +6.0% | -2.3% | 1.05 | 1.02 | -2.5% | -9.1% | +42.1% | 10/10/5/5 | | 100000 | home.page_tag (ms) | 52.4 | 52.5 | +0.3% | 52.5 | +0.1% | +0.3% | 57.3 | 56.1 | -2.2% | -4.8% | +4.1% | 10/10/5/5 | | 100000 | home.page_title (ms) | 0.859 | 0.858 | -0.2% | 0.783 | -8.9% | -4.0% | 1.08 | 1.06 | -2.2% | -15.3% | -7.6% | 10/10/5/5 | | 100000 | mem.after_first_page (MB) | 22.6 | 22.1 | -1.9% | 22 | -2.4% | +0.2% | 22.6 | 22.1 | -1.9% | -2.4% | +0.2% | 10/10/5/5 | | 100000 | mem.after_opens (MB) | 95.7 | 95.6 | -0.0% | 97.4 | +1.8% | -0.0% | 95.7 | 95.6 | -0.0% | +1.8% | -0.0% | 10/10/5/5 | | 100000 | mem.after_search (MB) | 103 | 91.4 | -11.1% | 93.6 | -9.0% | -2.9% | 103 | 91.4 | -11.1% | -9.0% | -2.9% | 10/10/5/5 | | 100000 | mem.idle (MB) | 18.4 | 18.8 | +2.0% | 18.7 | +1.9% | +0.8% | 18.4 | 18.8 | +2.0% | +1.9% | +0.8% | 10/10/5/5 | | 100000 | notes.append (ms) | 6.56 | 6.58 | +0.4% | 6.55 | -0.1% | +7.7% | 20.8 | 19 | -8.2% | -9.8% | -14.2% | 10/10/5/5 | | 100000 | notes.append.first (ms) | 990 | 1.15e+03 | +15.7% | 1.26e+03 | +27.0% | -3.5% | 990 | 1.15e+03 | +15.7% | +27.0% | -3.5% | 10/10/5/5 | | 100000 | open.api_get (ms) | 7.71 | 8.79 | +14.1% | 7.55 | -2.0% | -1.1% | 14 | 32.1 | +128.4% | +76.5% | -8.5% | 10/10/5/5 | | 100000 | open.api_get.warm (ms) | 0.165 | 0.165 | +0.2% | 0.171 | +3.7% | +5.9% | 0.212 | 0.21 | -1.1% | +1.1% | +4.1% | 10/10/5/5 | | 100000 | open.long_note (ms) | 68.8 | 67.3 | -2.3% | 74.9 | +8.8% | +2.5% | 68.8 | 67.3 | -2.3% | +8.8% | +2.5% | 10/10/5/5 | | 100000 | open.note (ms) | 1.47 | 0.905 | -38.5% | 0.896 | -39.1% | +21.2% | 4.85 | 4.1 | -15.5% | -13.2% | +40.7% | 10/10/5/5 | | 100000 | search.catch_up (s) | 46.1 | 46.1 | -0.0% | 46.1 | -0.1% | -0.0% | 46.1 | 46.1 | -0.0% | -0.1% | -0.0% | 10/10/5/5 | | 100000 | search.first_query (ms) | 27.4 | 26.7 | -2.3% | 27.4 | -0.0% | +1.6% | 27.4 | 26.7 | -2.3% | -0.0% | +1.6% | 10/10/5/5 | | 100000 | search.query (ms) | 2.51 | 1.92 | -23.4% | 1.97 | -21.5% | -29.8% | 25.4 | 20.2 | -20.3% | -19.3% | -24.3% | 10/10/5/5 | | 100000 | shutdown (ms) | 43.1 | 43.1 | -0.0% | 43.1 | -0.1% | -0.2% | 43.1 | 43.1 | -0.0% | -0.1% | -0.2% | 10/10/5/5 | | 100000 | startup.healthz (ms) | 4.86 | 4.99 | +2.7% | 4.86 | -0.1% | +1.3% | 4.86 | 4.99 | +2.7% | -0.1% | +1.3% | 10/10/5/5 | | 100000 | sync.durable_write (ms) | 24.8 | 25 | +0.7% | 24.8 | -0.2% | -0.3% | 32.5 | 31.8 | -2.4% | -3.2% | +4.7% | 10/10/5/5 | | 100000 | sync.round_trip (ms) | 0.216 | 0.208 | -3.4% | 0.217 | +0.6% | -3.1% | 0.337 | 0.344 | +2.1% | +3.3% | +5.6% | 10/10/5/5 | | 100000 | sync.save_ack (ms) | 3.7 | 3.67 | -1.0% | 3.73 | +0.8% | -1.3% | 5.29 | 4.46 | -15.7% | -7.1% | -5.7% | 10/10/5/5 |
kayg commented 2026-11-09 16:59:19 +00:00
Author
Owner
Copy link

Verdict after 3 rounds, and a simpler design (stopping the patching here per the 3-round rule)

What the three rounds fixed (in this PR)

Round Problem found Fix Effect (100k unless noted)
1 Every Notes row wrote 5 B-trees (child table + its index + a whole-table scan index) listing via the path index, child table dropped, partial scan index (1dae71c03) cold.disk_written +16.6% → +7.5% (now = #970)
1 The Files walk held the Home's index lock across all directory I/O per-call index borrow, one transaction per walk page (86f2411ba) open.api_get p95 +345% → +128%
2 Every walk stat'ed all SSD Notes/*.md only to skip them skip from the listed type before stat (0c99d74e9) fewer hot-start stats
2 Bash wrapper in the perf harness dropped (measurement artifact) startup.healthz flat at 1k/100k

What still fails, and whose it is

Metric head vs dev #970 vs dev Owner
cold.first_page 100k +128% +140% #970: identity commits on the cold path (13 × 8,192-row random-key commits; see #970 comment)
cold.index, cold.notes_per_s 10k/100k +14.5–15.6% / −12.7–13.5% same #970
cold.disk_written 100k +7.5% +7.3% #970 (WAL rewrite of the identity table)
home.page_big p50 100k +49% +56% #970
notes.append.first 100k +16% +27% #970
cold.first_page 1k/10k +21% / +56% +8.5% / +39% ~50 ms is #1010's: the Home open now runs ~7 more schema statements, each its own fsync'd autocommit (synchronous=FULL), on the first-page path
cold.index 1k +16.6% +6.7% #1010 (same cause)
open.api_get p95 100k +128% +77% both; #1010's share is the hot-start Files walk overlapping requests
startup.healthz 10k +14% −1% likely noise (1k/100k flat); recheck

Because #1010 is stacked on #970, it can't be ≤ dev on the cold metrics until #970 is. The remaining #1010-only costs are both fixed costs at Home open and start, so the simpler design removes them instead of tuning them:

Simpler design for the rest of #1010

  1. One schema transaction. Create every namespace table and index in a single BEGIN … COMMIT (one fsync per Home open instead of ~9; also below #970's 2). Expected: the ~50 ms at 1k/10k goes away, cold.first_page/cold.index at small sizes ≤ #970.
  2. No full walk at hot start. Persist each folder's (mtime, ctime, inode) from the last completed walk. At start, stat folders only: a folder whose stamp is unchanged keeps its rows without being listed. Cost goes from O(entries) to O(folders) (the 100k-note fixture has a handful). Watchers cover changes while running, the 15-minute reconcile stays the safety net, and overflow still forces a full walk. This mirrors the Notes warm scan (scan-state.tsv), which already makes hot start cheap.
  3. Start-up walk off the request path. Any walk that must run (first ever, overflow, a folder stamp changed) starts only after the Home has had no requests for ~2 s, and yields between 256-row pages. Requests never queue behind it.
  4. Identity guarantees are unchanged: markers, inode evidence and the collision journal keep the same code. Only when a folder is re-listed changes, and a changed folder is always re-listed.

Big-O after: Home open O(1) fsyncs. Hot start O(folders) stats plus O(changed folders × entries). Request latency is independent of Home size while idle detection holds.

Gates for that round: identity tests (incl. the 2,000-iteration rescan and legacy_attachment_ids_survive_every_retry_boundary), a new test that an unchanged folder is not re-listed and a changed one is, workspace fmt/clippy/tests, the 1M listing gate, and the same ABAB ×5 with #970 and control arms.

Dependency: #970 needs its cold identity-load fix (ID-ordered bulk load and/or deferred checkpoint, evidence linked above) before #1010 can be ≤ dev on cold.* and home.page_big.

## Verdict after 3 rounds, and a simpler design (stopping the patching here per the 3-round rule) ### What the three rounds fixed (in this PR) | Round | Problem found | Fix | Effect (100k unless noted) | |---|---|---|---| | 1 | Every Notes row wrote 5 B-trees (child table + its index + a whole-table scan index) | listing via the path index, child table dropped, partial scan index (`1dae71c03`) | `cold.disk_written` +16.6% → +7.5% (now = #970) | | 1 | The Files walk held the Home's index lock across all directory I/O | per-call index borrow, one transaction per walk page (`86f2411ba`) | `open.api_get` p95 +345% → +128% | | 2 | Every walk stat'ed all SSD `Notes/*.md` only to skip them | skip from the listed type before `stat` (`0c99d74e9`) | fewer hot-start stats | | 2 | Bash wrapper in the perf harness | dropped (measurement artifact) | `startup.healthz` flat at 1k/100k | ### What still fails, and whose it is | Metric | head vs dev | #970 vs dev | Owner | |---|---|---|---| | `cold.first_page` 100k | +128% | +140% | **#970**: identity commits on the cold path (13 × 8,192-row random-key commits; see [#970 comment](https://git.kayg.org/kayg/calternal/pulls/970#issuecomment-88985)) | | `cold.index`, `cold.notes_per_s` 10k/100k | +14.5–15.6% / −12.7–13.5% | same | **#970** | | `cold.disk_written` 100k | +7.5% | +7.3% | **#970** (WAL rewrite of the identity table) | | `home.page_big` p50 100k | +49% | +56% | **#970** | | `notes.append.first` 100k | +16% | +27% | **#970** | | `cold.first_page` 1k/10k | +21% / +56% | +8.5% / +39% | ~50 ms is **#1010's**: the Home open now runs ~7 more schema statements, each its own fsync'd autocommit (`synchronous=FULL`), on the first-page path | | `cold.index` 1k | +16.6% | +6.7% | **#1010** (same cause) | | `open.api_get` p95 100k | +128% | +77% | both; #1010's share is the hot-start Files walk overlapping requests | | `startup.healthz` 10k | +14% | −1% | likely noise (1k/100k flat); recheck | Because #1010 is stacked on #970, it can't be ≤ dev on the cold metrics until #970 is. The remaining #1010-only costs are both fixed costs at Home open and start, so the simpler design removes them instead of tuning them: ### Simpler design for the rest of #1010 1. **One schema transaction.** Create every namespace table and index in a single `BEGIN … COMMIT` (one fsync per Home open instead of ~9; also below #970's 2). Expected: the ~50 ms at 1k/10k goes away, `cold.first_page`/`cold.index` at small sizes ≤ #970. 2. **No full walk at hot start.** Persist each folder's `(mtime, ctime, inode)` from the last completed walk. At start, `stat` folders only: a folder whose stamp is unchanged keeps its rows without being listed. Cost goes from O(entries) to O(folders) (the 100k-note fixture has a handful). Watchers cover changes while running, the 15-minute reconcile stays the safety net, and overflow still forces a full walk. This mirrors the Notes warm scan (`scan-state.tsv`), which already makes hot start cheap. 3. **Start-up walk off the request path.** Any walk that must run (first ever, overflow, a folder stamp changed) starts only after the Home has had no requests for ~2 s, and yields between 256-row pages. Requests never queue behind it. 4. Identity guarantees are unchanged: markers, inode evidence and the collision journal keep the same code. Only *when* a folder is re-listed changes, and a changed folder is always re-listed. Big-O after: Home open O(1) fsyncs. Hot start O(folders) stats plus O(changed folders × entries). Request latency is independent of Home size while idle detection holds. Gates for that round: identity tests (incl. the 2,000-iteration rescan and `legacy_attachment_ids_survive_every_retry_boundary`), a new test that an unchanged folder is not re-listed and a changed one is, workspace fmt/clippy/tests, the 1M listing gate, and the same ABAB ×5 with #970 and control arms. **Dependency:** #970 needs its cold identity-load fix (ID-ordered bulk load and/or deferred checkpoint, evidence linked above) before #1010 can be ≤ dev on `cold.*` and `home.page_big`.
kayg commented 2026-11-09 19:12:47 +00:00
Author
Owner
Copy link

Round 4: the simpler design is built (head 018a90d00)

Commits since round 3:

  • 69ec4addb perf(files): create the namespace schema in one transaction. One WAL sync per new Home instead of ~9 on the first-page path; none on an up-to-date database.
  • 7c7586f87 feat(files): restart trusts folder stamps; Files walks run only while quiet.
    • Each full walk records file stamps (size, mtime, ctime, device, inode) and, before listing each folder, that folder's stamp. Unchanged stamps are not rewritten.
    • At restart: stat every stamped folder (O(folders), no listing). If none changed since the last completed walk, listings are complete at once.
    • Correctness rule (owner's correction): folder stamps can't see a file edited in place, so they only stand in for listing names. A full walk always follows each restart, and every other walk (overflow, changed path, periodic) runs on one Files task per Home. It starts after 2 s with no requests and pauses whenever one arrives (each 1,024 entries, each folder, each page commit). The Notes watcher loop no longer waits on Files walks.
    • Racy timestamps: a folder changed within 2 s of its stamp gets a stamp that never matches, as git does for its index. A change in the same timestamp tick (a few ms on ext4/tmpfs, up to 1–2 s elsewhere) can't pass as "unchanged". A test caught this case by timing.
    • Listing completeness is one flag: proven by the restart check or a finished walk, cleared by any watcher report, overflow or poll. Before either proof, a restarted Home's listing says incomplete.
  • 018a90d00 merges #970's head 381ce6750 (no content change), so the #970 perf arm is an exact ancestor.

Tests added (all pass, 10/10 repeats of the restart set on a loaded host):

  • in-place edit while stopped: names judged current, then the quiet walk updates size and ctime, same ID;
  • rename within a folder while stopped: check refuses, walk keeps the ID under the new name;
  • delete in a subfolder while stopped: check refuses, walk drops the row;
  • the restart check needs a completed walk;
  • a just-changed folder's stamp is never trusted;
  • the walk waits for quiet;
  • folder stamps page in order and retire with their walk.

Gates on compute02 at 018a90d00:

Gate Result
cargo fmt --all --check pass
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings pass
cargo test -j4 --workspace --locked --no-fail-fast pass
commit 69ec4addb alone: cargo test -p calternal-files pass

The 1M listing gate and the ABAB ×5 (dev, head, #970, dev-copy control) run in the next perf-turn slot (queued behind tiering-v3).

## Round 4: the simpler design is built (head `018a90d00`) Commits since round 3: - `69ec4addb` **perf(files): create the namespace schema in one transaction**. One WAL sync per new Home instead of ~9 on the first-page path; none on an up-to-date database. - `7c7586f87` **feat(files): restart trusts folder stamps; Files walks run only while quiet.** - Each full walk records file stamps (size, mtime, ctime, device, inode) and, *before* listing each folder, that folder's stamp. Unchanged stamps are not rewritten. - At restart: `stat` every stamped folder (O(folders), no listing). If none changed since the last completed walk, listings are complete at once. - **Correctness rule (owner's correction):** folder stamps can't see a file edited in place, so they only stand in for listing names. A full walk always follows each restart, and every other walk (overflow, changed path, periodic) runs on one Files task per Home. It starts after 2 s with no requests and pauses whenever one arrives (each 1,024 entries, each folder, each page commit). The Notes watcher loop no longer waits on Files walks. - **Racy timestamps:** a folder changed within 2 s of its stamp gets a stamp that never matches, as git does for its index. A change in the same timestamp tick (a few ms on ext4/tmpfs, up to 1–2 s elsewhere) can't pass as "unchanged". A test caught this case by timing. - Listing completeness is one flag: proven by the restart check or a finished walk, cleared by any watcher report, overflow or poll. Before either proof, a restarted Home's listing says incomplete. - `018a90d00` merges #970's head `381ce6750` (no content change), so the #970 perf arm is an exact ancestor. Tests added (all pass, 10/10 repeats of the restart set on a loaded host): - in-place edit while stopped: names judged current, then the quiet walk updates size and ctime, same ID; - rename within a folder while stopped: check refuses, walk keeps the ID under the new name; - delete in a subfolder while stopped: check refuses, walk drops the row; - the restart check needs a completed walk; - a just-changed folder's stamp is never trusted; - the walk waits for quiet; - folder stamps page in order and retire with their walk. Gates on compute02 at `018a90d00`: | Gate | Result | |---|---| | `cargo fmt --all --check` | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | | `cargo test -j4 --workspace --locked --no-fail-fast` | pass | | commit `69ec4addb` alone: `cargo test -p calternal-files` | pass | The 1M listing gate and the ABAB ×5 (dev, head, #970, dev-copy control) run in the next `perf-turn` slot (queued behind tiering-v3).
kayg commented 2026-11-09 23:45:35 +00:00
Author
Owner
Copy link

Perf round 4a: ABAB ×5, dev / head / #970 / dev-copy control (head 018a90d00)

Session 2026-11-09 19:41–20:37Z (01:11–02:07 IST), boot 7a65f072 (same epoch as rounds 1–3). Plain perf-turn files-1010 (FIFO) took perf-01's lock, then the control host took perf-02's; both released at the end. Per cycle: dev d4b32af94 (legacy), head, #970 381ce6750 (joined, an exact ancestor of head), dev, head, dev-copy control. Sizes 1k/10k/100k, --runs 1 --cold-runs 1 --hot-only --gate, fixtures generated from the same seed and verified by sha256. Medians of per-arm p50/p95. All 30 arms exited 0.

1M listing gate: files_listing entries=1000000 page_size=50 cold_first_us=101 p50_us=60 p99_us=91. Pass (first page 0.10 ms ≤ 100 ms).

#1010 share (head vs #970):

  • Cold path is flat: cold.first_page −1.0% / −1.1% / −0.4%, cold.index −0.9% / −0.3% / +1.6%, cold.disk_written +0.2% / −0.1% / +0.1% (1k/10k/100k). The one-transaction schema removed round 3's ~50 ms at 1k/10k.
  • Unresolved, list path: home.page_big p50 @100k +16% (p95 −5.6%) and home.page p50 +6–12%. Per-arm ranges overlap (page_big head 23.6–36.8 ms, #970 22.2–27.0 ms; rank test p≈0.06). The same #970 binary measured page_big at 29.5 ms in round 3 and 25.5 ms here, a swing larger than the share. These requests run in the first seconds after start under continuous load, when the deferred Files walk is still waiting for quiet. The dev-copy control can't size joined-layout noise, so round 4b adds a head-copy control and a second #970 arm per cycle (#970 n=10).
  • Healthz moves ±19% with a ±19% control: noise.

Head vs dev stays red on the cold path and on home.page_big @100k. Those match #970 alone (cold.first_page @100k +105% vs #970 +106%), so #1010 cannot go READY before #970 is ≤ dev. Status: #1010-share flat on cold; list path pending round 4b.

size metric dev p50 head p50 head vs dev #970 vs dev #1010 share (head vs #970) control vs dev head vs dev p95 #1010 share p95 control p95
1000 cold.disk_written (MB) 18.8 19 +1.2% +1.0% +0.2% -0.2% +1.2% +0.2% -0.2%
1000 cold.first_page (ms) 418 443 +6.1% +7.2% -1.0% +3.6% +6.1% -1.0% +3.6%
1000 cold.index (s) 0.52 0.545 +4.9% +5.8% -0.9% +3.0% +4.9% -0.9% +3.0%
1000 cold.mem_peak (MB) 46.4 47.7 +2.7% +4.0% -1.3% -0.2% +2.7% -1.3% -0.2%
1000 cold.notes_per_s (notes/s) 1.92e+03 1.83e+03 -4.7% -5.5% +0.9% -2.9% -4.7% +0.9% -2.9%
1000 cold.startup (ms) 19.8 20.5 +3.3% +0.6% +2.6% +4.0% +3.3% +2.6% +4.0%
1000 home.first_page (ms) 2.46 1.95 -20.6% -28.4% +10.9% -27.5% -20.6% +10.9% -27.5%
1000 home.page (ms) 0.68 0.779 +14.7% +7.4% +6.8% -3.0% +16.7% -2.0% +4.4%
1000 home.page_big (ms) 8.95 8.81 -1.6% -2.6% +1.1% -2.5% -1.6% +1.1% -2.5%
1000 home.page_next (ms) 0.74 0.726 -1.8% -2.6% +0.7% +0.1% -4.0% +3.5% -0.3%
1000 home.page_tag (ms) 1.27 1.25 -1.5% -4.0% +2.6% -4.3% -7.8% +0.4% -10.6%
1000 home.page_title (ms) 0.617 0.609 -1.2% -2.6% +1.4% -1.9% +7.3% +6.7% +0.6%
1000 mem.after_first_page (MB) 21.2 21 -0.9% -1.6% +0.7% +2.3% -0.9% +0.7% +2.3%
1000 mem.after_opens (MB) 34.7 35 +1.0% -0.1% +1.1% +0.2% +1.0% +1.1% +0.2%
1000 mem.after_search (MB) 48.1 48.1 +0.2% -0.4% +0.6% -0.3% +0.2% +0.6% -0.3%
1000 mem.idle (MB) 18.3 18.9 +3.3% +3.3% +0.0% +9.8% +3.3% +0.0% +9.8%
1000 notes.append (ms) 4.88 4.92 +0.8% +0.8% -0.0% -0.4% +0.3% +1.4% +0.3%
1000 notes.append.first (ms) 17 15.5 -8.6% +11.6% -18.1% +18.5% -8.6% -18.1% +18.5%
1000 open.api_get (ms) 7.45 7.31 -1.9% -3.0% +1.2% +0.5% -1.2% -9.8% +6.8%
1000 open.api_get.warm (ms) 0.165 0.17 +3.0% -1.6% +4.7% +4.5% -1.5% +1.7% +0.9%
1000 open.long_note (ms) 15.5 15.4 -0.8% -0.7% -0.1% -1.2% -0.8% -0.1% -1.2%
1000 open.note (ms) 0.707 0.679 -3.9% -4.5% +0.6% -6.1% -6.6% -1.4% -6.9%
1000 search.catch_up (s) 2 2 -0.0% -0.0% +0.0% +0.0% -0.0% +0.0% +0.0%
1000 search.first_query (ms) 27 26.8 -0.5% -2.4% +1.9% +19.5% -0.5% +1.9% +19.5%
1000 search.query (ms) 1.15 1.23 +6.9% +0.5% +6.4% +7.8% +6.4% +5.8% +9.3%
1000 shutdown (ms) 43.1 43 -0.2% -49.0% +95.5% -48.8% -0.2% +95.5% -48.8%
1000 startup.healthz (ms) 4.92 5.76 +17.2% -1.9% +19.5% +19.3% +17.2% +19.5% +19.3%
1000 sync.durable_write (ms) 24.6 24.7 +0.5% +0.7% -0.1% -2.4% +0.2% -11.6% -4.8%
1000 sync.round_trip (ms) 0.211 0.202 -4.5% -4.4% -0.1% -4.1% -5.4% -1.1% -10.1%
1000 sync.save_ack (ms) 3.68 3.68 -0.1% +0.6% -0.6% -2.2% -2.7% -8.9% +2.6%
10000 cold.disk_written (MB) 150 152 +1.4% +1.5% -0.1% +0.1% +1.4% -0.1% +0.1%
10000 cold.first_page (ms) 331 475 +43.3% +44.9% -1.1% -2.0% +43.3% -1.1% -2.0%
10000 cold.index (s) 1.47 1.63 +10.7% +11.0% -0.3% +0.1% +10.7% -0.3% +0.1%
10000 cold.mem_peak (MB) 144 145 +0.9% +0.7% +0.2% -0.2% +0.9% +0.2% -0.2%
10000 cold.notes_per_s (notes/s) 6.81e+03 6.15e+03 -9.6% -9.9% +0.3% -0.1% -9.6% +0.3% -0.1%
10000 cold.startup (ms) 20 20.3 +1.6% -0.1% +1.6% +0.9% +1.6% +1.6% +0.9%
10000 home.first_page (ms) 2.25 2.47 +9.7% +0.8% +8.8% +9.0% +9.7% +8.8% +9.0%
10000 home.page (ms) 0.745 0.89 +19.5% +12.5% +6.2% +1.4% -33.2% +11.9% -0.1%
10000 home.page_big (ms) 16.5 16.3 -1.3% -1.0% -0.3% +0.9% +1.1% +2.9% +1.6%
10000 home.page_next (ms) 0.696 0.724 +4.1% +4.8% -0.7% +1.7% +13.7% -5.7% -0.3%
10000 home.page_tag (ms) 5.26 5.47 +3.8% +1.8% +2.0% +1.1% +12.6% +10.4% +9.6%
10000 home.page_title (ms) 0.653 0.73 +11.9% +12.0% -0.1% -0.2% +75.2% +39.9% +7.5%
10000 mem.after_first_page (MB) 21.6 21.4 -0.9% -0.3% -0.5% -1.7% -0.9% -0.5% -1.7%
10000 mem.after_opens (MB) 40.7 40.5 -0.7% -1.7% +1.0% -0.4% -0.7% +1.0% -0.4%
10000 mem.after_search (MB) 75.4 75.6 +0.3% -1.1% +1.3% +0.3% +0.3% +1.3% +0.3%
10000 mem.idle (MB) 19.3 18.8 -2.8% -0.8% -2.0% -3.4% -2.8% -2.0% -3.4%
10000 notes.append (ms) 5.07 5.02 -1.0% -1.0% -0.0% -4.0% -3.1% +2.9% +1.7%
10000 notes.append.first (ms) 57 52.6 -7.7% -18.9% +13.8% -16.6% -7.7% +13.8% -16.6%
10000 open.api_get (ms) 7.47 7.49 +0.2% -4.4% +4.9% -0.2% -1.6% -8.2% -4.0%
10000 open.api_get.warm (ms) 0.163 0.166 +1.5% +0.8% +0.7% -0.8% +3.7% +5.4% +3.5%
10000 open.long_note (ms) 16.1 16.2 +1.0% +3.0% -2.0% +0.7% +1.0% -2.0% +0.7%
10000 open.note (ms) 0.615 0.614 -0.1% +1.1% -1.2% +3.0% -1.9% -0.4% -7.1%
10000 search.catch_up (s) 4.01 4.01 +0.1% +0.1% -0.1% +0.0% +0.1% -0.1% +0.0%
10000 search.first_query (ms) 28 28.4 +1.4% -3.9% +5.5% -4.3% +1.4% +5.5% -4.3%
10000 search.query (ms) 1.43 1.49 +4.4% +14.7% -9.0% -0.1% +62.2% -8.2% +79.9%
10000 shutdown (ms) 43.1 43.4 +0.6% -0.3% +0.8% -0.3% +0.6% +0.8% -0.3%
10000 startup.healthz (ms) 4.99 4.9 -1.7% +21.3% -19.0% -1.6% -1.7% -19.0% -1.6%
10000 sync.durable_write (ms) 24.2 24.6 +1.6% +1.0% +0.5% +0.9% -6.5% -4.4% +3.6%
10000 sync.round_trip (ms) 0.204 0.204 +0.1% +0.4% -0.2% +6.5% +1.6% +0.7% +2.8%
10000 sync.save_ack (ms) 3.63 3.71 +2.1% -0.2% +2.3% +2.3% +0.7% +8.0% -4.8%
100000 cold.disk_written (MB) 1.92e+03 2.06e+03 +7.5% +7.3% +0.1% -0.0% +7.5% +0.1% -0.0%
100000 cold.first_page (ms) 2.12e+03 4.35e+03 +105.1% +105.9% -0.4% -1.4% +105.1% -0.4% -1.4%
100000 cold.index (s) 15.5 17.8 +15.3% +13.5% +1.6% -0.2% +15.3% +1.6% -0.2%
100000 cold.mem_peak (MB) 293 285 -2.5% -1.0% -1.5% +0.2% -2.5% -1.5% +0.2%
100000 cold.notes_per_s (notes/s) 6.46e+03 5.61e+03 -13.3% -11.9% -1.5% +0.2% -13.3% -1.5% +0.2%
100000 cold.startup (ms) 20 19.9 -0.3% +0.0% -0.3% -0.7% -0.3% -0.3% -0.7%
100000 home.first_page (ms) 3.19 2.92 -8.4% -8.1% -0.4% +64.6% -8.4% -0.4% +64.6%
100000 home.page (ms) 0.898 0.973 +8.3% -3.0% +11.7% +17.0% -1.3% -0.3% +186.2%
100000 home.page_big (ms) 18.9 29.6 +56.8% +35.1% +16.1% +2.3% +63.2% -5.6% +11.1%
100000 home.page_next (ms) 0.745 0.854 +14.6% +16.0% -1.2% -0.6% -11.3% +15.6% -27.1%
100000 home.page_tag (ms) 52.7 51.8 -1.7% -1.5% -0.2% -0.7% -3.2% +2.6% +4.4%
100000 home.page_title (ms) 0.838 0.85 +1.4% -9.6% +12.1% +3.2% +8.7% +4.7% +8.8%
100000 mem.after_first_page (MB) 22.9 22.2 -3.1% -4.3% +1.2% +3.4% -3.1% +1.2% +3.4%
100000 mem.after_opens (MB) 98.4 98.9 +0.6% -1.5% +2.1% -2.3% +0.6% +2.1% -2.3%
100000 mem.after_search (MB) 98.8 95.6 -3.2% -3.4% +0.2% +1.3% -3.2% +0.2% +1.3%
100000 mem.idle (MB) 18.9 18.8 -0.4% -1.1% +0.7% +5.0% -0.4% +0.7% +5.0%
100000 notes.append (ms) 4.88 5.04 +3.2% +0.1% +3.1% +1.6% +1.3% -2.8% +5.4%
100000 notes.append.first (ms) 1.23e+03 1.21e+03 -1.9% -8.9% +7.7% +14.4% -1.9% +7.7% +14.4%
100000 open.api_get (ms) 7.6 7.73 +1.8% -3.0% +4.9% -0.7% +135.1% +6.3% -4.4%
100000 open.api_get.warm (ms) 0.168 0.169 +0.9% +0.1% +0.8% -2.0% +2.4% +4.7% -2.3%
100000 open.long_note (ms) 68.2 67 -1.8% -2.4% +0.6% -4.3% -1.8% +0.6% -4.3%
100000 open.note (ms) 0.794 0.903 +13.7% +18.5% -4.1% -4.8% +8.4% -10.1% -4.2%
100000 search.catch_up (s) 47.1 46.1 -2.1% -2.1% -0.0% -2.1% -2.1% -0.0% -2.1%
100000 search.first_query (ms) 27.5 27.6 +0.4% -1.4% +1.8% -0.4% +0.4% +1.8% -0.4%
100000 search.query (ms) 2 1.98 -0.7% +5.2% -5.7% +98.6% +7.0% +46.8% -2.0%
100000 shutdown (ms) 85.3 85.4 +0.1% +0.0% +0.1% +0.1% +0.1% +0.1% +0.1%
100000 startup.healthz (ms) 4.9 5.07 +3.4% -1.5% +5.0% +15.2% +3.4% +5.0% +15.2%
100000 sync.durable_write (ms) 24.6 24.9 +1.2% +3.9% -2.6% +0.6% -5.0% +13.7% +2.5%
100000 sync.round_trip (ms) 0.221 0.215 -2.5% -2.4% -0.1% +2.2% -1.3% -0.9% +3.7%
100000 sync.save_ack (ms) 3.72 3.68 -1.0% -0.1% -1.0% -1.6% -2.1% +0.0% -5.5%
## Perf round 4a: ABAB ×5, dev / head / #970 / dev-copy control (head `018a90d00`) Session 2026-11-09 19:41–20:37Z (01:11–02:07 IST), boot `7a65f072` (same epoch as rounds 1–3). Plain `perf-turn files-1010` (FIFO) took perf-01's lock, then the control host took perf-02's; both released at the end. Per cycle: dev `d4b32af94` (legacy), head, #970 `381ce6750` (joined, an exact ancestor of head), dev, head, dev-copy control. Sizes 1k/10k/100k, `--runs 1 --cold-runs 1 --hot-only --gate`, fixtures generated from the same seed and verified by sha256. Medians of per-arm p50/p95. All 30 arms exited 0. **1M listing gate:** `files_listing entries=1000000 page_size=50 cold_first_us=101 p50_us=60 p99_us=91`. Pass (first page 0.10 ms ≤ 100 ms). **#1010 share (head vs #970):** - **Cold path is flat:** `cold.first_page` −1.0% / −1.1% / −0.4%, `cold.index` −0.9% / −0.3% / +1.6%, `cold.disk_written` +0.2% / −0.1% / +0.1% (1k/10k/100k). The one-transaction schema removed round 3's ~50 ms at 1k/10k. - **Unresolved, list path:** `home.page_big` p50 @100k +16% (p95 −5.6%) and `home.page` p50 +6–12%. Per-arm ranges overlap (page_big head 23.6–36.8 ms, #970 22.2–27.0 ms; rank test p≈0.06). The same #970 binary measured page_big at 29.5 ms in round 3 and 25.5 ms here, a swing larger than the share. These requests run in the first seconds after start under continuous load, when the deferred Files walk is still waiting for quiet. The dev-copy control can't size joined-layout noise, so round 4b adds a **head-copy control** and a second #970 arm per cycle (#970 n=10). - Healthz moves ±19% with a ±19% control: noise. **Head vs dev** stays red on the cold path and on `home.page_big` @100k. Those match #970 alone (cold.first_page @100k +105% vs #970 +106%), so #1010 cannot go READY before #970 is ≤ dev. Status: **#1010-share flat on cold; list path pending round 4b.** | size | metric | dev p50 | head p50 | head vs dev | #970 vs dev | **#1010 share** (head vs #970) | control vs dev | head vs dev p95 | #1010 share p95 | control p95 | |---|---|---|---|---|---|---|---|---|---|---| | 1000 | cold.disk_written (MB) | 18.8 | 19 | +1.2% | +1.0% | +0.2% | -0.2% | +1.2% | +0.2% | -0.2% | | 1000 | cold.first_page (ms) | 418 | 443 | +6.1% | +7.2% | -1.0% | +3.6% | +6.1% | -1.0% | +3.6% | | 1000 | cold.index (s) | 0.52 | 0.545 | +4.9% | +5.8% | -0.9% | +3.0% | +4.9% | -0.9% | +3.0% | | 1000 | cold.mem_peak (MB) | 46.4 | 47.7 | +2.7% | +4.0% | -1.3% | -0.2% | +2.7% | -1.3% | -0.2% | | 1000 | cold.notes_per_s (notes/s) | 1.92e+03 | 1.83e+03 | -4.7% | -5.5% | +0.9% | -2.9% | -4.7% | +0.9% | -2.9% | | 1000 | cold.startup (ms) | 19.8 | 20.5 | +3.3% | +0.6% | +2.6% | +4.0% | +3.3% | +2.6% | +4.0% | | 1000 | home.first_page (ms) | 2.46 | 1.95 | -20.6% | -28.4% | +10.9% | -27.5% | -20.6% | +10.9% | -27.5% | | 1000 | home.page (ms) | 0.68 | 0.779 | +14.7% | +7.4% | +6.8% | -3.0% | +16.7% | -2.0% | +4.4% | | 1000 | home.page_big (ms) | 8.95 | 8.81 | -1.6% | -2.6% | +1.1% | -2.5% | -1.6% | +1.1% | -2.5% | | 1000 | home.page_next (ms) | 0.74 | 0.726 | -1.8% | -2.6% | +0.7% | +0.1% | -4.0% | +3.5% | -0.3% | | 1000 | home.page_tag (ms) | 1.27 | 1.25 | -1.5% | -4.0% | +2.6% | -4.3% | -7.8% | +0.4% | -10.6% | | 1000 | home.page_title (ms) | 0.617 | 0.609 | -1.2% | -2.6% | +1.4% | -1.9% | +7.3% | +6.7% | +0.6% | | 1000 | mem.after_first_page (MB) | 21.2 | 21 | -0.9% | -1.6% | +0.7% | +2.3% | -0.9% | +0.7% | +2.3% | | 1000 | mem.after_opens (MB) | 34.7 | 35 | +1.0% | -0.1% | +1.1% | +0.2% | +1.0% | +1.1% | +0.2% | | 1000 | mem.after_search (MB) | 48.1 | 48.1 | +0.2% | -0.4% | +0.6% | -0.3% | +0.2% | +0.6% | -0.3% | | 1000 | mem.idle (MB) | 18.3 | 18.9 | +3.3% | +3.3% | +0.0% | +9.8% | +3.3% | +0.0% | +9.8% | | 1000 | notes.append (ms) | 4.88 | 4.92 | +0.8% | +0.8% | -0.0% | -0.4% | +0.3% | +1.4% | +0.3% | | 1000 | notes.append.first (ms) | 17 | 15.5 | -8.6% | +11.6% | -18.1% | +18.5% | -8.6% | -18.1% | +18.5% | | 1000 | open.api_get (ms) | 7.45 | 7.31 | -1.9% | -3.0% | +1.2% | +0.5% | -1.2% | -9.8% | +6.8% | | 1000 | open.api_get.warm (ms) | 0.165 | 0.17 | +3.0% | -1.6% | +4.7% | +4.5% | -1.5% | +1.7% | +0.9% | | 1000 | open.long_note (ms) | 15.5 | 15.4 | -0.8% | -0.7% | -0.1% | -1.2% | -0.8% | -0.1% | -1.2% | | 1000 | open.note (ms) | 0.707 | 0.679 | -3.9% | -4.5% | +0.6% | -6.1% | -6.6% | -1.4% | -6.9% | | 1000 | search.catch_up (s) | 2 | 2 | -0.0% | -0.0% | +0.0% | +0.0% | -0.0% | +0.0% | +0.0% | | 1000 | search.first_query (ms) | 27 | 26.8 | -0.5% | -2.4% | +1.9% | +19.5% | -0.5% | +1.9% | +19.5% | | 1000 | search.query (ms) | 1.15 | 1.23 | +6.9% | +0.5% | +6.4% | +7.8% | +6.4% | +5.8% | +9.3% | | 1000 | shutdown (ms) | 43.1 | 43 | -0.2% | -49.0% | +95.5% | -48.8% | -0.2% | +95.5% | -48.8% | | 1000 | startup.healthz (ms) | 4.92 | 5.76 | +17.2% | -1.9% | +19.5% | +19.3% | +17.2% | +19.5% | +19.3% | | 1000 | sync.durable_write (ms) | 24.6 | 24.7 | +0.5% | +0.7% | -0.1% | -2.4% | +0.2% | -11.6% | -4.8% | | 1000 | sync.round_trip (ms) | 0.211 | 0.202 | -4.5% | -4.4% | -0.1% | -4.1% | -5.4% | -1.1% | -10.1% | | 1000 | sync.save_ack (ms) | 3.68 | 3.68 | -0.1% | +0.6% | -0.6% | -2.2% | -2.7% | -8.9% | +2.6% | | 10000 | cold.disk_written (MB) | 150 | 152 | +1.4% | +1.5% | -0.1% | +0.1% | +1.4% | -0.1% | +0.1% | | 10000 | cold.first_page (ms) | 331 | 475 | +43.3% | +44.9% | -1.1% | -2.0% | +43.3% | -1.1% | -2.0% | | 10000 | cold.index (s) | 1.47 | 1.63 | +10.7% | +11.0% | -0.3% | +0.1% | +10.7% | -0.3% | +0.1% | | 10000 | cold.mem_peak (MB) | 144 | 145 | +0.9% | +0.7% | +0.2% | -0.2% | +0.9% | +0.2% | -0.2% | | 10000 | cold.notes_per_s (notes/s) | 6.81e+03 | 6.15e+03 | -9.6% | -9.9% | +0.3% | -0.1% | -9.6% | +0.3% | -0.1% | | 10000 | cold.startup (ms) | 20 | 20.3 | +1.6% | -0.1% | +1.6% | +0.9% | +1.6% | +1.6% | +0.9% | | 10000 | home.first_page (ms) | 2.25 | 2.47 | +9.7% | +0.8% | +8.8% | +9.0% | +9.7% | +8.8% | +9.0% | | 10000 | home.page (ms) | 0.745 | 0.89 | +19.5% | +12.5% | +6.2% | +1.4% | -33.2% | +11.9% | -0.1% | | 10000 | home.page_big (ms) | 16.5 | 16.3 | -1.3% | -1.0% | -0.3% | +0.9% | +1.1% | +2.9% | +1.6% | | 10000 | home.page_next (ms) | 0.696 | 0.724 | +4.1% | +4.8% | -0.7% | +1.7% | +13.7% | -5.7% | -0.3% | | 10000 | home.page_tag (ms) | 5.26 | 5.47 | +3.8% | +1.8% | +2.0% | +1.1% | +12.6% | +10.4% | +9.6% | | 10000 | home.page_title (ms) | 0.653 | 0.73 | +11.9% | +12.0% | -0.1% | -0.2% | +75.2% | +39.9% | +7.5% | | 10000 | mem.after_first_page (MB) | 21.6 | 21.4 | -0.9% | -0.3% | -0.5% | -1.7% | -0.9% | -0.5% | -1.7% | | 10000 | mem.after_opens (MB) | 40.7 | 40.5 | -0.7% | -1.7% | +1.0% | -0.4% | -0.7% | +1.0% | -0.4% | | 10000 | mem.after_search (MB) | 75.4 | 75.6 | +0.3% | -1.1% | +1.3% | +0.3% | +0.3% | +1.3% | +0.3% | | 10000 | mem.idle (MB) | 19.3 | 18.8 | -2.8% | -0.8% | -2.0% | -3.4% | -2.8% | -2.0% | -3.4% | | 10000 | notes.append (ms) | 5.07 | 5.02 | -1.0% | -1.0% | -0.0% | -4.0% | -3.1% | +2.9% | +1.7% | | 10000 | notes.append.first (ms) | 57 | 52.6 | -7.7% | -18.9% | +13.8% | -16.6% | -7.7% | +13.8% | -16.6% | | 10000 | open.api_get (ms) | 7.47 | 7.49 | +0.2% | -4.4% | +4.9% | -0.2% | -1.6% | -8.2% | -4.0% | | 10000 | open.api_get.warm (ms) | 0.163 | 0.166 | +1.5% | +0.8% | +0.7% | -0.8% | +3.7% | +5.4% | +3.5% | | 10000 | open.long_note (ms) | 16.1 | 16.2 | +1.0% | +3.0% | -2.0% | +0.7% | +1.0% | -2.0% | +0.7% | | 10000 | open.note (ms) | 0.615 | 0.614 | -0.1% | +1.1% | -1.2% | +3.0% | -1.9% | -0.4% | -7.1% | | 10000 | search.catch_up (s) | 4.01 | 4.01 | +0.1% | +0.1% | -0.1% | +0.0% | +0.1% | -0.1% | +0.0% | | 10000 | search.first_query (ms) | 28 | 28.4 | +1.4% | -3.9% | +5.5% | -4.3% | +1.4% | +5.5% | -4.3% | | 10000 | search.query (ms) | 1.43 | 1.49 | +4.4% | +14.7% | -9.0% | -0.1% | +62.2% | -8.2% | +79.9% | | 10000 | shutdown (ms) | 43.1 | 43.4 | +0.6% | -0.3% | +0.8% | -0.3% | +0.6% | +0.8% | -0.3% | | 10000 | startup.healthz (ms) | 4.99 | 4.9 | -1.7% | +21.3% | -19.0% | -1.6% | -1.7% | -19.0% | -1.6% | | 10000 | sync.durable_write (ms) | 24.2 | 24.6 | +1.6% | +1.0% | +0.5% | +0.9% | -6.5% | -4.4% | +3.6% | | 10000 | sync.round_trip (ms) | 0.204 | 0.204 | +0.1% | +0.4% | -0.2% | +6.5% | +1.6% | +0.7% | +2.8% | | 10000 | sync.save_ack (ms) | 3.63 | 3.71 | +2.1% | -0.2% | +2.3% | +2.3% | +0.7% | +8.0% | -4.8% | | 100000 | cold.disk_written (MB) | 1.92e+03 | 2.06e+03 | +7.5% | +7.3% | +0.1% | -0.0% | +7.5% | +0.1% | -0.0% | | 100000 | cold.first_page (ms) | 2.12e+03 | 4.35e+03 | +105.1% | +105.9% | -0.4% | -1.4% | +105.1% | -0.4% | -1.4% | | 100000 | cold.index (s) | 15.5 | 17.8 | +15.3% | +13.5% | +1.6% | -0.2% | +15.3% | +1.6% | -0.2% | | 100000 | cold.mem_peak (MB) | 293 | 285 | -2.5% | -1.0% | -1.5% | +0.2% | -2.5% | -1.5% | +0.2% | | 100000 | cold.notes_per_s (notes/s) | 6.46e+03 | 5.61e+03 | -13.3% | -11.9% | -1.5% | +0.2% | -13.3% | -1.5% | +0.2% | | 100000 | cold.startup (ms) | 20 | 19.9 | -0.3% | +0.0% | -0.3% | -0.7% | -0.3% | -0.3% | -0.7% | | 100000 | home.first_page (ms) | 3.19 | 2.92 | -8.4% | -8.1% | -0.4% | +64.6% | -8.4% | -0.4% | +64.6% | | 100000 | home.page (ms) | 0.898 | 0.973 | +8.3% | -3.0% | +11.7% | +17.0% | -1.3% | -0.3% | +186.2% | | 100000 | home.page_big (ms) | 18.9 | 29.6 | +56.8% | +35.1% | +16.1% | +2.3% | +63.2% | -5.6% | +11.1% | | 100000 | home.page_next (ms) | 0.745 | 0.854 | +14.6% | +16.0% | -1.2% | -0.6% | -11.3% | +15.6% | -27.1% | | 100000 | home.page_tag (ms) | 52.7 | 51.8 | -1.7% | -1.5% | -0.2% | -0.7% | -3.2% | +2.6% | +4.4% | | 100000 | home.page_title (ms) | 0.838 | 0.85 | +1.4% | -9.6% | +12.1% | +3.2% | +8.7% | +4.7% | +8.8% | | 100000 | mem.after_first_page (MB) | 22.9 | 22.2 | -3.1% | -4.3% | +1.2% | +3.4% | -3.1% | +1.2% | +3.4% | | 100000 | mem.after_opens (MB) | 98.4 | 98.9 | +0.6% | -1.5% | +2.1% | -2.3% | +0.6% | +2.1% | -2.3% | | 100000 | mem.after_search (MB) | 98.8 | 95.6 | -3.2% | -3.4% | +0.2% | +1.3% | -3.2% | +0.2% | +1.3% | | 100000 | mem.idle (MB) | 18.9 | 18.8 | -0.4% | -1.1% | +0.7% | +5.0% | -0.4% | +0.7% | +5.0% | | 100000 | notes.append (ms) | 4.88 | 5.04 | +3.2% | +0.1% | +3.1% | +1.6% | +1.3% | -2.8% | +5.4% | | 100000 | notes.append.first (ms) | 1.23e+03 | 1.21e+03 | -1.9% | -8.9% | +7.7% | +14.4% | -1.9% | +7.7% | +14.4% | | 100000 | open.api_get (ms) | 7.6 | 7.73 | +1.8% | -3.0% | +4.9% | -0.7% | +135.1% | +6.3% | -4.4% | | 100000 | open.api_get.warm (ms) | 0.168 | 0.169 | +0.9% | +0.1% | +0.8% | -2.0% | +2.4% | +4.7% | -2.3% | | 100000 | open.long_note (ms) | 68.2 | 67 | -1.8% | -2.4% | +0.6% | -4.3% | -1.8% | +0.6% | -4.3% | | 100000 | open.note (ms) | 0.794 | 0.903 | +13.7% | +18.5% | -4.1% | -4.8% | +8.4% | -10.1% | -4.2% | | 100000 | search.catch_up (s) | 47.1 | 46.1 | -2.1% | -2.1% | -0.0% | -2.1% | -2.1% | -0.0% | -2.1% | | 100000 | search.first_query (ms) | 27.5 | 27.6 | +0.4% | -1.4% | +1.8% | -0.4% | +0.4% | +1.8% | -0.4% | | 100000 | search.query (ms) | 2 | 1.98 | -0.7% | +5.2% | -5.7% | +98.6% | +7.0% | +46.8% | -2.0% | | 100000 | shutdown (ms) | 85.3 | 85.4 | +0.1% | +0.0% | +0.1% | +0.1% | +0.1% | +0.1% | +0.1% | | 100000 | startup.healthz (ms) | 4.9 | 5.07 | +3.4% | -1.5% | +5.0% | +15.2% | +3.4% | +5.0% | +15.2% | | 100000 | sync.durable_write (ms) | 24.6 | 24.9 | +1.2% | +3.9% | -2.6% | +0.6% | -5.0% | +13.7% | +2.5% | | 100000 | sync.round_trip (ms) | 0.221 | 0.215 | -2.5% | -2.4% | -0.1% | +2.2% | -1.3% | -0.9% | +3.7% | | 100000 | sync.save_ack (ms) | 3.72 | 3.68 | -1.0% | -0.1% | -1.0% | -1.6% | -2.1% | +0.0% | -5.5% |
kayg commented 2026-11-10 05:13:38 +00:00
Author
Owner
Copy link

Round 4b result and round 5 (head 5517ce595)

Round 4b (2026-11-10 00:52–01:47Z, 06:22–07:17 IST; same epoch; plain perf-turn; 30/30 arms exit 0) added a head-copy control (the same head binary run twice) and a second #970 arm per cycle. 1M listing gate: cold_first_us=101 p50_us=60 p99_us=78, pass.

  • Joined-layout noise is large. Two runs of the same head binary differ by up to +10.5% on home.page_big @100k, −15% on home.page @10k and −33% on startup.healthz @100k. Round 4a's "+16% page_big share" is inside that band; 4b measured +3.9% (p95 −3.2%).
  • Pooling 4a+4b (head n=20 vs #970 n=15, rank test) left a few small shares that are statistically real: cold.index @100k +2.5%, cold.first_page @100k +2.9%, cold.disk_written @100k +0.1% (p≤0.01), and search.* / open.api_get.warm @1k +2–6% (p≈0.03–0.08). home.page_big @100k +11% at p=0.08 is not established.

Round 5, aimed at those shares:

  • 549871e17 perf(files): Notes rows keep #970's table bytes. #1010 had added kind and scan_generation columns to #970's shared current_files, so every Notes row carried two extra fields through every WAL rewrite. Those move to a Files-only current_walk_rows(id, kind, seen) (indexed by (seen, id) for the sweep); readers take kind from a LEFT JOIN. Notes puts, cold scans and deletes of Notes-owned rows never touch the side tables. The sweep now excludes exactly the rows Notes owns (SSD Notes/*.md), so a deleted non-Markdown file below Notes/ is swept too. A draft database migrates once.
  • 5517ce595 perf(notes): only requests and typing mark a Home busy. Home::index() no longer marks activity: it was a CAS loop plus a clock read on every index call, including the cold scan's hot loop. mark() is now a relaxed load and a store at most every 5 ms.

Gates on compute02 at 5517ce595:

Gate Result
cargo fmt --all --check pass
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings pass
cargo test -j4 --workspace --locked --no-fail-fast pass
release build pass

One first-run failure in an unrelated SyncClient durability test passed 10/10 on rerun; filed as #1049 (flaky rule, non-blocking).

Round 5's ABAB (dev, head, #970, head-copy, #970, dev-copy, ×5) is queued on perf-turn. Status remains #1010-share: cold flat within noise; small shares being re-measured. Head vs dev stays red on what #970 alone costs, so #1010 can't be READY before #970 is ≤ dev.

## Round 4b result and round 5 (head `5517ce595`) **Round 4b** (2026-11-10 00:52–01:47Z, 06:22–07:17 IST; same epoch; plain `perf-turn`; 30/30 arms exit 0) added a **head-copy control** (the same head binary run twice) and a second #970 arm per cycle. 1M listing gate: `cold_first_us=101 p50_us=60 p99_us=78`, pass. - **Joined-layout noise is large.** Two runs of the same head binary differ by up to +10.5% on `home.page_big` @100k, −15% on `home.page` @10k and −33% on `startup.healthz` @100k. Round 4a's "+16% page_big share" is inside that band; 4b measured +3.9% (p95 −3.2%). - **Pooling 4a+4b** (head n=20 vs #970 n=15, rank test) left a few small shares that are statistically real: `cold.index` @100k +2.5%, `cold.first_page` @100k +2.9%, `cold.disk_written` @100k +0.1% (p≤0.01), and `search.*` / `open.api_get.warm` @1k +2–6% (p≈0.03–0.08). `home.page_big` @100k +11% at p=0.08 is not established. **Round 5**, aimed at those shares: - `549871e17` **perf(files): Notes rows keep #970's table bytes.** #1010 had added `kind` and `scan_generation` columns to #970's shared `current_files`, so every Notes row carried two extra fields through every WAL rewrite. Those move to a Files-only `current_walk_rows(id, kind, seen)` (indexed by `(seen, id)` for the sweep); readers take kind from a `LEFT JOIN`. Notes puts, cold scans and deletes of Notes-owned rows never touch the side tables. The sweep now excludes exactly the rows Notes owns (SSD `Notes/*.md`), so a deleted non-Markdown file below `Notes/` is swept too. A draft database migrates once. - `5517ce595` **perf(notes): only requests and typing mark a Home busy.** `Home::index()` no longer marks activity: it was a CAS loop plus a clock read on every index call, including the cold scan's hot loop. `mark()` is now a relaxed load and a store at most every 5 ms. Gates on compute02 at `5517ce595`: | Gate | Result | |---|---| | `cargo fmt --all --check` | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | | `cargo test -j4 --workspace --locked --no-fail-fast` | pass | | release build | pass | One first-run failure in an unrelated SyncClient durability test passed 10/10 on rerun; filed as #1049 (flaky rule, non-blocking). Round 5's ABAB (dev, head, #970, head-copy, #970, dev-copy, ×5) is queued on `perf-turn`. Status remains **#1010-share: cold flat within noise; small shares being re-measured**. Head vs dev stays red on what #970 alone costs, so #1010 can't be READY before #970 is ≤ dev.
kayg commented 2026-11-10 11:10:02 +00:00
Author
Owner
Copy link

Perf round 5: #1010-share flat (head 5517ce595)

Session 2026-11-10 10:09–11:05Z (15:39–16:35 IST), boot 7a65f072 (same epoch as every earlier round). Plain perf-turn files-1010 (FIFO) took perf-01's lock, then the control host took perf-02's; both released at the end. Per cycle: dev d4b32af94 (legacy), head, #970 381ce6750 (an exact ancestor), head-copy, #970, dev-copy; ×5; 1k/10k/100k; --hot-only --gate; same seed-1 fixtures as rounds 4a/4b. All 30 arms exited 0.

1M listing gate: files_listing entries=1000000 page_size=50 cold_first_us=130 p50_us=69 p99_us=101. Pass (first page 0.13 ms ≤ 100 ms).

#1010 share (head vs #970), what round 5 targeted:

  • cold.index @100k +0.8% (pooled 4a/4b had +2.5%, p≤0.01), cold.disk_written @100k +0.0% (was +0.1%), cold.first_page @100k +3.6% (head-copy band −4.4%), home.page_big @100k −6.3%.
  • Rank test of every metric (head + head-copy n=10 vs #970 n=10, p50 and p95, ~120 comparisons): 13 have a worse median at p<0.1, about the ~12 chance alone predicts. They are scattered and none repeats the round-4 flags. Bimodal metrics explain the large ones: open.long_note @10k is ~16 ms or ~47 ms in every arm, dev included; search.query @100k swings +100% between dev and dev-copy.
  • startup.healthz: +0.4% / −1.9% / +0.7% share at 1k/10k/100k.

Head vs dev stays red only where #970 alone is red (cold.first_page @100k +107% vs #970 +100%; cold.index, cold.disk_written, home.page_big and open.note @100k track #970).

Status: #1010-share flat. Not READY: #1010 can only be READY once #970 is ≤ dev (its cold identity-load fix is in progress on the tiering lane). When that lands, I'll merge #970's head and re-run this ABAB.

Round 5 table (medians of per-arm p50/p95)

n arms: dev 5, head 5, head-copy 5, #970 10, dev-copy 5

size metric dev p50 head p50 head vs dev #970 vs dev #1010 share (head vs #970) control vs dev head-copy vs head head vs dev p95 #1010 share p95 head-copy vs head p95
1000 cold.disk_written (MB) 18.8 19 +1.4% +1.2% +0.2% +0.2% -0.1% +1.4% +0.2% -0.1%
1000 cold.first_page (ms) 408 437 +7.1% +8.5% -1.3% -4.9% +0.0% +7.1% -1.3% +0.0%
1000 cold.index (s) 0.51 0.539 +5.7% +6.8% -1.0% -3.9% +0.0% +5.7% -1.0% +0.0%
1000 cold.mem_peak (MB) 46.4 47.1 +1.5% +2.0% -0.5% -1.4% +1.8% +1.5% -0.5% +1.8%
1000 cold.notes_per_s (notes/s) 1.96e+03 1.86e+03 -5.4% -6.4% +1.0% +4.0% -0.0% -5.4% +1.0% -0.0%
1000 cold.startup (ms) 19.7 19.8 +0.5% +2.8% -2.2% +0.7% +0.8% +0.5% -2.2% +0.8%
1000 home.first_page (ms) 1.72 1.92 +11.3% +0.9% +10.3% -7.0% -5.8% +11.3% +10.3% -5.8%
1000 home.page (ms) 0.706 0.7 -0.8% +6.6% -7.0% -4.4% +3.8% -1.4% -14.6% -5.4%
1000 home.page_big (ms) 8.51 8.34 -1.9% +0.7% -2.6% -0.5% +4.0% -1.9% -2.6% +4.0%
1000 home.page_next (ms) 0.726 0.71 -2.2% -1.5% -0.7% -3.1% +1.3% -5.3% -3.2% +4.7%
1000 home.page_tag (ms) 1.23 1.22 -1.3% -0.6% -0.7% -0.4% +0.3% -6.3% -1.1% +5.9%
1000 home.page_title (ms) 0.594 0.577 -2.9% -1.4% -1.5% +1.6% -0.8% -3.2% +0.1% +5.2%
1000 mem.after_first_page (MB) 21.3 20.7 -2.8% -2.6% -0.2% -1.2% +0.0% -2.8% -0.2% +0.0%
1000 mem.after_opens (MB) 34.8 35.5 +2.1% +0.1% +2.0% +0.3% -1.3% +2.1% +2.0% -1.3%
1000 mem.after_search (MB) 47.5 48.2 +1.3% +0.7% +0.6% +1.0% +0.0% +1.3% +0.6% +0.0%
1000 mem.idle (MB) 18.5 18.9 +2.3% +1.6% +0.7% +1.2% -1.0% +2.3% +0.7% -1.0%
1000 notes.append (ms) 4.95 4.84 -2.3% -2.3% +0.0% -2.4% +0.8% +1.0% -1.1% +0.7%
1000 notes.append.first (ms) 24 15.6 -34.8% -3.5% -32.4% +0.4% +0.2% -34.8% -32.4% +0.2%
1000 open.api_get (ms) 7.48 7.3 -2.3% -2.6% +0.3% -1.5% -0.1% +8.0% +23.1% -2.5%
1000 open.api_get.warm (ms) 0.165 0.161 -2.4% -1.6% -0.8% -2.0% +1.7% +0.5% -0.7% +3.7%
1000 open.long_note (ms) 15 15.2 +1.5% -0.7% +2.2% -0.4% -0.5% +1.5% +2.2% -0.5%
1000 open.note (ms) 0.65 0.674 +3.8% -0.5% +4.3% +2.3% -0.3% -7.8% +0.1% +1.1%
1000 search.catch_up (s) 2 2 -0.0% +0.0% -0.0% +0.0% +0.0% -0.0% -0.0% +0.0%
1000 search.first_query (ms) 27.5 27.3 -0.8% -1.9% +1.1% +2.8% -2.6% -0.8% +1.1% -2.6%
1000 search.query (ms) 1.15 1.16 +1.3% +0.2% +1.1% -3.5% -0.2% +2.3% +5.2% +0.2%
1000 shutdown (ms) 21.9 43.1 +96.4% +96.0% +0.2% +95.1% -0.0% +96.4% +0.2% -0.0%
1000 startup.healthz (ms) 5.33 4.84 -9.1% -7.7% -1.5% -8.8% +0.9% -9.1% -1.5% +0.9%
1000 sync.durable_write (ms) 24.5 24.8 +0.9% +0.3% +0.6% -0.6% -1.3% +5.1% -1.3% -0.4%
1000 sync.round_trip (ms) 0.205 0.209 +1.8% +2.4% -0.5% +0.6% -2.1% -2.1% +0.7% -8.3%
1000 sync.save_ack (ms) 3.65 3.66 +0.2% +1.1% -0.9% -0.2% +0.3% -0.6% +3.7% -13.5%
10000 cold.disk_written (MB) 151 152 +1.1% +1.0% +0.0% -0.0% +0.2% +1.1% +0.0% +0.2%
10000 cold.first_page (ms) 323 463 +43.5% +43.6% -0.1% +3.9% +0.1% +43.5% -0.1% +0.1%
10000 cold.index (s) 1.48 1.62 +9.9% +9.6% +0.3% -0.4% -0.8% +9.9% +0.3% -0.8%
10000 cold.mem_peak (MB) 144 144 +0.4% +0.5% -0.1% -0.3% +0.7% +0.4% -0.1% +0.7%
10000 cold.notes_per_s (notes/s) 6.78e+03 6.17e+03 -9.0% -8.7% -0.3% +0.4% +0.8% -9.0% -0.3% +0.8%
10000 cold.startup (ms) 19.9 19.9 -0.1% +0.3% -0.4% +0.7% +0.0% -0.1% -0.4% +0.0%
10000 home.first_page (ms) 1.97 2.74 +39.5% +22.8% +13.6% +3.8% -1.7% +39.5% +13.6% -1.7%
10000 home.page (ms) 0.713 0.773 +8.4% +16.4% -6.9% +7.6% +5.1% +10.6% -19.3% +25.3%
10000 home.page_big (ms) 16.4 16.2 -1.1% -1.3% +0.2% +3.2% -1.6% +2.5% +4.2% -4.4%
10000 home.page_next (ms) 0.67 0.711 +6.1% +4.3% +1.7% +5.9% -0.5% +23.5% +3.8% -5.4%
10000 home.page_tag (ms) 5.2 5.22 +0.4% +5.6% -5.0% -0.8% +2.3% +1.5% -3.7% -3.0%
10000 home.page_title (ms) 0.652 0.709 +8.6% +7.6% +1.0% -0.6% +0.8% +15.3% -29.0% +9.3%
10000 mem.after_first_page (MB) 21.7 21.5 -1.0% -1.7% +0.7% -1.8% -0.8% -1.0% +0.7% -0.8%
10000 mem.after_opens (MB) 40.9 40.4 -1.2% -1.3% +0.1% -0.6% +2.2% -1.2% +0.1% +2.2%
10000 mem.after_search (MB) 75.3 75.7 +0.6% -0.2% +0.8% +0.8% -0.9% +0.6% +0.8% -0.9%
10000 mem.idle (MB) 19 18.7 -1.3% -0.4% -0.9% -1.8% -0.7% -1.3% -0.9% -0.7%
10000 notes.append (ms) 4.88 4.86 -0.4% -0.5% +0.1% -0.3% +1.2% +6.2% -3.7% +8.1%
10000 notes.append.first (ms) 47.7 23 -51.8% +6.4% -54.7% +0.5% +103.8% -51.8% -54.7% +103.8%
10000 open.api_get (ms) 7.48 7.29 -2.5% -1.2% -1.3% -2.5% +1.1% -4.9% -16.2% +18.7%
10000 open.api_get.warm (ms) 0.17 0.161 -5.7% -5.6% -0.1% -3.1% +2.3% -8.8% -7.4% +9.8%
10000 open.long_note (ms) 15.9 45.3 +185.5% +3.2% +176.6% +2.8% +1.3% +185.5% +176.6% +1.3%
10000 open.note (ms) 0.643 0.627 -2.6% -6.0% +3.6% -5.6% -3.7% +3.7% +3.3% -4.1%
10000 search.catch_up (s) 4.01 4.02 +0.2% +0.0% +0.1% -0.0% -0.1% +0.2% +0.1% -0.1%
10000 search.first_query (ms) 29.7 26.8 -9.6% -9.6% -0.0% -7.1% +1.5% -9.6% -0.0% +1.5%
10000 search.query (ms) 1.34 1.48 +10.2% -0.0% +10.3% +28.0% +16.6% +9.2% +32.9% -12.5%
10000 shutdown (ms) 43.1 43.1 +0.1% +0.1% +0.1% -0.1% -0.1% +0.1% +0.1% -0.1%
10000 startup.healthz (ms) 5.22 4.85 -7.1% -5.3% -1.8% -6.6% -0.2% -7.1% -1.8% -0.2%
10000 sync.durable_write (ms) 24.2 24.6 +1.8% +2.3% -0.4% +6.7% +0.0% -4.4% -7.1% +6.2%
10000 sync.round_trip (ms) 0.21 0.21 +0.0% +0.8% -0.8% -4.9% +3.8% -4.0% -3.5% +4.1%
10000 sync.save_ack (ms) 3.62 3.71 +2.4% +3.0% -0.6% +3.1% +1.4% +7.6% +0.9% +7.2%
100000 cold.disk_written (MB) 1.92e+03 2.06e+03 +7.4% +7.4% +0.0% -0.0% -0.0% +7.4% +0.0% -0.0%
100000 cold.first_page (ms) 2.08e+03 4.31e+03 +106.8% +99.7% +3.6% -3.2% -4.4% +106.8% +3.6% -4.4%
100000 cold.index (s) 15.4 17.6 +14.2% +13.3% +0.8% -0.8% -1.2% +14.2% +0.8% -1.2%
100000 cold.mem_peak (MB) 292 289 -1.0% -3.0% +2.0% +1.3% -0.2% -1.0% +2.0% -0.2%
100000 cold.notes_per_s (notes/s) 6.48e+03 5.68e+03 -12.4% -11.7% -0.8% +0.8% +1.2% -12.4% -0.8% +1.2%
100000 cold.startup (ms) 20 22.5 +12.7% +3.9% +8.6% +0.1% -10.7% +12.7% +8.6% -10.7%
100000 home.first_page (ms) 2.88 2.86 -0.6% +23.7% -19.6% +0.6% +5.4% -0.6% -19.6% +5.4%
100000 home.page (ms) 0.972 0.913 -6.1% -3.3% -2.9% +2.4% +5.1% -8.8% +8.7% -7.8%
100000 home.page_big (ms) 18.2 28.1 +54.6% +65.0% -6.3% -0.2% -4.9% +83.7% -9.4% -6.0%
100000 home.page_next (ms) 0.699 0.862 +23.3% +18.4% +4.2% +4.0% -0.3% -32.3% -3.1% +0.1%
100000 home.page_tag (ms) 51 51.7 +1.4% +2.8% -1.3% +2.2% -1.0% -2.2% -2.6% +0.8%
100000 home.page_title (ms) 0.869 0.825 -5.0% -8.0% +3.3% -10.7% -5.9% +8.5% +15.5% -16.4%
100000 mem.after_first_page (MB) 22.6 22.1 -2.1% -2.3% +0.3% +0.8% -0.9% -2.1% +0.3% -0.9%
100000 mem.after_opens (MB) 98.4 96.7 -1.8% -1.4% -0.3% -0.9% -1.5% -1.8% -0.3% -1.5%
100000 mem.after_search (MB) 99 97.5 -1.5% -4.3% +3.0% +2.5% -4.1% -1.5% +3.0% -4.1%
100000 mem.idle (MB) 19.7 18.9 -3.8% -3.3% -0.5% -4.1% -0.0% -3.8% -0.5% -0.0%
100000 notes.append (ms) 4.92 4.79 -2.7% +1.7% -4.3% -3.9% -0.4% -37.8% -39.0% +64.9%
100000 notes.append.first (ms) 1.29e+03 1.33e+03 +2.6% -9.2% +13.0% +4.5% +5.6% +2.6% +13.0% +5.6%
100000 open.api_get (ms) 7.36 7.52 +2.2% +2.0% +0.3% +2.4% +0.9% +93.8% -10.4% +1.6%
100000 open.api_get.warm (ms) 0.159 0.156 -1.6% +3.4% -4.9% +5.1% +7.7% +7.3% -2.5% +3.4%
100000 open.long_note (ms) 67.3 63.5 -5.6% -0.3% -5.3% -6.2% +4.9% -5.6% -5.3% +4.9%
100000 open.note (ms) 0.766 0.943 +23.0% +22.4% +0.5% +0.4% -4.0% +1.6% -19.0% +28.1%
100000 search.catch_up (s) 46.1 46.2 +0.1% -0.0% +0.2% +0.0% -0.1% +0.1% +0.2% -0.1%
100000 search.first_query (ms) 27 27.6 +2.2% -0.0% +2.2% +0.6% -1.8% +2.2% +2.2% -1.8%
100000 search.query (ms) 2.03 2.92 +43.6% -8.8% +57.4% +100.1% +39.2% -32.6% -28.8% +120.1%
100000 shutdown (ms) 84.4 85.4 +1.1% -23.8% +32.7% +1.0% -24.5% +1.1% +32.7% -24.5%
100000 startup.healthz (ms) 4.9 4.96 +1.3% +0.5% +0.7% -0.4% -0.3% +1.3% +0.7% -0.3%
100000 sync.durable_write (ms) 25.1 24.4 -2.9% -2.2% -0.8% -1.3% +0.4% -7.0% -10.1% +3.5%
100000 sync.round_trip (ms) 0.218 0.213 -2.1% +4.7% -6.5% +4.3% +7.3% -1.4% -0.2% -2.1%
100000 sync.save_ack (ms) 3.7 3.68 -0.5% -0.4% -0.2% -0.0% +0.5% -5.7% -0.8% +4.6%
## Perf round 5: #1010-share flat (head `5517ce595`) Session 2026-11-10 10:09–11:05Z (15:39–16:35 IST), boot `7a65f072` (same epoch as every earlier round). Plain `perf-turn files-1010` (FIFO) took perf-01's lock, then the control host took perf-02's; both released at the end. Per cycle: dev `d4b32af94` (legacy), head, #970 `381ce6750` (an exact ancestor), head-copy, #970, dev-copy; ×5; 1k/10k/100k; `--hot-only --gate`; same seed-1 fixtures as rounds 4a/4b. All 30 arms exited 0. **1M listing gate:** `files_listing entries=1000000 page_size=50 cold_first_us=130 p50_us=69 p99_us=101`. Pass (first page 0.13 ms ≤ 100 ms). **#1010 share (head vs #970), what round 5 targeted:** - `cold.index` @100k +0.8% (pooled 4a/4b had +2.5%, p≤0.01), `cold.disk_written` @100k +0.0% (was +0.1%), `cold.first_page` @100k +3.6% (head-copy band −4.4%), `home.page_big` @100k −6.3%. - **Rank test of every metric** (head + head-copy n=10 vs #970 n=10, p50 and p95, ~120 comparisons): 13 have a worse median at p<0.1, about the ~12 chance alone predicts. They are scattered and none repeats the round-4 flags. Bimodal metrics explain the large ones: `open.long_note` @10k is ~16 ms or ~47 ms in every arm, dev included; `search.query` @100k swings +100% between dev and dev-copy. - `startup.healthz`: +0.4% / −1.9% / +0.7% share at 1k/10k/100k. **Head vs dev** stays red only where #970 alone is red (`cold.first_page` @100k +107% vs #970 +100%; `cold.index`, `cold.disk_written`, `home.page_big` and `open.note` @100k track #970). **Status: #1010-share flat.** Not READY: #1010 can only be READY once #970 is ≤ dev (its cold identity-load fix is in progress on the tiering lane). When that lands, I'll merge #970's head and re-run this ABAB. ### Round 5 table (medians of per-arm p50/p95) n arms: dev 5, head 5, head-copy 5, #970 10, dev-copy 5 | size | metric | dev p50 | head p50 | head vs dev | #970 vs dev | **#1010 share** (head vs #970) | control vs dev | head-copy vs head | head vs dev p95 | #1010 share p95 | head-copy vs head p95 | |---|---|---|---|---|---|---|---|---|---|---|---| | 1000 | cold.disk_written (MB) | 18.8 | 19 | +1.4% | +1.2% | +0.2% | +0.2% | -0.1% | +1.4% | +0.2% | -0.1% | | 1000 | cold.first_page (ms) | 408 | 437 | +7.1% | +8.5% | -1.3% | -4.9% | +0.0% | +7.1% | -1.3% | +0.0% | | 1000 | cold.index (s) | 0.51 | 0.539 | +5.7% | +6.8% | -1.0% | -3.9% | +0.0% | +5.7% | -1.0% | +0.0% | | 1000 | cold.mem_peak (MB) | 46.4 | 47.1 | +1.5% | +2.0% | -0.5% | -1.4% | +1.8% | +1.5% | -0.5% | +1.8% | | 1000 | cold.notes_per_s (notes/s) | 1.96e+03 | 1.86e+03 | -5.4% | -6.4% | +1.0% | +4.0% | -0.0% | -5.4% | +1.0% | -0.0% | | 1000 | cold.startup (ms) | 19.7 | 19.8 | +0.5% | +2.8% | -2.2% | +0.7% | +0.8% | +0.5% | -2.2% | +0.8% | | 1000 | home.first_page (ms) | 1.72 | 1.92 | +11.3% | +0.9% | +10.3% | -7.0% | -5.8% | +11.3% | +10.3% | -5.8% | | 1000 | home.page (ms) | 0.706 | 0.7 | -0.8% | +6.6% | -7.0% | -4.4% | +3.8% | -1.4% | -14.6% | -5.4% | | 1000 | home.page_big (ms) | 8.51 | 8.34 | -1.9% | +0.7% | -2.6% | -0.5% | +4.0% | -1.9% | -2.6% | +4.0% | | 1000 | home.page_next (ms) | 0.726 | 0.71 | -2.2% | -1.5% | -0.7% | -3.1% | +1.3% | -5.3% | -3.2% | +4.7% | | 1000 | home.page_tag (ms) | 1.23 | 1.22 | -1.3% | -0.6% | -0.7% | -0.4% | +0.3% | -6.3% | -1.1% | +5.9% | | 1000 | home.page_title (ms) | 0.594 | 0.577 | -2.9% | -1.4% | -1.5% | +1.6% | -0.8% | -3.2% | +0.1% | +5.2% | | 1000 | mem.after_first_page (MB) | 21.3 | 20.7 | -2.8% | -2.6% | -0.2% | -1.2% | +0.0% | -2.8% | -0.2% | +0.0% | | 1000 | mem.after_opens (MB) | 34.8 | 35.5 | +2.1% | +0.1% | +2.0% | +0.3% | -1.3% | +2.1% | +2.0% | -1.3% | | 1000 | mem.after_search (MB) | 47.5 | 48.2 | +1.3% | +0.7% | +0.6% | +1.0% | +0.0% | +1.3% | +0.6% | +0.0% | | 1000 | mem.idle (MB) | 18.5 | 18.9 | +2.3% | +1.6% | +0.7% | +1.2% | -1.0% | +2.3% | +0.7% | -1.0% | | 1000 | notes.append (ms) | 4.95 | 4.84 | -2.3% | -2.3% | +0.0% | -2.4% | +0.8% | +1.0% | -1.1% | +0.7% | | 1000 | notes.append.first (ms) | 24 | 15.6 | -34.8% | -3.5% | -32.4% | +0.4% | +0.2% | -34.8% | -32.4% | +0.2% | | 1000 | open.api_get (ms) | 7.48 | 7.3 | -2.3% | -2.6% | +0.3% | -1.5% | -0.1% | +8.0% | +23.1% | -2.5% | | 1000 | open.api_get.warm (ms) | 0.165 | 0.161 | -2.4% | -1.6% | -0.8% | -2.0% | +1.7% | +0.5% | -0.7% | +3.7% | | 1000 | open.long_note (ms) | 15 | 15.2 | +1.5% | -0.7% | +2.2% | -0.4% | -0.5% | +1.5% | +2.2% | -0.5% | | 1000 | open.note (ms) | 0.65 | 0.674 | +3.8% | -0.5% | +4.3% | +2.3% | -0.3% | -7.8% | +0.1% | +1.1% | | 1000 | search.catch_up (s) | 2 | 2 | -0.0% | +0.0% | -0.0% | +0.0% | +0.0% | -0.0% | -0.0% | +0.0% | | 1000 | search.first_query (ms) | 27.5 | 27.3 | -0.8% | -1.9% | +1.1% | +2.8% | -2.6% | -0.8% | +1.1% | -2.6% | | 1000 | search.query (ms) | 1.15 | 1.16 | +1.3% | +0.2% | +1.1% | -3.5% | -0.2% | +2.3% | +5.2% | +0.2% | | 1000 | shutdown (ms) | 21.9 | 43.1 | +96.4% | +96.0% | +0.2% | +95.1% | -0.0% | +96.4% | +0.2% | -0.0% | | 1000 | startup.healthz (ms) | 5.33 | 4.84 | -9.1% | -7.7% | -1.5% | -8.8% | +0.9% | -9.1% | -1.5% | +0.9% | | 1000 | sync.durable_write (ms) | 24.5 | 24.8 | +0.9% | +0.3% | +0.6% | -0.6% | -1.3% | +5.1% | -1.3% | -0.4% | | 1000 | sync.round_trip (ms) | 0.205 | 0.209 | +1.8% | +2.4% | -0.5% | +0.6% | -2.1% | -2.1% | +0.7% | -8.3% | | 1000 | sync.save_ack (ms) | 3.65 | 3.66 | +0.2% | +1.1% | -0.9% | -0.2% | +0.3% | -0.6% | +3.7% | -13.5% | | 10000 | cold.disk_written (MB) | 151 | 152 | +1.1% | +1.0% | +0.0% | -0.0% | +0.2% | +1.1% | +0.0% | +0.2% | | 10000 | cold.first_page (ms) | 323 | 463 | +43.5% | +43.6% | -0.1% | +3.9% | +0.1% | +43.5% | -0.1% | +0.1% | | 10000 | cold.index (s) | 1.48 | 1.62 | +9.9% | +9.6% | +0.3% | -0.4% | -0.8% | +9.9% | +0.3% | -0.8% | | 10000 | cold.mem_peak (MB) | 144 | 144 | +0.4% | +0.5% | -0.1% | -0.3% | +0.7% | +0.4% | -0.1% | +0.7% | | 10000 | cold.notes_per_s (notes/s) | 6.78e+03 | 6.17e+03 | -9.0% | -8.7% | -0.3% | +0.4% | +0.8% | -9.0% | -0.3% | +0.8% | | 10000 | cold.startup (ms) | 19.9 | 19.9 | -0.1% | +0.3% | -0.4% | +0.7% | +0.0% | -0.1% | -0.4% | +0.0% | | 10000 | home.first_page (ms) | 1.97 | 2.74 | +39.5% | +22.8% | +13.6% | +3.8% | -1.7% | +39.5% | +13.6% | -1.7% | | 10000 | home.page (ms) | 0.713 | 0.773 | +8.4% | +16.4% | -6.9% | +7.6% | +5.1% | +10.6% | -19.3% | +25.3% | | 10000 | home.page_big (ms) | 16.4 | 16.2 | -1.1% | -1.3% | +0.2% | +3.2% | -1.6% | +2.5% | +4.2% | -4.4% | | 10000 | home.page_next (ms) | 0.67 | 0.711 | +6.1% | +4.3% | +1.7% | +5.9% | -0.5% | +23.5% | +3.8% | -5.4% | | 10000 | home.page_tag (ms) | 5.2 | 5.22 | +0.4% | +5.6% | -5.0% | -0.8% | +2.3% | +1.5% | -3.7% | -3.0% | | 10000 | home.page_title (ms) | 0.652 | 0.709 | +8.6% | +7.6% | +1.0% | -0.6% | +0.8% | +15.3% | -29.0% | +9.3% | | 10000 | mem.after_first_page (MB) | 21.7 | 21.5 | -1.0% | -1.7% | +0.7% | -1.8% | -0.8% | -1.0% | +0.7% | -0.8% | | 10000 | mem.after_opens (MB) | 40.9 | 40.4 | -1.2% | -1.3% | +0.1% | -0.6% | +2.2% | -1.2% | +0.1% | +2.2% | | 10000 | mem.after_search (MB) | 75.3 | 75.7 | +0.6% | -0.2% | +0.8% | +0.8% | -0.9% | +0.6% | +0.8% | -0.9% | | 10000 | mem.idle (MB) | 19 | 18.7 | -1.3% | -0.4% | -0.9% | -1.8% | -0.7% | -1.3% | -0.9% | -0.7% | | 10000 | notes.append (ms) | 4.88 | 4.86 | -0.4% | -0.5% | +0.1% | -0.3% | +1.2% | +6.2% | -3.7% | +8.1% | | 10000 | notes.append.first (ms) | 47.7 | 23 | -51.8% | +6.4% | -54.7% | +0.5% | +103.8% | -51.8% | -54.7% | +103.8% | | 10000 | open.api_get (ms) | 7.48 | 7.29 | -2.5% | -1.2% | -1.3% | -2.5% | +1.1% | -4.9% | -16.2% | +18.7% | | 10000 | open.api_get.warm (ms) | 0.17 | 0.161 | -5.7% | -5.6% | -0.1% | -3.1% | +2.3% | -8.8% | -7.4% | +9.8% | | 10000 | open.long_note (ms) | 15.9 | 45.3 | +185.5% | +3.2% | +176.6% | +2.8% | +1.3% | +185.5% | +176.6% | +1.3% | | 10000 | open.note (ms) | 0.643 | 0.627 | -2.6% | -6.0% | +3.6% | -5.6% | -3.7% | +3.7% | +3.3% | -4.1% | | 10000 | search.catch_up (s) | 4.01 | 4.02 | +0.2% | +0.0% | +0.1% | -0.0% | -0.1% | +0.2% | +0.1% | -0.1% | | 10000 | search.first_query (ms) | 29.7 | 26.8 | -9.6% | -9.6% | -0.0% | -7.1% | +1.5% | -9.6% | -0.0% | +1.5% | | 10000 | search.query (ms) | 1.34 | 1.48 | +10.2% | -0.0% | +10.3% | +28.0% | +16.6% | +9.2% | +32.9% | -12.5% | | 10000 | shutdown (ms) | 43.1 | 43.1 | +0.1% | +0.1% | +0.1% | -0.1% | -0.1% | +0.1% | +0.1% | -0.1% | | 10000 | startup.healthz (ms) | 5.22 | 4.85 | -7.1% | -5.3% | -1.8% | -6.6% | -0.2% | -7.1% | -1.8% | -0.2% | | 10000 | sync.durable_write (ms) | 24.2 | 24.6 | +1.8% | +2.3% | -0.4% | +6.7% | +0.0% | -4.4% | -7.1% | +6.2% | | 10000 | sync.round_trip (ms) | 0.21 | 0.21 | +0.0% | +0.8% | -0.8% | -4.9% | +3.8% | -4.0% | -3.5% | +4.1% | | 10000 | sync.save_ack (ms) | 3.62 | 3.71 | +2.4% | +3.0% | -0.6% | +3.1% | +1.4% | +7.6% | +0.9% | +7.2% | | 100000 | cold.disk_written (MB) | 1.92e+03 | 2.06e+03 | +7.4% | +7.4% | +0.0% | -0.0% | -0.0% | +7.4% | +0.0% | -0.0% | | 100000 | cold.first_page (ms) | 2.08e+03 | 4.31e+03 | +106.8% | +99.7% | +3.6% | -3.2% | -4.4% | +106.8% | +3.6% | -4.4% | | 100000 | cold.index (s) | 15.4 | 17.6 | +14.2% | +13.3% | +0.8% | -0.8% | -1.2% | +14.2% | +0.8% | -1.2% | | 100000 | cold.mem_peak (MB) | 292 | 289 | -1.0% | -3.0% | +2.0% | +1.3% | -0.2% | -1.0% | +2.0% | -0.2% | | 100000 | cold.notes_per_s (notes/s) | 6.48e+03 | 5.68e+03 | -12.4% | -11.7% | -0.8% | +0.8% | +1.2% | -12.4% | -0.8% | +1.2% | | 100000 | cold.startup (ms) | 20 | 22.5 | +12.7% | +3.9% | +8.6% | +0.1% | -10.7% | +12.7% | +8.6% | -10.7% | | 100000 | home.first_page (ms) | 2.88 | 2.86 | -0.6% | +23.7% | -19.6% | +0.6% | +5.4% | -0.6% | -19.6% | +5.4% | | 100000 | home.page (ms) | 0.972 | 0.913 | -6.1% | -3.3% | -2.9% | +2.4% | +5.1% | -8.8% | +8.7% | -7.8% | | 100000 | home.page_big (ms) | 18.2 | 28.1 | +54.6% | +65.0% | -6.3% | -0.2% | -4.9% | +83.7% | -9.4% | -6.0% | | 100000 | home.page_next (ms) | 0.699 | 0.862 | +23.3% | +18.4% | +4.2% | +4.0% | -0.3% | -32.3% | -3.1% | +0.1% | | 100000 | home.page_tag (ms) | 51 | 51.7 | +1.4% | +2.8% | -1.3% | +2.2% | -1.0% | -2.2% | -2.6% | +0.8% | | 100000 | home.page_title (ms) | 0.869 | 0.825 | -5.0% | -8.0% | +3.3% | -10.7% | -5.9% | +8.5% | +15.5% | -16.4% | | 100000 | mem.after_first_page (MB) | 22.6 | 22.1 | -2.1% | -2.3% | +0.3% | +0.8% | -0.9% | -2.1% | +0.3% | -0.9% | | 100000 | mem.after_opens (MB) | 98.4 | 96.7 | -1.8% | -1.4% | -0.3% | -0.9% | -1.5% | -1.8% | -0.3% | -1.5% | | 100000 | mem.after_search (MB) | 99 | 97.5 | -1.5% | -4.3% | +3.0% | +2.5% | -4.1% | -1.5% | +3.0% | -4.1% | | 100000 | mem.idle (MB) | 19.7 | 18.9 | -3.8% | -3.3% | -0.5% | -4.1% | -0.0% | -3.8% | -0.5% | -0.0% | | 100000 | notes.append (ms) | 4.92 | 4.79 | -2.7% | +1.7% | -4.3% | -3.9% | -0.4% | -37.8% | -39.0% | +64.9% | | 100000 | notes.append.first (ms) | 1.29e+03 | 1.33e+03 | +2.6% | -9.2% | +13.0% | +4.5% | +5.6% | +2.6% | +13.0% | +5.6% | | 100000 | open.api_get (ms) | 7.36 | 7.52 | +2.2% | +2.0% | +0.3% | +2.4% | +0.9% | +93.8% | -10.4% | +1.6% | | 100000 | open.api_get.warm (ms) | 0.159 | 0.156 | -1.6% | +3.4% | -4.9% | +5.1% | +7.7% | +7.3% | -2.5% | +3.4% | | 100000 | open.long_note (ms) | 67.3 | 63.5 | -5.6% | -0.3% | -5.3% | -6.2% | +4.9% | -5.6% | -5.3% | +4.9% | | 100000 | open.note (ms) | 0.766 | 0.943 | +23.0% | +22.4% | +0.5% | +0.4% | -4.0% | +1.6% | -19.0% | +28.1% | | 100000 | search.catch_up (s) | 46.1 | 46.2 | +0.1% | -0.0% | +0.2% | +0.0% | -0.1% | +0.1% | +0.2% | -0.1% | | 100000 | search.first_query (ms) | 27 | 27.6 | +2.2% | -0.0% | +2.2% | +0.6% | -1.8% | +2.2% | +2.2% | -1.8% | | 100000 | search.query (ms) | 2.03 | 2.92 | +43.6% | -8.8% | +57.4% | +100.1% | +39.2% | -32.6% | -28.8% | +120.1% | | 100000 | shutdown (ms) | 84.4 | 85.4 | +1.1% | -23.8% | +32.7% | +1.0% | -24.5% | +1.1% | +32.7% | -24.5% | | 100000 | startup.healthz (ms) | 4.9 | 4.96 | +1.3% | +0.5% | +0.7% | -0.4% | -0.3% | +1.3% | +0.7% | -0.3% | | 100000 | sync.durable_write (ms) | 25.1 | 24.4 | -2.9% | -2.2% | -0.8% | -1.3% | +0.4% | -7.0% | -10.1% | +3.5% | | 100000 | sync.round_trip (ms) | 0.218 | 0.213 | -2.1% | +4.7% | -6.5% | +4.3% | +7.3% | -1.4% | -0.2% | -2.1% | | 100000 | sync.save_ack (ms) | 3.7 | 3.68 | -0.5% | -0.4% | -0.2% | -0.0% | +0.5% | -5.7% | -0.8% | +4.6% |
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Merge conflict checking is in progress. Try again in few moments.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/files-s1-identity:feat/files-s1-identity
git switch feat/files-s1-identity

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff feat/files-s1-identity
git switch feat/files-s1-identity
git rebase dev
git switch dev
git merge --ff-only feat/files-s1-identity
git switch feat/files-s1-identity
git rebase dev
git switch dev
git merge --no-ff feat/files-s1-identity
git switch dev
git merge --squash feat/files-s1-identity
git switch dev
git merge --ff-only feat/files-s1-identity
git switch dev
git merge feat/files-s1-identity
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!1010
No description provided.