kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

Tiering v3 slice 1: one current namespace index #970

Open
kayg wants to merge 1 commit from feat/tiering-v3-s1-namespace into dev
pull from: feat/tiering-v3-s1-namespace
merge into: kayg:dev
Conversation 34 Commits 1 Files changed 1 +15
kayg commented 2026-11-05 17:06:54 +00:00
Owner
Copy link

Slice 1 plan: one current namespace index

Implements docs/plans/tiering-v3.md under ADR 0062's three-root contract: SSD Home tree, HDD Home tree, and SSD app state. All three roots are validated and locked together. Reuse #790 preflight, copy/verify, exclusive publish, and unsplit-recovery safeguards within that contract. There is no two-root userdata/appdata layout and no filesystem-level tiering.

Cost and size targets

  • One shared Files current namespace row owns each Notes ID, relative path, owner tree, and generation; no second identity source, pending overlay, projection writer, or request-path identity-log scan.
  • Active entries store ID, parent folder ID, name, owner tree, and generation. Folder names are stored once; full paths are rebuilt from parent chains when needed. The 56-byte target is a mean on realistic nested-path corpora at 100k and 1M entries, not a per-entry limit. Every valid path must work.
  • The hard namespace-cache cap is 64 MiB. If an active set would exceed it, release the least-recently-used idle Home map through the existing idle-release path and serve that Home from the current index on disk while its map is absent. Never fail a request because of cache pressure.
  • Warm room reads and append metadata refresh use their retained file reference without a second ID lookup. A cold room resolves once.
  • Lists and search retain their existing indexes; Slice 2 adds tag/order seeks for updated, created, and title order plus maintained counts.

Failure table

Failure Effect Detection Recovery Test
One of three roots is missing, swapped, replaced, or has a different install identity Requests could read or commit against the wrong Home/state generation Root role, pinned identity, mount and marker checks under all three locks Fail closed; restore the matching coordinated set before serving Missing/swapped/replaced roots and mismatched state
Crash between two-tree move steps Roots temporarily disagree Durable intent and per-root progress Replay the unfinished move before serving affected paths Inject failure at each rename and parent sync, then restart
Duplicate logical path or file/folder collision Joined namespace could hide bytes Unique path key and clash check Preserve both files and report conflict Case, Unicode, and file/folder collisions
Watcher gap or overflow Index may be stale Generation gap/overflow/restart marker Mark incomplete; bounded background reconciliation Dropped events, restart, outside cross-tree move
Current index/map is corrupt Identity is untrustworthy Schema, checksum, generation validation Rebuild from durable file markers before affected results are served Corrupt index and interrupted rebuild
64 MiB cache cap is reached A Home map cannot stay resident within budget Global byte counter and map leases Release the least-recently-used idle map; serve that Home from the current index on disk while its map is absent One 1M Home; ten active 100k Homes; idle release and disk-backed service
Restore has only one Home tree or state root Home generation is incomplete Three pinned roles and matching generation/manifest Restore the coordinated three-root generation before serving Restore interrupted during a move

Gates

  • Per applicable path at 1k/10k/100k/1M: read, append/write, list, and search; tagged page joins in Slice 2. Five ABAB pairs against exact dev plus a same-session dev-vs-dev control, with both perf locks held. Require p50 and p95 below paired dev and the 95% upper confidence bound of each ratio <1.00. Any tie, noisy control, missing size, or failed operation blocks the slice.
  • One 1M Home and ten active 100k Homes must stay within the 64 MiB namespace-cache cap; check the 56-byte mean at 100k and 1M. Include the first request after the 600-second idle release. Run the materialized 100×100k fixture on perf-01 and delete it afterwards.
  • Rust: cargo fmt --all --check, cargo clippy -j4 --workspace --all-targets --locked -- -D warnings, and cargo test -j4 --workspace --locked -- --test-threads=2.
  • Failure-table crash, restore, and watcher tests must each be observed failing before the fix and passing after.
  • Run Notes Playwright in Chromium on dev-01 and WebKit on dev-02 at 1440 and 390. Safari remains pending: std VMs paused for the Mac VM. Run svelte-check and web unit tests on the allowed VM.

Current evidence and exact gate commands will be added in PR comments after qualification. Nothing is merged by this PR.

## Slice 1 plan: one current namespace index Implements `docs/plans/tiering-v3.md` under ADR 0062's three-root contract: SSD Home tree, HDD Home tree, and SSD app state. All three roots are validated and locked together. Reuse #790 preflight, copy/verify, exclusive publish, and unsplit-recovery safeguards within that contract. There is no two-root `userdata`/`appdata` layout and no filesystem-level tiering. ### Cost and size targets - One shared Files current namespace row owns each Notes ID, relative path, owner tree, and generation; no second identity source, pending overlay, projection writer, or request-path identity-log scan. - Active entries store ID, parent folder ID, name, owner tree, and generation. Folder names are stored once; full paths are rebuilt from parent chains when needed. The 56-byte target is a mean on realistic nested-path corpora at 100k and 1M entries, not a per-entry limit. Every valid path must work. - The hard namespace-cache cap is 64 MiB. If an active set would exceed it, release the least-recently-used idle Home map through the existing idle-release path and serve that Home from the current index on disk while its map is absent. Never fail a request because of cache pressure. - Warm room reads and append metadata refresh use their retained file reference without a second ID lookup. A cold room resolves once. - Lists and search retain their existing indexes; Slice 2 adds tag/order seeks for updated, created, and title order plus maintained counts. ### Failure table | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | One of three roots is missing, swapped, replaced, or has a different install identity | Requests could read or commit against the wrong Home/state generation | Root role, pinned identity, mount and marker checks under all three locks | Fail closed; restore the matching coordinated set before serving | Missing/swapped/replaced roots and mismatched state | | Crash between two-tree move steps | Roots temporarily disagree | Durable intent and per-root progress | Replay the unfinished move before serving affected paths | Inject failure at each rename and parent sync, then restart | | Duplicate logical path or file/folder collision | Joined namespace could hide bytes | Unique path key and clash check | Preserve both files and report conflict | Case, Unicode, and file/folder collisions | | Watcher gap or overflow | Index may be stale | Generation gap/overflow/restart marker | Mark incomplete; bounded background reconciliation | Dropped events, restart, outside cross-tree move | | Current index/map is corrupt | Identity is untrustworthy | Schema, checksum, generation validation | Rebuild from durable file markers before affected results are served | Corrupt index and interrupted rebuild | | 64 MiB cache cap is reached | A Home map cannot stay resident within budget | Global byte counter and map leases | Release the least-recently-used idle map; serve that Home from the current index on disk while its map is absent | One 1M Home; ten active 100k Homes; idle release and disk-backed service | | Restore has only one Home tree or state root | Home generation is incomplete | Three pinned roles and matching generation/manifest | Restore the coordinated three-root generation before serving | Restore interrupted during a move | ### Gates - Per applicable path at 1k/10k/100k/1M: read, append/write, list, and search; tagged page joins in Slice 2. Five ABAB pairs against exact dev plus a same-session dev-vs-dev control, with both perf locks held. Require p50 and p95 below paired dev and the 95% upper confidence bound of each ratio <1.00. Any tie, noisy control, missing size, or failed operation blocks the slice. - One 1M Home and ten active 100k Homes must stay within the 64 MiB namespace-cache cap; check the 56-byte mean at 100k and 1M. Include the first request after the 600-second idle release. Run the materialized 100×100k fixture on perf-01 and delete it afterwards. - Rust: `cargo fmt --all --check`, `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`, and `cargo test -j4 --workspace --locked -- --test-threads=2`. - Failure-table crash, restore, and watcher tests must each be observed failing before the fix and passing after. - Run Notes Playwright in Chromium on dev-01 and WebKit on dev-02 at 1440 and 390. Safari remains `pending: std VMs paused` for the Mac VM. Run `svelte-check` and web unit tests on the allowed VM. Current evidence and exact gate commands will be added in PR comments after qualification. Nothing is merged by this PR.
test(storage): require the three-root serve contract
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
80fa24a54e
kayg commented 2026-11-06 05:28:07 +00:00
Author
Owner
Copy link

Slice 1 validation update — head 2cd5e68597e67b499641ac6d0ad16b4e54746713

  • cargo fmt --all --check: PASS after the formatting commit.
  • Focused contacts-actions suite: FAIL, 105 passed and 26 failed.

Exact command, run on dev-01 through ~/bin/slot and nix develop:

cargo fmt --all && cargo fmt --all --check && cargo test -j4 -p calternal-contacts-actions --locked -- --test-threads=2

The common failure is the active namespace exceeds its cache budget. For the valid path Notes/Contacts/Lakshmi Krishna.md, the active row uses 8 bytes, the UUID uses 16 bytes, and the NUL-terminated path uses 34 bytes: 58 bytes before hash slots. A one-row map therefore cannot meet the enforced 56-byte entry ceiling and refuses normal contact operations. The failed cases include tests::basics::a_new_name_renames_and_links_follow, several contact naming tests, and import/recovery tests that depend on those operations.

This exposes a contract conflict in Slice 1: the plan requires full path bytes to count toward a hard 56-byte-per-entry limit, while supported Notes paths can exceed the remaining 32 bytes after row and ID storage. The current contiguous FileRef.path: &str representation cannot satisfy both for this valid path. A design ruling is needed before continuing: clarify whether 56 bytes is an aggregate target-fixture average with the 64 MiB cache cap remaining hard, or change the path representation/API so shared path components can be represented without a full per-row copy. No fallback SQLite lookup should be added.

The earlier exact-head browser attempt stopped at Rust compilation before Playwright started; the compile mismatch was fixed in 131cac78c, but Chromium has not yet been rerun on this head. WebKit remains pending: std VMs paused. No perf comparison or memory qualification has run; those wait for the post-CPU-pinning window. Clippy, full workspace tests, crash/restore/watcher red-to-green gates, and the browser gates remain unqualified.

## Slice 1 validation update — head `2cd5e68597e67b499641ac6d0ad16b4e54746713` - `cargo fmt --all --check`: PASS after the formatting commit. - Focused contacts-actions suite: FAIL, 105 passed and 26 failed. Exact command, run on dev-01 through `~/bin/slot` and `nix develop`: ```sh cargo fmt --all && cargo fmt --all --check && cargo test -j4 -p calternal-contacts-actions --locked -- --test-threads=2 ``` The common failure is `the active namespace exceeds its cache budget`. For the valid path `Notes/Contacts/Lakshmi Krishna.md`, the active row uses 8 bytes, the UUID uses 16 bytes, and the NUL-terminated path uses 34 bytes: 58 bytes before hash slots. A one-row map therefore cannot meet the enforced 56-byte entry ceiling and refuses normal contact operations. The failed cases include `tests::basics::a_new_name_renames_and_links_follow`, several contact naming tests, and import/recovery tests that depend on those operations. This exposes a contract conflict in Slice 1: the plan requires full path bytes to count toward a hard 56-byte-per-entry limit, while supported Notes paths can exceed the remaining 32 bytes after row and ID storage. The current contiguous `FileRef.path: &str` representation cannot satisfy both for this valid path. A design ruling is needed before continuing: clarify whether 56 bytes is an aggregate target-fixture average with the 64 MiB cache cap remaining hard, or change the path representation/API so shared path components can be represented without a full per-row copy. No fallback SQLite lookup should be added. The earlier exact-head browser attempt stopped at Rust compilation before Playwright started; the compile mismatch was fixed in `131cac78c`, but Chromium has not yet been rerun on this head. WebKit remains `pending: std VMs paused`. No perf comparison or memory qualification has run; those wait for the post-CPU-pinning window. Clippy, full workspace tests, crash/restore/watcher red-to-green gates, and the browser gates remain unqualified.
kayg commented 2026-11-06 05:36:37 +00:00
Author
Owner
Copy link

Orchestrator ruling (2026-11-06): 56 B is a mean on realistic corpora, checked at 100k and 1M entries, not a per-entry limit. No valid path is rejected for size. Store each entry as parent folder ID plus its name, interned or held in a per-folder name arena; store folder names once and rebuild full paths from parent chains when needed. The hard limit is 64 MiB for the namespace cache. If loading the active set would exceed it, release the least-recently-used idle Home map through the existing idle-release path and serve that Home from the current index on disk until it is active again. Never fail a request because of the cache budget. Add coverage for Notes/Contacts/Lakshmi Krishna.md, a 40-level nested path with 255-byte names, and cache-pressure eviction and disk-backed service.

Orchestrator ruling (2026-11-06): 56 B is a mean on realistic corpora, checked at 100k and 1M entries, not a per-entry limit. No valid path is rejected for size. Store each entry as parent folder ID plus its name, interned or held in a per-folder name arena; store folder names once and rebuild full paths from parent chains when needed. The hard limit is 64 MiB for the namespace cache. If loading the active set would exceed it, release the least-recently-used idle Home map through the existing idle-release path and serve that Home from the current index on disk until it is active again. Never fail a request because of the cache budget. Add coverage for `Notes/Contacts/Lakshmi Krishna.md`, a 40-level nested path with 255-byte names, and cache-pressure eviction and disk-backed service.
kayg commented 2026-11-06 12:32:32 +00:00
Author
Owner
Copy link

Follow-up: incremental namespace-map growth (head 3ce5ed496)

  • Added incremental_map_grows_hash_tables_amortized.
  • RED: cargo test -j4 --locked -p calternal-files incremental_map_grows_hash_tables_amortized -- --nocapture failed with ID hash capacity grew 1,389 times for 10,000 rows.
  • The fix uses an O(1) minimum-slot check and geometric growth, so prime search and rehash happen only at capacity boundaries.
  • GREEN: the same command passed. cargo fmt --all --check passed on the current source.
  • The inherited full calternal-files run is still active on its pre-fix binary; it is not current-head evidence. Exact-head Rust gates remain pending.
## Follow-up: incremental namespace-map growth (head `3ce5ed496`) - Added `incremental_map_grows_hash_tables_amortized`. - RED: `cargo test -j4 --locked -p calternal-files incremental_map_grows_hash_tables_amortized -- --nocapture` failed with `ID hash capacity grew 1,389 times for 10,000 rows`. - The fix uses an O(1) minimum-slot check and geometric growth, so prime search and rehash happen only at capacity boundaries. - GREEN: the same command passed. `cargo fmt --all --check` passed on the current source. - The inherited full `calternal-files` run is still active on its pre-fix binary; it is not current-head evidence. Exact-head Rust gates remain pending.
kayg commented 2026-11-06 13:07:12 +00:00
Author
Owner
Copy link

Exact-head gate progress — 3ce5ed4 (2026-11-06)

  • cargo fmt --all --check: PASS.
  • cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2: PASS, 131 passed.
  • cargo clippy --workspace --all-targets --locked -j4 -- -D warnings: PASS.
  • cargo test -j4 --locked -p calternal-files -- --nocapture: PASS, 17 passed.
    • One Home, 1,000,000 entries: map 54,313,415 B (54.31 B/entry mean); total namespace cache 56,476,103 B.
    • Ten Homes, 100,000 entries each: maps 54,843,080 B (54.84 B/entry mean); total namespace cache 57,595,592 B.
    • The realistic Notes/Contacts/Lakshmi Krishna.md, 40-level path with 255-byte names, cache-pressure disk-backed lookup, and idle unload/reload tests passed.

Workspace, crash/restore/watcher integration tests, Notes Playwright, and perf comparisons remain to run on this exact head.

## Exact-head gate progress — `3ce5ed4` (2026-11-06) - `cargo fmt --all --check`: PASS. - `cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2`: PASS, 131 passed. - `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings`: PASS. - `cargo test -j4 --locked -p calternal-files -- --nocapture`: PASS, 17 passed. - One Home, 1,000,000 entries: map 54,313,415 B (54.31 B/entry mean); total namespace cache 56,476,103 B. - Ten Homes, 100,000 entries each: maps 54,843,080 B (54.84 B/entry mean); total namespace cache 57,595,592 B. - The realistic `Notes/Contacts/Lakshmi Krishna.md`, 40-level path with 255-byte names, cache-pressure disk-backed lookup, and idle unload/reload tests passed. Workspace, crash/restore/watcher integration tests, Notes Playwright, and perf comparisons remain to run on this exact head.
kayg commented 2026-11-06 20:08:42 +00:00
Author
Owner
Copy link

Three-root perf options — head 470cff6c

  • RED at 81dc2b324: cargo test -j4 --locked -p calternal-perf args::tests::parses_the_three_root_layout -- --nocapture failed because the runner rejected the new three-root flags.
  • Added root-mapping, partial-root and legacy compatibility assertions in 299e89a7; RED showed the missing root fields and mapping method.
  • GREEN on 470cff6c: cargo fmt --all --check && cargo test -j4 --locked -p calternal-perf args::tests -- --nocapture passed; 4 tests passed, 0 failed.
  • --data-dir still maps run state under its matching relative path.
## Three-root perf options — head `470cff6c` - RED at `81dc2b324`: `cargo test -j4 --locked -p calternal-perf args::tests::parses_the_three_root_layout -- --nocapture` failed because the runner rejected the new three-root flags. - Added root-mapping, partial-root and legacy compatibility assertions in `299e89a7`; RED showed the missing root fields and mapping method. - GREEN on `470cff6c`: `cargo fmt --all --check && cargo test -j4 --locked -p calternal-perf args::tests -- --nocapture` passed; 4 tests passed, 0 failed. - `--data-dir` still maps run state under its matching relative path.
kayg commented 2026-11-06 20:42:25 +00:00
Author
Owner
Copy link

Perf runner three-root wiring — head cc65c483

  • RED at e8572ba5: cargo test -j4 --locked -p calternal-perf args::tests -- --nocapture did not compile because Options::fixture_root and server_args were missing.
  • RED at a76f10f4: cargo test -j4 --locked -p calternal-perf roots::tests::promotion_moves_each_root_to_its_matching_destination -- --nocapture failed with NotFound because the indexed destination parents were absent.
  • GREEN on dev-01 at cc65c483: ~/bin/slot nix develop -c bash -c 'cargo fmt --all --check && cargo test -j4 --locked -p calternal-perf -- --nocapture && sh -n perf/fixture-server.sh' passed. All 14 perf unit tests passed and the shell syntax check completed.

The runner now keeps SSD Home, HDD Home and app state separate through fixture setup, cold-to-warm promotion and cleanup, while retaining legacy --data-dir mapping. These are harness checks, not performance comparisons. Perf gates remain pending for a post-14:16 IST session with both perf locks held.

## Perf runner three-root wiring — head `cc65c483` - RED at `e8572ba5`: `cargo test -j4 --locked -p calternal-perf args::tests -- --nocapture` did not compile because `Options::fixture_root` and `server_args` were missing. - RED at `a76f10f4`: `cargo test -j4 --locked -p calternal-perf roots::tests::promotion_moves_each_root_to_its_matching_destination -- --nocapture` failed with `NotFound` because the indexed destination parents were absent. - GREEN on dev-01 at `cc65c483`: `~/bin/slot nix develop -c bash -c 'cargo fmt --all --check && cargo test -j4 --locked -p calternal-perf -- --nocapture && sh -n perf/fixture-server.sh'` passed. All 14 perf unit tests passed and the shell syntax check completed. The runner now keeps SSD Home, HDD Home and app state separate through fixture setup, cold-to-warm promotion and cleanup, while retaining legacy `--data-dir` mapping. These are harness checks, not performance comparisons. Perf gates remain pending for a post-14:16 IST session with both perf locks held.
kayg commented 2026-11-06 20:50:38 +00:00
Author
Owner
Copy link

Notes append harness — head fc638907

The perf runner now records notes.append.first for the first append to an unopened room and notes.append for 200 repeated appends through that room. Each request uses the public { "id": ..., "text": ... } input and a fresh idempotency key.

Exact-head dev-01 command, run through ~/bin/slot and nix develop:

cargo fmt --all --check && cargo clippy -j4 --locked -p calternal-perf --all-targets -- -D warnings && cargo test -j4 --locked -p calternal-perf -- --nocapture && sh -n perf/fixture-server.sh

Result: PASS; 15 tests passed, 0 failed. These checks do not qualify latency. The ABAB ×5 matrix and both perf locks remain required in the post-14:16 IST epoch.

## Notes append harness — head `fc638907` The perf runner now records `notes.append.first` for the first append to an unopened room and `notes.append` for 200 repeated appends through that room. Each request uses the public `{ "id": ..., "text": ... }` input and a fresh idempotency key. Exact-head dev-01 command, run through `~/bin/slot` and `nix develop`: ```sh cargo fmt --all --check && cargo clippy -j4 --locked -p calternal-perf --all-targets -- -D warnings && cargo test -j4 --locked -p calternal-perf -- --nocapture && sh -n perf/fixture-server.sh ``` Result: PASS; 15 tests passed, 0 failed. These checks do not qualify latency. The ABAB ×5 matrix and both perf locks remain required in the post-14:16 IST epoch.
kayg commented 2026-11-06 21:02:57 +00:00
Author
Owner
Copy link

Slice 1 owner ruling (2026-11-06), reflected under “Cost and memory” in docs/plans/tiering-v3.md:

  • 56 B is the mean on realistic nested paths at 100k and 1M entries, never a per-entry ceiling. Every valid path works.
  • Paths are stored as parent folder ID plus component name; folder names are stored once and full paths are rebuilt only when needed.
  • 64 MiB is the hard total namespace-cache cap. At pressure, release the least-recently-used idle Home map and serve that Home from the current index on disk until it is active again; cache pressure never fails a request.
  • Coverage includes Notes/Contacts/Lakshmi Krishna.md, a 40-level path with 255-byte names, cache pressure, disk-backed service, and idle release/reload.

This is the implementation contract for the remaining Slice 1 gates.

Slice 1 owner ruling (2026-11-06), reflected under “Cost and memory” in `docs/plans/tiering-v3.md`: - 56 B is the mean on realistic nested paths at 100k and 1M entries, never a per-entry ceiling. Every valid path works. - Paths are stored as parent folder ID plus component name; folder names are stored once and full paths are rebuilt only when needed. - 64 MiB is the hard total namespace-cache cap. At pressure, release the least-recently-used idle Home map and serve that Home from the current index on disk until it is active again; cache pressure never fails a request. - Coverage includes `Notes/Contacts/Lakshmi Krishna.md`, a 40-level path with 255-byte names, cache pressure, disk-backed service, and idle release/reload. This is the implementation contract for the remaining Slice 1 gates.
kayg commented 2026-11-06 23:51:26 +00:00
Author
Owner
Copy link

Slice 1 exact-head gate update — 3a9ae1198cceabef38ee45d7a7779d497ff46fb6

Rust gates on dev-01:

  • cargo fmt --all --check && cargo test -j4 -p calternal-contacts-actions --locked -- --test-threads=2: PASS; 131 passed.
  • cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS.
  • cargo test -j4 --workspace --locked -- --test-threads=2: PASS; 1,637 passed, 0 failed, 21 ignored.

Failure-table red/green evidence:

  • RED on baseline 789fdaa90: delete, restore, outside-rename and restart assertions each exited 101.
  • GREEN on 164e1e664: all four assertions passed.
  • Exact-head rerun on 3a9ae1198: tests::kill9::delete_killed_after_its_answer_finishes_once, tests::kill9::restore_killed_after_its_answer_finishes_once, tests::outside::outside_rename_keeps_the_id, and tests::outside::restart_after_outside_rename_rebuilds_shared_current_row each passed (1 test per command).

Web gates:

  • bun run check && bun run test: PASS; svelte-check reported 0 errors and 16 warnings in 10 files; Vitest passed 109 tests in 11 files. The warnings are in source files unchanged by this PR.
  • Notes Playwright on dev-01 Chromium and dev-02 WebKit, both at 1440 and 390, using e2e/day.spec.ts, journal-trash.spec.ts, list-search.spec.ts, search-corpus.spec.ts, tags.spec.ts and trash.spec.ts: each browser matrix passed 48, skipped 20, failed 0.

Safari on the Mac remains pending while that std VM is paused. Perf and cache qualification remain pending: next session will be entirely in the post-14:16 IST epoch, with both perf locks held, dev-vs-dev control, ABAB ×5 at 1k/10k/100k/1M, and the materialized fixture cleanup.

## Slice 1 exact-head gate update — `3a9ae1198cceabef38ee45d7a7779d497ff46fb6` Rust gates on dev-01: - `cargo fmt --all --check && cargo test -j4 -p calternal-contacts-actions --locked -- --test-threads=2`: PASS; 131 passed. - `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`: PASS. - `cargo test -j4 --workspace --locked -- --test-threads=2`: PASS; 1,637 passed, 0 failed, 21 ignored. Failure-table red/green evidence: - RED on baseline `789fdaa90`: delete, restore, outside-rename and restart assertions each exited 101. - GREEN on `164e1e664`: all four assertions passed. - Exact-head rerun on `3a9ae1198`: `tests::kill9::delete_killed_after_its_answer_finishes_once`, `tests::kill9::restore_killed_after_its_answer_finishes_once`, `tests::outside::outside_rename_keeps_the_id`, and `tests::outside::restart_after_outside_rename_rebuilds_shared_current_row` each passed (1 test per command). Web gates: - `bun run check && bun run test`: PASS; `svelte-check` reported 0 errors and 16 warnings in 10 files; Vitest passed 109 tests in 11 files. The warnings are in source files unchanged by this PR. - Notes Playwright on dev-01 Chromium and dev-02 WebKit, both at 1440 and 390, using `e2e/day.spec.ts`, `journal-trash.spec.ts`, `list-search.spec.ts`, `search-corpus.spec.ts`, `tags.spec.ts` and `trash.spec.ts`: each browser matrix passed 48, skipped 20, failed 0. Safari on the Mac remains pending while that std VM is paused. Perf and cache qualification remain pending: next session will be entirely in the post-14:16 IST epoch, with both perf locks held, dev-vs-dev control, ABAB ×5 at 1k/10k/100k/1M, and the materialized fixture cleanup.
kayg commented 2026-11-07 00:40:19 +00:00
Author
Owner
Copy link

Exact-head namespace cache gate — 3a9ae1198

Ran on dev-01 via ~/bin/slot nix develop.

Command: cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture

PASS; 1 test passed in 94.40 s.

Active set Map bytes Mean bytes/entry Total namespace cache
One Home, 1,000,000 entries 54,313,415 B 54.31 B 56,476,103 B
Ten Homes, 100,000 entries each 54,843,080 B 54.84 B 57,595,592 B

Both mean targets are below 56 B/entry and both totals are below the 64 MiB cap. The 100 x 100k materialized fixture and post-epoch perf matrix remain pending.

## Exact-head namespace cache gate — 3a9ae1198 Ran on dev-01 via ~/bin/slot nix develop. Command: cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture PASS; 1 test passed in 94.40 s. | Active set | Map bytes | Mean bytes/entry | Total namespace cache | |---|---:|---:|---:| | One Home, 1,000,000 entries | 54,313,415 B | 54.31 B | 56,476,103 B | | Ten Homes, 100,000 entries each | 54,843,080 B | 54.84 B | 57,595,592 B | Both mean targets are below 56 B/entry and both totals are below the 64 MiB cap. The 100 x 100k materialized fixture and post-epoch perf matrix remain pending.
kayg commented 2026-11-07 06:15:03 +00:00
Author
Owner
Copy link

Exact-head progress — aeaf8d422a

  • Notes WebKit on dev-02: PASS, 48 passed, 20 skipped, 0 failed. Ran the six Notes specs (day, journal-trash, list-search, search-corpus, tags, trash) in webkit-1440 and webkit-390.
  • Cold-fill test on an idle dev-02: PASS, 1 passed in 3.74 s:
    cargo test -j4 --locked -p calternal-tasks a_cold_fill_gives_the_same_rows_as_the_usual_way -- --nocapture
  • A dev-01 attempt timed out in the 10 s watcher-fill wait while an unrelated workspace suite was active. This is not counted as a gate pass or code failure; the full exact-head Rust gates remain pending on a clear slot.
  • The first dev-02 WebKit process was cut off by the SSH 60 s command timeout after 29 cases; it was rerun detached and completed successfully. No process from that first run remained.

The exact-head contacts-actions, clippy, workspace, crash/restore/watcher, Chromium and perf gates remain pending.

## Exact-head progress — aeaf8d422a4252d4f255ab2ff1e9ae920ddd409e - Notes WebKit on dev-02: PASS, 48 passed, 20 skipped, 0 failed. Ran the six Notes specs (day, journal-trash, list-search, search-corpus, tags, trash) in webkit-1440 and webkit-390. - Cold-fill test on an idle dev-02: PASS, 1 passed in 3.74 s: cargo test -j4 --locked -p calternal-tasks a_cold_fill_gives_the_same_rows_as_the_usual_way -- --nocapture - A dev-01 attempt timed out in the 10 s watcher-fill wait while an unrelated workspace suite was active. This is not counted as a gate pass or code failure; the full exact-head Rust gates remain pending on a clear slot. - The first dev-02 WebKit process was cut off by the SSH 60 s command timeout after 29 cases; it was rerun detached and completed successfully. No process from that first run remained. The exact-head contacts-actions, clippy, workspace, crash/restore/watcher, Chromium and perf gates remain pending.
kayg commented 2026-11-07 06:31:23 +00:00
Author
Owner
Copy link

Notes Chromium exact-head gate — aeaf8d422a

  • PASS on dev-01: 48 passed, 20 skipped, 0 failed across day, journal-trash, list-search, search-corpus, tags and trash in chromium-1440 and chromium-390.
  • Command path: ~/bin/slot nix develop; bun install --frozen-lockfile; cargo build -j4 --locked -p calternal; then bunx playwright test with --workers=1 and the six specs above.

WebKit passed on dev-02 in the preceding progress comment. The exact-head Rust workspace run is still in progress on dev-02.

## Notes Chromium exact-head gate — aeaf8d422a4252d4f255ab2ff1e9ae920ddd409e - PASS on dev-01: 48 passed, 20 skipped, 0 failed across day, journal-trash, list-search, search-corpus, tags and trash in chromium-1440 and chromium-390. - Command path: ~/bin/slot nix develop; bun install --frozen-lockfile; cargo build -j4 --locked -p calternal; then bunx playwright test with --workers=1 and the six specs above. WebKit passed on dev-02 in the preceding progress comment. The exact-head Rust workspace run is still in progress on dev-02.
kayg commented 2026-11-07 06:35:16 +00:00
Author
Owner
Copy link

Exact-head Rust gates — aeaf8d422a

Ran on dev-02, through ~/bin/slot and nix develop:

  • cargo fmt --all --check: PASS.
  • cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2: PASS, 131 passed.
  • cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS.
  • cargo test -j4 --workspace --locked -- --test-threads=2: PASS, 1,640 passed, 0 failed, 21 ignored (workspace test binaries and doctests).
  • Exact-head cold-fill test: PASS on the idle dev-02 in 3.74 s. Its dev-01 attempt timed out while an unrelated workspace job was active, as recorded above.

The four failure-table crash/restore/watcher cases and the memory test with output are being rerun individually at this head.

## Exact-head Rust gates — aeaf8d422a4252d4f255ab2ff1e9ae920ddd409e Ran on dev-02, through ~/bin/slot and nix develop: - cargo fmt --all --check: PASS. - cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2: PASS, 131 passed. - cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS. - cargo test -j4 --workspace --locked -- --test-threads=2: PASS, 1,640 passed, 0 failed, 21 ignored (workspace test binaries and doctests). - Exact-head cold-fill test: PASS on the idle dev-02 in 3.74 s. Its dev-01 attempt timed out while an unrelated workspace job was active, as recorded above. The four failure-table crash/restore/watcher cases and the memory test with output are being rerun individually at this head.
kayg commented 2026-11-07 07:02:06 +00:00
Author
Owner
Copy link

Exact-head failure-table and namespace-cache rerun — aeaf8d422a4252d4f255ab2ff1e9ae920ddd409e

The four focused failure-table assertions are green on dev-02 at this head (one test per invocation, each 1 passed, 0 failed): delete killed after answer, restore killed after answer, outside rename preserves the ID, and restart after outside rename rebuilds the shared current row. Their baseline RED evidence remains in comment 83259.

The namespace-cache gate ran on dev-01 through ~/bin/slot nix develop:

cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture

Result: PASS, 1 passed, 0 failed, in 114.33 s.

Active set Map bytes Mean bytes/entry Total namespace cache
One Home, 1,000,000 entries 54,313,415 B 54.31 B 56,476,103 B
Ten Homes, 100,000 entries each 54,843,080 B 54.84 B 57,595,592 B

Both means are below 56 B/entry and both total cache measurements are below 64 MiB. The materialized 100 × 100k fixture and the ABAB ×5 performance matrix with both perf locks remain pending; no comparison numbers are claimed here.

## Exact-head failure-table and namespace-cache rerun — `aeaf8d422a4252d4f255ab2ff1e9ae920ddd409e` The four focused failure-table assertions are green on dev-02 at this head (one test per invocation, each 1 passed, 0 failed): delete killed after answer, restore killed after answer, outside rename preserves the ID, and restart after outside rename rebuilds the shared current row. Their baseline RED evidence remains in comment 83259. The namespace-cache gate ran on dev-01 through `~/bin/slot nix develop`: ```sh cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture ``` Result: PASS, 1 passed, 0 failed, in 114.33 s. | Active set | Map bytes | Mean bytes/entry | Total namespace cache | |---|---:|---:|---:| | One Home, 1,000,000 entries | 54,313,415 B | 54.31 B | 56,476,103 B | | Ten Homes, 100,000 entries each | 54,843,080 B | 54.84 B | 57,595,592 B | Both means are below 56 B/entry and both total cache measurements are below 64 MiB. The materialized 100 × 100k fixture and the ABAB ×5 performance matrix with both perf locks remain pending; no comparison numbers are claimed here.
kayg commented 2026-11-07 15:17:42 +00:00
Author
Owner
Copy link

Slice 1 gate update — head 2034d17d036dad97039f8516f16b7dd58dd43348

The Rust gates ran at b209787, which is the Rust source state in this head; the later sync with current dev changed UI/spec and budget files, with no Rust-source changes.

  • cargo fmt --all --check: PASS.
  • cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2: PASS, 131 passed.
  • Warm-read perf harness test: PASS, 1 passed.
  • cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS.
  • cargo test -j4 --workspace --locked -- --test-threads=2: PASS, 1,779 passed, 0 failed, 21 ignored.

Crash, restore and watcher cases

The four failure-table assertions were RED on baseline 789fdaa90 (each exited 101) and GREEN in the current Rust suite: delete killed after its answer, restore killed after its answer, outside rename preserves the ID, and restart after outside rename rebuilds the current row. The workspace run also passed the split-publish boundary, partial-publish recovery, three-root unsplit recovery, and namespace cache-pressure/reload tests.

Namespace memory

The exact cache test reports:

Active set Map bytes Mean Total namespace cache
One Home, 1,000,000 entries 54,313,415 B 54.31 B/entry 56,476,103 B
Ten Homes, 100,000 entries each 54,843,080 B 54.84 B/entry 57,595,592 B

Both means are below 56 B/entry, and both totals are below 64 MiB. The current workspace run also passed the cache-pressure, disk-backed lookup and idle unload/reload tests.

Notes Playwright

At 2034d17, the six Notes specs (day, journal-trash, list-search, search-corpus, tags, trash) passed in Chromium on dev-01 and WebKit on dev-02. Each browser matrix covered 1440 and 390 projects: 48 passed, 20 skipped, 0 failed.

Perf

No comparison has started yet. The shared locks on both perf VMs are held by another active run. I will recheck disk space after the locks are released, hold both locks for the full session, run the dev-vs-dev control and ABAB ×5 matrix, then post the results separately. No fixture has been created yet.

## Slice 1 gate update — head `2034d17d036dad97039f8516f16b7dd58dd43348` The Rust gates ran at `b209787`, which is the Rust source state in this head; the later sync with current `dev` changed UI/spec and budget files, with no Rust-source changes. - `cargo fmt --all --check`: PASS. - `cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2`: PASS, 131 passed. - Warm-read perf harness test: PASS, 1 passed. - `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`: PASS. - `cargo test -j4 --workspace --locked -- --test-threads=2`: PASS, 1,779 passed, 0 failed, 21 ignored. ### Crash, restore and watcher cases The four failure-table assertions were RED on baseline `789fdaa90` (each exited 101) and GREEN in the current Rust suite: delete killed after its answer, restore killed after its answer, outside rename preserves the ID, and restart after outside rename rebuilds the current row. The workspace run also passed the split-publish boundary, partial-publish recovery, three-root unsplit recovery, and namespace cache-pressure/reload tests. ### Namespace memory The exact cache test reports: | Active set | Map bytes | Mean | Total namespace cache | |---|---:|---:|---:| | One Home, 1,000,000 entries | 54,313,415 B | 54.31 B/entry | 56,476,103 B | | Ten Homes, 100,000 entries each | 54,843,080 B | 54.84 B/entry | 57,595,592 B | Both means are below 56 B/entry, and both totals are below 64 MiB. The current workspace run also passed the cache-pressure, disk-backed lookup and idle unload/reload tests. ### Notes Playwright At `2034d17`, the six Notes specs (`day`, `journal-trash`, `list-search`, `search-corpus`, `tags`, `trash`) passed in Chromium on dev-01 and WebKit on dev-02. Each browser matrix covered 1440 and 390 projects: 48 passed, 20 skipped, 0 failed. ### Perf No comparison has started yet. The shared locks on both perf VMs are held by another active run. I will recheck disk space after the locks are released, hold both locks for the full session, run the dev-vs-dev control and ABAB ×5 matrix, then post the results separately. No fixture has been created yet.
kayg commented 2026-11-07 17:32:29 +00:00
Author
Owner
Copy link

Exact-head Rust gate update — 9a5563595297d1acd502ffd48c552d1f5c911165

Ran on dev-01 in tiering-v3-s1-sync-1e903 through ~/bin/slot and nix develop.

cargo fmt --all --check &&
cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2 &&
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings &&
cargo test -j4 --workspace --locked -- --test-threads=2
  • Formatting: PASS.
  • calternal-contacts-actions: PASS, 131 passed.
  • Workspace clippy with -D warnings: PASS.
  • Workspace tests: PASS, 1,644 passed, 0 failed, 22 ignored.
  • The workspace run covered the current namespace tests, including the contact path, 40-level deep path, idle LRU release, and disk-backed lookup cases. The large 100 × 100k materialized fixture remains intentionally ignored here and will run on perf-01.

The four crash/restore/watcher assertions were recorded RED on baseline 789fdaa90 and GREEN after the implementation; the exact-head workspace rerun passed them again. The latest commits after the previous browser run only register/add the materialized Rust test, with no application or browser-spec changes.

Still pending: the 100 × 100k perf-01 fixture and the locked post-14:16 IST dev/dev control plus ABAB ×5 hot-path matrix.

## Exact-head Rust gate update — `9a5563595297d1acd502ffd48c552d1f5c911165` Ran on dev-01 in `tiering-v3-s1-sync-1e903` through `~/bin/slot` and `nix develop`. ```sh cargo fmt --all --check && cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2 && cargo clippy -j4 --workspace --all-targets --locked -- -D warnings && cargo test -j4 --workspace --locked -- --test-threads=2 ``` - Formatting: PASS. - `calternal-contacts-actions`: PASS, 131 passed. - Workspace clippy with `-D warnings`: PASS. - Workspace tests: PASS, 1,644 passed, 0 failed, 22 ignored. - The workspace run covered the current namespace tests, including the contact path, 40-level deep path, idle LRU release, and disk-backed lookup cases. The large 100 × 100k materialized fixture remains intentionally ignored here and will run on perf-01. The four crash/restore/watcher assertions were recorded RED on baseline `789fdaa90` and GREEN after the implementation; the exact-head workspace rerun passed them again. The latest commits after the previous browser run only register/add the materialized Rust test, with no application or browser-spec changes. Still pending: the 100 × 100k perf-01 fixture and the locked post-14:16 IST dev/dev control plus ABAB ×5 hot-path matrix.
kayg commented 2026-11-07 17:48:27 +00:00
Author
Owner
Copy link

Notes browser gate update — PR head 9a5563595297d1acd502ffd48c552d1f5c911165

Six Notes specs ran in both viewport projects on each browser, with --workers=1.

  • Chromium on dev-01 at PR head 9a556359: 48 passed, 20 skipped, 0 failed.
  • WebKit on dev-02 at 2034d17d: 48 passed, 20 skipped, 0 failed.

The current head's only changes after 2034d17d are the test-only namespace module registration and the materialized Rust cache test, so the Notes app code and Playwright specs are unchanged.

Commands, run through ~/bin/slot and nix develop:

bun install --frozen-lockfile &&
cargo build -j4 --locked -p calternal &&
cd apps/web &&
CALTERNAL_BIN=<worktree>/target/debug/calternal bunx playwright test \
  e2e/day.spec.ts e2e/journal-trash.spec.ts e2e/list-search.spec.ts \
  e2e/search-corpus.spec.ts e2e/tags.spec.ts e2e/trash.spec.ts \
  --project=<browser>-1440 --project=<browser>-390 --workers=1
## Notes browser gate update — PR head `9a5563595297d1acd502ffd48c552d1f5c911165` Six Notes specs ran in both viewport projects on each browser, with `--workers=1`. - Chromium on dev-01 at PR head `9a556359`: 48 passed, 20 skipped, 0 failed. - WebKit on dev-02 at `2034d17d`: 48 passed, 20 skipped, 0 failed. The current head's only changes after `2034d17d` are the test-only namespace module registration and the materialized Rust cache test, so the Notes app code and Playwright specs are unchanged. Commands, run through `~/bin/slot` and `nix develop`: ```sh bun install --frozen-lockfile && cargo build -j4 --locked -p calternal && cd apps/web && CALTERNAL_BIN=<worktree>/target/debug/calternal bunx playwright test \ e2e/day.spec.ts e2e/journal-trash.spec.ts e2e/list-search.spec.ts \ e2e/search-corpus.spec.ts e2e/tags.spec.ts e2e/trash.spec.ts \ --project=<browser>-1440 --project=<browser>-390 --workers=1 ```
kayg commented 2026-11-07 22:35:07 +00:00
Author
Owner
Copy link

Root-bundle fixture update — head 8fd37d37

  • RED: the new materialized_homes_cycle_across_root_bundles test failed at the missing root_bundle_index helper.
  • GREEN on dev-01 at 8fd37d37: cargo fmt --all --check, the root-bundle cycle test (1 passed), and small_materialized_homes_stay_serviceable_through_one_cache (1 passed).
  • The small real-file fixture materialized 600 files across three Homes and stayed within the cache cap (2,390,388 B cache use; 403.85 B/entry at this small size).

The focused commands ran through ~/bin/slot nix develop. Exact-head contacts-actions, clippy, workspace, failure-table and Notes browser gates remain queued behind current dev VM slot jobs; memory/perf qualification is still pending.

## Root-bundle fixture update — head `8fd37d37` - RED: the new `materialized_homes_cycle_across_root_bundles` test failed at the missing `root_bundle_index` helper. - GREEN on dev-01 at `8fd37d37`: `cargo fmt --all --check`, the root-bundle cycle test (1 passed), and `small_materialized_homes_stay_serviceable_through_one_cache` (1 passed). - The small real-file fixture materialized 600 files across three Homes and stayed within the cache cap (2,390,388 B cache use; 403.85 B/entry at this small size). The focused commands ran through `~/bin/slot nix develop`. Exact-head contacts-actions, clippy, workspace, failure-table and Notes browser gates remain queued behind current dev VM slot jobs; memory/perf qualification is still pending.
kayg commented 2026-11-07 23:51:26 +00:00
Author
Owner
Copy link

Slice 1 gate update — head 879e398f

The approved 56-byte mean, parent-ID-plus-name path storage, 64 MiB LRU eviction and disk-backed lookup ruling is already recorded in comment #82999 and in docs/plans/tiering-v3.md under “Cost and memory”.

Exact-head checks

  • Contacts actions: PASS, 131 passed.
  • cargo clippy --workspace --all-targets --locked -j4 -- -D warnings: PASS.
  • Workspace: PASS, 1,649 passed, 22 ignored, 0 failed.
  • Crash/restore/watcher: delete-after-answer, restore-after-answer, outside rename preserves ID, and restart reconciliation each passed at this head; their baseline RED results are recorded in comment #83259.
  • Notes Playwright: Chromium 48 passed / 20 skipped; WebKit 48 passed / 20 skipped.
  • Fresh namespace memory run on dev-01:
    cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture: PASS, 1 passed in 115.96 s.
Active set Map bytes Mean bytes/entry Namespace cache bytes
One Home, 1,000,000 entries 54,313,415 54.31 56,476,103
Ten Homes, 100,000 entries each 54,843,080 54.84 57,595,592

Both means are below 56 B/entry and both caches are below 64 MiB.

Pending

The 100 × 100k materialized fixture and the post-epoch ABAB ×5 plus dev-vs-dev matrix have not started. Both perf locks are currently held by other jobs; I am waiting for them before writing the fixture or starting any comparison. No perf measurement was made in this session.

## Slice 1 gate update — head `879e398f` The approved 56-byte mean, parent-ID-plus-name path storage, 64 MiB LRU eviction and disk-backed lookup ruling is already recorded in comment #82999 and in `docs/plans/tiering-v3.md` under “Cost and memory”. ### Exact-head checks - Contacts actions: PASS, 131 passed. - `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings`: PASS. - Workspace: PASS, 1,649 passed, 22 ignored, 0 failed. - Crash/restore/watcher: delete-after-answer, restore-after-answer, outside rename preserves ID, and restart reconciliation each passed at this head; their baseline RED results are recorded in comment #83259. - Notes Playwright: Chromium 48 passed / 20 skipped; WebKit 48 passed / 20 skipped. - Fresh namespace memory run on dev-01: `cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture`: PASS, 1 passed in 115.96 s. | Active set | Map bytes | Mean bytes/entry | Namespace cache bytes | |---|---:|---:|---:| | One Home, 1,000,000 entries | 54,313,415 | 54.31 | 56,476,103 | | Ten Homes, 100,000 entries each | 54,843,080 | 54.84 | 57,595,592 | Both means are below 56 B/entry and both caches are below 64 MiB. ### Pending The 100 × 100k materialized fixture and the post-epoch ABAB ×5 plus dev-vs-dev matrix have not started. Both perf locks are currently held by other jobs; I am waiting for them before writing the fixture or starting any comparison. No perf measurement was made in this session.
kayg commented 2026-11-08 00:23:05 +00:00
Author
Owner
Copy link

1k hot-run offset fixture — head aa6b22ba2d2bd0336b9fede2025299bd45590779

  • RED at c7f4a8f7: ten_hot_run_offsets_fit_the_smallest_fixture failed as expected because the 1k fixture has 600 samples and offset 9 requested 640.
  • The per-run sample consumers fit slots 0–59 after reducing PER_RUN to 60; the idle-only slot moved from 60 to 58, leaving append's final slot at 59.
  • GREEN on dev-01: cargo test -j4 --locked -p calternal-perf ten_hot_run_offsets_fit_the_smallest_fixture -- --nocapture — 1 passed.
  • Pushed commit aa6b22ba to the PR branch. Exact-head Rust, failure-table, and Notes browser gates are running next.
## 1k hot-run offset fixture — head `aa6b22ba2d2bd0336b9fede2025299bd45590779` - RED at `c7f4a8f7`: `ten_hot_run_offsets_fit_the_smallest_fixture` failed as expected because the 1k fixture has 600 samples and offset 9 requested 640. - The per-run sample consumers fit slots 0–59 after reducing `PER_RUN` to 60; the idle-only slot moved from 60 to 58, leaving append's final slot at 59. - GREEN on dev-01: `cargo test -j4 --locked -p calternal-perf ten_hot_run_offsets_fit_the_smallest_fixture -- --nocapture` — 1 passed. - Pushed commit `aa6b22ba` to the PR branch. Exact-head Rust, failure-table, and Notes browser gates are running next.
kayg commented 2026-11-08 00:42:48 +00:00
Author
Owner
Copy link

Exact-head gates — aa6b22ba2d2bd0336b9fede2025299bd45590779

  • cargo fmt --all --check: PASS.
  • cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2: PASS, 131 passed.
  • cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS.
  • cargo test -j4 --workspace --locked -- --test-threads=2: PASS, 1,650 passed, 22 ignored, 0 failed.
  • Failure-table RED evidence remains on this PR at baseline 789fdaa90 (comment #83259). At this head, delete-after-answer, restore-after-answer, outside rename preserves ID, and restart reconciliation each passed (1 test per invocation).
  • Notes Chromium on dev-01: 48 passed, 20 skipped, 0 failed. Notes WebKit on dev-02: 48 passed, 20 skipped, 0 failed. Both used the six Notes specs in desktop and phone projects with one worker.
  • The 1M/ten-Home namespace cache test passed again in the workspace run. The recorded table below is from 879e398f, whose namespace implementation is unchanged at this head:
Active set Map bytes Mean bytes/entry Total namespace cache
One Home, 1,000,000 entries 54,313,415 B 54.31 B 56,476,103 B
Ten Homes, 100,000 entries each 54,843,080 B 54.84 B 57,595,592 B

The materialized 100 × 100k fixture and post-epoch ABAB/control comparisons remain pending; both perf locks are currently held. No fixture was created.

## Exact-head gates — `aa6b22ba2d2bd0336b9fede2025299bd45590779` - `cargo fmt --all --check`: PASS. - `cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2`: PASS, 131 passed. - `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`: PASS. - `cargo test -j4 --workspace --locked -- --test-threads=2`: PASS, 1,650 passed, 22 ignored, 0 failed. - Failure-table RED evidence remains on this PR at baseline `789fdaa90` (comment #83259). At this head, delete-after-answer, restore-after-answer, outside rename preserves ID, and restart reconciliation each passed (1 test per invocation). - Notes Chromium on dev-01: 48 passed, 20 skipped, 0 failed. Notes WebKit on dev-02: 48 passed, 20 skipped, 0 failed. Both used the six Notes specs in desktop and phone projects with one worker. - The 1M/ten-Home namespace cache test passed again in the workspace run. The recorded table below is from `879e398f`, whose namespace implementation is unchanged at this head: | Active set | Map bytes | Mean bytes/entry | Total namespace cache | |---|---:|---:|---:| | One Home, 1,000,000 entries | 54,313,415 B | 54.31 B | 56,476,103 B | | Ten Homes, 100,000 entries each | 54,843,080 B | 54.84 B | 57,595,592 B | The materialized 100 × 100k fixture and post-epoch ABAB/control comparisons remain pending; both perf locks are currently held. No fixture was created.
kayg commented 2026-11-08 13:01:32 +00:00
Author
Owner
Copy link

Slice 1 cache ruling and materialized fixture update — c060cfcf

The approved cache contract is recorded under “Cost and memory” in docs/plans/tiering-v3.md: 56 B is a mean on realistic nested paths at 100k and 1M entries; paths use parent folder IDs plus names; the 64 MiB cap applies to the whole namespace cache; idle Home maps are released LRU and served from the current index on disk until active again. Valid paths, including Notes/Contacts/Lakshmi Krishna.md and the 40-level 255-byte-name path, remain supported.

The materialized fixture now accepts any nonempty set of existing parent directories. Commits 7f09a6f9 and 12197eef added variable-root coverage and validation; the namespace materialized test suite passed 5 tests with the 10-million-file test ignored. c060cfcf fixes the test’s stale comment that still said the fixture required exactly five parents.

The Rust gates last ran at code head 12197eef: fmt passed, contacts-actions passed (131), clippy passed with -D warnings, and the workspace passed (1,653 passed, 22 ignored). c060cfcf changes only the test comment.

The next session is in the post-14:16 IST epoch. Both perf locks are active. The materialized 100 × 100k fixture and ABAB ×5 plus dev-vs-dev comparisons have not started; no new perf numbers are claimed here.

## Slice 1 cache ruling and materialized fixture update — `c060cfcf` The approved cache contract is recorded under “Cost and memory” in `docs/plans/tiering-v3.md`: 56 B is a mean on realistic nested paths at 100k and 1M entries; paths use parent folder IDs plus names; the 64 MiB cap applies to the whole namespace cache; idle Home maps are released LRU and served from the current index on disk until active again. Valid paths, including `Notes/Contacts/Lakshmi Krishna.md` and the 40-level 255-byte-name path, remain supported. The materialized fixture now accepts any nonempty set of existing parent directories. Commits `7f09a6f9` and `12197eef` added variable-root coverage and validation; the namespace materialized test suite passed 5 tests with the 10-million-file test ignored. `c060cfcf` fixes the test’s stale comment that still said the fixture required exactly five parents. The Rust gates last ran at code head `12197eef`: fmt passed, contacts-actions passed (131), clippy passed with `-D warnings`, and the workspace passed (1,653 passed, 22 ignored). `c060cfcf` changes only the test comment. The next session is in the post-14:16 IST epoch. Both perf locks are active. The materialized 100 × 100k fixture and ABAB ×5 plus dev-vs-dev comparisons have not started; no new perf numbers are claimed here.
kayg commented 2026-11-08 14:52:34 +00:00
Author
Owner
Copy link

Exact-head slice 1 gates — 3e8dbfeb2fb76ca4665e8ee89eae90cd4c5bf3a4

Ran on dev-01 in tiering-v3-s1-sync-1e903, through ~/bin/slot and nix develop.

  • cargo fmt --all --check: PASS.
  • cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2: PASS, 131 passed.
  • cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS.
  • cargo test -j4 --workspace --locked -- --test-threads=2: PASS, 1,653 passed, 0 failed, 22 ignored.
  • The workspace run includes the Lakshmi contact path, 40-level path, cache-pressure disk lookup, and idle unload/reload tests.

The four failure-table assertions each passed individually at this head: delete_killed_after_its_answer_finishes_once, restore_killed_after_its_answer_finishes_once, outside_rename_keeps_the_id, and restart_after_outside_rename_rebuilds_shared_current_row. Their baseline RED evidence on 789fdaa90 remains in comment #83259.

The exact-head namespace-memory command passed in 134.61 s:

cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture
Active set Map bytes Mean bytes/entry Total namespace cache
One Home, 1,000,000 entries 54,313,415 B 54.31 B 56,476,103 B
Ten Homes, 100,000 entries each 54,843,080 B 54.84 B 57,595,592 B

Both means are below 56 B/entry and both totals are below 64 MiB.

Notes Chromium on dev-01 and WebKit on dev-02 are running now at this head. The materialized 100 × 100k fixture and post-epoch dev/dev control plus ABAB ×5 remain pending. Both perf locks were held by another run at preflight; perf-01 then showed 84 GB available, so I will recheck disk after the locks are free before creating the fixture. No perf comparison numbers are claimed here.

## Exact-head slice 1 gates — `3e8dbfeb2fb76ca4665e8ee89eae90cd4c5bf3a4` Ran on dev-01 in `tiering-v3-s1-sync-1e903`, through `~/bin/slot` and `nix develop`. - `cargo fmt --all --check`: PASS. - `cargo test -j4 --locked -p calternal-contacts-actions -- --test-threads=2`: PASS, 131 passed. - `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`: PASS. - `cargo test -j4 --workspace --locked -- --test-threads=2`: PASS, 1,653 passed, 0 failed, 22 ignored. - The workspace run includes the Lakshmi contact path, 40-level path, cache-pressure disk lookup, and idle unload/reload tests. The four failure-table assertions each passed individually at this head: `delete_killed_after_its_answer_finishes_once`, `restore_killed_after_its_answer_finishes_once`, `outside_rename_keeps_the_id`, and `restart_after_outside_rename_rebuilds_shared_current_row`. Their baseline RED evidence on `789fdaa90` remains in comment #83259. The exact-head namespace-memory command passed in 134.61 s: ```sh cargo test -j4 --locked -p calternal-files realistic_nested_maps_fit_one_million_and_ten_home_cache_budgets -- --nocapture ``` | Active set | Map bytes | Mean bytes/entry | Total namespace cache | |---|---:|---:|---:| | One Home, 1,000,000 entries | 54,313,415 B | 54.31 B | 56,476,103 B | | Ten Homes, 100,000 entries each | 54,843,080 B | 54.84 B | 57,595,592 B | Both means are below 56 B/entry and both totals are below 64 MiB. Notes Chromium on dev-01 and WebKit on dev-02 are running now at this head. The materialized 100 × 100k fixture and post-epoch dev/dev control plus ABAB ×5 remain pending. Both perf locks were held by another run at preflight; perf-01 then showed 84 GB available, so I will recheck disk after the locks are free before creating the fixture. No perf comparison numbers are claimed here.
kayg commented 2026-11-08 14:57:10 +00:00
Author
Owner
Copy link

Notes browser gates — 3e8dbfeb2fb76ca4665e8ee89eae90cd4c5bf3a4

Ran the six Notes specs in the 1440 and 390 projects with one worker, after bun install --frozen-lockfile and cargo build -j4 --locked -p calternal, through ~/bin/slot and nix develop.

  • Chromium on dev-01: 48 passed, 20 skipped, 0 failed.
  • WebKit on dev-02: 48 passed, 20 skipped, 0 failed.

Specs: day, journal-trash, list-search, search-corpus, tags, and trash.

## Notes browser gates — `3e8dbfeb2fb76ca4665e8ee89eae90cd4c5bf3a4` Ran the six Notes specs in the 1440 and 390 projects with one worker, after `bun install --frozen-lockfile` and `cargo build -j4 --locked -p calternal`, through `~/bin/slot` and `nix develop`. - Chromium on dev-01: 48 passed, 20 skipped, 0 failed. - WebKit on dev-02: 48 passed, 20 skipped, 0 failed. Specs: `day`, `journal-trash`, `list-search`, `search-corpus`, `tags`, and `trash`.
kayg commented 2026-11-08 17:07:49 +00:00
Author
Owner
Copy link

100 × 100k materialized namespace-cache gate — exact head 3e8dbfeb2f

Command, run on perf-01 through the slot helper with both perf locks held:

CALTERNAL_MATERIALIZED_ROOTS=<25 existing parent directories> \
  /home/claude/bin/slot /home/claude/perf/tiering-v3-s1-run-20261108T1646Z/staged/v3/calternal_files_test \
  materialized_100_homes_of_100k_notes_keep_ten_maps_within_cache_cap --ignored --nocapture

The 25 unique parents were split across 13 directories on / and 12 on /srv/homes to fit the available inode counts. The test materialized all 100 Homes and cleaned their contents on teardown; the empty parent directories were then removed.

Result: PASS, 1 passed, 0 failed, 0 ignored, 22 filtered out, 738.84 s.

Active set Files Map bytes Mean bytes/entry Cache bytes Cache under pressure
Ten Homes, 100,000 entries each 10,000,000 54,839,960 54.84 B 57,595,592 57,661,128

Both cache values are below 64 MiB. After cleanup, perf-01 returned to 78 GB free on / and 26 GB free on /srv/homes. The earlier one-Home 1M result is recorded in comment #86638.

## 100 × 100k materialized namespace-cache gate — exact head 3e8dbfeb2fb76ca4665e8ee89eae90cd4c5bf3a4 Command, run on perf-01 through the slot helper with both perf locks held: ~~~sh CALTERNAL_MATERIALIZED_ROOTS=<25 existing parent directories> \ /home/claude/bin/slot /home/claude/perf/tiering-v3-s1-run-20261108T1646Z/staged/v3/calternal_files_test \ materialized_100_homes_of_100k_notes_keep_ten_maps_within_cache_cap --ignored --nocapture ~~~ The 25 unique parents were split across 13 directories on / and 12 on /srv/homes to fit the available inode counts. The test materialized all 100 Homes and cleaned their contents on teardown; the empty parent directories were then removed. Result: PASS, 1 passed, 0 failed, 0 ignored, 22 filtered out, 738.84 s. | Active set | Files | Map bytes | Mean bytes/entry | Cache bytes | Cache under pressure | |---|---:|---:|---:|---:|---:| | Ten Homes, 100,000 entries each | 10,000,000 | 54,839,960 | 54.84 B | 57,595,592 | 57,661,128 | Both cache values are below 64 MiB. After cleanup, perf-01 returned to 78 GB free on / and 26 GB free on /srv/homes. The earlier one-Home 1M result is recorded in comment #86638.
kayg commented 2026-11-09 06:01:43 +00:00
Author
Owner
Copy link

Synced with dev, Rust gates: head 381ce675064f0c425676ffea16fc63cac3afaff9

Takeover from the Codex agent that stopped at 09:16 IST. Its stale perf holds (tiering-v3-s1-postpin-hold0{1,2}-20261109-0912) are released.

  • 88ccb3e5d merges dev 2fa9e18fb (branch was 100 commits behind). Conflicts were docs plus one real overlap: dev's loaded and this branch's initialized typing-gate flags in notes_crash_typing.rs are the same thing, kept as one initialized. Dev's recount_with_after_walk test hook kept next to the multi-root recount. Dev has since moved to d4b32af94, web only (no Rust change).
  • 381ce6750 rustfmt fix for that rename.

Gates on dev-01 (nix develop, ~/bin/slot):

  • cargo fmt --all --check: PASS
  • cargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASS
  • cargo test -j4 --workspace --locked -- --test-threads=2: PASS, 1,654 passed, 0 failed, 22 ignored

Perf gate: not met yet

The head-841 session (11-09 01:08–03:13 CET, before the merge; dev/dev control + ABAB ×5) at 100k shows v3 slower than dev with the whole 95% CI above 1.00 (geometric mean ratio of v3/dev):

Path @100k p50 ratio [95% CI] p95 ratio [95% CI]
home.page_big 1.318 [1.164, 1.493] 1.465 [1.302, 1.649]
home.page_next 1.172 [1.086, 1.265] 1.326 [0.808, 2.176]
home.page_title 1.134 [1.039, 1.238] 1.035 [0.652, 1.643]
mem.idle 1.043 [1.022, 1.065]

At 1k/10k almost every path is a tie (CI straddles 1.00), which also fails the plan's strict rule. The 1M setup failed with a retryable 500 (fixed in 841f55525). The paging SQL is unchanged from dev, so the 100k delta is likely the meta.db produced by the v3 cold scan, not the query; unconfirmed. A fresh session at this head starts once the Mail lane (#1011) and the queued lanes release the perf locks.

Flaky auth test seen on #1008, not blocking: #1021.

## Synced with dev, Rust gates: head `381ce675064f0c425676ffea16fc63cac3afaff9` Takeover from the Codex agent that stopped at 09:16 IST. Its stale perf holds (`tiering-v3-s1-postpin-hold0{1,2}-20261109-0912`) are released. - `88ccb3e5d` merges dev `2fa9e18fb` (branch was 100 commits behind). Conflicts were docs plus one real overlap: dev's `loaded` and this branch's `initialized` typing-gate flags in `notes_crash_typing.rs` are the same thing, kept as one `initialized`. Dev's `recount_with_after_walk` test hook kept next to the multi-root recount. Dev has since moved to `d4b32af94`, web only (no Rust change). - `381ce6750` rustfmt fix for that rename. Gates on dev-01 (`nix develop`, `~/bin/slot`): - `cargo fmt --all --check`: PASS - `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`: PASS - `cargo test -j4 --workspace --locked -- --test-threads=2`: PASS, 1,654 passed, 0 failed, 22 ignored ### Perf gate: not met yet The head-841 session (11-09 01:08–03:13 CET, before the merge; dev/dev control + ABAB ×5) at 100k shows v3 slower than dev with the whole 95% CI above 1.00 (geometric mean ratio of v3/dev): | Path @100k | p50 ratio [95% CI] | p95 ratio [95% CI] | |---|---|---| | home.page_big | 1.318 [1.164, 1.493] | 1.465 [1.302, 1.649] | | home.page_next | 1.172 [1.086, 1.265] | 1.326 [0.808, 2.176] | | home.page_title | 1.134 [1.039, 1.238] | 1.035 [0.652, 1.643] | | mem.idle | 1.043 [1.022, 1.065] | | At 1k/10k almost every path is a tie (CI straddles 1.00), which also fails the plan's strict rule. The 1M setup failed with a retryable 500 (fixed in `841f55525`). The paging SQL is unchanged from dev, so the 100k delta is likely the `meta.db` produced by the v3 cold scan, not the query; unconfirmed. A fresh session at this head starts once the Mail lane (#1011) and the queued lanes release the perf locks. Flaky auth test seen on #1008, not blocking: #1021.
kayg commented 2026-11-09 06:22:40 +00:00
Author
Owner
Copy link

Perf gate reading (orchestrator, 11-09) + head-841 page slowdown root cause

How "strictly faster on every path" is read

The plan's rule would fail paths v3 does not touch (for example search.catch_up sits at exactly 1.000). The gate for this lane is now:

  • Paths v3 changes must beat dev: the 95% upper bound of v3/dev for p50 and p95 is below 1.00. For #970 these are open.api_get, open.api_get.warm, open.note, open.long_note, notes.append, notes.append.first and home.first_page (first page after idle release). #1008 adds home.page_tag, home.page_tag_created and home.page_tag_title.
  • Paths v3 does not change must not regress: the v3/dev point ratio for p50 and p95 is no higher than the dev/dev control's 95% upper bound for that path (or 1.00 if that is higher). That covers list paging (home.page* without tags, same SQL as dev), search, and memory.
  • Ratios are geometric means over 5 interleaved pairs, with a t-interval (df 4) on the log ratios, at 1k, 10k, 100k and 1M. A missing size or a failed run fails the gate.

kayg: please override this if you meant the strict reading.

The +32 % / +17 % / +13 % home.page_big, home.page_next, home.page_title at 100k (head 841)

Root-caused off the perf VMs, on dev-01, with a 100k Home built by each of dev d4b32af94 and slice 1 381ce6750 (same harness, same seed):

meta.db after the cold scan dev s1
size 846,741,504 B 847,233,024 B
page_count / freelist 206,724 / 88 206,844 / 89
WAL left after setup none none
sqlite_stat1 absent absent
user_version, indexes, row counts (notes 100,000, tags 446,233, links 611,392) identical identical

The two files match. Neither side runs ANALYZE, and the query plans and paging SQL are unchanged from dev. So stale stats, fragmentation and an un-checkpointed WAL are ruled out.

A local A/B on these Homes (5 interleaved pairs, hot-only, 100k, dev-01) does not reproduce the slowdown:

Path s1/dev p50 [95% CI] p95 [95% CI]
home.page_big 0.880 [0.605, 1.278] 0.944 [0.558, 1.597]
home.page_next 0.799 [0.540, 1.184] 0.582 [0.319, 1.060]
home.page_title 0.984 [0.565, 1.714] 1.051 [0.359, 3.082]

dev-01 is a shared build host, so these numbers only show direction; the perf VMs decide. Conclusion: the head-841 delta came from that session, not from the code. The most likely cause is the perf VM's 7.7 GB page cache shared by four 20–29 GB Homes and ~850 MB meta.db files. No code change was made for it. The fresh session (control-a, s1, control-b, s2 interleaved per pair, both locks) is the proof and is queued behind #1011 and the lanes already waiting.

## Perf gate reading (orchestrator, 11-09) + head-841 page slowdown root cause ### How "strictly faster on every path" is read The plan's rule would fail paths v3 does not touch (for example `search.catch_up` sits at exactly 1.000). The gate for this lane is now: - **Paths v3 changes must beat dev:** the 95% upper bound of v3/dev for p50 and p95 is below 1.00. For #970 these are `open.api_get`, `open.api_get.warm`, `open.note`, `open.long_note`, `notes.append`, `notes.append.first` and `home.first_page` (first page after idle release). #1008 adds `home.page_tag`, `home.page_tag_created` and `home.page_tag_title`. - **Paths v3 does not change must not regress:** the v3/dev point ratio for p50 and p95 is no higher than the dev/dev control's 95% upper bound for that path (or 1.00 if that is higher). That covers list paging (`home.page*` without tags, same SQL as dev), search, and memory. - Ratios are geometric means over 5 interleaved pairs, with a t-interval (df 4) on the log ratios, at 1k, 10k, 100k and 1M. A missing size or a failed run fails the gate. kayg: please override this if you meant the strict reading. ### The +32 % / +17 % / +13 % `home.page_big`, `home.page_next`, `home.page_title` at 100k (head 841) Root-caused off the perf VMs, on dev-01, with a 100k Home built by each of dev `d4b32af94` and slice 1 `381ce6750` (same harness, same seed): | `meta.db` after the cold scan | dev | s1 | |---|---:|---:| | size | 846,741,504 B | 847,233,024 B | | page_count / freelist | 206,724 / 88 | 206,844 / 89 | | WAL left after setup | none | none | | `sqlite_stat1` | absent | absent | | user_version, indexes, row counts (notes 100,000, tags 446,233, links 611,392) | identical | identical | The two files match. Neither side runs ANALYZE, and the query plans and paging SQL are unchanged from dev. So stale stats, fragmentation and an un-checkpointed WAL are ruled out. A local A/B on these Homes (5 interleaved pairs, hot-only, 100k, dev-01) does not reproduce the slowdown: | Path | s1/dev p50 [95% CI] | p95 [95% CI] | |---|---|---| | home.page_big | 0.880 [0.605, 1.278] | 0.944 [0.558, 1.597] | | home.page_next | 0.799 [0.540, 1.184] | 0.582 [0.319, 1.060] | | home.page_title | 0.984 [0.565, 1.714] | 1.051 [0.359, 3.082] | dev-01 is a shared build host, so these numbers only show direction; the perf VMs decide. Conclusion: the head-841 delta came from that session, not from the code. The most likely cause is the perf VM's 7.7 GB page cache shared by four 20–29 GB Homes and ~850 MB `meta.db` files. No code change was made for it. The fresh session (control-a, s1, control-b, s2 interleaved per pair, both locks) is the proof and is queued behind #1011 and the lanes already waiting.
kayg commented 2026-11-09 11:16:57 +00:00
Author
Owner
Copy link

Fresh session H1 (1k/10k/100k): s1 vs dev

Session H1: perf-01, 11-09 11:07–11:37 CET (15:37–16:07 IST). Both locks were held: perf-01 through perf-turn, perf-02 through perf-lock. Binaries (sha256): dev d4b32af94 b99ace57…6859, s1 381ce6750 8b5292d6…bf0e, s2 eaeb417e7 f7e5ad0f…10d0, perf harness b3682687…e4a2. Each Home had its own setup. Every pair ran in the order control-a, s1, control-b, s2 (--hot-only --reuse-indexed, offset = pair). The gate reading is the one in the earlier comment: changed paths need a 95% upper bound below 1.00; unchanged paths need the point ratio no higher than the control's upper bound (or 1.00 if higher). Before H1, I deleted the predecessor's stale fixtures (96 GB work-head841, 18 GB of abab-20261107 Homes; reports kept). That took 7 min at the start of the turn.

Result for #970: FAIL on the gate as written, but no real regression. The paths s1 changes are level with dev; most CIs straddle 1.00. They don't beat it, except open.api_get p50 at 100k, 0.877 [0.773, 0.995]. Two unchanged paths miss by a hair: home.page_title p50 at 100k is 1.105 against a control bound of 1.101, and search.query p95 is 1.086 against 0.996. The earlier +32 % page_big at 100k did not come back: it is in the PASS rows below.

Full gate table (s1)
size path kind s1/dev p50 [CI] s1/dev p95 [CI] control UCB p50/p95 verdict
1000 home.first_page changed 0.802 [0.609, 1.057] 0.802 [0.609, 1.057] 1.392/1.392 FAIL
1000 home.page unchanged 0.883 [0.727, 1.074] 0.620 [0.398, 0.965] 1.088/2.375 PASS
1000 home.page_big unchanged 1.078 [0.758, 1.534] 1.078 [0.758, 1.534] 1.111/1.111 PASS
1000 home.page_next unchanged 0.971 [0.816, 1.155] 0.910 [0.750, 1.105] 1.214/1.568 PASS
1000 home.page_tag unchanged 0.968 [0.811, 1.155] 1.004 [0.688, 1.467] 1.253/1.489 PASS
1000 home.page_tag_created unchanged 0.982 [0.746, 1.293] 0.928 [0.434, 1.980] 1.205/1.755 PASS
1000 home.page_tag_title unchanged 0.940 [0.778, 1.136] 0.930 [0.744, 1.163] 1.154/1.275 PASS
1000 home.page_title unchanged 0.916 [0.749, 1.120] 1.033 [0.855, 1.250] 1.136/1.082 PASS
1000 mem.after_first_page memory 1.004 [0.970, 1.038] 1.004 [0.970, 1.038] 1.050/1.050 PASS
1000 mem.after_opens memory 0.996 [0.972, 1.021] 0.996 [0.972, 1.021] 1.009/1.009 PASS
1000 mem.after_search memory 1.004 [0.959, 1.052] 1.004 [0.959, 1.052] 1.058/1.058 PASS
1000 mem.idle memory 1.024 [0.992, 1.057] 1.024 [0.992, 1.057] 1.034/1.034 PASS
1000 notes.append changed 1.230 [0.969, 1.562] 1.112 [0.784, 1.577] 1.296/1.572 FAIL
1000 notes.append.first changed 1.162 [0.433, 3.119] 1.162 [0.433, 3.119] 1.584/1.584 FAIL
1000 open.api_get changed 1.137 [0.608, 2.125] 1.858 [0.308, 11.206] 1.525/2.480 FAIL
1000 open.api_get.warm changed 1.034 [0.762, 1.403] 1.060 [0.657, 1.710] 1.530/2.529 FAIL
1000 open.long_note changed 1.345 [0.313, 5.779] 1.345 [0.313, 5.779] 1.737/1.737 FAIL
1000 open.note changed 1.025 [0.797, 1.318] 0.951 [0.785, 1.153] 1.311/1.217 FAIL
1000 search.catch_up unchanged 1.000 [0.999, 1.001] 1.000 [0.999, 1.001] 11.489/11.489 PASS
1000 search.first_query unchanged 1.075 [0.811, 1.424] 1.075 [0.811, 1.424] 1.495/1.495 PASS
1000 search.query unchanged 0.955 [0.901, 1.011] 0.965 [0.877, 1.061] 1.163/1.367 PASS
10000 home.first_page changed 0.822 [0.238, 2.840] 0.822 [0.238, 2.840] 3.002/3.002 FAIL
10000 home.page unchanged 1.030 [0.765, 1.385] 0.695 [0.368, 1.310] 1.053/2.822 PASS
10000 home.page_big unchanged 0.976 [0.876, 1.088] 0.954 [0.828, 1.100] 1.056/1.106 PASS
10000 home.page_next unchanged 1.102 [0.923, 1.314] 1.147 [0.853, 1.544] 1.179/1.388 PASS
10000 home.page_tag unchanged 0.947 [0.885, 1.014] 0.926 [0.720, 1.191] 1.033/1.162 PASS
10000 home.page_tag_created unchanged 0.975 [0.875, 1.086] 0.971 [0.886, 1.065] 1.082/1.119 PASS
10000 home.page_tag_title unchanged 1.020 [0.862, 1.207] 0.999 [0.808, 1.234] 1.095/1.961 PASS
10000 home.page_title unchanged 1.049 [0.919, 1.198] 0.794 [0.220, 2.869] 1.051/1.194 PASS
10000 mem.after_first_page memory 0.980 [0.956, 1.005] 0.980 [0.956, 1.005] 1.052/1.052 PASS
10000 mem.after_opens memory 0.964 [0.929, 1.001] 0.964 [0.929, 1.001] 1.031/1.031 PASS
10000 mem.after_search memory 0.971 [0.931, 1.012] 0.971 [0.931, 1.012] 1.064/1.064 PASS
10000 mem.idle memory 0.998 [0.940, 1.061] 0.998 [0.940, 1.061] 1.021/1.021 PASS
10000 notes.append changed 0.840 [0.574, 1.229] 0.896 [0.648, 1.240] 1.397/1.271 FAIL
10000 notes.append.first changed 0.818 [0.392, 1.711] 0.818 [0.392, 1.711] 1.623/1.623 FAIL
10000 open.api_get changed 0.986 [0.499, 1.946] 0.925 [0.326, 2.628] 1.171/1.602 FAIL
10000 open.api_get.warm changed 1.002 [0.846, 1.186] 1.022 [0.893, 1.169] 1.261/2.003 FAIL
10000 open.long_note changed 1.016 [0.431, 2.393] 1.016 [0.431, 2.393] 2.555/2.555 FAIL
10000 open.note changed 0.637 [0.424, 0.957] 0.628 [0.363, 1.086] 3.427/3.254 FAIL
10000 search.catch_up unchanged 0.998 [0.992, 1.004] 0.998 [0.992, 1.004] 1.002/1.002 PASS
10000 search.first_query unchanged 0.542 [0.136, 2.167] 0.542 [0.136, 2.167] 2.707/2.707 PASS
10000 search.query unchanged 1.033 [0.965, 1.106] 0.754 [0.343, 1.656] 1.295/1.872 PASS
100000 home.first_page changed 0.870 [0.586, 1.293] 0.870 [0.586, 1.293] 1.262/1.262 FAIL
100000 home.page unchanged 0.981 [0.830, 1.159] 0.974 [0.511, 1.854] 1.136/4.568 PASS
100000 home.page_big unchanged 0.847 [0.809, 0.887] 0.851 [0.739, 0.980] 0.975/1.009 PASS
100000 home.page_next unchanged 0.935 [0.871, 1.005] 0.894 [0.752, 1.063] 1.105/1.730 PASS
100000 home.page_tag unchanged 0.978 [0.952, 1.005] 1.033 [0.924, 1.155] 1.012/1.063 PASS
100000 home.page_tag_created unchanged 0.925 [0.884, 0.968] 0.950 [0.919, 0.982] 0.934/0.934 PASS
100000 home.page_tag_title unchanged 0.988 [0.970, 1.006] 0.942 [0.888, 0.998] 1.024/0.992 PASS
100000 home.page_title unchanged 1.105 [0.906, 1.348] 1.449 [0.719, 2.922] 1.101/1.083 FAIL
100000 mem.after_first_page memory 0.894 [0.866, 0.924] 0.894 [0.866, 0.924] 1.036/1.036 PASS
100000 mem.after_opens memory 0.921 [0.891, 0.953] 0.921 [0.891, 0.953] 0.981/0.981 PASS
100000 mem.after_search memory 0.975 [0.916, 1.039] 0.975 [0.916, 1.039] 1.111/1.111 PASS
100000 mem.idle memory 1.002 [0.959, 1.046] 1.002 [0.959, 1.046] 1.148/1.148 PASS
100000 notes.append changed 0.993 [0.951, 1.038] 1.101 [0.849, 1.428] 1.094/1.135 FAIL
100000 notes.append.first changed 0.981 [0.889, 1.083] 0.981 [0.889, 1.083] 1.203/1.203 FAIL
100000 open.api_get changed 0.877 [0.773, 0.995] 0.987 [0.871, 1.119] 1.093/1.141 FAIL
100000 open.api_get.warm changed 0.916 [0.841, 0.997] 0.943 [0.777, 1.145] 0.982/1.111 FAIL
100000 open.long_note changed 0.892 [0.760, 1.045] 0.892 [0.760, 1.045] 1.090/1.090 FAIL
100000 open.note changed 0.903 [0.719, 1.135] 0.968 [0.856, 1.095] 1.143/1.188 FAIL
100000 search.catch_up unchanged 0.805 [0.543, 1.194] 0.805 [0.543, 1.194] 1.194/1.194 PASS
100000 search.first_query unchanged 1.033 [0.948, 1.125] 1.033 [0.948, 1.125] 1.132/1.132 PASS
100000 search.query unchanged 0.971 [0.901, 1.046] 1.086 [0.726, 1.625] 1.027/0.996 FAIL

OVERALL FAIL

1M plan and disk (per orchestrator, 11-09)

perf-01 has 123 GB free on / now (my ≤100k Homes use 13 GB). Mail measured about 45 GB free needed for a 1M cold setup; a settled dev 1M Home is about 25 GB. Four 1M Homes (~100 GB settled, plus the 45 GB peak and the 15 GB floor) do not fit, so 1M uses three Homes: dev control-a, s1 and s2. Its dev/dev control is control-a run again in the same turn, ordered A, s1, A′, s2. Before each 1M turn I check for ≥ 60 GB free (the 45 GB cold peak plus the 15 GB floor) and stop instead of filling the disk. Each turn is one ticket, taken only once binaries are staged: setups take one Home per turn (~40 min each), then one measuring turn.

## Fresh session H1 (1k/10k/100k): s1 vs dev Session H1: perf-01, 11-09 11:07–11:37 CET (15:37–16:07 IST). Both locks were held: perf-01 through `perf-turn`, perf-02 through `perf-lock`. Binaries (sha256): dev `d4b32af94` `b99ace57…6859`, s1 `381ce6750` `8b5292d6…bf0e`, s2 `eaeb417e7` `f7e5ad0f…10d0`, perf harness `b3682687…e4a2`. Each Home had its own setup. Every pair ran in the order control-a, s1, control-b, s2 (`--hot-only --reuse-indexed`, offset = pair). The gate reading is the one in the earlier comment: changed paths need a 95% upper bound below 1.00; unchanged paths need the point ratio no higher than the control's upper bound (or 1.00 if higher). Before H1, I deleted the predecessor's stale fixtures (96 GB `work-head841`, 18 GB of `abab-20261107` Homes; reports kept). That took 7 min at the start of the turn. **Result for #970: FAIL on the gate as written, but no real regression.** The paths s1 changes are level with dev; most CIs straddle 1.00. They don't beat it, except `open.api_get` p50 at 100k, 0.877 [0.773, 0.995]. Two unchanged paths miss by a hair: `home.page_title` p50 at 100k is 1.105 against a control bound of 1.101, and `search.query` p95 is 1.086 against 0.996. The earlier +32 % `page_big` at 100k did not come back: it is in the PASS rows below. <details><summary>Full gate table (s1)</summary> | size | path | kind | s1/dev p50 [CI] | s1/dev p95 [CI] | control UCB p50/p95 | verdict | |---|---|---|---|---|---|---| | 1000 | home.first_page | changed | 0.802 [0.609, 1.057] | 0.802 [0.609, 1.057] | 1.392/1.392 | **FAIL** | | 1000 | home.page | unchanged | 0.883 [0.727, 1.074] | 0.620 [0.398, 0.965] | 1.088/2.375 | PASS | | 1000 | home.page_big | unchanged | 1.078 [0.758, 1.534] | 1.078 [0.758, 1.534] | 1.111/1.111 | PASS | | 1000 | home.page_next | unchanged | 0.971 [0.816, 1.155] | 0.910 [0.750, 1.105] | 1.214/1.568 | PASS | | 1000 | home.page_tag | unchanged | 0.968 [0.811, 1.155] | 1.004 [0.688, 1.467] | 1.253/1.489 | PASS | | 1000 | home.page_tag_created | unchanged | 0.982 [0.746, 1.293] | 0.928 [0.434, 1.980] | 1.205/1.755 | PASS | | 1000 | home.page_tag_title | unchanged | 0.940 [0.778, 1.136] | 0.930 [0.744, 1.163] | 1.154/1.275 | PASS | | 1000 | home.page_title | unchanged | 0.916 [0.749, 1.120] | 1.033 [0.855, 1.250] | 1.136/1.082 | PASS | | 1000 | mem.after_first_page | memory | 1.004 [0.970, 1.038] | 1.004 [0.970, 1.038] | 1.050/1.050 | PASS | | 1000 | mem.after_opens | memory | 0.996 [0.972, 1.021] | 0.996 [0.972, 1.021] | 1.009/1.009 | PASS | | 1000 | mem.after_search | memory | 1.004 [0.959, 1.052] | 1.004 [0.959, 1.052] | 1.058/1.058 | PASS | | 1000 | mem.idle | memory | 1.024 [0.992, 1.057] | 1.024 [0.992, 1.057] | 1.034/1.034 | PASS | | 1000 | notes.append | changed | 1.230 [0.969, 1.562] | 1.112 [0.784, 1.577] | 1.296/1.572 | **FAIL** | | 1000 | notes.append.first | changed | 1.162 [0.433, 3.119] | 1.162 [0.433, 3.119] | 1.584/1.584 | **FAIL** | | 1000 | open.api_get | changed | 1.137 [0.608, 2.125] | 1.858 [0.308, 11.206] | 1.525/2.480 | **FAIL** | | 1000 | open.api_get.warm | changed | 1.034 [0.762, 1.403] | 1.060 [0.657, 1.710] | 1.530/2.529 | **FAIL** | | 1000 | open.long_note | changed | 1.345 [0.313, 5.779] | 1.345 [0.313, 5.779] | 1.737/1.737 | **FAIL** | | 1000 | open.note | changed | 1.025 [0.797, 1.318] | 0.951 [0.785, 1.153] | 1.311/1.217 | **FAIL** | | 1000 | search.catch_up | unchanged | 1.000 [0.999, 1.001] | 1.000 [0.999, 1.001] | 11.489/11.489 | PASS | | 1000 | search.first_query | unchanged | 1.075 [0.811, 1.424] | 1.075 [0.811, 1.424] | 1.495/1.495 | PASS | | 1000 | search.query | unchanged | 0.955 [0.901, 1.011] | 0.965 [0.877, 1.061] | 1.163/1.367 | PASS | | 10000 | home.first_page | changed | 0.822 [0.238, 2.840] | 0.822 [0.238, 2.840] | 3.002/3.002 | **FAIL** | | 10000 | home.page | unchanged | 1.030 [0.765, 1.385] | 0.695 [0.368, 1.310] | 1.053/2.822 | PASS | | 10000 | home.page_big | unchanged | 0.976 [0.876, 1.088] | 0.954 [0.828, 1.100] | 1.056/1.106 | PASS | | 10000 | home.page_next | unchanged | 1.102 [0.923, 1.314] | 1.147 [0.853, 1.544] | 1.179/1.388 | PASS | | 10000 | home.page_tag | unchanged | 0.947 [0.885, 1.014] | 0.926 [0.720, 1.191] | 1.033/1.162 | PASS | | 10000 | home.page_tag_created | unchanged | 0.975 [0.875, 1.086] | 0.971 [0.886, 1.065] | 1.082/1.119 | PASS | | 10000 | home.page_tag_title | unchanged | 1.020 [0.862, 1.207] | 0.999 [0.808, 1.234] | 1.095/1.961 | PASS | | 10000 | home.page_title | unchanged | 1.049 [0.919, 1.198] | 0.794 [0.220, 2.869] | 1.051/1.194 | PASS | | 10000 | mem.after_first_page | memory | 0.980 [0.956, 1.005] | 0.980 [0.956, 1.005] | 1.052/1.052 | PASS | | 10000 | mem.after_opens | memory | 0.964 [0.929, 1.001] | 0.964 [0.929, 1.001] | 1.031/1.031 | PASS | | 10000 | mem.after_search | memory | 0.971 [0.931, 1.012] | 0.971 [0.931, 1.012] | 1.064/1.064 | PASS | | 10000 | mem.idle | memory | 0.998 [0.940, 1.061] | 0.998 [0.940, 1.061] | 1.021/1.021 | PASS | | 10000 | notes.append | changed | 0.840 [0.574, 1.229] | 0.896 [0.648, 1.240] | 1.397/1.271 | **FAIL** | | 10000 | notes.append.first | changed | 0.818 [0.392, 1.711] | 0.818 [0.392, 1.711] | 1.623/1.623 | **FAIL** | | 10000 | open.api_get | changed | 0.986 [0.499, 1.946] | 0.925 [0.326, 2.628] | 1.171/1.602 | **FAIL** | | 10000 | open.api_get.warm | changed | 1.002 [0.846, 1.186] | 1.022 [0.893, 1.169] | 1.261/2.003 | **FAIL** | | 10000 | open.long_note | changed | 1.016 [0.431, 2.393] | 1.016 [0.431, 2.393] | 2.555/2.555 | **FAIL** | | 10000 | open.note | changed | 0.637 [0.424, 0.957] | 0.628 [0.363, 1.086] | 3.427/3.254 | **FAIL** | | 10000 | search.catch_up | unchanged | 0.998 [0.992, 1.004] | 0.998 [0.992, 1.004] | 1.002/1.002 | PASS | | 10000 | search.first_query | unchanged | 0.542 [0.136, 2.167] | 0.542 [0.136, 2.167] | 2.707/2.707 | PASS | | 10000 | search.query | unchanged | 1.033 [0.965, 1.106] | 0.754 [0.343, 1.656] | 1.295/1.872 | PASS | | 100000 | home.first_page | changed | 0.870 [0.586, 1.293] | 0.870 [0.586, 1.293] | 1.262/1.262 | **FAIL** | | 100000 | home.page | unchanged | 0.981 [0.830, 1.159] | 0.974 [0.511, 1.854] | 1.136/4.568 | PASS | | 100000 | home.page_big | unchanged | 0.847 [0.809, 0.887] | 0.851 [0.739, 0.980] | 0.975/1.009 | PASS | | 100000 | home.page_next | unchanged | 0.935 [0.871, 1.005] | 0.894 [0.752, 1.063] | 1.105/1.730 | PASS | | 100000 | home.page_tag | unchanged | 0.978 [0.952, 1.005] | 1.033 [0.924, 1.155] | 1.012/1.063 | PASS | | 100000 | home.page_tag_created | unchanged | 0.925 [0.884, 0.968] | 0.950 [0.919, 0.982] | 0.934/0.934 | PASS | | 100000 | home.page_tag_title | unchanged | 0.988 [0.970, 1.006] | 0.942 [0.888, 0.998] | 1.024/0.992 | PASS | | 100000 | home.page_title | unchanged | 1.105 [0.906, 1.348] | 1.449 [0.719, 2.922] | 1.101/1.083 | **FAIL** | | 100000 | mem.after_first_page | memory | 0.894 [0.866, 0.924] | 0.894 [0.866, 0.924] | 1.036/1.036 | PASS | | 100000 | mem.after_opens | memory | 0.921 [0.891, 0.953] | 0.921 [0.891, 0.953] | 0.981/0.981 | PASS | | 100000 | mem.after_search | memory | 0.975 [0.916, 1.039] | 0.975 [0.916, 1.039] | 1.111/1.111 | PASS | | 100000 | mem.idle | memory | 1.002 [0.959, 1.046] | 1.002 [0.959, 1.046] | 1.148/1.148 | PASS | | 100000 | notes.append | changed | 0.993 [0.951, 1.038] | 1.101 [0.849, 1.428] | 1.094/1.135 | **FAIL** | | 100000 | notes.append.first | changed | 0.981 [0.889, 1.083] | 0.981 [0.889, 1.083] | 1.203/1.203 | **FAIL** | | 100000 | open.api_get | changed | 0.877 [0.773, 0.995] | 0.987 [0.871, 1.119] | 1.093/1.141 | **FAIL** | | 100000 | open.api_get.warm | changed | 0.916 [0.841, 0.997] | 0.943 [0.777, 1.145] | 0.982/1.111 | **FAIL** | | 100000 | open.long_note | changed | 0.892 [0.760, 1.045] | 0.892 [0.760, 1.045] | 1.090/1.090 | **FAIL** | | 100000 | open.note | changed | 0.903 [0.719, 1.135] | 0.968 [0.856, 1.095] | 1.143/1.188 | **FAIL** | | 100000 | search.catch_up | unchanged | 0.805 [0.543, 1.194] | 0.805 [0.543, 1.194] | 1.194/1.194 | PASS | | 100000 | search.first_query | unchanged | 1.033 [0.948, 1.125] | 1.033 [0.948, 1.125] | 1.132/1.132 | PASS | | 100000 | search.query | unchanged | 0.971 [0.901, 1.046] | 1.086 [0.726, 1.625] | 1.027/0.996 | **FAIL** | OVERALL FAIL </details> ### 1M plan and disk (per orchestrator, 11-09) perf-01 has 123 GB free on / now (my ≤100k Homes use 13 GB). Mail measured about 45 GB free needed for a 1M cold setup; a settled dev 1M Home is about 25 GB. Four 1M Homes (~100 GB settled, plus the 45 GB peak and the 15 GB floor) **do not fit**, so 1M uses three Homes: dev control-a, s1 and s2. Its dev/dev control is control-a run again in the same turn, ordered A, s1, A′, s2. Before each 1M turn I check for ≥ 60 GB free (the 45 GB cold peak plus the 15 GB floor) and stop instead of filling the disk. Each turn is one ticket, taken only once binaries are staged: setups take one Home per turn (~40 min each), then one measuring turn.
kayg commented 2026-11-09 16:03:17 +00:00
Author
Owner
Copy link

From #1010's perf lane: where the cold first page goes at 100k (for #970's owner)

While fixing #1010 (stacked on this PR) I timed the cold Notes scan phase by phase at 100k notes. Temporary eprintln! timers, never committed; release builds of dev d4b32af94 and #1010 0c99d74e9, which has #970's identity store with #1010's extra B-trees removed. compute02, so the numbers show direction only; 2 runs each.

Phase (seconds since scan start) dev run 1 dev run 2 #1010 run 1 #1010 run 2
stat all + load state 0.99 1.48 0.99 0.89
compare with index 1.26 1.98 1.24 1.11
read_ids done 3.29 6.97 2.60 3.22
identity loop + commits done 3.96 8.06 8.16 9.67
identity phase alone 0.67 1.09 5.56 6.46

The first page can't appear before the identity phase ends (events/meta_ids run after it), so this phase is the cold first-page gap. On #970's storage, 13 commit_batch calls of 8,192 rows take 4.4 s of it, growing with table size (138, 143, 251, 203, 267, 492, 478, 414, 340, 794, 405, 299, 138 ms). The rows arrive in path order, so their IDs land at random across the WITHOUT ROWID table: each commit touches most of the table's leaf pages through a 64 KiB page cache and rewrites them into the WAL. That also matches the extra cold.disk_written (~+140 MB at 100k vs dev's append-only ids.jsonl).

Dev's equivalent phase is one ids.jsonl append per row and one durability wait. Stack samples of a full #970 cold run show commit_batch at ~2% of total CPU, so the cost is I/O and page churn in this one phase, not CPU across the run.

Possible directions (not implemented here; this is #970's storage):

  • insert each cold batch in ID order, or the whole cold load in ID order;
  • defer WAL checkpoints until the bulk load ends;
  • a transient larger page cache only during the cold bulk load (memory budget permitting).

#1010's perf comments will show its arms next to a #970 arm, so each delta can be attributed.

## From #1010's perf lane: where the cold first page goes at 100k (for #970's owner) While fixing #1010 (stacked on this PR) I timed the cold Notes scan phase by phase at 100k notes. Temporary `eprintln!` timers, never committed; release builds of dev `d4b32af94` and #1010 `0c99d74e9`, which has #970's identity store with #1010's extra B-trees removed. compute02, so the numbers show direction only; 2 runs each. | Phase (seconds since scan start) | dev run 1 | dev run 2 | #1010 run 1 | #1010 run 2 | |---|---:|---:|---:|---:| | stat all + load state | 0.99 | 1.48 | 0.99 | 0.89 | | compare with index | 1.26 | 1.98 | 1.24 | 1.11 | | `read_ids` done | 3.29 | 6.97 | 2.60 | 3.22 | | identity loop + commits done | 3.96 | 8.06 | **8.16** | **9.67** | | identity phase alone | **0.67** | **1.09** | **5.56** | **6.46** | The first page can't appear before the identity phase ends (events/`meta_ids` run after it), so this phase is the cold first-page gap. On #970's storage, 13 `commit_batch` calls of 8,192 rows take 4.4 s of it, growing with table size (138, 143, 251, 203, 267, 492, 478, 414, 340, 794, 405, 299, 138 ms). The rows arrive in path order, so their IDs land at random across the `WITHOUT ROWID` table: each commit touches most of the table's leaf pages through a 64 KiB page cache and rewrites them into the WAL. That also matches the extra `cold.disk_written` (~+140 MB at 100k vs dev's append-only `ids.jsonl`). Dev's equivalent phase is one `ids.jsonl` append per row and one durability wait. Stack samples of a full #970 cold run show `commit_batch` at ~2% of total CPU, so the cost is I/O and page churn in this one phase, not CPU across the run. Possible directions (not implemented here; this is #970's storage): - insert each cold batch in ID order, or the whole cold load in ID order; - defer WAL checkpoints until the bulk load ends; - a transient larger page cache only during the cold bulk load (memory budget permitting). #1010's perf comments will show its arms next to a #970 arm, so each delta can be attributed.
kayg commented 2026-11-09 16:59:43 +00:00
Author
Owner
Copy link

H1b re-measure (1k/10k/100k): the H1 near-misses were noise

Session H1b: perf-01, 11-09 16:48–16:54 CET (21:18–21:24 IST). Both locks were held: perf-01 through perf-turn, perf-02 through perf-lock. The Homes from H1 were reused (no new setups). Pairs ran in the order control-a, s1, control-b, s2fix (ee00d638c, sha256 84e5efcb…17d6), at offsets 0–4.

  • At 100k, home.page_title p50 is now 0.880 (control bound 1.286) and search.query p95 is 0.940 (bound 1.179). Both PASS; the H1 misses were noise.
  • New wobbles on other unchanged paths, all within about ±5 % of the control: 10k page_title p50 1.134 vs a bound of 1.116; 100k page_tag_title p50 1.052 vs 1.021. They flip between sessions in both directions. No path regresses in both H1 and H1b.
  • The paths s1 changes are level with dev again. Per the orchestrator (11-09), a tie at ≤100k is not a regression; the result at 1M decides.
Gate table (s1, H1b)
size path kind s1/dev p50 [CI] s1/dev p95 [CI] control UCB p50/p95 verdict
1000 home.first_page changed 1.047 [0.664, 1.652] 1.047 [0.664, 1.652] 1.121/1.121 FAIL
1000 home.page unchanged 0.940 [0.802, 1.101] 1.123 [0.622, 2.028] 1.109/2.003 PASS
1000 home.page_big unchanged 0.936 [0.880, 0.995] 0.936 [0.880, 0.995] 1.185/1.185 PASS
1000 home.page_next unchanged 0.981 [0.952, 1.010] 1.016 [0.866, 1.193] 1.031/1.213 PASS
1000 home.page_tag unchanged 0.999 [0.935, 1.068] 1.166 [0.906, 1.499] 1.127/1.497 PASS
1000 home.page_tag_created unchanged 0.979 [0.925, 1.036] 0.924 [0.821, 1.040] 1.062/1.124 PASS
1000 home.page_tag_title unchanged 1.003 [0.961, 1.046] 0.985 [0.952, 1.018] 1.085/1.166 PASS
1000 home.page_title unchanged 1.019 [0.946, 1.099] 1.089 [0.748, 1.586] 1.198/1.384 PASS
1000 mem.after_first_page memory 0.957 [0.945, 0.969] 0.957 [0.945, 0.969] 1.024/1.024 PASS
1000 mem.after_opens memory 1.000 [0.989, 1.010] 1.000 [0.989, 1.010] 1.014/1.014 PASS
1000 mem.after_search memory 1.004 [0.947, 1.064] 1.004 [0.947, 1.064] 1.072/1.072 PASS
1000 mem.idle memory 0.993 [0.963, 1.024] 0.993 [0.963, 1.024] 1.042/1.042 PASS
1000 notes.append changed 1.012 [0.935, 1.095] 1.013 [0.894, 1.147] 1.047/1.089 FAIL
1000 notes.append.first changed 1.162 [0.850, 1.588] 1.162 [0.850, 1.588] 2.589/2.589 FAIL
1000 open.api_get changed 1.035 [0.978, 1.096] 0.902 [0.719, 1.131] 1.093/1.235 FAIL
1000 open.api_get.warm changed 0.984 [0.840, 1.151] 0.969 [0.818, 1.148] 1.104/1.224 FAIL
1000 open.long_note changed 1.082 [0.868, 1.350] 1.082 [0.868, 1.350] 1.339/1.339 FAIL
1000 open.note changed 0.990 [0.902, 1.086] 1.022 [0.930, 1.122] 1.225/1.166 FAIL
1000 search.catch_up unchanged 1.000 [0.999, 1.000] 1.000 [0.999, 1.000] 1.001/1.001 PASS
1000 search.first_query unchanged 1.046 [0.762, 1.437] 1.046 [0.762, 1.437] 1.423/1.423 PASS
1000 search.query unchanged 0.933 [0.863, 1.009] 0.988 [0.923, 1.058] 1.106/1.102 PASS
10000 home.first_page changed 0.986 [0.662, 1.467] 0.986 [0.662, 1.467] 1.574/1.574 FAIL
10000 home.page unchanged 1.099 [0.969, 1.246] 0.773 [0.294, 2.029] 1.162/1.723 PASS
10000 home.page_big unchanged 1.004 [0.975, 1.033] 1.020 [0.988, 1.054] 1.044/1.150 PASS
10000 home.page_next unchanged 0.999 [0.918, 1.089] 1.105 [0.974, 1.255] 1.157/1.257 PASS
10000 home.page_tag unchanged 1.034 [0.954, 1.122] 0.991 [0.867, 1.133] 1.060/1.130 PASS
10000 home.page_tag_created unchanged 1.024 [0.999, 1.050] 1.221 [1.068, 1.396] 1.047/1.116 FAIL
10000 home.page_tag_title unchanged 1.010 [0.984, 1.037] 1.114 [0.849, 1.460] 1.093/1.449 PASS
10000 home.page_title unchanged 1.134 [0.985, 1.305] 1.579 [0.732, 3.406] 1.116/3.605 FAIL
10000 mem.after_first_page memory 0.988 [0.942, 1.036] 0.988 [0.942, 1.036] 1.028/1.028 PASS
10000 mem.after_opens memory 0.999 [0.948, 1.053] 0.999 [0.948, 1.053] 1.088/1.088 PASS
10000 mem.after_search memory 1.007 [0.944, 1.074] 1.007 [0.944, 1.074] 1.051/1.051 PASS
10000 mem.idle memory 1.015 [0.965, 1.067] 1.015 [0.965, 1.067] 1.024/1.024 PASS
10000 notes.append changed 1.014 [0.934, 1.100] 0.977 [0.721, 1.325] 1.061/1.286 FAIL
10000 notes.append.first changed 0.937 [0.620, 1.416] 0.937 [0.620, 1.416] 1.804/1.804 FAIL
10000 open.api_get changed 1.008 [0.981, 1.035] 1.065 [0.816, 1.389] 1.176/1.229 FAIL
10000 open.api_get.warm changed 1.028 [0.953, 1.109] 0.976 [0.920, 1.034] 1.060/1.020 FAIL
10000 open.long_note changed 1.022 [0.926, 1.127] 1.022 [0.926, 1.127] 1.128/1.128 FAIL
10000 open.note changed 0.996 [0.933, 1.064] 1.018 [0.876, 1.182] 1.178/1.120 FAIL
10000 search.catch_up unchanged 1.000 [0.999, 1.001] 1.000 [0.999, 1.001] 1.000/1.000 PASS
10000 search.first_query unchanged 0.881 [0.729, 1.065] 0.881 [0.729, 1.065] 1.150/1.150 PASS
10000 search.query unchanged 0.980 [0.948, 1.012] 0.856 [0.602, 1.216] 1.117/1.311 PASS
100000 home.first_page changed 0.677 [0.431, 1.062] 0.677 [0.431, 1.062] 1.034/1.034 FAIL
100000 home.page unchanged 0.850 [0.625, 1.157] 0.625 [0.326, 1.198] 1.226/1.287 PASS
100000 home.page_big unchanged 1.024 [0.962, 1.090] 0.920 [0.860, 0.983] 0.999/0.861 FAIL
100000 home.page_next unchanged 1.033 [0.967, 1.103] 0.978 [0.821, 1.164] 1.040/1.189 PASS
100000 home.page_tag unchanged 0.997 [0.991, 1.003] 1.030 [0.972, 1.091] 1.027/1.150 PASS
100000 home.page_tag_created unchanged 1.012 [0.984, 1.042] 1.020 [0.995, 1.046] 0.967/0.994 FAIL
100000 home.page_tag_title unchanged 1.052 [1.027, 1.077] 1.079 [1.036, 1.123] 1.021/1.005 FAIL
100000 home.page_title unchanged 0.880 [0.685, 1.132] 0.686 [0.341, 1.380] 1.286/1.684 PASS
100000 mem.after_first_page memory 0.866 [0.810, 0.926] 0.866 [0.810, 0.926] 1.041/1.041 PASS
100000 mem.after_opens memory 0.981 [0.926, 1.039] 0.981 [0.926, 1.039] 1.020/1.020 PASS
100000 mem.after_search memory 1.045 [0.984, 1.109] 1.045 [0.984, 1.109] 1.238/1.238 PASS
100000 mem.idle memory 0.999 [0.958, 1.042] 0.999 [0.958, 1.042] 1.121/1.121 PASS
100000 notes.append changed 0.982 [0.949, 1.017] 0.969 [0.764, 1.229] 1.064/1.017 FAIL
100000 notes.append.first changed 1.056 [0.853, 1.308] 1.056 [0.853, 1.308] 1.672/1.672 FAIL
100000 open.api_get changed 0.912 [0.800, 1.039] 0.991 [0.947, 1.038] 0.939/1.270 FAIL
100000 open.api_get.warm changed 0.983 [0.883, 1.093] 1.009 [0.903, 1.128] 1.166/1.183 FAIL
100000 open.long_note changed 1.018 [0.923, 1.123] 1.018 [0.923, 1.123] 1.055/1.055 FAIL
100000 open.note changed 0.923 [0.837, 1.017] 1.135 [0.894, 1.442] 0.941/1.033 FAIL
100000 search.catch_up unchanged 0.575 [0.391, 0.844] 0.575 [0.391, 0.844] 0.844/0.844 PASS
100000 search.first_query unchanged 1.079 [0.837, 1.391] 1.079 [0.837, 1.391] 1.996/1.996 PASS
100000 search.query unchanged 0.984 [0.930, 1.041] 0.940 [0.816, 1.082] 1.125/1.179 PASS

OVERALL FAIL

Next: one more ≤100k turn for #1008, then the 1M turns (three Homes, ≥ 60 GB free check before each).

## H1b re-measure (1k/10k/100k): the H1 near-misses were noise Session H1b: perf-01, 11-09 16:48–16:54 CET (21:18–21:24 IST). Both locks were held: perf-01 through `perf-turn`, perf-02 through `perf-lock`. The Homes from H1 were reused (no new setups). Pairs ran in the order control-a, s1, control-b, s2fix (`ee00d638c`, sha256 `84e5efcb…17d6`), at offsets 0–4. - At 100k, `home.page_title` p50 is now 0.880 (control bound 1.286) and `search.query` p95 is 0.940 (bound 1.179). Both PASS; the H1 misses were noise. - New wobbles on other unchanged paths, all within about ±5 % of the control: 10k `page_title` p50 1.134 vs a bound of 1.116; 100k `page_tag_title` p50 1.052 vs 1.021. They flip between sessions in both directions. No path regresses in both H1 and H1b. - The paths s1 changes are level with dev again. Per the orchestrator (11-09), a tie at ≤100k is not a regression; the result at 1M decides. <details><summary>Gate table (s1, H1b)</summary> | size | path | kind | s1/dev p50 [CI] | s1/dev p95 [CI] | control UCB p50/p95 | verdict | |---|---|---|---|---|---|---| | 1000 | home.first_page | changed | 1.047 [0.664, 1.652] | 1.047 [0.664, 1.652] | 1.121/1.121 | **FAIL** | | 1000 | home.page | unchanged | 0.940 [0.802, 1.101] | 1.123 [0.622, 2.028] | 1.109/2.003 | PASS | | 1000 | home.page_big | unchanged | 0.936 [0.880, 0.995] | 0.936 [0.880, 0.995] | 1.185/1.185 | PASS | | 1000 | home.page_next | unchanged | 0.981 [0.952, 1.010] | 1.016 [0.866, 1.193] | 1.031/1.213 | PASS | | 1000 | home.page_tag | unchanged | 0.999 [0.935, 1.068] | 1.166 [0.906, 1.499] | 1.127/1.497 | PASS | | 1000 | home.page_tag_created | unchanged | 0.979 [0.925, 1.036] | 0.924 [0.821, 1.040] | 1.062/1.124 | PASS | | 1000 | home.page_tag_title | unchanged | 1.003 [0.961, 1.046] | 0.985 [0.952, 1.018] | 1.085/1.166 | PASS | | 1000 | home.page_title | unchanged | 1.019 [0.946, 1.099] | 1.089 [0.748, 1.586] | 1.198/1.384 | PASS | | 1000 | mem.after_first_page | memory | 0.957 [0.945, 0.969] | 0.957 [0.945, 0.969] | 1.024/1.024 | PASS | | 1000 | mem.after_opens | memory | 1.000 [0.989, 1.010] | 1.000 [0.989, 1.010] | 1.014/1.014 | PASS | | 1000 | mem.after_search | memory | 1.004 [0.947, 1.064] | 1.004 [0.947, 1.064] | 1.072/1.072 | PASS | | 1000 | mem.idle | memory | 0.993 [0.963, 1.024] | 0.993 [0.963, 1.024] | 1.042/1.042 | PASS | | 1000 | notes.append | changed | 1.012 [0.935, 1.095] | 1.013 [0.894, 1.147] | 1.047/1.089 | **FAIL** | | 1000 | notes.append.first | changed | 1.162 [0.850, 1.588] | 1.162 [0.850, 1.588] | 2.589/2.589 | **FAIL** | | 1000 | open.api_get | changed | 1.035 [0.978, 1.096] | 0.902 [0.719, 1.131] | 1.093/1.235 | **FAIL** | | 1000 | open.api_get.warm | changed | 0.984 [0.840, 1.151] | 0.969 [0.818, 1.148] | 1.104/1.224 | **FAIL** | | 1000 | open.long_note | changed | 1.082 [0.868, 1.350] | 1.082 [0.868, 1.350] | 1.339/1.339 | **FAIL** | | 1000 | open.note | changed | 0.990 [0.902, 1.086] | 1.022 [0.930, 1.122] | 1.225/1.166 | **FAIL** | | 1000 | search.catch_up | unchanged | 1.000 [0.999, 1.000] | 1.000 [0.999, 1.000] | 1.001/1.001 | PASS | | 1000 | search.first_query | unchanged | 1.046 [0.762, 1.437] | 1.046 [0.762, 1.437] | 1.423/1.423 | PASS | | 1000 | search.query | unchanged | 0.933 [0.863, 1.009] | 0.988 [0.923, 1.058] | 1.106/1.102 | PASS | | 10000 | home.first_page | changed | 0.986 [0.662, 1.467] | 0.986 [0.662, 1.467] | 1.574/1.574 | **FAIL** | | 10000 | home.page | unchanged | 1.099 [0.969, 1.246] | 0.773 [0.294, 2.029] | 1.162/1.723 | PASS | | 10000 | home.page_big | unchanged | 1.004 [0.975, 1.033] | 1.020 [0.988, 1.054] | 1.044/1.150 | PASS | | 10000 | home.page_next | unchanged | 0.999 [0.918, 1.089] | 1.105 [0.974, 1.255] | 1.157/1.257 | PASS | | 10000 | home.page_tag | unchanged | 1.034 [0.954, 1.122] | 0.991 [0.867, 1.133] | 1.060/1.130 | PASS | | 10000 | home.page_tag_created | unchanged | 1.024 [0.999, 1.050] | 1.221 [1.068, 1.396] | 1.047/1.116 | **FAIL** | | 10000 | home.page_tag_title | unchanged | 1.010 [0.984, 1.037] | 1.114 [0.849, 1.460] | 1.093/1.449 | PASS | | 10000 | home.page_title | unchanged | 1.134 [0.985, 1.305] | 1.579 [0.732, 3.406] | 1.116/3.605 | **FAIL** | | 10000 | mem.after_first_page | memory | 0.988 [0.942, 1.036] | 0.988 [0.942, 1.036] | 1.028/1.028 | PASS | | 10000 | mem.after_opens | memory | 0.999 [0.948, 1.053] | 0.999 [0.948, 1.053] | 1.088/1.088 | PASS | | 10000 | mem.after_search | memory | 1.007 [0.944, 1.074] | 1.007 [0.944, 1.074] | 1.051/1.051 | PASS | | 10000 | mem.idle | memory | 1.015 [0.965, 1.067] | 1.015 [0.965, 1.067] | 1.024/1.024 | PASS | | 10000 | notes.append | changed | 1.014 [0.934, 1.100] | 0.977 [0.721, 1.325] | 1.061/1.286 | **FAIL** | | 10000 | notes.append.first | changed | 0.937 [0.620, 1.416] | 0.937 [0.620, 1.416] | 1.804/1.804 | **FAIL** | | 10000 | open.api_get | changed | 1.008 [0.981, 1.035] | 1.065 [0.816, 1.389] | 1.176/1.229 | **FAIL** | | 10000 | open.api_get.warm | changed | 1.028 [0.953, 1.109] | 0.976 [0.920, 1.034] | 1.060/1.020 | **FAIL** | | 10000 | open.long_note | changed | 1.022 [0.926, 1.127] | 1.022 [0.926, 1.127] | 1.128/1.128 | **FAIL** | | 10000 | open.note | changed | 0.996 [0.933, 1.064] | 1.018 [0.876, 1.182] | 1.178/1.120 | **FAIL** | | 10000 | search.catch_up | unchanged | 1.000 [0.999, 1.001] | 1.000 [0.999, 1.001] | 1.000/1.000 | PASS | | 10000 | search.first_query | unchanged | 0.881 [0.729, 1.065] | 0.881 [0.729, 1.065] | 1.150/1.150 | PASS | | 10000 | search.query | unchanged | 0.980 [0.948, 1.012] | 0.856 [0.602, 1.216] | 1.117/1.311 | PASS | | 100000 | home.first_page | changed | 0.677 [0.431, 1.062] | 0.677 [0.431, 1.062] | 1.034/1.034 | **FAIL** | | 100000 | home.page | unchanged | 0.850 [0.625, 1.157] | 0.625 [0.326, 1.198] | 1.226/1.287 | PASS | | 100000 | home.page_big | unchanged | 1.024 [0.962, 1.090] | 0.920 [0.860, 0.983] | 0.999/0.861 | **FAIL** | | 100000 | home.page_next | unchanged | 1.033 [0.967, 1.103] | 0.978 [0.821, 1.164] | 1.040/1.189 | PASS | | 100000 | home.page_tag | unchanged | 0.997 [0.991, 1.003] | 1.030 [0.972, 1.091] | 1.027/1.150 | PASS | | 100000 | home.page_tag_created | unchanged | 1.012 [0.984, 1.042] | 1.020 [0.995, 1.046] | 0.967/0.994 | **FAIL** | | 100000 | home.page_tag_title | unchanged | 1.052 [1.027, 1.077] | 1.079 [1.036, 1.123] | 1.021/1.005 | **FAIL** | | 100000 | home.page_title | unchanged | 0.880 [0.685, 1.132] | 0.686 [0.341, 1.380] | 1.286/1.684 | PASS | | 100000 | mem.after_first_page | memory | 0.866 [0.810, 0.926] | 0.866 [0.810, 0.926] | 1.041/1.041 | PASS | | 100000 | mem.after_opens | memory | 0.981 [0.926, 1.039] | 0.981 [0.926, 1.039] | 1.020/1.020 | PASS | | 100000 | mem.after_search | memory | 1.045 [0.984, 1.109] | 1.045 [0.984, 1.109] | 1.238/1.238 | PASS | | 100000 | mem.idle | memory | 0.999 [0.958, 1.042] | 0.999 [0.958, 1.042] | 1.121/1.121 | PASS | | 100000 | notes.append | changed | 0.982 [0.949, 1.017] | 0.969 [0.764, 1.229] | 1.064/1.017 | **FAIL** | | 100000 | notes.append.first | changed | 1.056 [0.853, 1.308] | 1.056 [0.853, 1.308] | 1.672/1.672 | **FAIL** | | 100000 | open.api_get | changed | 0.912 [0.800, 1.039] | 0.991 [0.947, 1.038] | 0.939/1.270 | **FAIL** | | 100000 | open.api_get.warm | changed | 0.983 [0.883, 1.093] | 1.009 [0.903, 1.128] | 1.166/1.183 | **FAIL** | | 100000 | open.long_note | changed | 1.018 [0.923, 1.123] | 1.018 [0.923, 1.123] | 1.055/1.055 | **FAIL** | | 100000 | open.note | changed | 0.923 [0.837, 1.017] | 1.135 [0.894, 1.442] | 0.941/1.033 | **FAIL** | | 100000 | search.catch_up | unchanged | 0.575 [0.391, 0.844] | 0.575 [0.391, 0.844] | 0.844/0.844 | PASS | | 100000 | search.first_query | unchanged | 1.079 [0.837, 1.391] | 1.079 [0.837, 1.391] | 1.996/1.996 | PASS | | 100000 | search.query | unchanged | 0.984 [0.930, 1.041] | 0.940 [0.816, 1.082] | 1.125/1.179 | PASS | OVERALL FAIL </details> Next: one more ≤100k turn for #1008, then the 1M turns (three Homes, ≥ 60 GB free check before each).
kayg commented 2026-11-09 18:56:37 +00:00
Author
Owner
Copy link

Reconciling with #1010's cold-path numbers (comment 88985): they're right; my sessions never measured cold

My H1 and H1b sessions ran the harness with --hot-only --reuse-indexed. Each Home was indexed during setup, and setup used --cold-runs 0: the harness still indexes once, but cold.* is not recorded. So every number I posted is a warm path, and "s1 ties dev at ≤100k" holds for warm paths only. cold.first_page, cold.index, cold.notes_per_s and cold.disk_written were never measured here. The #1010 lane measured exactly those, and the regression is real.

It reproduces on dev-01 (cold, 100k; loaded host, direction only):

run dev cold.first_page s1 cold.first_page dev disk_written s1 disk_written
1 4,116 ms 9,570 ms 1,920 MB 2,064 MB
2 5,021 ms 7,453 ms 1,918 MB 2,059 MB

Fix under test (wip/tiering-v3-cold, 1e8a8adef):

  • Namespace::put_many sorts each batch by ID, so primary-key inserts walk the B-tree in order instead of touching leaves at random.
  • Batches of ≥1,024 rows get a 16 MiB SQLite page cache for that one transaction, then the resting 64 KiB (or 0) setting is restored.
  • Cold discovery batches go from 8,192 to 65,536 rows, so 100k notes means 2 commits and fsyncs instead of 13.

The single dev-01 sample of this build was taken while the host was swamped (startup 686 ms against a usual 35 ms), so it tells nothing yet. A perf-01 turn is queued: cold ABAB ×5 with dev, s1, a dev copy and fixed s1 per round, all three sizes per arm. If that doesn't close the gap, the next steps are the ones the orchestrator listed: serve the first page before the identity backfill finishes, and defer the rest to idle.

Further 1M work is paused until the cold path shows no regression. #1010 is blocked on this.

## Reconciling with #1010's cold-path numbers (comment 88985): they're right; my sessions never measured cold My H1 and H1b sessions ran the harness with `--hot-only --reuse-indexed`. Each Home was indexed during setup, and setup used `--cold-runs 0`: the harness still indexes once, but `cold.*` is not recorded. So every number I posted is a **warm** path, and "s1 ties dev at ≤100k" holds for warm paths only. `cold.first_page`, `cold.index`, `cold.notes_per_s` and `cold.disk_written` were never measured here. The #1010 lane measured exactly those, and the regression is real. It reproduces on dev-01 (cold, 100k; loaded host, direction only): | run | dev cold.first_page | s1 cold.first_page | dev disk_written | s1 disk_written | |---|---:|---:|---:|---:| | 1 | 4,116 ms | 9,570 ms | 1,920 MB | 2,064 MB | | 2 | 5,021 ms | 7,453 ms | 1,918 MB | 2,059 MB | **Fix under test (`wip/tiering-v3-cold`, `1e8a8adef`):** - `Namespace::put_many` sorts each batch by ID, so primary-key inserts walk the B-tree in order instead of touching leaves at random. - Batches of ≥1,024 rows get a 16 MiB SQLite page cache for that one transaction, then the resting 64 KiB (or 0) setting is restored. - Cold discovery batches go from 8,192 to 65,536 rows, so 100k notes means 2 commits and fsyncs instead of 13. The single dev-01 sample of this build was taken while the host was swamped (startup 686 ms against a usual 35 ms), so it tells nothing yet. A perf-01 turn is queued: cold ABAB ×5 with dev, s1, a dev copy and fixed s1 per round, all three sizes per arm. If that doesn't close the gap, the next steps are the ones the orchestrator listed: serve the first page before the identity backfill finishes, and defer the rest to idle. Further 1M work is paused until the cold path shows no regression. #1010 is blocked on this.
kayg commented 2026-11-10 04:59:48 +00:00
Author
Owner
Copy link

Cold path: fixes 1–3 measured, root cause narrowed, fix 4 queued

Cold ABAB ×5 on perf-01, 11-10 00:41–01:20 CET (05:11–05:50 IST), both locks held (perf-turn, then perf-lock on perf-02). Each round ran dev (control-a), fix 2, the dev copy (control-b) and fix 3; each arm covers 1k/10k/100k as a cold run plus one hot-only warm run. Gate: point ratio ≤ max(1.00, upper bound of the dev-copy/dev ratio).

100k before (s1 381ce6750, 4 pairs) fix 1 (1e8a8adef, 4 pairs) fix 2 (858debdfb) fix 3 (cecbd15df)
cold.first_page 2.50× 1.86× 1.62× [1.36, 1.93] 1.73×
cold.index 1.23× 1.10× 1.01 (pass) 1.09×
cold.disk_written 1.074× 1.022× 1.022× 1.012×
  • Fix 1: sort each bulk write by ID; a 16 MiB page cache only for that transaction; 65,536-row cold batches (2 commits at 100k instead of 13).
  • Fix 2: ActiveMap::insert_row re-sorted its whole case_masks / long_ids vectors after every push. Most titles have capitals, so a cold load was O(n²) there. It now inserts at the sorted position, normally an append. Test: reused_rows_keep_case_sidecars_ordered.
  • Fix 3: rollback-journal mode for bulk loads (pages written once instead of twice). It cut bytes but made the first page and indexing slower, so it's dropped.

1k and 10k pass on every metric except disk_written (+0.7–0.8 %). home.page_big and open.api_get pass at all sizes.

Where the rest of the 100k first page goes. The first notes.list is never retried; it blocks until the first described batch, which only starts after the whole identity loop. A scratch timer probe (dev-01, uncommitted) shows inside put_many:

batch per-row checks SQLite write map update
65,536 rows 72 ms 751 ms 1,207 ms
34,464 rows 62 ms 617 ms 479 ms

Fix 4 (373a72dff, on fix 2, without fix 3): map inserts hash the caller's id and path instead of decoding the ID and rebuilding the path from the folder chain, and the map remembers the last parent folder, so rows in path order skip folding and probing every prefix. Files+notes tests: 586 passed, 0 failed. Its cold ABAB ×5 is queued in perf-turn.

disk_written (+2.2 % at 100k) is structural: files/current.db is 18 MB, where dev's ids.jsonl is 7.3 MB. The next lever there is a smaller row (the path_key column duplicates the path for plain-ASCII names), if fix 4 lands the timing.

Fix 2 cold gate table

pairs used: [0, 1, 2, 3, 4]

size metric s1fix2/dev [95% CI] dev copy/dev UCB verdict
1000 cold.first_page p50 0.891 [0.446, 1.783] 1.651 PASS
1000 cold.index p50 0.897 [0.490, 1.644] 1.544 PASS
1000 cold.notes_per_s p50 (1/x) 0.897 [0.490, 1.644] 1.544 PASS
1000 cold.disk_written p50 1.008 [1.001, 1.014] 1.003 FAIL
1000 home.page_big p50 1.025 [0.878, 1.197] 1.102 PASS
1000 home.page_big p95 1.025 [0.878, 1.197] 1.102 PASS
1000 open.api_get p50 0.860 [0.435, 1.701] 1.607 PASS
1000 open.api_get p95 0.844 [0.353, 2.018] 1.991 PASS
10000 cold.first_page p50 1.091 [0.504, 2.361] 1.606 PASS
10000 cold.index p50 0.877 [0.517, 1.488] 1.353 PASS
10000 cold.notes_per_s p50 (1/x) 0.877 [0.517, 1.488] 1.353 PASS
10000 cold.disk_written p50 1.007 [1.004, 1.010] 1.001 FAIL
10000 home.page_big p50 0.981 [0.908, 1.061] 1.150 PASS
10000 home.page_big p95 0.966 [0.922, 1.012] 1.259 PASS
10000 open.api_get p50 0.783 [0.370, 1.659] 1.568 PASS
10000 open.api_get p95 0.761 [0.259, 2.232] 1.826 PASS
100000 cold.first_page p50 1.621 [1.359, 1.934] 1.102 FAIL
100000 cold.index p50 1.011 [0.809, 1.263] 1.057 PASS
100000 cold.notes_per_s p50 (1/x) 1.011 [0.809, 1.263] 1.057 PASS
100000 cold.disk_written p50 1.022 [1.021, 1.023] 1.000 FAIL
100000 home.page_big p50 0.963 [0.915, 1.014] 1.030 PASS
100000 home.page_big p95 0.967 [0.905, 1.032] 1.076 PASS
100000 open.api_get p50 1.062 [0.755, 1.494] 1.142 PASS
100000 open.api_get p95 0.949 [0.657, 1.373] 1.244 PASS

OVERALL FAIL

Fix 3 cold gate table

pairs used: [0, 1, 2, 3, 4]

size metric s1fix3/dev [95% CI] dev copy/dev UCB verdict
1000 cold.first_page p50 0.894 [0.452, 1.771] 1.651 PASS
1000 cold.index p50 0.900 [0.495, 1.636] 1.544 PASS
1000 cold.notes_per_s p50 (1/x) 0.900 [0.495, 1.636] 1.544 PASS
1000 cold.disk_written p50 1.012 [1.006, 1.018] 1.003 FAIL
1000 home.page_big p50 0.971 [0.853, 1.105] 1.102 PASS
1000 home.page_big p95 0.971 [0.853, 1.105] 1.102 PASS
1000 open.api_get p50 0.810 [0.437, 1.501] 1.607 PASS
1000 open.api_get p95 0.763 [0.329, 1.769] 1.991 PASS
10000 cold.first_page p50 1.201 [0.561, 2.572] 1.606 PASS
10000 cold.index p50 0.928 [0.587, 1.466] 1.353 PASS
10000 cold.notes_per_s p50 (1/x) 0.928 [0.587, 1.466] 1.353 PASS
10000 cold.disk_written p50 1.008 [1.002, 1.014] 1.001 FAIL
10000 home.page_big p50 0.960 [0.844, 1.091] 1.150 PASS
10000 home.page_big p95 0.943 [0.886, 1.004] 1.259 PASS
10000 open.api_get p50 0.750 [0.360, 1.560] 1.568 PASS
10000 open.api_get p95 0.761 [0.262, 2.205] 1.826 PASS
100000 cold.first_page p50 1.730 [1.642, 1.823] 1.102 FAIL
100000 cold.index p50 1.086 [1.053, 1.121] 1.057 FAIL
100000 cold.notes_per_s p50 (1/x) 1.086 [1.053, 1.121] 1.057 FAIL
100000 cold.disk_written p50 1.012 [1.011, 1.013] 1.000 FAIL
100000 home.page_big p50 0.976 [0.864, 1.103] 1.030 PASS
100000 home.page_big p95 1.023 [0.831, 1.259] 1.076 PASS
100000 open.api_get p50 0.817 [0.526, 1.268] 1.142 PASS
100000 open.api_get p95 0.928 [0.721, 1.194] 1.244 PASS

OVERALL FAIL

## Cold path: fixes 1–3 measured, root cause narrowed, fix 4 queued Cold ABAB ×5 on perf-01, 11-10 00:41–01:20 CET (05:11–05:50 IST), both locks held (`perf-turn`, then `perf-lock` on perf-02). Each round ran dev (`control-a`), fix 2, the dev copy (`control-b`) and fix 3; each arm covers 1k/10k/100k as a cold run plus one hot-only warm run. Gate: point ratio ≤ max(1.00, upper bound of the dev-copy/dev ratio). | 100k | before (s1 `381ce6750`, 4 pairs) | fix 1 (`1e8a8adef`, 4 pairs) | fix 2 (`858debdfb`) | fix 3 (`cecbd15df`) | |---|---:|---:|---:|---:| | cold.first_page | 2.50× | 1.86× | **1.62×** [1.36, 1.93] | 1.73× | | cold.index | 1.23× | 1.10× | **1.01** (pass) | 1.09× | | cold.disk_written | 1.074× | 1.022× | 1.022× | 1.012× | - **Fix 1:** sort each bulk write by ID; a 16 MiB page cache only for that transaction; 65,536-row cold batches (2 commits at 100k instead of 13). - **Fix 2:** `ActiveMap::insert_row` re-sorted its whole `case_masks` / `long_ids` vectors after every push. Most titles have capitals, so a cold load was O(n²) there. It now inserts at the sorted position, normally an append. Test: `reused_rows_keep_case_sidecars_ordered`. - **Fix 3:** rollback-journal mode for bulk loads (pages written once instead of twice). It cut bytes but made the first page and indexing slower, so it's **dropped**. 1k and 10k pass on every metric except `disk_written` (+0.7–0.8 %). `home.page_big` and `open.api_get` pass at all sizes. **Where the rest of the 100k first page goes.** The first `notes.list` is never retried; it blocks until the first described batch, which only starts after the whole identity loop. A scratch timer probe (dev-01, uncommitted) shows inside `put_many`: | batch | per-row checks | SQLite write | map update | |---|---:|---:|---:| | 65,536 rows | 72 ms | 751 ms | **1,207 ms** | | 34,464 rows | 62 ms | 617 ms | 479 ms | **Fix 4** (`373a72dff`, on fix 2, without fix 3): map inserts hash the caller's `id` and `path` instead of decoding the ID and rebuilding the path from the folder chain, and the map remembers the last parent folder, so rows in path order skip folding and probing every prefix. Files+notes tests: 586 passed, 0 failed. Its cold ABAB ×5 is queued in `perf-turn`. `disk_written` (+2.2 % at 100k) is structural: `files/current.db` is 18 MB, where dev's `ids.jsonl` is 7.3 MB. The next lever there is a smaller row (the `path_key` column duplicates the path for plain-ASCII names), if fix 4 lands the timing. <details><summary>Fix 2 cold gate table</summary> pairs used: [0, 1, 2, 3, 4] | size | metric | s1fix2/dev [95% CI] | dev copy/dev UCB | verdict | |---|---|---|---|---| | 1000 | cold.first_page p50 | 0.891 [0.446, 1.783] | 1.651 | PASS | | 1000 | cold.index p50 | 0.897 [0.490, 1.644] | 1.544 | PASS | | 1000 | cold.notes_per_s p50 (1/x) | 0.897 [0.490, 1.644] | 1.544 | PASS | | 1000 | cold.disk_written p50 | 1.008 [1.001, 1.014] | 1.003 | **FAIL** | | 1000 | home.page_big p50 | 1.025 [0.878, 1.197] | 1.102 | PASS | | 1000 | home.page_big p95 | 1.025 [0.878, 1.197] | 1.102 | PASS | | 1000 | open.api_get p50 | 0.860 [0.435, 1.701] | 1.607 | PASS | | 1000 | open.api_get p95 | 0.844 [0.353, 2.018] | 1.991 | PASS | | 10000 | cold.first_page p50 | 1.091 [0.504, 2.361] | 1.606 | PASS | | 10000 | cold.index p50 | 0.877 [0.517, 1.488] | 1.353 | PASS | | 10000 | cold.notes_per_s p50 (1/x) | 0.877 [0.517, 1.488] | 1.353 | PASS | | 10000 | cold.disk_written p50 | 1.007 [1.004, 1.010] | 1.001 | **FAIL** | | 10000 | home.page_big p50 | 0.981 [0.908, 1.061] | 1.150 | PASS | | 10000 | home.page_big p95 | 0.966 [0.922, 1.012] | 1.259 | PASS | | 10000 | open.api_get p50 | 0.783 [0.370, 1.659] | 1.568 | PASS | | 10000 | open.api_get p95 | 0.761 [0.259, 2.232] | 1.826 | PASS | | 100000 | cold.first_page p50 | 1.621 [1.359, 1.934] | 1.102 | **FAIL** | | 100000 | cold.index p50 | 1.011 [0.809, 1.263] | 1.057 | PASS | | 100000 | cold.notes_per_s p50 (1/x) | 1.011 [0.809, 1.263] | 1.057 | PASS | | 100000 | cold.disk_written p50 | 1.022 [1.021, 1.023] | 1.000 | **FAIL** | | 100000 | home.page_big p50 | 0.963 [0.915, 1.014] | 1.030 | PASS | | 100000 | home.page_big p95 | 0.967 [0.905, 1.032] | 1.076 | PASS | | 100000 | open.api_get p50 | 1.062 [0.755, 1.494] | 1.142 | PASS | | 100000 | open.api_get p95 | 0.949 [0.657, 1.373] | 1.244 | PASS | OVERALL FAIL </details> <details><summary>Fix 3 cold gate table</summary> pairs used: [0, 1, 2, 3, 4] | size | metric | s1fix3/dev [95% CI] | dev copy/dev UCB | verdict | |---|---|---|---|---| | 1000 | cold.first_page p50 | 0.894 [0.452, 1.771] | 1.651 | PASS | | 1000 | cold.index p50 | 0.900 [0.495, 1.636] | 1.544 | PASS | | 1000 | cold.notes_per_s p50 (1/x) | 0.900 [0.495, 1.636] | 1.544 | PASS | | 1000 | cold.disk_written p50 | 1.012 [1.006, 1.018] | 1.003 | **FAIL** | | 1000 | home.page_big p50 | 0.971 [0.853, 1.105] | 1.102 | PASS | | 1000 | home.page_big p95 | 0.971 [0.853, 1.105] | 1.102 | PASS | | 1000 | open.api_get p50 | 0.810 [0.437, 1.501] | 1.607 | PASS | | 1000 | open.api_get p95 | 0.763 [0.329, 1.769] | 1.991 | PASS | | 10000 | cold.first_page p50 | 1.201 [0.561, 2.572] | 1.606 | PASS | | 10000 | cold.index p50 | 0.928 [0.587, 1.466] | 1.353 | PASS | | 10000 | cold.notes_per_s p50 (1/x) | 0.928 [0.587, 1.466] | 1.353 | PASS | | 10000 | cold.disk_written p50 | 1.008 [1.002, 1.014] | 1.001 | **FAIL** | | 10000 | home.page_big p50 | 0.960 [0.844, 1.091] | 1.150 | PASS | | 10000 | home.page_big p95 | 0.943 [0.886, 1.004] | 1.259 | PASS | | 10000 | open.api_get p50 | 0.750 [0.360, 1.560] | 1.568 | PASS | | 10000 | open.api_get p95 | 0.761 [0.262, 2.205] | 1.826 | PASS | | 100000 | cold.first_page p50 | 1.730 [1.642, 1.823] | 1.102 | **FAIL** | | 100000 | cold.index p50 | 1.086 [1.053, 1.121] | 1.057 | **FAIL** | | 100000 | cold.notes_per_s p50 (1/x) | 1.086 [1.053, 1.121] | 1.057 | **FAIL** | | 100000 | cold.disk_written p50 | 1.012 [1.011, 1.013] | 1.000 | **FAIL** | | 100000 | home.page_big p50 | 0.976 [0.864, 1.103] | 1.030 | PASS | | 100000 | home.page_big p95 | 1.023 [0.831, 1.259] | 1.076 | PASS | | 100000 | open.api_get p50 | 0.817 [0.526, 1.268] | 1.142 | PASS | | 100000 | open.api_get p95 | 0.928 [0.721, 1.194] | 1.244 | PASS | OVERALL FAIL </details>
kayg commented 2026-11-10 10:08:58 +00:00
Author
Owner
Copy link

Cold path after fix 4: still behind dev at every size; stopping patch rounds (blocker + design options)

Cold ABAB ×5 on perf-01, 11-10 10:41–11:08 CET (15:11–15:38 IST), both locks held. Each round ran dev (control-a), fix 4 (373a72dff = fixes 1+2 plus map inserts that reuse the given id/path and the last parent folder; fix 3 dropped) and the dev copy (control-b). This session's control was tight (dev-copy bounds 0.99–1.11). Fix 2's "pass" at 1k/10k in the previous session was noise: its control bounds were 1.35–1.65.

cold, fix 4 / dev [95% CI] 1k 10k 100k
cold.first_page 1.105 [1.013, 1.204] 1.542 [1.439, 1.652] 1.775 [1.543, 2.042]
cold.index 1.085 1.141 1.105
cold.disk_written 1.009 1.009 1.024

Warm home.page_big and open.api_get are flat, apart from noisy p95s.

Why patching has stopped. Four rounds took the 100k first page from 2.5× to about 1.6–1.8×, but none reached dev. The remaining gap is structural. The first notes.list returns only once the first described batch commits, and describing starts only after every discovered identity is committed to files/current.db. That is an indexed SQLite table (ID primary key plus a (path_key, id) index) with fsynced commits, where dev appends to ids.jsonl. At 100k the SQLite write alone is about 0.6–0.75 s per 65k-row batch on a quiet host, and the extra bytes (about +42 MB at 100k) come from the same table plus its WAL.

Options (per the 3-rounds rule; this needs the orchestrator's or kayg's call):

  1. Overlap identity commit with describing (the orchestrator's "first page before backfill"). Start describing as soon as the in-memory map has the batch, run the current.db commit concurrently, and keep the rule that no meta row or event is published before its identity is durable. First page ≈ max(commit, first describe batch) instead of their sum. Moderate change in scan.rs; the crash table needs one more row (crash after describe, before commit, so the meta rows must not be published yet). This doesn't help disk_written.
  2. Cold bulk build. When the current table is empty (a true cold Home), insert without the (path_key, id) index and CREATE INDEX once at the end (a sorted build), like meta.db's fresh fill. Cuts SQL time and some WAL bytes; small, local change.
  3. Smaller rows. Store UUID IDs as 16-byte blobs in current.db (the table is new in this PR, so there's no migration). That's about 40 B less per row across the key and the index, which narrows disk_written; whether it reaches dev's bytes is uncertain.

Recommendation: 2 + 3 first, because they're small and local; then 1 if the first page is still above dev. Each step gets the same cold ABAB ×5 with a dev-copy control.

Fix commits are on wip/tiering-v3-cold4 (not on this PR's branch yet). Files+notes tests pass there: 586 passed, 0 failed.

Fix 4 cold gate table

pairs used: [0, 1, 2, 3, 4]

size metric s1fix4/dev [95% CI] dev copy/dev UCB verdict
1000 cold.first_page p50 1.105 [1.013, 1.204] 0.992 FAIL
1000 cold.index p50 1.085 [1.011, 1.164] 0.994 FAIL
1000 cold.notes_per_s p50 (1/x) 1.085 [1.011, 1.164] 0.994 FAIL
1000 cold.disk_written p50 1.009 [1.004, 1.015] 1.004 FAIL
1000 home.page_big p50 0.960 [0.849, 1.084] 1.051 PASS
1000 home.page_big p95 0.960 [0.849, 1.084] 1.051 PASS
1000 open.api_get p50 1.006 [0.877, 1.154] 1.031 PASS
1000 open.api_get p95 1.061 [0.921, 1.221] 1.116 PASS
10000 cold.first_page p50 1.542 [1.439, 1.652] 1.109 FAIL
10000 cold.index p50 1.141 [1.054, 1.236] 1.052 FAIL
10000 cold.notes_per_s p50 (1/x) 1.141 [1.054, 1.236] 1.052 FAIL
10000 cold.disk_written p50 1.009 [1.006, 1.011] 1.007 FAIL
10000 home.page_big p50 1.028 [0.942, 1.123] 1.032 PASS
10000 home.page_big p95 1.044 [0.944, 1.154] 1.163 PASS
10000 open.api_get p50 1.057 [0.980, 1.140] 1.046 FAIL
10000 open.api_get p95 0.936 [0.809, 1.082] 1.379 PASS
100000 cold.first_page p50 1.775 [1.543, 2.042] 1.082 FAIL
100000 cold.index p50 1.105 [1.018, 1.199] 1.075 FAIL
100000 cold.notes_per_s p50 (1/x) 1.105 [1.018, 1.199] 1.075 FAIL
100000 cold.disk_written p50 1.024 [1.018, 1.030] 1.002 FAIL
100000 home.page_big p50 1.004 [0.883, 1.141] 1.095 PASS
100000 home.page_big p95 1.029 [0.795, 1.332] 1.084 PASS
100000 open.api_get p50 1.035 [0.961, 1.115] 1.123 PASS
100000 open.api_get p95 1.737 [0.380, 7.932] 1.082 FAIL

OVERALL FAIL

## Cold path after fix 4: still behind dev at every size; stopping patch rounds (blocker + design options) Cold ABAB ×5 on perf-01, 11-10 10:41–11:08 CET (15:11–15:38 IST), both locks held. Each round ran dev (`control-a`), fix 4 (`373a72dff` = fixes 1+2 plus map inserts that reuse the given id/path and the last parent folder; fix 3 dropped) and the dev copy (`control-b`). This session's control was tight (dev-copy bounds 0.99–1.11). Fix 2's "pass" at 1k/10k in the previous session was noise: its control bounds were 1.35–1.65. | cold, fix 4 / dev [95% CI] | 1k | 10k | 100k | |---|---|---|---| | cold.first_page | 1.105 [1.013, 1.204] | 1.542 [1.439, 1.652] | 1.775 [1.543, 2.042] | | cold.index | 1.085 | 1.141 | 1.105 | | cold.disk_written | 1.009 | 1.009 | 1.024 | Warm `home.page_big` and `open.api_get` are flat, apart from noisy p95s. **Why patching has stopped.** Four rounds took the 100k first page from 2.5× to about 1.6–1.8×, but none reached dev. The remaining gap is structural. The first `notes.list` returns only once the first described batch commits, and describing starts only after **every** discovered identity is committed to `files/current.db`. That is an indexed SQLite table (ID primary key plus a `(path_key, id)` index) with fsynced commits, where dev appends to `ids.jsonl`. At 100k the SQLite write alone is about 0.6–0.75 s per 65k-row batch on a quiet host, and the extra bytes (about +42 MB at 100k) come from the same table plus its WAL. **Options (per the 3-rounds rule; this needs the orchestrator's or kayg's call):** 1. **Overlap identity commit with describing** (the orchestrator's "first page before backfill"). Start describing as soon as the in-memory map has the batch, run the `current.db` commit concurrently, and keep the rule that no meta row or event is published before its identity is durable. First page ≈ max(commit, first describe batch) instead of their sum. Moderate change in `scan.rs`; the crash table needs one more row (crash after describe, before commit, so the meta rows must not be published yet). This doesn't help `disk_written`. 2. **Cold bulk build.** When the current table is empty (a true cold Home), insert without the `(path_key, id)` index and `CREATE INDEX` once at the end (a sorted build), like `meta.db`'s fresh fill. Cuts SQL time and some WAL bytes; small, local change. 3. **Smaller rows.** Store UUID IDs as 16-byte blobs in `current.db` (the table is new in this PR, so there's no migration). That's about 40 B less per row across the key and the index, which narrows `disk_written`; whether it reaches dev's bytes is uncertain. **Recommendation:** 2 + 3 first, because they're small and local; then 1 if the first page is still above dev. Each step gets the same cold ABAB ×5 with a dev-copy control. Fix commits are on `wip/tiering-v3-cold4` (not on this PR's branch yet). Files+notes tests pass there: 586 passed, 0 failed. <details><summary>Fix 4 cold gate table</summary> pairs used: [0, 1, 2, 3, 4] | size | metric | s1fix4/dev [95% CI] | dev copy/dev UCB | verdict | |---|---|---|---|---| | 1000 | cold.first_page p50 | 1.105 [1.013, 1.204] | 0.992 | **FAIL** | | 1000 | cold.index p50 | 1.085 [1.011, 1.164] | 0.994 | **FAIL** | | 1000 | cold.notes_per_s p50 (1/x) | 1.085 [1.011, 1.164] | 0.994 | **FAIL** | | 1000 | cold.disk_written p50 | 1.009 [1.004, 1.015] | 1.004 | **FAIL** | | 1000 | home.page_big p50 | 0.960 [0.849, 1.084] | 1.051 | PASS | | 1000 | home.page_big p95 | 0.960 [0.849, 1.084] | 1.051 | PASS | | 1000 | open.api_get p50 | 1.006 [0.877, 1.154] | 1.031 | PASS | | 1000 | open.api_get p95 | 1.061 [0.921, 1.221] | 1.116 | PASS | | 10000 | cold.first_page p50 | 1.542 [1.439, 1.652] | 1.109 | **FAIL** | | 10000 | cold.index p50 | 1.141 [1.054, 1.236] | 1.052 | **FAIL** | | 10000 | cold.notes_per_s p50 (1/x) | 1.141 [1.054, 1.236] | 1.052 | **FAIL** | | 10000 | cold.disk_written p50 | 1.009 [1.006, 1.011] | 1.007 | **FAIL** | | 10000 | home.page_big p50 | 1.028 [0.942, 1.123] | 1.032 | PASS | | 10000 | home.page_big p95 | 1.044 [0.944, 1.154] | 1.163 | PASS | | 10000 | open.api_get p50 | 1.057 [0.980, 1.140] | 1.046 | **FAIL** | | 10000 | open.api_get p95 | 0.936 [0.809, 1.082] | 1.379 | PASS | | 100000 | cold.first_page p50 | 1.775 [1.543, 2.042] | 1.082 | **FAIL** | | 100000 | cold.index p50 | 1.105 [1.018, 1.199] | 1.075 | **FAIL** | | 100000 | cold.notes_per_s p50 (1/x) | 1.105 [1.018, 1.199] | 1.075 | **FAIL** | | 100000 | cold.disk_written p50 | 1.024 [1.018, 1.030] | 1.002 | **FAIL** | | 100000 | home.page_big p50 | 1.004 [0.883, 1.141] | 1.095 | PASS | | 100000 | home.page_big p95 | 1.029 [0.795, 1.332] | 1.084 | PASS | | 100000 | open.api_get p50 | 1.035 [0.961, 1.115] | 1.123 | PASS | | 100000 | open.api_get p95 | 1.737 [0.380, 7.932] | 1.082 | **FAIL** | OVERALL FAIL </details>
kayg commented 2026-11-10 12:06:46 +00:00
Author
Owner
Copy link

Handoff to Codex: cold path, fix 5 + option 1 (Claude stops here)

Branch: wip/tiering-v3-cold4 at b0ee390c9 (the same head is on wip/tiering-v3-cold5). It sits on this PR's 381ce6750. Nothing is on the PR branch yet. All my perf holds are released; no tiering units are left on perf-01/02.

What fix 5 changed (34141810d…1625521a5):

  • files/current.db stores canonical lowercase UUIDs as 16-byte BLOB keys and any other spelling as TEXT (crates/files/src/namespace/sql_id.rs). On open, a one-time migration (user_version 1) packs old text UUIDs.
  • A bulk batch at least as large as the table (a cold load) drops current_files_by_path and rebuilds it once, in the same transaction.
  • The notes test helper shared_current_file reads both key forms.

Fix 5 cold ABAB ×5 (perf-01, 11-10 16:36–17:03 IST, both locks held; dev = control-a, dev copy = control-b; gate = ratio ≤ max(1.00, the dev copy's upper bound)):

fix 5 / dev 1k 10k 100k
cold.first_page pass 1.423 (ctl 1.185) 1.682 (ctl 1.043)
cold.index pass 1.092 (ctl 1.079) pass
cold.disk_written 1.010 pass 1.016 (fix 4: 1.024)

Warm home.page_big and open.api_get pass at every size.

Option 1 (approved), implemented, not yet measured (0dba6fdde and later, ending at b0ee390c9):

  • scan::identify_many commits the first 1,024 cold identities (50 in tests), then describes and announces only that slice via the new publish(), then continues. The rest is committed and published in one bulk fill. No meta row or event comes before its identity commit.
  • MetaDb::begin_fresh(batch) now also allows a store of at most 4,096 rows. It first removes seeded rows that the batch describes again, because the fill writes rows as new.
  • New crash tests in notes/src/tests/scan_crash.rs kill the child process at identity (after the first commit, before the describe) and at published (after the first slice is listed). Both pass: no undurable row is listed, and after a restart every file is listed once under its claimed ID.
  • first_empty_cache_batch_does_not_wait_for_later_read_chunks now accepts an earlier first commit (1..=READ_CHUNK).

Gate state at b0ee390c9: fmt and clippy pass. In the workspace tests, notes passed, including both scan_crash tests, but calternal --test notes_crash_typing::kill_nine_while_people_type_loses_and_doubles_nothing failed (run stopped there; the cause isn't diagnosed). I haven't checked whether it fails on 381ce6750 or dev too. Next step 1 is to run it --repeat-style on dev, then on this head.

Exact next steps

  1. Diagnose kill_nine_while_people_type_loses_and_doubles_nothing at b0ee390c9: flake (10× on dev and on this head) or a real effect of the first-slice publish.
  2. Run the full gates (cargo fmt --all --check, cargo clippy -j4 --workspace --all-targets --locked -- -D warnings, cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2) and build release calternal.
  3. Stage it as ~/perf/tiering-v3-0909/staged/s1fix6 on perf-01 (check sha256). The runner ~/perf/tiering-v3-0909/coldrun6.sh is ready (dev / s1fix6 / dev copy × 5; 1k, 10k, 100k cold plus one warm run each). Run it under plain ~/bin/perf-turn, with perf-02 held by perf-lock. Analyse with the same gate: cold.first_page, index, notes_per_s, disk_written, warm page_big and open.api_get.
  4. If it's flat, rewrite as clean commits on this PR (RED test, then fix, per change), merge into #1008, run its gates, then the 1M plan (three Homes, ≥ 60 GB free check before each turn).
## Handoff to Codex: cold path, fix 5 + option 1 (Claude stops here) **Branch:** `wip/tiering-v3-cold4` at `b0ee390c9` (the same head is on `wip/tiering-v3-cold5`). It sits on this PR's `381ce6750`. Nothing is on the PR branch yet. All my perf holds are released; no tiering units are left on perf-01/02. **What fix 5 changed** (`34141810d`…`1625521a5`): - `files/current.db` stores canonical lowercase UUIDs as 16-byte `BLOB` keys and any other spelling as `TEXT` (`crates/files/src/namespace/sql_id.rs`). On open, a one-time migration (`user_version` 1) packs old text UUIDs. - A bulk batch at least as large as the table (a cold load) drops `current_files_by_path` and rebuilds it once, in the same transaction. - The notes test helper `shared_current_file` reads both key forms. **Fix 5 cold ABAB ×5** (perf-01, 11-10 16:36–17:03 IST, both locks held; dev = `control-a`, dev copy = `control-b`; gate = ratio ≤ max(1.00, the dev copy's upper bound)): | fix 5 / dev | 1k | 10k | 100k | |---|---|---|---| | cold.first_page | pass | **1.423** (ctl 1.185) | **1.682** (ctl 1.043) | | cold.index | pass | 1.092 (ctl 1.079) | pass | | cold.disk_written | 1.010 | pass | **1.016** (fix 4: 1.024) | Warm `home.page_big` and `open.api_get` pass at every size. **Option 1 (approved), implemented, not yet measured** (`0dba6fdde` and later, ending at `b0ee390c9`): - `scan::identify_many` commits the first 1,024 cold identities (50 in tests), then describes and announces only that slice via the new `publish()`, then continues. The rest is committed and published in one bulk fill. No meta row or event comes before its identity commit. - `MetaDb::begin_fresh(batch)` now also allows a store of at most 4,096 rows. It first removes seeded rows that the batch describes again, because the fill writes rows as new. - New crash tests in `notes/src/tests/scan_crash.rs` kill the child process at `identity` (after the first commit, before the describe) and at `published` (after the first slice is listed). Both pass: no undurable row is listed, and after a restart every file is listed once under its claimed ID. - `first_empty_cache_batch_does_not_wait_for_later_read_chunks` now accepts an earlier first commit (1..=`READ_CHUNK`). **Gate state at `b0ee390c9`:** fmt and clippy pass. In the workspace tests, notes passed, including both `scan_crash` tests, but `calternal --test notes_crash_typing::kill_nine_while_people_type_loses_and_doubles_nothing` **failed** (run stopped there; the cause isn't diagnosed). I haven't checked whether it fails on `381ce6750` or dev too. Next step 1 is to run it `--repeat`-style on dev, then on this head. **Exact next steps** 1. Diagnose `kill_nine_while_people_type_loses_and_doubles_nothing` at `b0ee390c9`: flake (10× on dev and on this head) or a real effect of the first-slice publish. 2. Run the full gates (`cargo fmt --all --check`, `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`, `cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2`) and build release `calternal`. 3. Stage it as `~/perf/tiering-v3-0909/staged/s1fix6` on perf-01 (check sha256). The runner `~/perf/tiering-v3-0909/coldrun6.sh` is ready (dev / s1fix6 / dev copy × 5; 1k, 10k, 100k cold plus one warm run each). Run it under plain `~/bin/perf-turn`, with perf-02 held by `perf-lock`. Analyse with the same gate: cold.first_page, index, notes_per_s, disk_written, warm page_big and open.api_get. 4. If it's flat, rewrite as clean commits on this PR (RED test, then fix, per change), merge into #1008, run its gates, then the 1M plan (three Homes, ≥ 60 GB free check before each turn).
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Merge conflict checking is in progress. Try again in few moments.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/tiering-v3-s1-namespace:feat/tiering-v3-s1-namespace
git switch feat/tiering-v3-s1-namespace

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff feat/tiering-v3-s1-namespace
git switch feat/tiering-v3-s1-namespace
git rebase dev
git switch dev
git merge --ff-only feat/tiering-v3-s1-namespace
git switch feat/tiering-v3-s1-namespace
git rebase dev
git switch dev
git merge --no-ff feat/tiering-v3-s1-namespace
git switch dev
git merge --squash feat/tiering-v3-s1-namespace
git switch dev
git merge --ff-only feat/tiering-v3-s1-namespace
git switch dev
git merge feat/tiering-v3-s1-namespace
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!970
No description provided.