kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

Mail P1b: search save-log token and paged reconcile #1011

Open
kayg wants to merge 54 commits from t3/paged-search-reconcile-final into dev
pull from: t3/paged-search-reconcile-final
merge into: kayg:dev
Conversation 9 Commits 54 Files changed 28 +5169 -239
kayg commented 2026-11-07 18:44:57 +00:00
Owner
Copy link

Rethink after the 2026-11-07 follow-up review

The review found that the legacy-cursor path scanned pages to find the final page, then checked only that page. A pre-fence cursor can already cover a stale notes.saved entry several pages earlier, so this tail check cannot prove the whole cursor safe. The generation fence for new receipts remains unchanged; legacy cursor recovery is revised below.

The implementation review found that ordinary room saves publish notes.saved through a separate path from Home executor actions. Both paths now attach the accepted per-note History version to the event and publish the same version with the accepted source. Room saves reuse the already durable Room.version; the version rides on the existing live_meta generation commit, with no added file read, wait, or fsync. The ordinary_room_save_receipt_matches_published_source_version regression proves that the receipt and provider agree.

Perf run note for 2026-11-08

The perf VMs require a password for sudo, so the measurement user cannot run calternal-perf --drop-caches. The dev/dev and candidate/dev samples therefore use identical warm page-cache conditions. cold.startup measures a fresh unindexed Home with warm file pages; it does not qualify cold page-cache I/O. search.catch_up still compares the candidate's token catch-up with dev's full reconcile.

Revised restart path

  • A current cursor with persisted page-start metadata is validated by Engine::position; the indexer re-reads its last journal page and checks each known source generation before trusting the cursor.
  • A cursor without page-start metadata is wholly untrusted. The indexer runs one full paged reconcile and scans retained save receipts from the oldest retained row through the captured head, so no earlier receipt is skipped. It uses the legacy token only as the use-score replay start, then publishes current documents, use scores, and the head token atomically. It does not upgrade legacy cursor metadata after checking only the tail.
  • The legacy reconcile is read-side only. It uses at most 256 journal entries, 256 provider rows, 64 document fetches, 512 compare metadata records, and a 2,000 KiB SQLite writer cache with fence state in a temporary file. Memory stays bounded by those page and batch caps; temporary disk is O(S) for changed rows and fences.
  • Normal current-cursor startup remains incremental: O(K log K + R + D) time and O(256 + R + 64) memory. The one-time legacy repair costs O(K + N log M + M) time, matching the existing full paged path, with bounded memory. Later starts use the normal incremental path.
  • If migration crashes before publish, the old cursor and index stay in place and the next start repeats the full reconcile. Notes saves do not change.
Failure Effect Detection Recovery Test
Legacy cursor has a raced receipt outside its final journal page A tail-only check would leave stale text searchable Missing page-start metadata triggers a receipt scan from the oldest retained row; unknown receipts are re-read from a pinned provider snapshot Full paged reconcile; atomically publish documents and cursor after receipt and snapshot validation legacy_stale_receipt_before_final_page_forces_paged_reindex
Ordinary room-save receipt lacks source_version The indexer cannot compare the receipt with the exact published Notes generation The real-save test compares journal and provider versions New room saves encode Room.version and publish it with live_meta; older versionless receipts stay on the unknown re-read path ordinary_room_save_receipt_matches_published_source_version
Crash during legacy reconcile No mixed documents/cursor become visible Reconcile has not committed Retry the full paged reconcile on restart Legacy stale-receipt recovery and page-boundary kill/restart tests

This one-time legacy path can take as long as today's full reconcile because its old checkpoint carries no proof of which earlier pages were verified. The performance table will separate this compatibility repair from current-cursor startup and report both against the same-epoch control.

Design after the 2026-11-07 generation-fence ruling

P1b stays a shared, backend-only search change. Notes remains the current provider; Mail can use the same cursor and page API later. Notes search, save behavior, result order, and UI stay unchanged. The indexer owns the fence: it will not index a source older than a notes.saved receipt or checkpoint past that receipt.

Fence and restart rules

History::install_executor already returns the accepted per-note version. Append that value as source_version to each new notes.saved event (at most 37 serialized bytes including its comma). Publish the same version with the accepted Notes source and expose it to search. Existing events without the field are unknown: read the item again from a fresh pinned Notes generation, compare its source stamp, and validate the snapshot before checkpointing.

When a journal page touches an item, compare every saved version in that page with the published source version. If the source is behind, drop the private page stage, keep the previous cursor, and park the entry. Subscribe to the existing MetaDb.populated watch before pinning the provider snapshot; its publication notification retries the same page without a polling loop. If the watch produces no notification, allow five bounded exponential-backoff rereads (10, 20, 40, 80, 160 ms), then wait on the watch. A still-pending entry never advances the cursor.

Every commit covers a contiguous run of verified entries, with documents, use scores, and the page token in one SQLite transaction. On startup, validate a current cursor and re-read the final journal page before trusting it. Store that page's starting token beside each new cursor. A legacy cursor without that metadata cannot identify which earlier receipts may have raced, so treat it as wholly untrusted and run a full paged provider reconcile. Publish the verified documents, use scores and current cursor together; a crash leaves the legacy cursor untrusted and the reconcile repeats.

A missing or expired cursor does a full paged reconcile. Before publishing its head token, scan the retained journal through that head and stage the latest per-item fence in SQLite's temporary file. Compare each provider page against those fences; a pending source drops the whole private reconcile. A restored or truncated log is detected only through the engine contract: P1b uses Engine::position and Engine::journal; #928 remains the only owner of signed token proof and high-water restore detection. P1b does not read the high-water file or add/parse token formats. If #928 lands first, rebase and keep this API boundary.

Sizes and cost

Work Bound Cost
Incremental catch-up 256 journal entries; 1,024 related IDs; 64 document payloads O(K log K + R + D); memory O(256 + R + 64)
Cursor-tail recheck for a current cursor 256 entries; one provider page O(256 + D) time and memory
One-time legacy cursor repair Same page, fetch, SQLite cache and disk-fence caps as full reconcile O(K + N log M + M) time, bounded memory
Full reconcile 256 provider rows; 64 document payloads; 512 compare metadata records; fence state in SQLite temp_store=FILE, 2,000 KiB writer cache O(K + N log M + M) for K journal entries, N provider rows, M indexed rows; RAM independent of Home size; temp disk O(S) for changed rows and per-item fences
Version metadata ≤37 serialized bytes per event and accepted-source row At most 37 MB raw for 1M fully versioned rows, before SQLite page overhead; no extra fsync or save-path read

No page uses OFFSET or retains a Home-sized vector. The version is returned by the existing history install; Notes does no extra work on the notes-save path beyond encoding the appended field.

Failure table

Failure Effect Detection Recovery Test
notes.saved is visible before source publication The old source cannot enter the index and the cursor stays before the receipt Published source version is below the event version Drop the page, wait on the existing publication watch, retry the same page Real Notes executor delayed-publication race, incremental and missing-index cases
Crash after receipt, before source publication Receipt remains; cursor cannot pass it Restart replays the durable executor record and rechecks the receipt version Recovery publishes source; index resumes from the prior cursor Crash/restart between journal receipt and publication
Crash or kill at a page boundary Private stage is discarded; committed documents and cursor agree Reopen index and inspect the saved cursor Replay the uncommitted page kill_restart_at_every_provider_page_boundary_converges
Legacy cursor lacks page-start metadata A stale save may be older than the final journal page, so a tail-only check can miss it Missing page-start metadata identifies the legacy cursor Treat the cursor as untrusted and run a full paged reconcile before publishing any cursor legacy_stale_receipt_before_final_page_forces_paged_reindex
Cursor came from a restored older backup Engine cannot prove it against the restored log Engine::position returns SyncTokenExpired Full paged reconcile; never construct a token locally Restored-backup cursor test; #928 owns token-proof coverage
Cursor is ahead or behind Ahead/expired cursor cannot be used; retained behind cursor needs only its tail pages position result plus page sequence checks Full reconcile if expired; otherwise apply bounded pages ahead_cursor_reindexes_from_current_head, valid_behind_cursor_uses_pages_without_listing
Save log is truncated or index is missing No trusted checkpoint remains position/page read expires, or cursor is absent Full paged reconcile with disk-backed fences and one atomic publish truncated_log_reindexes_from_pages, missing_index_rebuilds_in_pages
Publication notification is absent or the snapshot changes No unverified page becomes visible Bounded fallback retry; provider snapshot validation Retry after a notification/backoff or discard and start a fresh reconcile Watch and snapshot-change cases in the generation property test

Tests

  • Ordinary room save: edit and save a real Note, then prove the receipt version matches the published Notes source version.

  • Real Notes race: pause EventRecorder::record_once after the durable receipt and before it returns to BatchPublished; run incremental catch-up and missing-index reconcile; prove neither indexes the old text nor passes the cursor; release publication, restart the indexer, and prove the new text is indexed.

  • Crash before publication: stop after the receipt, reopen Notes so executor recovery replays it, and prove the cursor never skips the entry.

  • Reopen after every journal/provider page boundary and compare the final index to a clean build. A legacy cursor without page-start metadata must also repair a stale receipt more than 256 entries before the cursor.

  • Property test over save versions, delayed/reordered publication notifications, duplicate touches, arbitrary page cuts, and restarts; assert one indexed row per item, latest text, and a cursor no later than the contiguous verified prefix.

  • Restored-older-backup cursor, cursor ahead/behind, truncated log, missing index, and 1M-entry paged reconcile with a 256-row maximum and bounded memory.

Gates

Gate Exact command or run Pass condition
Format CARGO_BUILD_JOBS=4 CARGO_INCREMENTAL=0 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc && cargo fmt --check' Pass
Clippy CARGO_BUILD_JOBS=4 CARGO_INCREMENTAL=0 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc && cargo clippy --workspace --all-targets -- -D warnings' Pass
Rust tests CARGO_BUILD_JOBS=4 CARGO_INCREMENTAL=0 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc && cargo test --workspace --no-fail-fast' Pass
Crash-typing flake cargo test -p calternal --test notes_crash_typing kill_nine_while_people_type_loses_and_doubles_nothing -- --exact 20 candidate and 20 dev runs interleaved; report both pass rates
Chromium Notes search/save, 1440 and 390 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc/apps/web && CALTERNAL_BIN=/home/claude/Developer/Worktrees/mail-p1b-test-e0fc/target/debug/calternal bunx playwright test e2e/search-palette.spec.ts e2e/search-corpus.spec.ts e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1' Pass
WebKit Notes search/save, 1440 and 390 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/mail-p1b-webkit-b822165c6/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/mail-p1b-webkit-b822165c6/target/debug/calternal bunx playwright test e2e/search-palette.spec.ts e2e/search-corpus.spec.ts e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=webkit-1440 --project=webkit-390 --workers=1' Pass
Perf Hold ~/bin/perf-lock on perf-01 then perf-02 for the complete same-epoch session. Run dev-vs-dev control and candidate/dev ABAB ×5; each sample runs calternal-perf --bin "$SERVER_BIN" --work "$WORK" --data-dir "$DATA" --state-dir "$STATE" --budgets perf/budgets.json --sizes 100k,1000000 --runs 1 --cold-runs 1 --idle-secs 0 --json "$OUT/run.json" --md "$OUT/run.md" and storage-bench save --dir "$SAVE_DIR" --saves 200. Compare first query, query, catch-up, notes-save p50/p99, typing, and startup at 100k/1M beyond the dev-vs-dev spread. Every path is flat or better. The 1M reconcile must beat today's full path or the PR records the measured reason it does not.

Any regression beyond the control spread blocks qualification. Post commands, rates, all metric deltas, and the same-epoch control in PR comments. Never merge.

Design note: the indexer never waits forever on a receipt fence (liveness fix, 2026-11-09)

Problem. Search holds a save receipt back until the provider has published the version the receipt names (the "fence"). If that version is never published, PublicationWait::retry waits on the publication signal with no limit, and the full rebuild enforces the same fences, so it waits too. Search then reports indexing: true for ever. Seen on compute02 and perf-01 with the gap stage (idle server, DIAG waiting for publication signal).

"Receipt ahead of provider". A receipt for item X names source version N (SourceFence::Known(N)), but the provider's published document for X has version M < N (or none), and no further publication arrives. Real ways to get there: a Home restored from an older backup, or note history rebuilt, while the save-log keeps its receipts.

The cap. Options::publication_cap, default 30 s, counted as time stuck on one page: it starts at the first reread for that page, restarts when the page is staged, and is not extended by publication signals for other items (so steady typing elsewhere cannot keep it alive). Tests set it to milliseconds.

After the cap (both places). One relaxed attempt for that page or rebuild, using the same pinned provider snapshot:

  • journal path (catch_up_to, recheck_cursor_tail, apply_live_batch): the page is staged without its Known fences;
  • rebuild path (reconcile_pages): the attempt ignores comparison.pending, passes no fences to the refresh, and skips has_unseen_known_fence.
    The receipt is superseded: the provider's current document is indexed, the cursor advances, indexing goes false. One WARN per pass names the provider and the time waited, and the process-wide counter calternal_search::fences_superseded() goes up by one per relaxed page or rebuild. Relaxing lasts for that page or attempt only; the next page is strict again.

Why this does not publish stale search results.

  • The index never gets anything the provider has not published: every document is read from the provider's pinned snapshot, exactly as a rebuild reads it. Nothing is invented from the receipt.
  • For a receipt that is ahead for good, the provider's current document is the newest that exists, so the index equals the source.
  • A strict fence is still used for the whole 30 s, which covers ordinary publication delay (milliseconds) and a long first scan (every batch publishes, so the page stages as soon as its note appears).
  • Known limit: if the provider publishes N only after 30 s of silence, the index shows the older text of that one note until it changes again or the Home reconciles at restart. I am not adding a deferred re-check in this PR; it will be a follow-up issue if wanted.

Tests. (1) A receipt ahead of any publication: sync_to_head finishes, the doc is the provider's current one, and fences_superseded() rose, within cap + slack, for both the catch-up path and the rebuild path. (2) A publication that arrives inside the cap is honoured: the indexed doc has the receipt's version and the counter did not rise. Plus the in-repo gap stage in the perf runner and its ssh retry.

Update (same day, after coding). Two details that were not in the note above:

  • After the first give-up in a pass, later pages in that pass wait only a tenth of the cap (3 s by default), because the Home is then known to hold receipts the provider never publishes.
  • The harness gap stage is --gap-saves N in crates/perf (with --search-only for the smoke set plus search). It makes a real gap through the real server: build the index (search.cold_reconcile), keep a copy, save N edits so every receipt is valid, then put the older index back. The earlier out-of-tree stage could not be recovered; this is a new, reproducible one.
## Rethink after the 2026-11-07 follow-up review The review found that the legacy-cursor path scanned pages to find the final page, then checked only that page. A pre-fence cursor can already cover a stale `notes.saved` entry several pages earlier, so this tail check cannot prove the whole cursor safe. The generation fence for new receipts remains unchanged; legacy cursor recovery is revised below. The implementation review found that ordinary room saves publish notes.saved through a separate path from Home executor actions. Both paths now attach the accepted per-note History version to the event and publish the same version with the accepted source. Room saves reuse the already durable Room.version; the version rides on the existing live_meta generation commit, with no added file read, wait, or fsync. The ordinary_room_save_receipt_matches_published_source_version regression proves that the receipt and provider agree. > ### Perf run note for 2026-11-08 > The perf VMs require a password for `sudo`, so the measurement user cannot run `calternal-perf --drop-caches`. The dev/dev and candidate/dev samples therefore use identical warm page-cache conditions. `cold.startup` measures a fresh unindexed Home with warm file pages; it does not qualify cold page-cache I/O. `search.catch_up` still compares the candidate's token catch-up with dev's full reconcile. ### Revised restart path - A current cursor with persisted page-start metadata is validated by `Engine::position`; the indexer re-reads its last journal page and checks each known source generation before trusting the cursor. - A cursor without page-start metadata is wholly untrusted. The indexer runs one full paged reconcile and scans retained save receipts from the oldest retained row through the captured head, so no earlier receipt is skipped. It uses the legacy token only as the use-score replay start, then publishes current documents, use scores, and the head token atomically. It does not upgrade legacy cursor metadata after checking only the tail. - The legacy reconcile is read-side only. It uses at most 256 journal entries, 256 provider rows, 64 document fetches, 512 compare metadata records, and a 2,000 KiB SQLite writer cache with fence state in a temporary file. Memory stays bounded by those page and batch caps; temporary disk is O(S) for changed rows and fences. - Normal current-cursor startup remains incremental: O(K log K + R + D) time and O(256 + R + 64) memory. The one-time legacy repair costs O(K + N log M + M) time, matching the existing full paged path, with bounded memory. Later starts use the normal incremental path. - If migration crashes before publish, the old cursor and index stay in place and the next start repeats the full reconcile. Notes saves do not change. | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | Legacy cursor has a raced receipt outside its final journal page | A tail-only check would leave stale text searchable | Missing page-start metadata triggers a receipt scan from the oldest retained row; unknown receipts are re-read from a pinned provider snapshot | Full paged reconcile; atomically publish documents and cursor after receipt and snapshot validation | `legacy_stale_receipt_before_final_page_forces_paged_reindex` | | Ordinary room-save receipt lacks source_version | The indexer cannot compare the receipt with the exact published Notes generation | The real-save test compares journal and provider versions | New room saves encode Room.version and publish it with live_meta; older versionless receipts stay on the unknown re-read path | ordinary_room_save_receipt_matches_published_source_version | | Crash during legacy reconcile | No mixed documents/cursor become visible | Reconcile has not committed | Retry the full paged reconcile on restart | Legacy stale-receipt recovery and page-boundary kill/restart tests | This one-time legacy path can take as long as today's full reconcile because its old checkpoint carries no proof of which earlier pages were verified. The performance table will separate this compatibility repair from current-cursor startup and report both against the same-epoch control. ## Design after the 2026-11-07 generation-fence ruling P1b stays a shared, backend-only search change. Notes remains the current provider; Mail can use the same cursor and page API later. Notes search, save behavior, result order, and UI stay unchanged. The indexer owns the fence: it will not index a source older than a `notes.saved` receipt or checkpoint past that receipt. ### Fence and restart rules `History::install_executor` already returns the accepted per-note version. Append that value as `source_version` to each new `notes.saved` event (at most 37 serialized bytes including its comma). Publish the same version with the accepted Notes source and expose it to search. Existing events without the field are `unknown`: read the item again from a fresh pinned Notes generation, compare its source stamp, and validate the snapshot before checkpointing. When a journal page touches an item, compare every saved version in that page with the published source version. If the source is behind, drop the private page stage, keep the previous cursor, and park the entry. Subscribe to the existing `MetaDb.populated` watch before pinning the provider snapshot; its publication notification retries the same page without a polling loop. If the watch produces no notification, allow five bounded exponential-backoff rereads (10, 20, 40, 80, 160 ms), then wait on the watch. A still-pending entry never advances the cursor. Every commit covers a contiguous run of verified entries, with documents, use scores, and the page token in one SQLite transaction. On startup, validate a current cursor and re-read the final journal page before trusting it. Store that page's starting token beside each new cursor. A legacy cursor without that metadata cannot identify which earlier receipts may have raced, so treat it as wholly untrusted and run a full paged provider reconcile. Publish the verified documents, use scores and current cursor together; a crash leaves the legacy cursor untrusted and the reconcile repeats. A missing or expired cursor does a full paged reconcile. Before publishing its head token, scan the retained journal through that head and stage the latest per-item fence in SQLite's temporary file. Compare each provider page against those fences; a pending source drops the whole private reconcile. A restored or truncated log is detected only through the engine contract: P1b uses `Engine::position` and `Engine::journal`; #928 remains the only owner of signed token proof and high-water restore detection. P1b does not read the high-water file or add/parse token formats. If #928 lands first, rebase and keep this API boundary. ### Sizes and cost | Work | Bound | Cost | |---|---:|---| | Incremental catch-up | 256 journal entries; 1,024 related IDs; 64 document payloads | `O(K log K + R + D)`; memory `O(256 + R + 64)` | | Cursor-tail recheck for a current cursor | 256 entries; one provider page | `O(256 + D)` time and memory | | One-time legacy cursor repair | Same page, fetch, SQLite cache and disk-fence caps as full reconcile | `O(K + N log M + M)` time, bounded memory | | Full reconcile | 256 provider rows; 64 document payloads; 512 compare metadata records; fence state in SQLite `temp_store=FILE`, 2,000 KiB writer cache | `O(K + N log M + M)` for `K` journal entries, `N` provider rows, `M` indexed rows; RAM independent of Home size; temp disk `O(S)` for changed rows and per-item fences | | Version metadata | ≤37 serialized bytes per event and accepted-source row | At most 37 MB raw for 1M fully versioned rows, before SQLite page overhead; no extra fsync or save-path read | No page uses `OFFSET` or retains a Home-sized vector. The version is returned by the existing history install; Notes does no extra work on the notes-save path beyond encoding the appended field. ### Failure table | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | `notes.saved` is visible before source publication | The old source cannot enter the index and the cursor stays before the receipt | Published source version is below the event version | Drop the page, wait on the existing publication watch, retry the same page | Real Notes executor delayed-publication race, incremental and missing-index cases | | Crash after receipt, before source publication | Receipt remains; cursor cannot pass it | Restart replays the durable executor record and rechecks the receipt version | Recovery publishes source; index resumes from the prior cursor | Crash/restart between journal receipt and publication | | Crash or kill at a page boundary | Private stage is discarded; committed documents and cursor agree | Reopen index and inspect the saved cursor | Replay the uncommitted page | `kill_restart_at_every_provider_page_boundary_converges` | | Legacy cursor lacks page-start metadata | A stale save may be older than the final journal page, so a tail-only check can miss it | Missing page-start metadata identifies the legacy cursor | Treat the cursor as untrusted and run a full paged reconcile before publishing any cursor | `legacy_stale_receipt_before_final_page_forces_paged_reindex` | | Cursor came from a restored older backup | Engine cannot prove it against the restored log | `Engine::position` returns `SyncTokenExpired` | Full paged reconcile; never construct a token locally | Restored-backup cursor test; #928 owns token-proof coverage | | Cursor is ahead or behind | Ahead/expired cursor cannot be used; retained behind cursor needs only its tail pages | `position` result plus page sequence checks | Full reconcile if expired; otherwise apply bounded pages | `ahead_cursor_reindexes_from_current_head`, `valid_behind_cursor_uses_pages_without_listing` | | Save log is truncated or index is missing | No trusted checkpoint remains | `position`/page read expires, or cursor is absent | Full paged reconcile with disk-backed fences and one atomic publish | `truncated_log_reindexes_from_pages`, `missing_index_rebuilds_in_pages` | | Publication notification is absent or the snapshot changes | No unverified page becomes visible | Bounded fallback retry; provider snapshot validation | Retry after a notification/backoff or discard and start a fresh reconcile | Watch and snapshot-change cases in the generation property test | ### Tests - Ordinary room save: edit and save a real Note, then prove the receipt version matches the published Notes source version. - Real Notes race: pause `EventRecorder::record_once` after the durable receipt and before it returns to `BatchPublished`; run incremental catch-up and missing-index reconcile; prove neither indexes the old text nor passes the cursor; release publication, restart the indexer, and prove the new text is indexed. - Crash before publication: stop after the receipt, reopen Notes so executor recovery replays it, and prove the cursor never skips the entry. - Reopen after every journal/provider page boundary and compare the final index to a clean build. A legacy cursor without page-start metadata must also repair a stale receipt more than 256 entries before the cursor. - Property test over save versions, delayed/reordered publication notifications, duplicate touches, arbitrary page cuts, and restarts; assert one indexed row per item, latest text, and a cursor no later than the contiguous verified prefix. - Restored-older-backup cursor, cursor ahead/behind, truncated log, missing index, and 1M-entry paged reconcile with a 256-row maximum and bounded memory. ### Gates | Gate | Exact command or run | Pass condition | |---|---|---| | Format | `CARGO_BUILD_JOBS=4 CARGO_INCREMENTAL=0 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc && cargo fmt --check'` | Pass | | Clippy | `CARGO_BUILD_JOBS=4 CARGO_INCREMENTAL=0 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc && cargo clippy --workspace --all-targets -- -D warnings'` | Pass | | Rust tests | `CARGO_BUILD_JOBS=4 CARGO_INCREMENTAL=0 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc && cargo test --workspace --no-fail-fast'` | Pass | | Crash-typing flake | `cargo test -p calternal --test notes_crash_typing kill_nine_while_people_type_loses_and_doubles_nothing -- --exact` | 20 candidate and 20 dev runs interleaved; report both pass rates | | Chromium Notes search/save, 1440 and 390 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/mail-p1b-test-e0fc/apps/web && CALTERNAL_BIN=/home/claude/Developer/Worktrees/mail-p1b-test-e0fc/target/debug/calternal bunx playwright test e2e/search-palette.spec.ts e2e/search-corpus.spec.ts e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1'` | Pass | | WebKit Notes search/save, 1440 and 390 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/mail-p1b-webkit-b822165c6/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/mail-p1b-webkit-b822165c6/target/debug/calternal bunx playwright test e2e/search-palette.spec.ts e2e/search-corpus.spec.ts e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=webkit-1440 --project=webkit-390 --workers=1'` | Pass | | Perf | Hold `~/bin/perf-lock` on perf-01 then perf-02 for the complete same-epoch session. Run dev-vs-dev control and candidate/dev ABAB ×5; each sample runs `calternal-perf --bin "$SERVER_BIN" --work "$WORK" --data-dir "$DATA" --state-dir "$STATE" --budgets perf/budgets.json --sizes 100k,1000000 --runs 1 --cold-runs 1 --idle-secs 0 --json "$OUT/run.json" --md "$OUT/run.md"` and `storage-bench save --dir "$SAVE_DIR" --saves 200`. | Compare first query, query, catch-up, notes-save p50/p99, typing, and startup at 100k/1M beyond the dev-vs-dev spread. Every path is flat or better. The 1M reconcile must beat today's full path or the PR records the measured reason it does not. | Any regression beyond the control spread blocks qualification. Post commands, rates, all metric deltas, and the same-epoch control in PR comments. Never merge. ## Design note: the indexer never waits forever on a receipt fence (liveness fix, 2026-11-09) **Problem.** Search holds a save receipt back until the provider has published the version the receipt names (the "fence"). If that version is never published, `PublicationWait::retry` waits on the publication signal with no limit, and the full rebuild enforces the same fences, so it waits too. Search then reports `indexing: true` for ever. Seen on compute02 and perf-01 with the gap stage (idle server, `DIAG waiting for publication signal`). **"Receipt ahead of provider".** A receipt for item X names source version N (`SourceFence::Known(N)`), but the provider's published document for X has version M < N (or none), and no further publication arrives. Real ways to get there: a Home restored from an older backup, or note history rebuilt, while the save-log keeps its receipts. **The cap.** `Options::publication_cap`, default **30 s**, counted as time stuck on one page: it starts at the first reread for that page, restarts when the page is staged, and is not extended by publication signals for other items (so steady typing elsewhere cannot keep it alive). Tests set it to milliseconds. **After the cap (both places).** One relaxed attempt for that page or rebuild, using the same pinned provider snapshot: - journal path (`catch_up_to`, `recheck_cursor_tail`, `apply_live_batch`): the page is staged without its `Known` fences; - rebuild path (`reconcile_pages`): the attempt ignores `comparison.pending`, passes no fences to the refresh, and skips `has_unseen_known_fence`. The receipt is **superseded**: the provider's current document is indexed, the cursor advances, `indexing` goes false. One `WARN` per pass names the provider and the time waited, and the process-wide counter `calternal_search::fences_superseded()` goes up by one per relaxed page or rebuild. Relaxing lasts for that page or attempt only; the next page is strict again. **Why this does not publish stale search results.** - The index never gets anything the provider has not published: every document is read from the provider's pinned snapshot, exactly as a rebuild reads it. Nothing is invented from the receipt. - For a receipt that is ahead for good, the provider's current document is the newest that exists, so the index equals the source. - A strict fence is still used for the whole 30 s, which covers ordinary publication delay (milliseconds) and a long first scan (every batch publishes, so the page stages as soon as its note appears). - Known limit: if the provider publishes N only after 30 s of silence, the index shows the older text of that one note until it changes again or the Home reconciles at restart. I am not adding a deferred re-check in this PR; it will be a follow-up issue if wanted. **Tests.** (1) A receipt ahead of any publication: `sync_to_head` finishes, the doc is the provider's current one, and `fences_superseded()` rose, within cap + slack, for both the catch-up path and the rebuild path. (2) A publication that arrives inside the cap is honoured: the indexed doc has the receipt's version and the counter did not rise. Plus the in-repo gap stage in the perf runner and its ssh retry. **Update (same day, after coding).** Two details that were not in the note above: - After the first give-up in a pass, later pages in that pass wait only a tenth of the cap (3 s by default), because the Home is then known to hold receipts the provider never publishes. - The harness gap stage is `--gap-saves N` in `crates/perf` (with `--search-only` for the smoke set plus search). It makes a real gap through the real server: build the index (`search.cold_reconcile`), keep a copy, save N edits so every receipt is valid, then put the older index back. The earlier out-of-tree stage could not be recovered; this is a new, reproducible one.
kayg added 54 commits 2026-11-07 18:44:57 +00:00
chore(search): lock fixture filesystem dependency
Some checks failed
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
000873f49e
kayg commented 2026-11-08 01:11:41 +00:00
Author
Owner
Copy link

Crash-typing flake comparison

The first branch-native run reported candidate 18/20 and dev 2/20, but those runs used different test sources: the candidate test retries the documented retryable not_saved_yet response, while dev's older test expects an immediate 200. I reran the test interleaved with the exact candidate test source on both revisions (SHA-256 f9947402f29e4bacafbc5dc803c4236228411cd923bf63a80e2e65969b7d2555).

Revision Passes Failure
Candidate 0efea4a1 20/20 None
dev cd7f8443 19/20 One b never converged

Each arm ran the exact command CARGO_BUILD_JOBS=4 cargo test -p calternal --test notes_crash_typing kill_nine_while_people_type_loses_and_doubles_nothing -- --exact, alternating candidate then dev for 20 rounds. The dev-only failure is retained in the log; the candidate had no failures.

## Crash-typing flake comparison The first branch-native run reported candidate 18/20 and dev 2/20, but those runs used different test sources: the candidate test retries the documented retryable `not_saved_yet` response, while dev's older test expects an immediate 200. I reran the test interleaved with the exact candidate test source on both revisions (SHA-256 `f9947402f29e4bacafbc5dc803c4236228411cd923bf63a80e2e65969b7d2555`). | Revision | Passes | Failure | |---|---:|---| | Candidate `0efea4a1` | 20/20 | None | | dev `cd7f8443` | 19/20 | One `b never converged` | Each arm ran the exact command `CARGO_BUILD_JOBS=4 cargo test -p calternal --test notes_crash_typing kill_nine_while_people_type_loses_and_doubles_nothing -- --exact`, alternating candidate then dev for 20 rounds. The dev-only failure is retained in the log; the candidate had no failures.
kayg commented 2026-11-08 07:31:08 +00:00
Author
Owner
Copy link

macOS gate rerun on 2026-11-08

#995 is closed and marked superseded by #1011. The open #1011 head 5a83ac274504b0b5fdb9e657cafa6a82e2a6544f has origin/dev merge-base 93d9a321a39044afddfe9075db2e1c1f729e3b78, so it already includes #1013.

The first Mac run exposed a test-fixture path collision: Notes::new colocates data and state, and case-insensitive APFS aliases <user>/Notes with <user>/notes. The write executor then correctly refused a journal inside Notes. Commits f73e93209 and 5a83ac274 change the P1b test fixtures to use separate data and state roots through Notes::with_state; Notes runtime behavior is unchanged.

Each Cargo gate below ran as its own Nix/slot step on /Users/lume/Developer/Worktrees/mail-p1b-mac-1011-eb1a4294, with a 40-minute per-step timeout. The helper acquired /tmp/mac-bench.lock, wrote its holder, and removed it in its EXIT trap before the next step. Other Mac lanes used the lock between our steps.

Gate Command inside the Nix dev shell Result
Format timeout --kill-after=30s 40m cargo fmt --check PASS
Clippy timeout --kill-after=30s 40m cargo clippy --workspace --all-targets -- -D warnings PASS
Search library timeout --kill-after=30s 40m cargo test -p calternal-search --lib PASS: 39 passed, 0 failed; includes the legacy cursor and delayed-publication cases
Workspace tests timeout --kill-after=30s 40m cargo test --workspace --no-fail-fast PASS: 1,608 passed, 0 failed, 22 ignored

The full suite's first Mac attempt also had one kill_nine_while_people_type_loses_and_doubles_nothing failure (b never converged). It passed on the final full-suite rerun. The requested interleaved dev-02 repetitions passed: candidate 20/20, dev 20/20, using cargo test -p calternal --test notes_crash_typing kill_nine_while_people_type_loses_and_doubles_nothing -- --exact.

P1b perf remains pending; these results make no performance claim.

## macOS gate rerun on 2026-11-08 #995 is closed and marked superseded by #1011. The open #1011 head `5a83ac274504b0b5fdb9e657cafa6a82e2a6544f` has `origin/dev` merge-base `93d9a321a39044afddfe9075db2e1c1f729e3b78`, so it already includes #1013. The first Mac run exposed a test-fixture path collision: `Notes::new` colocates data and state, and case-insensitive APFS aliases `<user>/Notes` with `<user>/notes`. The write executor then correctly refused a journal inside Notes. Commits `f73e93209` and `5a83ac274` change the P1b test fixtures to use separate data and state roots through `Notes::with_state`; Notes runtime behavior is unchanged. Each Cargo gate below ran as its own Nix/slot step on `/Users/lume/Developer/Worktrees/mail-p1b-mac-1011-eb1a4294`, with a 40-minute per-step timeout. The helper acquired `/tmp/mac-bench.lock`, wrote its holder, and removed it in its EXIT trap before the next step. Other Mac lanes used the lock between our steps. | Gate | Command inside the Nix dev shell | Result | |---|---|---| | Format | `timeout --kill-after=30s 40m cargo fmt --check` | PASS | | Clippy | `timeout --kill-after=30s 40m cargo clippy --workspace --all-targets -- -D warnings` | PASS | | Search library | `timeout --kill-after=30s 40m cargo test -p calternal-search --lib` | PASS: 39 passed, 0 failed; includes the legacy cursor and delayed-publication cases | | Workspace tests | `timeout --kill-after=30s 40m cargo test --workspace --no-fail-fast` | PASS: 1,608 passed, 0 failed, 22 ignored | The full suite's first Mac attempt also had one `kill_nine_while_people_type_loses_and_doubles_nothing` failure (`b never converged`). It passed on the final full-suite rerun. The requested interleaved dev-02 repetitions passed: candidate 20/20, dev 20/20, using `cargo test -p calternal --test notes_crash_typing kill_nine_while_people_type_loses_and_doubles_nothing -- --exact`. P1b perf remains pending; these results make no performance claim.
kayg commented 2026-11-09 09:45:05 +00:00
Author
Owner
Copy link

P1b perf ABAB status, 2026-11-09 ~10:45 CET (no numbers yet)

Head under test: 02baeb54212095d956d1fcce49594d6c36d36ea2. Staged binaries verified with sha256sum -c.

Time (CET) Event
05:51-06:14 First matrix: stopped itself at the 15 GB disk floor (exit 70). I freed my predecessor's data/state dirs (exact paths).
06:17-06:49 Second matrix: dev control and block-1-a1-dev finished. block-1-a2-candidate then stalled at "100000 notes, warm run 1" from 06:49. Harness and server were both idle, no sockets, every thread in futex/epoll wait. I stopped my own unit. Cause not found: the same stage passed on both dev arms.
08:55-09:11 Third matrix: the 1M-note cold run fails with notes.list 500 on dev too, at the same step as before. The 100k cold index writes 4.5 GB, so 1M needs about 45 GB, and the shared perf disk has 16-25 GB free. Unconfirmed as ENOSPC.

Changes to the rig: 100k only (no 1M), 25-minute per-arm timeout with one retry.

Not done: the ABAB x5 and the dev-vs-dev control. Another lane held perf-01 for over 3 hours after 09:11, and I did not get a turn. I cancelled my queued unit and hold no lock.

The candidate stall is unresolved. If it recurs under the new timeout, treat it as a PR bug: capture its stack and fix it before merge.

## P1b perf ABAB status, 2026-11-09 ~10:45 CET (no numbers yet) Head under test: `02baeb54212095d956d1fcce49594d6c36d36ea2`. Staged binaries verified with `sha256sum -c`. | Time (CET) | Event | |---|---| | 05:51-06:14 | First matrix: stopped itself at the 15 GB disk floor (exit 70). I freed my predecessor's `data`/`state` dirs (exact paths). | | 06:17-06:49 | Second matrix: dev control and `block-1-a1-dev` finished. `block-1-a2-candidate` then stalled at "100000 notes, warm run 1" from 06:49. Harness and server were both idle, no sockets, every thread in futex/epoll wait. I stopped my own unit. Cause not found: the same stage passed on both dev arms. | | 08:55-09:11 | Third matrix: the 1M-note cold run fails with `notes.list` 500 on **dev** too, at the same step as before. The 100k cold index writes 4.5 GB, so 1M needs about 45 GB, and the shared perf disk has 16-25 GB free. Unconfirmed as ENOSPC. | Changes to the rig: 100k only (no 1M), 25-minute per-arm timeout with one retry. Not done: the ABAB x5 and the dev-vs-dev control. Another lane held perf-01 for over 3 hours after 09:11, and I did not get a turn. I cancelled my queued unit and hold no lock. The candidate stall is unresolved. If it recurs under the new timeout, treat it as a PR bug: capture its stack and fix it before merge.
kayg commented 2026-11-09 13:17:30 +00:00
Author
Owner
Copy link

P1b perf turn 2, 2026-11-09 (FIFO): candidate warm-run stall reproduced, no ABAB numbers

PR head (API head.sha): 02baeb54212095d956d1fcce49594d6c36d36ea2. refs/pull/1011/head still shows 000873f49…, which is the stale-ref gotcha; the branch t3/paged-search-reconcile-final is at 02baeb542.

Item Result
Turn Held perf-01 and perf-02 from 11:32 CET (FIFO). 100k only; 1M dropped (see #1028).
block-1-a1-dev Finished: 6 min.
block-1-a2-candidate Stalled again at 100000 notes, warm run 1 from 11:39:35 CET, same as the 06:49 stall in the first session.
Stall state Server healthy: GET /healthz returned 200 on its real port. Server idle: search-writer parked on an empty std::sync::mpmc::Channel<calternal_search::index::Job>, all tokio workers in futex wait. Harness: main thread parked in tokio::runtime::park::Inner::park under calternal_perf::main, all workers parked, one worker in epoll_wait, no TCP connection open, no timer armed.
Stacks gdb -p <pid> thread apply all bt of both processes, kept on perf-01 under results-turn-patched-harness/block-1-a2-candidate/stall-gdb-*.txt.
Stopped I stopped my own unit; both locks released at 11:46 CET.

Reading: the harness is waiting on something that never arrives, while the server has finished its work. Dev arms pass the same stage on the same harness build. The staged harness (harness/*/calternal-perf, sha 21c0679b… candidate, e02f9fc2… dev) contains an "offline saves for a warm search catch-up gap" edit that is not in 02baeb542, and its source is in no worktree I can find. So I cannot yet say whether the stall is in the candidate server or in that edit.

Next test, queued but not run: the clean harnesses built from each tree (bin/*/calternal-perf, c6149a26… candidate, fa53d86f… dev). If the stall disappears, it is the edit. If it stays, it is a PR bug and needs a fix on compute02 before merge.

Not run: the ABAB x5 and any per-metric verdict.

Process notes: ~/bin/perf-turn was edited while my waiting copy ran, so bash hit syntax error near unexpected token done at 13:56 CET and my ticket (2 h into the wait) was dropped. A ticket named 0000000000-782670-sharing-842b (timestamp 0) was at the head of the line for about an hour. I now hold no ticket and no lock.

## P1b perf turn 2, 2026-11-09 (FIFO): candidate warm-run stall reproduced, no ABAB numbers PR head (API `head.sha`): `02baeb54212095d956d1fcce49594d6c36d36ea2`. `refs/pull/1011/head` still shows `000873f49…`, which is the stale-ref gotcha; the branch `t3/paged-search-reconcile-final` is at `02baeb542`. | Item | Result | |---|---| | Turn | Held perf-01 and perf-02 from 11:32 CET (FIFO). 100k only; 1M dropped (see #1028). | | `block-1-a1-dev` | Finished: 6 min. | | `block-1-a2-candidate` | **Stalled again** at `100000 notes, warm run 1` from 11:39:35 CET, same as the 06:49 stall in the first session. | | Stall state | Server healthy: `GET /healthz` returned 200 on its real port. Server idle: `search-writer` parked on an empty `std::sync::mpmc::Channel<calternal_search::index::Job>`, all tokio workers in futex wait. Harness: main thread parked in `tokio::runtime::park::Inner::park` under `calternal_perf::main`, all workers parked, one worker in `epoll_wait`, no TCP connection open, no timer armed. | | Stacks | `gdb -p <pid> thread apply all bt` of both processes, kept on perf-01 under `results-turn-patched-harness/block-1-a2-candidate/stall-gdb-*.txt`. | | Stopped | I stopped my own unit; both locks released at 11:46 CET. | Reading: the harness is waiting on something that never arrives, while the server has finished its work. Dev arms pass the same stage on the same harness build. The staged harness (`harness/*/calternal-perf`, sha `21c0679b…` candidate, `e02f9fc2…` dev) contains an "offline saves for a warm search catch-up gap" edit that is not in `02baeb542`, and its source is in no worktree I can find. So I cannot yet say whether the stall is in the candidate server or in that edit. Next test, queued but not run: the clean harnesses built from each tree (`bin/*/calternal-perf`, `c6149a26…` candidate, `fa53d86f…` dev). If the stall disappears, it is the edit. If it stays, it is a PR bug and needs a fix on compute02 before merge. Not run: the ABAB x5 and any per-metric verdict. Process notes: `~/bin/perf-turn` was edited while my waiting copy ran, so bash hit `syntax error near unexpected token done` at 13:56 CET and my ticket (2 h into the wait) was dropped. A ticket named `0000000000-782670-sharing-842b` (timestamp 0) was at the head of the line for about an hour. I now hold no ticket and no lock.
kayg commented 2026-11-09 16:06:21 +00:00
Author
Owner
Copy link

P1b: stall root-caused off-lock (compute02, no perf lock), Mac gates re-run at 02baeb542

PR head (API): 02baeb54212095d956d1fcce49594d6c36d36ea2. No perf numbers yet; I did not take the perf turn because the stall is a design question (below).

Mac gates at 02baeb542 (macos-01, one lock hold per step, each released)

Gate Command (inside the Nix dev shell) Result
Format cargo fmt --check PASS
Clippy cargo clippy --workspace --all-targets -- -D warnings PASS
Search lib cargo test -p calternal-search --lib PASS: 40 passed, 0 failed
Workspace cargo test --workspace --no-fail-fast PASS: 1,609 passed, 0 failed, 22 ignored

Stall repro, compute02 (12 cores), 100k warm, 20 min cap, no perf lock

Harness Server Result
clean (built from each tree) dev pass, rc=0
clean candidate pass, rc=0 (6 min)
edited dev harness (e02f9fc2…, has the save-log gap stage) dev pass, rc=0
edited candidate harness (21c0679b…, gap stage) candidate stalls at "warm run 1", rc=124, 2 of 2 runs
edited candidate harness dev slow, not stuck: notes.list incomplete: true, host load ~35 (rc=124 at the cap)

The clean harness has no gap stage, so search.catch_up is about 0 s and search.cold_reconcile is absent. It does not measure what #1011 changes. The edited harnesses' gap source is in no repo or worktree I could find.

Root cause (candidate server)

At the stall the server answers /healthz 200 and notes.list returns incomplete: false, but search.query returns indexing: true indefinitely, with every thread idle. A diagnostic build (extra warn! only) logged DIAG waiting for publication signal provider="notes" and never logged that it arrived. The indexer is parked in PublicationWait::retry (crates/search/src/indexer/sync.rs): after its 5 backoff rereads it waits on receiver.changed().await with no cap, for a receipt whose source_version is ahead of what the notes provider publishes. Nothing publishes again, because the notes scan is already finished.

What I tried and why I stopped

I capped that wait at 30 s and returned SyncTokenExpired so the caller rebuilds. The stall changed shape but did not go away: the rebuild (reconcile_pages) enforces the same receipt fences, so it waits and times out too, and the log repeats search receipt was never published; rebuilding / search could not synchronize its index. Diff kept on compute02 at ~/perf/mail-1011-repro/bounded-wait-experiment.diff, not pushed.

So the fence design has no escape when a receipt's version is never published. Decision needed from the owner, not a quick patch:

  1. If the gap stage's receipts are not valid (version ahead of anything the notes history can reach), fix the harness gap data and keep the fence as designed. Its source is needed for that.
  2. If a real Home can reach that state (history rebuilt, restored from an older backup), the fence needs a defined exit, e.g. after a cap the rebuild accepts the provider's current state, drops that receipt's fence and logs it, with a test.

Cleanup: no ticket, no lock, no unit left on perf-01, perf-02 or compute02. Repro files kept at ~/perf/mail-1011-repro on compute02 (data 1.5 GB).

## P1b: stall root-caused off-lock (compute02, no perf lock), Mac gates re-run at `02baeb542` PR head (API): `02baeb54212095d956d1fcce49594d6c36d36ea2`. No perf numbers yet; I did not take the perf turn because the stall is a design question (below). ### Mac gates at `02baeb542` (macos-01, one lock hold per step, each released) | Gate | Command (inside the Nix dev shell) | Result | |---|---|---| | Format | `cargo fmt --check` | PASS | | Clippy | `cargo clippy --workspace --all-targets -- -D warnings` | PASS | | Search lib | `cargo test -p calternal-search --lib` | PASS: 40 passed, 0 failed | | Workspace | `cargo test --workspace --no-fail-fast` | PASS: 1,609 passed, 0 failed, 22 ignored | ### Stall repro, compute02 (12 cores), 100k warm, 20 min cap, no perf lock | Harness | Server | Result | |---|---|---| | clean (built from each tree) | dev | pass, rc=0 | | clean | candidate | pass, rc=0 (6 min) | | edited dev harness (`e02f9fc2…`, has the save-log gap stage) | dev | pass, rc=0 | | edited candidate harness (`21c0679b…`, gap stage) | candidate | **stalls at "warm run 1", rc=124, 2 of 2 runs** | | edited candidate harness | dev | slow, not stuck: `notes.list incomplete: true`, host load ~35 (rc=124 at the cap) | The clean harness has no gap stage, so `search.catch_up` is about 0 s and `search.cold_reconcile` is absent. It does not measure what #1011 changes. The edited harnesses' gap source is in no repo or worktree I could find. ### Root cause (candidate server) At the stall the server answers `/healthz` 200 and `notes.list` returns `incomplete: false`, but `search.query` returns `indexing: true` indefinitely, with every thread idle. A diagnostic build (extra `warn!` only) logged `DIAG waiting for publication signal provider="notes"` and never logged that it arrived. The indexer is parked in `PublicationWait::retry` (`crates/search/src/indexer/sync.rs`): after its 5 backoff rereads it waits on `receiver.changed().await` with no cap, for a receipt whose `source_version` is ahead of what the notes provider publishes. Nothing publishes again, because the notes scan is already finished. ### What I tried and why I stopped I capped that wait at 30 s and returned `SyncTokenExpired` so the caller rebuilds. The stall changed shape but did not go away: the rebuild (`reconcile_pages`) enforces the same receipt fences, so it waits and times out too, and the log repeats `search receipt was never published; rebuilding` / `search could not synchronize its index`. Diff kept on compute02 at `~/perf/mail-1011-repro/bounded-wait-experiment.diff`, not pushed. So the fence design has no escape when a receipt's version is never published. Decision needed from the owner, not a quick patch: 1. If the gap stage's receipts are not valid (version ahead of anything the notes history can reach), fix the harness gap data and keep the fence as designed. Its source is needed for that. 2. If a real Home can reach that state (history rebuilt, restored from an older backup), the fence needs a defined exit, e.g. after a cap the rebuild accepts the provider's current state, drops that receipt's fence and logs it, with a test. Cleanup: no ticket, no lock, no unit left on perf-01, perf-02 or compute02. Repro files kept at `~/perf/mail-1011-repro` on compute02 (data 1.5 GB).
kayg commented 2026-11-09 18:31:45 +00:00
Author
Owner
Copy link

P1b perf ABAB x5 with the save-log gap stage, 100k, perf-01/perf-02 (2026-11-09 18:52-19:30 CET)

PR head dcb50cda8df62e2cad6d3fdd9218f7b3538f6352 (candidate server built from it; harness --gap-saves 600 --search-only). Dev = 93d9a321 build. Run through plain FIFO perf-turn (front-of-line grant), both perf locks held for the whole run, released after.

Order per block: dev, candidate, dev, candidate; 5 blocks, 20 arms, every arm passed (no stall, no failure). The control is dev vs dev: "spread" is the median relative gap between the two dev arms of a block.

arms: 20 failures: []
| metric | stat | dev median | cand median | delta | dev-vs-dev spread (median |a1-a3|) | cand/dev blocks worse | verdict |
|---|---|---:|---:|---:|---:|---:|---|
| search.catch_up | p50 | 2.01 | 2.01 | -0.3% | 0.4% | 1/5 | better |
| search.cold_reconcile | p50 | 47.9 | 49.2 | +2.8% | 2.3% | 4/5 | SLOWER beyond noise |
| search.query | p50 | 1.61 | 3.2 | +98.1% | 8.1% | 5/5 | SLOWER beyond noise |
| search.query | p95 | 3.94 | 7.46 | +89.6% | 8.2% | 5/5 | SLOWER beyond noise |
| search.first_query | p50 | 3.44e+03 | 2.09e+03 | -39.1% | 18.7% | 0/5 | better |
| startup.healthz | p50 | 5.15 | 5.28 | +2.5% | 31.3% | 2/5 | flat/better |
| open.api_get | p50 | 2.49 | 2.94 | +18.1% | 6.4% | 4/5 | SLOWER beyond noise |
| open.api_get | p95 | 21.9 | 20.5 | -6.5% | 10.9% | 1/5 | better |
| open.note | p50 | 2.53 | 3.27 | +29.2% | 6.1% | 5/5 | SLOWER beyond noise |
| open.note | p95 | 9.61 | 98.7 | +927.0% | 23.6% | 5/5 | SLOWER beyond noise |

This is not mergeable as it stands. search.query is consistently about 2x slower on the candidate in all 5 blocks (dev 1.5-1.7 ms, candidate 3.0-3.7 ms at p50; p95 3.9 vs 7.4 ms), far outside the dev-vs-dev spread. open.note p50 is a little higher (about +0.5 ms) and its p95 is outlier-driven (one ~100 ms open in 7 of 10 candidate arms against 1 of 10 dev arms). search.cold_reconcile is +2.8% against a 2.3% spread (borderline). search.first_query is better and search.catch_up is flat.

Caveats on this table:

  • search.catch_up reads 2.01 s in both arms because the harness polled "still indexing?" every 2 s, so any catch-up under 2 s is invisible. The poll is now 50 ms in ae2383a7d (harness only); a second turn with the fine harness is needed for a real catch-up number.
  • 1M notes is not measured (#1028).

Investigating the search.query slowdown next (index shape after the paged rebuild is the first suspect; no read-path code changed in the PR).

## P1b perf ABAB x5 with the save-log gap stage, 100k, perf-01/perf-02 (2026-11-09 18:52-19:30 CET) PR head `dcb50cda8df62e2cad6d3fdd9218f7b3538f6352` (candidate server built from it; harness `--gap-saves 600 --search-only`). Dev = `93d9a321` build. Run through plain FIFO `perf-turn` (front-of-line grant), both perf locks held for the whole run, released after. Order per block: dev, candidate, dev, candidate; 5 blocks, 20 arms, every arm passed (no stall, no failure). The control is dev vs dev: "spread" is the median relative gap between the two dev arms of a block. arms: 20 failures: [] | metric | stat | dev median | cand median | delta | dev-vs-dev spread (median |a1-a3|) | cand/dev blocks worse | verdict | |---|---|---:|---:|---:|---:|---:|---| | search.catch_up | p50 | 2.01 | 2.01 | -0.3% | 0.4% | 1/5 | better | | search.cold_reconcile | p50 | 47.9 | 49.2 | +2.8% | 2.3% | 4/5 | SLOWER beyond noise | | search.query | p50 | 1.61 | 3.2 | +98.1% | 8.1% | 5/5 | SLOWER beyond noise | | search.query | p95 | 3.94 | 7.46 | +89.6% | 8.2% | 5/5 | SLOWER beyond noise | | search.first_query | p50 | 3.44e+03 | 2.09e+03 | -39.1% | 18.7% | 0/5 | better | | startup.healthz | p50 | 5.15 | 5.28 | +2.5% | 31.3% | 2/5 | flat/better | | open.api_get | p50 | 2.49 | 2.94 | +18.1% | 6.4% | 4/5 | SLOWER beyond noise | | open.api_get | p95 | 21.9 | 20.5 | -6.5% | 10.9% | 1/5 | better | | open.note | p50 | 2.53 | 3.27 | +29.2% | 6.1% | 5/5 | SLOWER beyond noise | | open.note | p95 | 9.61 | 98.7 | +927.0% | 23.6% | 5/5 | SLOWER beyond noise | **This is not mergeable as it stands.** `search.query` is consistently about 2x slower on the candidate in all 5 blocks (dev 1.5-1.7 ms, candidate 3.0-3.7 ms at p50; p95 3.9 vs 7.4 ms), far outside the dev-vs-dev spread. `open.note` p50 is a little higher (about +0.5 ms) and its p95 is outlier-driven (one ~100 ms open in 7 of 10 candidate arms against 1 of 10 dev arms). `search.cold_reconcile` is +2.8% against a 2.3% spread (borderline). `search.first_query` is better and `search.catch_up` is flat. Caveats on this table: - `search.catch_up` reads 2.01 s in both arms because the harness polled "still indexing?" every 2 s, so any catch-up under 2 s is invisible. The poll is now 50 ms in `ae2383a7d` (harness only); a second turn with the fine harness is needed for a real catch-up number. - 1M notes is not measured (#1028). Investigating the `search.query` slowdown next (index shape after the paged rebuild is the first suspect; no read-path code changed in the PR).
kayg commented 2026-11-09 23:42:42 +00:00
Author
Owner
Copy link

P1b: liveness fix done; perf regression on search.query found, plan below (stopping, not patching)

PR head dcb50cda8df62e2cad6d3fdd9218f7b3538f6352 (the liveness fix and the gap stage are pushed to it; ae2383a7d on the side branch t3/mail-1011-liveness adds a 50 ms catch-up poll to the harness and is not on the PR yet).

Done

  • Stall fixed in both places (crates/search/src/indexer/publication.rs, sync.rs): the wait is capped (Options::publication_cap, 30 s) and the page or rebuild is then staged without its fences. Counter fences_superseded(), one warn per pass.
  • Tests: 4 new in fence_liveness.rs (receipt ahead of provider finishes on the catch-up path and on the rebuild path; late publication inside the cap is honoured on both). Search lib: 44 passed.
  • Gap stage in the repo: --gap-saves N, --search-only. Candidate with gap stage: 3/3 warm runs pass on compute02 (the stall was 2/2 before).
  • Mac gates at dcb50cda8 (macos-01): fmt, clippy pass; search lib 44 passed; workspace 1,614 passed, 0 failed, 22 ignored. The first Mac run at b4cb96a7e had one kill_nine_while_people_type_loses_and_doubles_nothing failure; ten reruns at dcb50cda8 gave 8 pass, 2 FAIL (13.6 s, the same failure shape). It is the known flaky test (see the 11-08 comment: dev also failed 1 of 20 on Linux), but I have not shown it is unrelated on macOS, so I am not calling it clean.
  • ABAB x5 with gap stage: table in the comment above. All 20 arms passed.

Not mergeable: search.query about 2x slower (5 of 5 blocks)

p50 dev 1.5-1.7 ms vs candidate 3.0-3.7 ms; p95 3.9 vs 7.4 ms. Also open.note p50 +0.5 ms (p95 outliers of ~100 ms in 7 of 10 candidate arms), search.cold_reconcile +2.8% (spread 2.3%). Better: search.first_query (-39%); flat: search.catch_up (but the poll was too coarse to see it).

What I narrowed down (compute02, loaded, so ratios only)

  • The query path is unchanged in the PR (Search::query and index.search are the same code).
  • Final index shape is identical between arms (100,000 docs, same FTS/title/trigram row counts, same page count). Only the WAL differs (81 MB candidate, 61 MB dev).
  • Without the gap stage the candidate is not slower (search.query p50 5.1 / 3.6 ms vs dev 7.5 / 8.2 ms). So the slowdown comes from the save-log catch-up path leaving the index in a state that queries more slowly, not from the cold build or the query code.

Plan (needs a decision: this is round 2 of "new problem found", so no blind patch)

  1. Reproduce on perf-01 with the fine harness (ae2383a7d), gap on, and read PRAGMA wal_checkpoint and WAL size at query time for both arms (one front-of-line turn, 30 min).
  2. If the WAL is the cause (candidate publishes each catch-up page as its own WAL commit, and wal_autocheckpoint = 0): checkpoint (passive) once after catch-up finishes, in the indexer, with a test that WAL size is bounded after a catch-up of N pages.
  3. If not the WAL: compare fts segment structure and ANALYZE/stat tables between arms, then the generation/vocab cache invalidation counts (candidate bumps generation once per page).
  4. Re-run the ABAB with search.query as the gate. Nothing merges until search.query, open.note and cold_reconcile are flat against dev.

Cleanup done: compute02 repro data and targets deleted (59 MB of scripts left), perf-01 run data deleted, no ticket, lock or unit held on either perf VM.

## P1b: liveness fix done; perf regression on `search.query` found, plan below (stopping, not patching) PR head `dcb50cda8df62e2cad6d3fdd9218f7b3538f6352` (the liveness fix and the gap stage are pushed to it; `ae2383a7d` on the side branch `t3/mail-1011-liveness` adds a 50 ms catch-up poll to the harness and is not on the PR yet). ### Done - **Stall fixed in both places** (`crates/search/src/indexer/publication.rs`, `sync.rs`): the wait is capped (`Options::publication_cap`, 30 s) and the page or rebuild is then staged without its fences. Counter `fences_superseded()`, one warn per pass. - **Tests:** 4 new in `fence_liveness.rs` (receipt ahead of provider finishes on the catch-up path and on the rebuild path; late publication inside the cap is honoured on both). Search lib: 44 passed. - **Gap stage in the repo:** `--gap-saves N`, `--search-only`. Candidate with gap stage: 3/3 warm runs pass on compute02 (the stall was 2/2 before). - **Mac gates at `dcb50cda8`** (macos-01): fmt, clippy pass; search lib 44 passed; workspace 1,614 passed, 0 failed, 22 ignored. The first Mac run at `b4cb96a7e` had one `kill_nine_while_people_type_loses_and_doubles_nothing` failure; ten reruns at `dcb50cda8` gave 8 pass, 2 FAIL (13.6 s, the same failure shape). It is the known flaky test (see the 11-08 comment: dev also failed 1 of 20 on Linux), but I have not shown it is unrelated on macOS, so I am not calling it clean. - **ABAB x5 with gap stage**: table in the comment above. All 20 arms passed. ### Not mergeable: `search.query` about 2x slower (5 of 5 blocks) p50 dev 1.5-1.7 ms vs candidate 3.0-3.7 ms; p95 3.9 vs 7.4 ms. Also `open.note` p50 +0.5 ms (p95 outliers of ~100 ms in 7 of 10 candidate arms), `search.cold_reconcile` +2.8% (spread 2.3%). Better: `search.first_query` (-39%); flat: `search.catch_up` (but the poll was too coarse to see it). ### What I narrowed down (compute02, loaded, so ratios only) - The query path is unchanged in the PR (`Search::query` and `index.search` are the same code). - Final index shape is identical between arms (100,000 docs, same FTS/title/trigram row counts, same page count). Only the WAL differs (81 MB candidate, 61 MB dev). - **Without the gap stage the candidate is not slower** (`search.query` p50 5.1 / 3.6 ms vs dev 7.5 / 8.2 ms). So the slowdown comes from the save-log catch-up path leaving the index in a state that queries more slowly, not from the cold build or the query code. ### Plan (needs a decision: this is round 2 of "new problem found", so no blind patch) 1. Reproduce on perf-01 with the fine harness (`ae2383a7d`), gap on, and read `PRAGMA wal_checkpoint` and WAL size at query time for both arms (one front-of-line turn, 30 min). 2. If the WAL is the cause (candidate publishes each catch-up page as its own WAL commit, and `wal_autocheckpoint = 0`): checkpoint (passive) once after catch-up finishes, in the indexer, with a test that WAL size is bounded after a catch-up of N pages. 3. If not the WAL: compare `fts` segment structure and `ANALYZE`/stat tables between arms, then the `generation`/vocab cache invalidation counts (candidate bumps `generation` once per page). 4. Re-run the ABAB with `search.query` as the gate. Nothing merges until `search.query`, `open.note` and `cold_reconcile` are flat against dev. Cleanup done: compute02 repro data and targets deleted (59 MB of scripts left), perf-01 run data deleted, no ticket, lock or unit held on either perf VM.
kayg commented 2026-11-10 09:41:50 +00:00
Author
Owner
Copy link

P1b round 2: search.query regression explained (harness artifact); ABAB x5 re-run; two gates still open

PR head 3d5999c7a775725b842ae85fbec24d08f139d78f (candidate server = the PR's search code; the commits since dcb50cda8 are harness-only: 50 ms catch-up poll, populated-baseline gap stage).

Why search.query looked 2x slower

  • Cross-test on compute02: the index state decides it, not the server binary. A state built by the candidate's catch-up answers in ~7 ms under either binary; the dev-built state in ~3.4 ms under either.
  • Table pages, fill, row counts and FTS segments are identical. The one difference: the dev-built state has 0 frecency rows, the candidate-built one 593. The candidate's catch-up replays the save "uses" (correct); dev's legacy-cursor path deliberately skips them.
  • read.rs::frecent() returns at once when frecency is empty; otherwise it reads the 64 most recent used notes for every query. A fixed cost that saturates at 64 rows, which every Home in use pays on dev too.
  • Proof by swap: with 593 rows inserted into the dev state and the candidate state emptied, the slow/fast pattern inverted under both binaries.
  • WAL is not involved (the writer already checkpoints passively after 250 ms of quiet); no checkpoint change is needed, so I did not add one.
  • Fix is in the harness only (3d5999c7a): the gap stage opens 100 notes before the gap so both arms start with a used-notes table (checked: dev arm 100 rows, candidate 593).

ABAB x5, 100k, gap stage, populated baseline (perf-01/02, 2026-11-10 06:36-~07:30 CET), 20 arms, 0 failures

arms: 20 failures: []
| metric | stat | dev median | cand median | delta | dev-vs-dev spread (median |a1-a3|) | cand/dev blocks worse | verdict |
|---|---|---:|---:|---:|---:|---:|---|
| search.catch_up | p50 | 1.27 | 0.85 | -33.0% | 3.0% | 0/5 | better |
| search.cold_reconcile | p50 | 45.6 | 47.3 | +3.5% | 1.2% | 5/5 | SLOWER beyond noise |
| search.query | p50 | 3.34 | 3.23 | -3.4% | 7.3% | 2/5 | better |
| search.query | p95 | 7.39 | 7.48 | +1.2% | 8.4% | 3/5 | flat/better |
| search.first_query | p50 | 5.62 | 7.43 | +32.2% | 11.0% | 5/5 | SLOWER beyond noise |
| startup.healthz | p50 | 5.55 | 5.05 | -9.1% | 47.2% | 0/5 | better |
| open.api_get | p50 | 0.851 | 0.819 | -3.8% | 6.8% | 3/5 | better |
| open.api_get | p95 | 3.89 | 4.61 | +18.5% | 238.4% | 4/5 | flat/better |
| open.note | p50 | 0.947 | 0.851 | -10.1% | 16.3% | 2/5 | better |
| open.note | p95 | 36.7 | 4.86 | -86.8% | 16.1% | 0/5 | better |

Gate "search.query, open.note, cold_reconcile all <= dev"

  • search.query p50 3.34 -> 3.23 ms, p95 7.39 -> 7.48 ms: flat (inside the 7-8% spread). Pass.
  • open.note p50 -10%, p95 36.7 -> 4.9 ms: pass (dev p95 had a ~90 ms outlier in 3 of 10 arms).
  • search.catch_up 1.27 -> 0.85 s (-33%, 0/5 blocks worse): better.
  • search.cold_reconcile 45.6 -> 47.3 s (+3.5%, 5 of 5 blocks worse, spread 1.2%): FAIL, a steady ~+2 s. Likely the receipt-fence scan the rebuild now does before listing providers (stage_source_fences, then fence joins in compare_page); not yet profiled.
  • search.first_query p50 5.6 -> 7.4 ms (+32%, 5 of 5 blocks): mostly 4 of 10 candidate arms at 9.9-17.7 ms; the other arms are 5-7 ms like dev. Small in absolute terms but not flat: FAIL by the no-regression rule until explained.

macOS flake kill_nine_while_people_type_loses_and_doubles_nothing (macos-01, same command, 20 runs each)

  • dev 9438918f3: 20/20 pass.
  • PR head: 2/20 fail (13.6 s, ... never converged, the rejoined-tab text check at notes_crash_typing.rs:413). Earlier PR runs: 1 of 1 and 2 of 10 failed. Pooled that is 5 failures in ~31 PR runs vs 0 in 20 on dev.
  • So I can no longer call it pre-existing on macOS. The PR changes the Notes room/publish path (room/save.rs, room/mod.rs listed now keyed by version, meta.rs live_meta gets source_version). Not diagnosed.

State

Liveness fix, tests and gap stage are pushed. Not mergeable: cold_reconcile and first_query regress, and the Mac flake looks PR-specific. Two follow-ups for a decision (the 3-round rule applies, so I am stopping here):

  1. Profile the rebuild's fence scan on perf (cold_reconcile +2 s), then either speed it up or accept it as the cost of fenced receipts, with kayg's call since the rule is "no regression".
  2. Bisect the Mac flake over the Notes-side commits (20 runs per step on macos-01).
    Also worth its own issue: search.query pays a fixed frecent() cost for any Home with 64+ used notes, on dev too.

Cleanup: perf-01/perf-02 locks and units released, compute02 and perf-01 run data removed.

## P1b round 2: `search.query` regression explained (harness artifact); ABAB x5 re-run; two gates still open PR head `3d5999c7a775725b842ae85fbec24d08f139d78f` (candidate server = the PR's search code; the commits since `dcb50cda8` are harness-only: 50 ms catch-up poll, populated-baseline gap stage). ### Why `search.query` looked 2x slower - Cross-test on compute02: the **index state** decides it, not the server binary. A state built by the candidate's catch-up answers in ~7 ms under either binary; the dev-built state in ~3.4 ms under either. - Table pages, fill, row counts and FTS segments are identical. The one difference: the dev-built state has **0 `frecency` rows**, the candidate-built one 593. The candidate's catch-up replays the save "uses" (correct); dev's legacy-cursor path deliberately skips them. - `read.rs::frecent()` returns at once when `frecency` is empty; otherwise it reads the 64 most recent used notes for every query. A fixed cost that saturates at 64 rows, which every Home in use pays on dev too. - Proof by swap: with 593 rows inserted into the dev state and the candidate state emptied, the slow/fast pattern inverted under both binaries. - WAL is not involved (the writer already checkpoints passively after 250 ms of quiet); no checkpoint change is needed, so I did not add one. - Fix is in the harness only (`3d5999c7a`): the gap stage opens 100 notes before the gap so both arms start with a used-notes table (checked: dev arm 100 rows, candidate 593). ### ABAB x5, 100k, gap stage, populated baseline (perf-01/02, 2026-11-10 06:36-~07:30 CET), 20 arms, 0 failures arms: 20 failures: [] | metric | stat | dev median | cand median | delta | dev-vs-dev spread (median |a1-a3|) | cand/dev blocks worse | verdict | |---|---|---:|---:|---:|---:|---:|---| | search.catch_up | p50 | 1.27 | 0.85 | -33.0% | 3.0% | 0/5 | better | | search.cold_reconcile | p50 | 45.6 | 47.3 | +3.5% | 1.2% | 5/5 | SLOWER beyond noise | | search.query | p50 | 3.34 | 3.23 | -3.4% | 7.3% | 2/5 | better | | search.query | p95 | 7.39 | 7.48 | +1.2% | 8.4% | 3/5 | flat/better | | search.first_query | p50 | 5.62 | 7.43 | +32.2% | 11.0% | 5/5 | SLOWER beyond noise | | startup.healthz | p50 | 5.55 | 5.05 | -9.1% | 47.2% | 0/5 | better | | open.api_get | p50 | 0.851 | 0.819 | -3.8% | 6.8% | 3/5 | better | | open.api_get | p95 | 3.89 | 4.61 | +18.5% | 238.4% | 4/5 | flat/better | | open.note | p50 | 0.947 | 0.851 | -10.1% | 16.3% | 2/5 | better | | open.note | p95 | 36.7 | 4.86 | -86.8% | 16.1% | 0/5 | better | ### Gate "search.query, open.note, cold_reconcile all <= dev" - `search.query` p50 3.34 -> 3.23 ms, p95 7.39 -> 7.48 ms: **flat** (inside the 7-8% spread). Pass. - `open.note` p50 -10%, p95 36.7 -> 4.9 ms: **pass** (dev p95 had a ~90 ms outlier in 3 of 10 arms). - `search.catch_up` 1.27 -> 0.85 s (-33%, 0/5 blocks worse): **better**. - `search.cold_reconcile` 45.6 -> 47.3 s (**+3.5%**, 5 of 5 blocks worse, spread 1.2%): **FAIL**, a steady ~+2 s. Likely the receipt-fence scan the rebuild now does before listing providers (`stage_source_fences`, then fence joins in `compare_page`); not yet profiled. - `search.first_query` p50 5.6 -> 7.4 ms (+32%, 5 of 5 blocks): mostly 4 of 10 candidate arms at 9.9-17.7 ms; the other arms are 5-7 ms like dev. Small in absolute terms but not flat: **FAIL** by the no-regression rule until explained. ### macOS flake `kill_nine_while_people_type_loses_and_doubles_nothing` (macos-01, same command, 20 runs each) - dev `9438918f3`: **20/20 pass**. - PR head: **2/20 fail** (13.6 s, `... never converged`, the rejoined-tab text check at `notes_crash_typing.rs:413`). Earlier PR runs: 1 of 1 and 2 of 10 failed. Pooled that is 5 failures in ~31 PR runs vs 0 in 20 on dev. - So I can no longer call it pre-existing on macOS. The PR changes the Notes room/publish path (`room/save.rs`, `room/mod.rs` `listed` now keyed by version, `meta.rs live_meta` gets `source_version`). Not diagnosed. ### State Liveness fix, tests and gap stage are pushed. Not mergeable: `cold_reconcile` and `first_query` regress, and the Mac flake looks PR-specific. Two follow-ups for a decision (the 3-round rule applies, so I am stopping here): 1. Profile the rebuild's fence scan on perf (`cold_reconcile` +2 s), then either speed it up or accept it as the cost of fenced receipts, with kayg's call since the rule is "no regression". 2. Bisect the Mac flake over the Notes-side commits (20 runs per step on macos-01). Also worth its own issue: `search.query` pays a fixed `frecent()` cost for any Home with 64+ used notes, on dev too. Cleanup: perf-01/perf-02 locks and units released, compute02 and perf-01 run data removed.
kayg commented 2026-11-10 12:00:52 +00:00
Author
Owner
Copy link

P1b handoff to Codex (Claude usage limit), PR head 015546c00f9e3896a2088c89bc937ab5b36b5dcb

Pushed: everything is on the PR branch. Last commits: 015546c00 (crash-test fix), 2949f1e26 (first_query breakdown metrics), 3d5999c7a (gap stage opens 100 notes), ae2383a7d (50 ms catch-up poll). No perf lock, ticket or Mac lock held; my compute02 units are stopped.

1. macOS flake kill_nine_while_people_type…: found and fixed (test-only)

Mac runs, 20 each (re-run once at 11:13-11:16Z with no window announced in infraternal#260):

Code Test file Result
base 93d9a321a PR's rewritten test 4/20 fail, then 8/20 on re-run
PR head base's old test 0/20 fail
base 93d9a321a fixed test (015546c00) 20/20 pass, twice
PR head fixed test 20/20 pass, twice
Dev's old test passing 20/20 compared different test bodies. The PR's runtime code is not the cause. The fix moves the retrying notes.save before the first notes.get (as the original test did). The exact mechanism is likely the read racing the room's post-restart replay, not proven; the comment in the test says only what is established.

2. search.cold_reconcile +2 s (45.6 -> 47.3 s, 5/5 blocks): NOT fixed, profile not finished

  • Hypothesis: a cold rebuild has no cursor, so stage_source_fences (crates/search/src/indexer/sync.rs) scans the whole retained save-log (~200k entries at 100k notes) before listing; dev does no such scan.
  • I built a timing-only server (warn! lines PROF fence_scan, fence_scan_detail with journal-read vs record_fences ms, list_and_compare, attempt_total) from 3d5999c7a on compute02 (diff is uncommitted in ~/Developer/Worktrees/mail-1011-fix; it is 24 lines in sync.rs). Its first run was just launched and stopped, so there are no numbers yet.
  • Ideas that keep the fence semantics (need your design call): skip the per-entry item.clone() for entries with no fence (created entries), batch record_fences, or pipeline the scan with provider listing. A scan avoided only for a brand-new empty index would change semantics, so no.

3. search.first_query outliers: not explained yet

Candidate arms 4/10 at 9.9-17.7 ms vs 5-7 ms otherwise (dev 4.6-6.5). Queries use a pooled reader, so the first one opens a fresh connection (cold SQLite cache). 2949f1e26 adds search.first_query_sql (server took_ms), search.second_query and search.third_query to the harness, so the next ABAB separates SQL time from open/queue time. I did not test the cold-page-cache idea itself.

4. frecent() fixed cost: issue #1053

Evidence and the proposed top-64 in-memory cache are there. It explains the earlier 2x search.query (empty frecency table in the dev arm); harness fixed, search.query is now flat.

Next ABAB

Use plain perf-turn; the harness needs calternal-perf-fine rebuilt at 2949f1e26. Perf-02 handshake note: ssh from perf-01 to perf-02 fails inside systemd units, and my poller must not delete the runner's own ready file (it did, once).

## P1b handoff to Codex (Claude usage limit), PR head `015546c00f9e3896a2088c89bc937ab5b36b5dcb` Pushed: everything is on the PR branch. Last commits: `015546c00` (crash-test fix), `2949f1e26` (first_query breakdown metrics), `3d5999c7a` (gap stage opens 100 notes), `ae2383a7d` (50 ms catch-up poll). No perf lock, ticket or Mac lock held; my compute02 units are stopped. ### 1. macOS flake `kill_nine_while_people_type…`: found and fixed (test-only) Mac runs, 20 each (re-run once at 11:13-11:16Z with no window announced in infraternal#260): | Code | Test file | Result | |---|---|---| | base `93d9a321a` | PR's rewritten test | 4/20 fail, then 8/20 on re-run | | PR head | base's old test | 0/20 fail | | base `93d9a321a` | **fixed** test (`015546c00`) | 20/20 pass, twice | | PR head | **fixed** test | 20/20 pass, twice | Dev's old test passing 20/20 compared different test bodies. The PR's runtime code is not the cause. The fix moves the retrying `notes.save` before the first `notes.get` (as the original test did). The exact mechanism is likely the read racing the room's post-restart replay, not proven; the comment in the test says only what is established. ### 2. `search.cold_reconcile` +2 s (45.6 -> 47.3 s, 5/5 blocks): NOT fixed, profile not finished - Hypothesis: a cold rebuild has no cursor, so `stage_source_fences` (`crates/search/src/indexer/sync.rs`) scans the whole retained save-log (~200k entries at 100k notes) before listing; dev does no such scan. - I built a timing-only server (`warn!` lines `PROF fence_scan`, `fence_scan_detail` with journal-read vs `record_fences` ms, `list_and_compare`, `attempt_total`) from `3d5999c7a` on compute02 (diff is uncommitted in `~/Developer/Worktrees/mail-1011-fix`; it is 24 lines in `sync.rs`). Its first run was just launched and stopped, so there are no numbers yet. - Ideas that keep the fence semantics (need your design call): skip the per-entry `item.clone()` for entries with no fence (created entries), batch `record_fences`, or pipeline the scan with provider listing. A scan avoided only for a brand-new empty index would change semantics, so no. ### 3. `search.first_query` outliers: not explained yet Candidate arms 4/10 at 9.9-17.7 ms vs 5-7 ms otherwise (dev 4.6-6.5). Queries use a pooled reader, so the first one opens a fresh connection (cold SQLite cache). `2949f1e26` adds `search.first_query_sql` (server `took_ms`), `search.second_query` and `search.third_query` to the harness, so the next ABAB separates SQL time from open/queue time. I did not test the cold-page-cache idea itself. ### 4. `frecent()` fixed cost: issue #1053 Evidence and the proposed top-64 in-memory cache are there. It explains the earlier 2x `search.query` (empty `frecency` table in the dev arm); harness fixed, `search.query` is now flat. ### Next ABAB Use plain `perf-turn`; the harness needs `calternal-perf-fine` rebuilt at `2949f1e26`. Perf-02 handshake note: ssh from perf-01 to perf-02 fails inside systemd units, and my poller must not delete the runner's own `ready` file (it did, once).
Some checks failed
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Merge conflict checking is in progress. Try again in few moments.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin t3/paged-search-reconcile-final:t3/paged-search-reconcile-final
git switch t3/paged-search-reconcile-final

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff t3/paged-search-reconcile-final
git switch t3/paged-search-reconcile-final
git rebase dev
git switch dev
git merge --ff-only t3/paged-search-reconcile-final
git switch t3/paged-search-reconcile-final
git rebase dev
git switch dev
git merge --no-ff t3/paged-search-reconcile-final
git switch dev
git merge --squash t3/paged-search-reconcile-final
git switch dev
git merge --ff-only t3/paged-search-reconcile-final
git switch dev
git merge t3/paged-search-reconcile-final
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!1011
No description provided.