kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

notes: add hide and archive actions with exact batch undo #887

Open
kayg wants to merge 1 commit from notes-archive-hide into dev
pull from: notes-archive-hide
merge into: kayg:dev
Conversation 33 Commits 1 Files changed 2 +88
kayg commented 2026-11-03 05:04:45 +00:00
Owner
Copy link

implementation plan

Add file-level hidden: true and archived: true through typed Notes actions and the strict frontmatter splice planner into the shared Home writer. Remove keys on unhide/unarchive. Bulk is one bounded exact-byte transaction and one undo step. Add conservative Home inverse coordination for these actions, refusing the entire undo after any later edit.

Use a sparse visibility index in meta.db, committed with list/source rows. Ordinary lists and views exclude both flags; show_hidden reveals hidden rows with a flag for the later faded UI; place: archived selects archived rows. Archived sources stay available to other consumers, but the Notes search provider omits them during reconcile and refresh, so FTS query cost is unchanged. Search snapshots captured during pending shared-writer publication remain readable, but an index-only validation hook forces the indexer to retry before marking their journal events complete.

sizes and complexity

At most 100 unique ids per request, existing 8 MiB per-note and 64 MiB transaction caps. Planning is O(total selected bytes); placement/publication is one batch. Visibility storage is O(flagged notes). Ordinary first-page ordering retains the existing index; total subtracts a union of sparse excluded ids, never scans the entire Notes table. Search query SQL stays unchanged.

failure table

failure effect detection recovery test
unsupported metadata or a value that isn't true/false (an old hidden: block list is the exception, see H1) that note skipped with a reason, others change strict current read + planner edit source visibility_skips_values_that_are_not_flags_byte_for_byte, unit table
oversized request no admitted batch caps smaller selection bulk cap tests
another device or file sync edits a note mid-action batch planned again from fresh bytes (max 3) lease / room revision refusal before the journal automatic visibility_plans_again_when_a_note_changes_mid_action
later room/file edit, rename or trash before undo whole undo refused as BlockChanged (never 500) room revision, binding and exact byte guards refresh; preserve newer text changed-post, rename and trash undo tests
interrupted placement/publication writer keeps recovery intent shared executor journal existing Home recovery executor regression suite
journal receipt precedes accepted metadata old snapshot cannot mark FTS fresh provider index-publication validation pin and retry held-publication search regression
duplicate request original result executor request fingerprint + action idempotency replay retry test
foreign note id no foreign Home access owner-scoped lookup use authorized owner path isolation test

decisions

Hidden notes leave every ordinary list/view but remain searchable; this orchestrator-selected behavior is reversible. Show hidden is per-query. Archived notes leave search and appear in the built-in Archived place. Hidden+archived notes require Show hidden even there. These flags edit the FILE, so a shared-note implementation needs Edit permission, never a personal Read-only preference. This dev base retains owner-scoped actions; cross-owner access remains refused.

review fixes (76637), decisions

  • Per-note results. Replies add done, skipped and results: [{id, state: done|skipped, changed, reason?}], so the UI can say "Archived 99 · 1 couldn't be changed". The changed notes are still one exact batch with one undo. Notes already in place are not guarded as dependencies, so an edit to one of them can't refuse the rest. A note deleted mid-action is skipped. A note that keeps changing through 3 plans still refuses the whole batch (BlockChanged), because the executor doesn't name the participant that refused.
  • Reader direction: strict. The reader got stricter rather than the writer more lenient. Teaching the writer CRLF/BOM/True would mean a second splice path through lossy decoding for a data-safety feature. A file like that simply isn't flagged until it's edited in source mode.
  • Tasks. Tasks in hidden or archived notes give no task rows (TasksDescriber::describe), so Tasks lists, counts, Untracked and task tag counts leave them out, the way archived notes leave search. Hidden counts too: Tasks views are views. One consequence: a checkbox in a visible note that links a hidden/archived task note counts as an ordinary (untracked) checkbox.
  • Backlinks. The backlinks panel leaves out hidden and archived source notes. Rename and Trash relinking (sources_of) still include them, so their links stay correct. Unlinked mentions leave out hidden and archived notes too, even though search keeps hidden ones.
  • Tag counts. Notes tag counts are computed by the web store from list rows, which already exclude hidden/archived. notes.tag-preview still counts every note, because a tag rename also edits hidden and archived notes.
  • Hidden rule. Hidden notes are out of every list and view (Notes, Tasks, backlinks) and still in search.
  • Other devices. notes.saved from these actions carries visibility (read from the exact post-image bytes). The web store drops a row that became hidden/archived and reads the list again when one comes back (notes-visibility.ts, unit-tested). UI in #877.
  • Search freshness. A publishing executor write pauses search refreshes only for its own notes (validate_index(Some(items))). A full reconcile still waits, because it marks every journal entry up to head done.
  • Archived place. It drives from note_visibility via visibility_archived (CROSS JOIN order) and looks notes up by primary key: O(archived) rows plus a sort of those rows. Without the index hint during a fresh fill. Query-plan test included.
  • Undo errors. A changed room, rename, trash or consumed undo returns BlockChanged; another author gets Forbidden.
  • Not changed here: key names (hidden: vs ADR 0056) and what Archive means for shared notes; both are with kayg.

validation

Exact commands and results: gate evidence and dev-01 performance comparison.

UI wiring belongs to #877. Never merge.

One grammar decides both sides (review 76637 P1/P2): strict_frontmatter::current reads a key with exactly the planner's grammar. A note counts as hidden/archived only when that read gives a plain true and its raw bytes are plain (no BOM, valid UTF-8, no \r in the metadata). The decoded text the list and search readers see has lost the BOM and CRLF, so every reader site carries that raw bit. The actions only change a key that is absent or a plain true/false. Everything else (ADR 0056 hidden: block lists, yes, True, quoted strings, numbers, maps, CRLF/BOM metadata, comments) is neither a flag nor writable: that note is skipped with a reason and stays byte-exact. So nothing can be archived but impossible to unarchive. Supported files keep every non-target byte, including body comments and a missing final newline. Client controls and selected-note existence belong to #877: a hidden search hit must be opened through Notes read access, not treated as deleted because it's absent from the ordinary list.

H1: hidden-block list key (kayg, 2026-11-03)

  • Note flag stays hidden: true (Hide/Unhide write only that key, or remove it).
  • ADR 0056's block list moves to hidden blocks:. Block hide/unhide write only that key.
  • Read: the old hidden: list and hidden blocks: both count as block references (union). An old list never means "note hidden".
  • Move: the next block hide/unhide on a note with an old hidden: list moves it to hidden blocks: in the same write; Hide on such a note moves it too and sets the flag. Unhide on it is a no-op. Nothing else in the file changes; no bulk rewrite. This replaces the earlier "skip: value isn't a boolean" for lists.
  • Search skips lines whose ^id is in either list (line numbers kept).
  • Writer: the strict writer now prints plain-word keys bare, spaced ones included (hidden blocks:, like last edited:), for every caller. Words YAML reads as values (true, no, null…) and id keys stay quoted. Lists keep the writer's block-list style, the same as tags:. A quoted "hidden blocks": already in a file is read, kept byte for byte, and printed bare only when its own value is rewritten. The tasks status tests were updated to expect bare keys.
  • Code: crates/notes/src/block_visibility.rs (read + change set), visibility.rs (decide_all), search.rs. There is no block hide action in the backend yet; block_visibility::changes is the write side it will call.
  • H2 (what Archive means for shared notes) is unchanged.
## implementation plan Add file-level `hidden: true` and `archived: true` through typed Notes actions and the strict frontmatter splice planner into the shared Home writer. Remove keys on unhide/unarchive. Bulk is one bounded exact-byte transaction and one undo step. Add conservative Home inverse coordination for these actions, refusing the entire undo after any later edit. Use a sparse visibility index in meta.db, committed with list/source rows. Ordinary lists and views exclude both flags; `show_hidden` reveals hidden rows with a flag for the later faded UI; `place: archived` selects archived rows. Archived sources stay available to other consumers, but the Notes search provider omits them during reconcile and refresh, so FTS query cost is unchanged. Search snapshots captured during pending shared-writer publication remain readable, but an index-only validation hook forces the indexer to retry before marking their journal events complete. ### sizes and complexity At most 100 unique ids per request, existing 8 MiB per-note and 64 MiB transaction caps. Planning is O(total selected bytes); placement/publication is one batch. Visibility storage is O(flagged notes). Ordinary first-page ordering retains the existing index; total subtracts a union of sparse excluded ids, never scans the entire Notes table. Search query SQL stays unchanged. ### failure table | failure | effect | detection | recovery | test | |---|---|---|---|---| | unsupported metadata or a value that isn't `true`/`false` (an old `hidden:` block list is the exception, see H1) | that note skipped with a reason, others change | strict `current` read + planner | edit source | `visibility_skips_values_that_are_not_flags_byte_for_byte`, unit table | | oversized request | no admitted batch | caps | smaller selection | bulk cap tests | | another device or file sync edits a note mid-action | batch planned again from fresh bytes (max 3) | lease / room revision refusal before the journal | automatic | `visibility_plans_again_when_a_note_changes_mid_action` | | later room/file edit, rename or trash before undo | whole undo refused as `BlockChanged` (never 500) | room revision, binding and exact byte guards | refresh; preserve newer text | changed-post, rename and trash undo tests | | interrupted placement/publication | writer keeps recovery intent | shared executor journal | existing Home recovery | executor regression suite | | journal receipt precedes accepted metadata | old snapshot cannot mark FTS fresh | provider index-publication validation | pin and retry | held-publication search regression | | duplicate request | original result | executor request fingerprint + action idempotency | replay | retry test | | foreign note id | no foreign Home access | owner-scoped lookup | use authorized owner path | isolation test | ### decisions Hidden notes leave every ordinary list/view but remain searchable; this orchestrator-selected behavior is reversible. Show hidden is per-query. Archived notes leave search and appear in the built-in Archived place. Hidden+archived notes require Show hidden even there. These flags edit the FILE, so a shared-note implementation needs Edit permission, never a personal Read-only preference. This dev base retains owner-scoped actions; cross-owner access remains refused. ### review fixes (76637), decisions - **Per-note results.** Replies add `done`, `skipped` and `results: [{id, state: done|skipped, changed, reason?}]`, so the UI can say "Archived 99 · 1 couldn't be changed". The changed notes are still one exact batch with one undo. Notes already in place are not guarded as dependencies, so an edit to one of them can't refuse the rest. A note deleted mid-action is skipped. A note that keeps changing through 3 plans still refuses the whole batch (`BlockChanged`), because the executor doesn't name the participant that refused. - **Reader direction: strict.** The reader got stricter rather than the writer more lenient. Teaching the writer CRLF/BOM/`True` would mean a second splice path through lossy decoding for a data-safety feature. A file like that simply isn't flagged until it's edited in source mode. - **Tasks.** Tasks in hidden or archived notes give no task rows (`TasksDescriber::describe`), so Tasks lists, counts, Untracked and task tag counts leave them out, the way archived notes leave search. Hidden counts too: Tasks views are views. One consequence: a checkbox in a visible note that links a hidden/archived task note counts as an ordinary (untracked) checkbox. - **Backlinks.** The backlinks panel leaves out hidden and archived source notes. Rename and Trash relinking (`sources_of`) still include them, so their links stay correct. Unlinked mentions leave out hidden and archived notes too, even though search keeps hidden ones. - **Tag counts.** Notes tag counts are computed by the web store from list rows, which already exclude hidden/archived. `notes.tag-preview` still counts every note, because a tag rename also edits hidden and archived notes. - **Hidden rule.** Hidden notes are out of every list and view (Notes, Tasks, backlinks) and still in search. - **Other devices.** `notes.saved` from these actions carries `visibility` (read from the exact post-image bytes). The web store drops a row that became hidden/archived and reads the list again when one comes back (`notes-visibility.ts`, unit-tested). UI in #877. - **Search freshness.** A publishing executor write pauses search refreshes only for its own notes (`validate_index(Some(items))`). A full reconcile still waits, because it marks every journal entry up to `head` done. - **Archived place.** It drives from `note_visibility` via `visibility_archived` (`CROSS JOIN` order) and looks notes up by primary key: O(archived) rows plus a sort of those rows. Without the index hint during a fresh fill. Query-plan test included. - **Undo errors.** A changed room, rename, trash or consumed undo returns `BlockChanged`; another author gets `Forbidden`. - **Not changed here:** key names (`hidden:` vs ADR 0056) and what Archive means for shared notes; both are with kayg. ### validation Exact commands and results: [gate evidence](https://git.kayg.org/kayg/calternal/pulls/887#issuecomment-76584) and [dev-01 performance comparison](https://git.kayg.org/kayg/calternal/pulls/887#issuecomment-76625). UI wiring belongs to #877. Never merge. One grammar decides both sides (review [76637](https://git.kayg.org/kayg/calternal/pulls/887#issuecomment-76637) P1/P2): `strict_frontmatter::current` reads a key with exactly the planner's grammar. A note counts as hidden/archived only when that read gives a plain `true` **and** its raw bytes are plain (no BOM, valid UTF-8, no `\r` in the metadata). The decoded text the list and search readers see has lost the BOM and CRLF, so every reader site carries that raw bit. The actions only change a key that is absent or a plain `true`/`false`. Everything else (ADR 0056 `hidden:` block lists, `yes`, `True`, quoted strings, numbers, maps, CRLF/BOM metadata, comments) is neither a flag nor writable: that note is skipped with a reason and stays byte-exact. So nothing can be archived but impossible to unarchive. Supported files keep every non-target byte, including body comments and a missing final newline. Client controls and selected-note existence belong to #877: a hidden search hit must be opened through Notes read access, not treated as deleted because it's absent from the ordinary list. ### H1: hidden-block list key (kayg, 2026-11-03) - Note flag stays `hidden: true` (Hide/Unhide write only that key, or remove it). - ADR 0056's block list moves to `hidden blocks:`. Block hide/unhide write only that key. - Read: the old `hidden:` list and `hidden blocks:` both count as block references (union). An old list never means "note hidden". - Move: the next block hide/unhide on a note with an old `hidden:` list moves it to `hidden blocks:` in the same write; Hide on such a note moves it too and sets the flag. Unhide on it is a no-op. Nothing else in the file changes; no bulk rewrite. This replaces the earlier "skip: value isn't a boolean" for lists. - Search skips lines whose `^id` is in either list (line numbers kept). - Writer: the strict writer now prints plain-word keys bare, spaced ones included (`hidden blocks:`, like `last edited:`), for every caller. Words YAML reads as values (`true`, `no`, `null`…) and id keys stay quoted. Lists keep the writer's block-list style, the same as `tags:`. A quoted `"hidden blocks":` already in a file is read, kept byte for byte, and printed bare only when its own value is rewritten. The tasks status tests were updated to expect bare keys. - Code: `crates/notes/src/block_visibility.rs` (read + change set), `visibility.rs` (`decide_all`), `search.rs`. There is no block hide action in the backend yet; `block_visibility::changes` is the write side it will call. - H2 (what Archive means for shared notes) is unchanged.
notes-archive-hide-base.json
207 KiB
notes-archive-hide-candidate.json
206 KiB
test: specify note hide and archive actions
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
7306f64e10
kayg commented 2026-11-03 05:18:01 +00:00
Author
Owner
Copy link

Implementation checkpoint at 746e18ed: four Notes actions, strict frontmatter batch writes, opt-in exact Home undo guarded by committed room revisions, sparse list visibility and Archived place, archived search-source exclusion, and client visibility event handling are pushed. Tests cover all action directions, bulk undo, BOM/CRLF preservation, owner isolation, malformed batch refusal, hidden+archived queries, restart undo, and live FTS removal/restoration.

Verification is queued through the ordinary slot behind the active BM14 train on dev-01. No gate or performance pass is claimed yet. Generated action contracts and TS verification remain pending the VM run.

Implementation checkpoint at `746e18ed`: four Notes actions, strict frontmatter batch writes, opt-in exact Home undo guarded by committed room revisions, sparse list visibility and Archived place, archived search-source exclusion, and client visibility event handling are pushed. Tests cover all action directions, bulk undo, BOM/CRLF preservation, owner isolation, malformed batch refusal, hidden+archived queries, restart undo, and live FTS removal/restoration. Verification is queued through the ordinary slot behind the active BM14 train on dev-01. No gate or performance pass is claimed yet. Generated action contracts and TS verification remain pending the VM run.
kayg commented 2026-11-03 05:33:51 +00:00
Author
Owner
Copy link

verification queue update, 11:03 ist: dev-01 still has active train lock bm14 3233084. its browser command has finished; its owner is waiting for the train release marker. the notes baseline-red, candidate-green and subsequent contract/rust/clippy/client gate jobs are waiting through the ordinary slot. dev-02 also has an active bm14 train lock. no lock was changed and no passing gate or performance result is claimed. the exact-base 41f2efa3 versus candidate dev-01 comparison is prepared for 10k and 100k homes with five warm restarts per version.

verification queue update, 11:03 ist: dev-01 still has active train lock `bm14 3233084`. its browser command has finished; its owner is waiting for the train release marker. the notes baseline-red, candidate-green and subsequent contract/rust/clippy/client gate jobs are waiting through the ordinary slot. dev-02 also has an active bm14 train lock. no lock was changed and no passing gate or performance result is claimed. the exact-base 41f2efa3 versus candidate dev-01 comparison is prepared for 10k and 100k homes with five warm restarts per version.
kayg commented 2026-11-03 05:50:28 +00:00
Author
Owner
Copy link

targeted verification on dev-01:

  • baseline test-only checkout: cargo test --locked -j4 -p calternal-notes visibility -- --nocapture: 65 passed, 2 expected failures, both UnknownAction for notes.hide.
  • candidate 1c0e5ebb: cargo test --locked -j4 -p calternal-notes tests::visibility:: -- --nocapture: 8 passed.
  • cargo test --locked -j4 -p calternal-search --test note_visibility -- --nocapture: 1 passed, covering hidden search retention, archive FTS removal, unarchive and undo restoration.

all commands ran in nix through the ordinary slot after the bm14 owner released its lock. the earlier checkpoint wording about BOM/CRLF preservation was an unverified fixture assumption: the established strict planner refuses those envelopes, metadata comments and unsupported scalar forms. the final tests prove whole-batch byte-exact refusal for them, and exact forward/inverse bytes for supported metadata, unicode body comments and no final newline.

#877 integration note: ordinary list absence can now mean hidden or archived, so selected-note existence must be resolved separately (Notes read/get remains available). this branch keeps TS wire/action types and server events; client controls and cache/selection integration remain with #877. broader gates and dev-01 performance are still running/pending.

targeted verification on dev-01: - baseline test-only checkout: `cargo test --locked -j4 -p calternal-notes visibility -- --nocapture`: 65 passed, 2 expected failures, both `UnknownAction` for `notes.hide`. - candidate `1c0e5ebb`: `cargo test --locked -j4 -p calternal-notes tests::visibility:: -- --nocapture`: 8 passed. - `cargo test --locked -j4 -p calternal-search --test note_visibility -- --nocapture`: 1 passed, covering hidden search retention, archive FTS removal, unarchive and undo restoration. all commands ran in nix through the ordinary slot after the bm14 owner released its lock. the earlier checkpoint wording about BOM/CRLF preservation was an unverified fixture assumption: the established strict planner refuses those envelopes, metadata comments and unsupported scalar forms. the final tests prove whole-batch byte-exact refusal for them, and exact forward/inverse bytes for supported metadata, unicode body comments and no final newline. #877 integration note: ordinary list absence can now mean hidden or archived, so selected-note existence must be resolved separately (Notes read/get remains available). this branch keeps TS wire/action types and server events; client controls and cache/selection integration remain with #877. broader gates and dev-01 performance are still running/pending.
kayg commented 2026-11-03 06:00:40 +00:00
Author
Owner
Copy link

additional concurrency regression: holding the writer after the durable journal receipt but before metadata publication reproduced stale archived FTS rows at 9f660180. cargo test --locked -j4 -p calternal-search --test note_visibility -- --nocapture failed its eventual-consistency assertion (0 passed, 1 failed). the test gate releases after a refresh opportunity; its timeout is synchronization, not a latency budget.

fd5fd769 adds a provider index-publication validation hook. Notes snapshots captured while a shared writer is pending remain readable as accepted generations, but cannot mark a search refresh complete; the indexer retries against a newly pinned snapshot. search query SQL is unchanged.

the first full touched-crate Rust run also found the cold-fill index-count assertion (554 passed, 1 failed, 6 ignored in Notes): the two new visibility indexes were missing from the existing fresh-fill drop/rebuild inventory. 9f660180 adds them. full gates are rerunning with calternal-core included.

additional concurrency regression: holding the writer after the durable journal receipt but before metadata publication reproduced stale archived FTS rows at `9f660180`. `cargo test --locked -j4 -p calternal-search --test note_visibility -- --nocapture` failed its eventual-consistency assertion (0 passed, 1 failed). the test gate releases after a refresh opportunity; its timeout is synchronization, not a latency budget. `fd5fd769` adds a provider index-publication validation hook. Notes snapshots captured while a shared writer is pending remain readable as accepted generations, but cannot mark a search refresh complete; the indexer retries against a newly pinned snapshot. search query SQL is unchanged. the first full touched-crate Rust run also found the cold-fill index-count assertion (554 passed, 1 failed, 6 ignored in Notes): the two new visibility indexes were missing from the existing fresh-fill drop/rebuild inventory. `9f660180` adds them. full gates are rerunning with calternal-core included.
kayg commented 2026-11-03 06:09:50 +00:00
Author
Owner
Copy link

gates at 32b8533d5 on dev-01 (all in nix through ~/bin/slot):

command result
cargo test --locked -j4 -p calternal-notes tests::visibility:: -- --nocapture pass, 8 tests
cargo test --locked -j4 -p calternal-search --test note_visibility -- --nocapture pass, including held journal-before-metadata publication
cargo test --locked -j4 -p calternal-core -p calternal-notes -p calternal-engine -p calternal-search -p calternal-tasks -p calternal pass; Notes 555 passed, 6 ignored; capability inventory and cross-user isolation pass
cargo clippy --locked -j4 -p calternal-core -p calternal-notes -p calternal-engine -p calternal-search -p calternal-tasks -p calternal --all-targets -- -D warnings pass
cargo fmt --all -- --check pass
bun install --frozen-lockfile pass
bun run --cwd apps/web test pass, 96 tests in 8 files
cargo test --locked -j4 -p calternal contract_matches_the_server pass
node ci/check-case-paths.mjs pass
node ci/check-no-evidence.mjs pass
bun run --cwd apps/web check 2 errors and 15 warnings, confirmed on exact baseline

client baseline proof: the unmodified 41f2efa387b74175971a838f602535089d0d5fb4 checkout's normal check stops first on stale generated action types. running cd apps/web && bunx svelte-check --tsconfig ./tsconfig.json directly on that checkout produces exactly the same 2 errors and 15 warnings as this branch. both errors are in apps/web/e2e/review4-signout.spec.ts:225-226: expect(logoutStatus) narrows to never, so its comparison matchers are unavailable. those files are unchanged. the branch's generated action types pass their drift check and there are no diagnostics in the changed client files.

release binaries for the performance comparison are building. no performance pass is claimed yet.

gates at `32b8533d5` on dev-01 (all in nix through `~/bin/slot`): | command | result | |---|---| | `cargo test --locked -j4 -p calternal-notes tests::visibility:: -- --nocapture` | pass, 8 tests | | `cargo test --locked -j4 -p calternal-search --test note_visibility -- --nocapture` | pass, including held journal-before-metadata publication | | `cargo test --locked -j4 -p calternal-core -p calternal-notes -p calternal-engine -p calternal-search -p calternal-tasks -p calternal` | pass; Notes 555 passed, 6 ignored; capability inventory and cross-user isolation pass | | `cargo clippy --locked -j4 -p calternal-core -p calternal-notes -p calternal-engine -p calternal-search -p calternal-tasks -p calternal --all-targets -- -D warnings` | pass | | `cargo fmt --all -- --check` | pass | | `bun install --frozen-lockfile` | pass | | `bun run --cwd apps/web test` | pass, 96 tests in 8 files | | `cargo test --locked -j4 -p calternal contract_matches_the_server` | pass | | `node ci/check-case-paths.mjs` | pass | | `node ci/check-no-evidence.mjs` | pass | | `bun run --cwd apps/web check` | 2 errors and 15 warnings, confirmed on exact baseline | client baseline proof: the unmodified `41f2efa387b74175971a838f602535089d0d5fb4` checkout's normal check stops first on stale generated action types. running `cd apps/web && bunx svelte-check --tsconfig ./tsconfig.json` directly on that checkout produces exactly the same 2 errors and 15 warnings as this branch. both errors are in `apps/web/e2e/review4-signout.spec.ts:225-226`: `expect(logoutStatus)` narrows to `never`, so its comparison matchers are unavailable. those files are unchanged. the branch's generated action types pass their drift check and there are no diagnostics in the changed client files. release binaries for the performance comparison are building. no performance pass is claimed yet.
kayg commented 2026-11-03 06:30:47 +00:00
Author
Owner
Copy link

dev-01 performance comparison: exact baseline 41f2efa387b74175971a838f602535089d0d5fb4 versus head 32b8533d5fa49b22fb0ffe4b12f3cff841f19e13.

both reports contain zero scenario failures. no measured regression outside the harness-reported noise. the 10k differences are small relative to variation; the 100k results are lower. this does not establish a speedup or precisely identical latency.

notes metric baseline p50 / p95 ms candidate p50 / p95 ms baseline / candidate noise
10,000 home.first_page 3.877 / 4.850 4.514 / 5.240 46% / 33%
10,000 home.page 1.293 / 1.731 1.574 / 2.018 49% / 35%
10,000 search.query 2.644 / 6.088 2.971 / 6.412 42% / 118%
100,000 home.first_page 5.834 / 9.072 4.102 / 4.808 82% / 59%
100,000 home.page 2.267 / 2.993 1.743 / 2.120 21% / 48%
100,000 search.query 3.631 / 8.429 2.473 / 5.374 59% / 130%

machine: Homes: Linux 6.12.113+deb13-amd64 · 12 CPUs · 94Gi RAM · disk /dev/sda1 ext4 (non-rotational) · state: Linux 6.12.113+deb13-amd64 · 12 CPUs · 94Gi RAM · disk /dev/sda1 ext4 (non-rotational). ordinary slot, sequential versions, same fixture seed and harness. indexed fixture homes have no visibility flags set; this measures steady restart behavior, not the one-time schema upgrade. first page: five server restarts, one sample each. ordinary page: five runs x 20 samples. search: five runs x 55 samples. multi-sample metrics use median per-run p95; first page uses the tail across restarts.

build in each checkout, nix + ordinary slot:

cargo build --locked --release -j4 -p calternal -p calternal-perf --bin calternal --bin calternal-perf

the candidate reused the disposable baseline target directory through CARGO_TARGET_DIR; baseline binaries were copied before rebuilding. server sha256:

baseline dd671333af8353dcfb9b9a0d8fc3f5003a93330e1f0db07c19e5215ced094fbb
candidate b545ca5b66ee0b0be6118ea528df043521636c7f2a6f1871f8a992ae32384bdc

measurement command in nix (version = base, then candidate):

~/bin/slot ~/notes-archive-hide-perf/$version-perf --bin ~/notes-archive-hide-perf/$version-calternal --work ~/notes-archive-hide-perf/$version-work --sizes 10k,100k --runs 5 --cold-runs 0 --idle-secs 0 --storm-edits 1 --json ~/notes-archive-hide-perf/$version.json --md ~/notes-archive-hide-perf/$version.md

raw artifacts: baseline JSON / candidate JSON

gate evidence and exact-baseline client-check exception: #887 (comment). UI controls, Archived navigation, Show hidden fading, and cache/selected-note integration remain with #877. file flags require Edit access, not a personal Read-only preference; current actions are owner scoped. hidden-search semantics remain the reversible orchestrator decision.

dev-01 performance comparison: exact baseline `41f2efa387b74175971a838f602535089d0d5fb4` versus head `32b8533d5fa49b22fb0ffe4b12f3cff841f19e13`. both reports contain zero scenario failures. no measured regression outside the harness-reported noise. the 10k differences are small relative to variation; the 100k results are lower. this does not establish a speedup or precisely identical latency. | notes | metric | baseline p50 / p95 ms | candidate p50 / p95 ms | baseline / candidate noise | |---:|---|---:|---:|---:| | 10,000 | `home.first_page` | 3.877 / 4.850 | 4.514 / 5.240 | 46% / 33% | | 10,000 | `home.page` | 1.293 / 1.731 | 1.574 / 2.018 | 49% / 35% | | 10,000 | `search.query` | 2.644 / 6.088 | 2.971 / 6.412 | 42% / 118% | | 100,000 | `home.first_page` | 5.834 / 9.072 | 4.102 / 4.808 | 82% / 59% | | 100,000 | `home.page` | 2.267 / 2.993 | 1.743 / 2.120 | 21% / 48% | | 100,000 | `search.query` | 3.631 / 8.429 | 2.473 / 5.374 | 59% / 130% | machine: Homes: Linux 6.12.113+deb13-amd64 · 12 CPUs · 94Gi RAM · disk /dev/sda1 ext4 (non-rotational) · state: Linux 6.12.113+deb13-amd64 · 12 CPUs · 94Gi RAM · disk /dev/sda1 ext4 (non-rotational). ordinary slot, sequential versions, same fixture seed and harness. indexed fixture homes have no visibility flags set; this measures steady restart behavior, not the one-time schema upgrade. first page: five server restarts, one sample each. ordinary page: five runs x 20 samples. search: five runs x 55 samples. multi-sample metrics use median per-run p95; first page uses the tail across restarts. build in each checkout, nix + ordinary slot: ```sh cargo build --locked --release -j4 -p calternal -p calternal-perf --bin calternal --bin calternal-perf ``` the candidate reused the disposable baseline target directory through `CARGO_TARGET_DIR`; baseline binaries were copied before rebuilding. server sha256: ```text baseline dd671333af8353dcfb9b9a0d8fc3f5003a93330e1f0db07c19e5215ced094fbb candidate b545ca5b66ee0b0be6118ea528df043521636c7f2a6f1871f8a992ae32384bdc ``` measurement command in nix (`version` = `base`, then `candidate`): ```sh ~/bin/slot ~/notes-archive-hide-perf/$version-perf --bin ~/notes-archive-hide-perf/$version-calternal --work ~/notes-archive-hide-perf/$version-work --sizes 10k,100k --runs 5 --cold-runs 0 --idle-secs 0 --storm-edits 1 --json ~/notes-archive-hide-perf/$version.json --md ~/notes-archive-hide-perf/$version.md ``` raw artifacts: [baseline JSON](https://git.kayg.org/attachments/2875e20b-7782-4c70-941a-8ca71810e6e1) / [candidate JSON](https://git.kayg.org/attachments/0d7bb411-c0a6-4fdf-9c86-4d3efa5e79d7) gate evidence and exact-baseline client-check exception: https://git.kayg.org/kayg/calternal/pulls/887#issuecomment-76584. UI controls, Archived navigation, Show hidden fading, and cache/selected-note integration remain with #877. file flags require Edit access, not a personal Read-only preference; current actions are owner scoped. hidden-search semantics remain the reversible orchestrator decision.
kayg changed title from WIP: notes hide and archive through the shared writer to notes: add hide and archive actions with exact batch undo 2026-11-03 06:30:48 +00:00
kayg commented 2026-11-03 06:38:56 +00:00
Author
Owner
Copy link

independent review at 32b8533d (data safety + correctness)

read-only review of the diff against 41f2efa3, built nothing. the exact-splice path, room leases, ExpectedRoom guards and whole-batch undo refusal look right to me. the problems below are mostly at the edges: what the action does to values that aren't the flag, and where the read side and the write side disagree.

P1

1. hide/unhide/archive/unarchive overwrite or delete values that flag() says are not flags
crates/notes/src/actions/visibility.rs:71-76 always sends Change { path: [key], value: true | None }. the strict planner accepts a scalar or block-list leaf as a target (strict_frontmatter.rs:177-181 only refuses Map/Opaque), so it replaces or removes whatever is there. but visibility::flag (visibility.rs:119-132) deliberately treats lists and strings as "not hidden" and keeps them as data.

  • note with legacy block refs in block form: hidden:\n - "^e7k3f9x2m8q1w4zr". lists show it as not hidden, so the UI would offer Hide. Hide rewrites it to hidden: true and the block refs are gone. Unhide on the same note deletes the list outright.
  • archived: yes / archived: old: flag() says not archived, so the note stays in lists and search. notes.unarchive deletes the key; notes.archive rewrites the user's value to true.
    the only way back is the Undo token, and only until the next edit; after that you need Versions. fix: when planning, refuse unless the current value is absent, true (for unhide/unarchive) or false (for hide/archive). add tests for the list, yes, "true" and false cases.

P2

2. the read grammar and the write grammar differ, so a note can be archived but impossible to unarchive
flag() reads through lenient props::entries and serde_yaml 0.9, which accepts True/TRUE, CRLF, BOM, comments, flow lists and uppercase keys. the planner refuses all of those (README: True refuses as a target; CRLF/BOM/comments/flow lists refuse the whole envelope).
scenario: a file synced in over WebDAV or Obsidian with archived: True, or archived: true with CRLF line endings. meta.rs:1149 / generation.rs:150 mark it archived, it leaves search and shows up in Archived, and notes.unarchive returns couldn't plan these property changes (Grammar). the note is stuck out of search until someone edits the source by hand. fix: one grammar. either flag() recognises only what the planner can round-trip, or add a test proving unarchive works for every value flag() accepts.

3. one unsupported note refuses the whole bulk batch, and many ordinary notes are unsupported
visibility.rs:75-76 returns early on the first refusal. any note with an uppercase key, aliases: [Work, Office], a frontmatter comment, a recurring log: map, legacy hidden: [^id], CRLF or BOM can never be hidden or archived, and one such note in a 100-note selection fails all 100. this is byte-safe, but it is user-visible breakage on imported Homes. this needs a decision (per-note results, or skip with a count), not just a test. visibility_unsupported_frontmatter_refuses_the_whole_batch currently locks in the all-or-nothing behaviour.

4. sharing Q10 mismatch: Archived is modelled as a place, but placing is personal
meta.rs adds Place::Archived, and the PR body says "built-in Archived place". sharing Q10 says "pinning and placing a shared note are personal; text, tags and cover are edits to the file". the 11-03 decision says archive is the frontmatter archived: true, which is a file edit. so once sharing lands, a recipient with Edit who archives a note archives it for the owner and every other recipient, and a Read-only recipient can't archive a note shared with them at all. there's no live bug yet because cross-owner access is refused, but the two decisions conflict and kayg has to pick one before sharing ships. hidden has the same question.

5. archived/hidden notes still show up outside the Notes list and Notes FTS
filtering only happens in MetaDb::page (meta.rs:669+) and NotesGeneration::{list,doc}. crates/tasks gets the new fields but never reads them (only the test fixture changes), so tasks inside archived notes still appear in Tasks/Today. backlinks and tag counts still count archived and hidden notes, so a tag reads "3" and lists 2. either filter them or record that this scope is intentional.

6. no UI, no Archived view, and other devices don't react
the client diff is only actions.gen.ts, api.typetest.ts and two optional fields in notes-wire.ts. there's no Archived view and no Hide/Archive entry point. the new visibility payload in notes.saved (home.rs events) has no consumer, so after device A archives a note, device B's open list keeps showing it until a refetch. the earlier checkpoint comment mentioned "client visibility event handling", and 92902f2a removed it. per the 10-26 process rule (no dev landing without UI), this either needs its UI lane or should say it is backend-only.

P3

  1. search freshness stalls Home-wide. notes/src/search.rs:81 sets index_pending whenever any executor write is blocked in the Home, not just one that touches the items being indexed. while writes keep coming (100-note batches, the Tasks B1 adapter), every refresh/reconcile publish fails NotSavedYet and retries. there's no bound and no test for continuous writes. scope it to related()/touched ids.
  2. Archived place is O(N) on large Homes. n.id IN (SELECT ... archived = 1) plus the forced order index (meta.rs:711) walks notes in updated order. with 50 old archived notes in a 100k Home, the first Archived page scans about 100k rows. the bench only covered home.first_page; add place: archived and show_hidden at 100k.
  3. undo refusals come back as internal errors. home.rs:179,182,221 map "changed since", "already undone" and "undo history released" to internal(..), so the client can't tell an ordinary "this note changed since" from a server fault. use BlockChanged/Refused.
  4. upgrade path for notes that were already archived. generation.rs:135 adds searchable with DEFAULT 1, and visibility_upgrade_revalidates_saved_rows_before_returning_a_list (tests/visibility.rs:383) checks only the list. no test shows that a pre-existing archived: true note's FTS rows are dropped after upgrade.
  5. missing failure tests: an edit from a second device (room) that lands between self.get and execute_note_write, an external file write picked up by the watcher in that same window, undo after the note is renamed or trashed, a note with no frontmatter (envelope insert), and a retry without an idempotency key (it gets a new no-op job with undo_available: false, which is fine but untested).

counts: P1 1, P2 5, P3 5.

## independent review at `32b8533d` (data safety + correctness) read-only review of the diff against `41f2efa3`, built nothing. the exact-splice path, room leases, ExpectedRoom guards and whole-batch undo refusal look right to me. the problems below are mostly at the edges: what the action does to values that aren't the flag, and where the read side and the write side disagree. ### P1 **1. hide/unhide/archive/unarchive overwrite or delete values that `flag()` says are not flags** `crates/notes/src/actions/visibility.rs:71-76` always sends `Change { path: [key], value: true | None }`. the strict planner accepts a scalar or block-list leaf as a target (`strict_frontmatter.rs:177-181` only refuses Map/Opaque), so it replaces or removes whatever is there. but `visibility::flag` (`visibility.rs:119-132`) deliberately treats lists and strings as "not hidden" and keeps them as data. - note with legacy block refs in block form: `hidden:\n - "^e7k3f9x2m8q1w4zr"`. lists show it as not hidden, so the UI would offer Hide. Hide rewrites it to `hidden: true` and the block refs are gone. Unhide on the same note deletes the list outright. - `archived: yes` / `archived: old`: `flag()` says not archived, so the note stays in lists and search. `notes.unarchive` deletes the key; `notes.archive` rewrites the user's value to `true`. the only way back is the Undo token, and only until the next edit; after that you need Versions. fix: when planning, refuse unless the current value is absent, `true` (for unhide/unarchive) or `false` (for hide/archive). add tests for the list, `yes`, `"true"` and `false` cases. ### P2 **2. the read grammar and the write grammar differ, so a note can be archived but impossible to unarchive** `flag()` reads through lenient `props::entries` and `serde_yaml` 0.9, which accepts `True`/`TRUE`, CRLF, BOM, comments, flow lists and uppercase keys. the planner refuses all of those (README: `True` refuses as a target; CRLF/BOM/comments/flow lists refuse the whole envelope). scenario: a file synced in over WebDAV or Obsidian with `archived: True`, or `archived: true` with CRLF line endings. `meta.rs:1149` / `generation.rs:150` mark it archived, it leaves search and shows up in Archived, and `notes.unarchive` returns `couldn't plan these property changes (Grammar)`. the note is stuck out of search until someone edits the source by hand. fix: one grammar. either `flag()` recognises only what the planner can round-trip, or add a test proving unarchive works for every value `flag()` accepts. **3. one unsupported note refuses the whole bulk batch, and many ordinary notes are unsupported** `visibility.rs:75-76` returns early on the first refusal. any note with an uppercase key, `aliases: [Work, Office]`, a frontmatter comment, a recurring `log:` map, legacy `hidden: [^id]`, CRLF or BOM can never be hidden or archived, and one such note in a 100-note selection fails all 100. this is byte-safe, but it is user-visible breakage on imported Homes. this needs a decision (per-note results, or skip with a count), not just a test. `visibility_unsupported_frontmatter_refuses_the_whole_batch` currently locks in the all-or-nothing behaviour. **4. sharing Q10 mismatch: Archived is modelled as a place, but placing is personal** `meta.rs` adds `Place::Archived`, and the PR body says "built-in Archived place". sharing Q10 says "pinning and placing a shared note are personal; text, tags and cover are edits to the file". the 11-03 decision says archive is the frontmatter `archived: true`, which is a file edit. so once sharing lands, a recipient with Edit who archives a note archives it for the owner and every other recipient, and a Read-only recipient can't archive a note shared with them at all. there's no live bug yet because cross-owner access is refused, but the two decisions conflict and kayg has to pick one before sharing ships. hidden has the same question. **5. archived/hidden notes still show up outside the Notes list and Notes FTS** filtering only happens in `MetaDb::page` (`meta.rs:669+`) and `NotesGeneration::{list,doc}`. `crates/tasks` gets the new fields but never reads them (only the test fixture changes), so tasks inside archived notes still appear in Tasks/Today. backlinks and tag counts still count archived and hidden notes, so a tag reads "3" and lists 2. either filter them or record that this scope is intentional. **6. no UI, no Archived view, and other devices don't react** the client diff is only `actions.gen.ts`, `api.typetest.ts` and two optional fields in `notes-wire.ts`. there's no Archived view and no Hide/Archive entry point. the new `visibility` payload in `notes.saved` (`home.rs` events) has no consumer, so after device A archives a note, device B's open list keeps showing it until a refetch. the earlier checkpoint comment mentioned "client visibility event handling", and `92902f2a` removed it. per the 10-26 process rule (no dev landing without UI), this either needs its UI lane or should say it is backend-only. ### P3 7. **search freshness stalls Home-wide.** `notes/src/search.rs:81` sets `index_pending` whenever *any* executor write is blocked in the Home, not just one that touches the items being indexed. while writes keep coming (100-note batches, the Tasks B1 adapter), every refresh/reconcile publish fails `NotSavedYet` and retries. there's no bound and no test for continuous writes. scope it to `related()`/touched ids. 8. **Archived place is O(N) on large Homes.** `n.id IN (SELECT ... archived = 1)` plus the forced order index (`meta.rs:711`) walks `notes` in `updated` order. with 50 old archived notes in a 100k Home, the first Archived page scans about 100k rows. the bench only covered `home.first_page`; add `place: archived` and `show_hidden` at 100k. 9. **undo refusals come back as internal errors.** `home.rs:179,182,221` map "changed since", "already undone" and "undo history released" to `internal(..)`, so the client can't tell an ordinary "this note changed since" from a server fault. use `BlockChanged`/`Refused`. 10. **upgrade path for notes that were already archived.** `generation.rs:135` adds `searchable` with `DEFAULT 1`, and `visibility_upgrade_revalidates_saved_rows_before_returning_a_list` (`tests/visibility.rs:383`) checks only the list. no test shows that a pre-existing `archived: true` note's FTS rows are dropped after upgrade. 11. **missing failure tests:** an edit from a second device (room) that lands between `self.get` and `execute_note_write`, an external file write picked up by the watcher in that same window, undo after the note is renamed or trashed, a note with no frontmatter (envelope insert), and a retry without an idempotency key (it gets a new no-op job with `undo_available: false`, which is fine but untested). counts: **P1 1, P2 5, P3 5.**
kayg commented 2026-11-03 06:45:41 +00:00
Author
Owner
Copy link

perf-VM retest is in progress. the earlier dev-01 measurements in #76625 do not qualify as perf-gate evidence.

baseline is exact origin/dev 41f2efa387b74175971a838f602535089d0d5fb4; original PR head 32b8533d5fa49b22fb0ffe4b12f3cff841f19e13 already contains that dev, so merging adds no product changes. test-only head 3d544cb94 adds calternal-perf --archived-percent N --hidden-percent N, reserves live scenario samples, and reports prefix and typo search separately. the mixed fixture will use 10k notes, 20% archived and a disjoint 10% hidden. no measurement artifacts are committed.

builds and targeted harness verification are queued through ordinary ~/bin/slot on dev-01 behind BM14b. binaries will be relayed through the control host with SHA256 checks. server measurements are planned for perf-02, browser boot/first paint for perf-01, each through the installed ~/bin/perf-lock. disk checks currently show 260 GiB free on dev-01, 31 GiB on perf-02, 48 GiB on perf-01.

policy decision: this checkout's standard gate uses fixed p95 budgets in perf/budgets.json, not a global percentage-regression threshold. those budgets and the documented >25% noise rerun rule are retained unchanged. prefix/typo subseries will also be compared with the existing 100 ms search budget. the old perf/run-vm.sh transport/build wrapper names the retired host and builds remotely, so its calternal-perf gate invocation will be run using the current brief's build/relay/lock rules on the named perf VMs. any unavailable storage/cache-drop capabilities will be stated with the final commands rather than silently claimed. never merge.

perf-VM retest is in progress. the earlier dev-01 measurements in #76625 do not qualify as perf-gate evidence. baseline is exact `origin/dev` `41f2efa387b74175971a838f602535089d0d5fb4`; original PR head `32b8533d5fa49b22fb0ffe4b12f3cff841f19e13` already contains that dev, so merging adds no product changes. test-only head `3d544cb94` adds `calternal-perf --archived-percent N --hidden-percent N`, reserves live scenario samples, and reports prefix and typo search separately. the mixed fixture will use 10k notes, 20% archived and a disjoint 10% hidden. no measurement artifacts are committed. builds and targeted harness verification are queued through ordinary `~/bin/slot` on dev-01 behind BM14b. binaries will be relayed through the control host with SHA256 checks. server measurements are planned for perf-02, browser boot/first paint for perf-01, each through the installed `~/bin/perf-lock`. disk checks currently show 260 GiB free on dev-01, 31 GiB on perf-02, 48 GiB on perf-01. policy decision: this checkout's standard gate uses fixed p95 budgets in `perf/budgets.json`, not a global percentage-regression threshold. those budgets and the documented >25% noise rerun rule are retained unchanged. prefix/typo subseries will also be compared with the existing 100 ms search budget. the old `perf/run-vm.sh` transport/build wrapper names the retired host and builds remotely, so its `calternal-perf` gate invocation will be run using the current brief's build/relay/lock rules on the named perf VMs. any unavailable storage/cache-drop capabilities will be stated with the final commands rather than silently claimed. never merge.
kayg commented 2026-11-03 08:57:32 +00:00
Author
Owner
Copy link

review 76637 fixes, gates at 404cbff7 (dev-01, nix, ~/bin/slot)

commit fixes
dd7bd0e8 P1 never destroy a non-flag value; P2-2 one grammar for reader and writer; P2-3 per-note results, re-plan on concurrent edits
e9bd21c7 P3-9 undo refusals → BlockChanged (Forbidden for another author); P3-11 mid-action edit (device + file sync), undo after rename/trash, no-frontmatter tests
a5e31014 P3-7 search pause scoped to the pending notes; P3-8 Archived place driven by visibility_archived (query-plan test); P3-10 upgrade drops pre-archived notes from search
cf64f552, aaa80da6 P2-5 Tasks, backlinks and unlinked mentions leave out hidden/archived notes; P2-6 web store follows notes.saved visibility
64e4141c test fixture fixes; a no-op execution reports changed: false per note
78add232, 1ca15b9a, 404cbff7 rustfmt (the 1ca15b9a one formats the perf job's 3d544cb9 so the fmt gate passes)
command result
cargo fmt --all -- --check pass
cargo clippy --locked -j4 -p calternal-core -p calternal-notes -p calternal-engine -p calternal-search -p calternal-tasks -p calternal --all-targets -- -D warnings pass
cargo test --locked -j4 -p calternal-notes visibility pass, 85
cargo test --locked -j4 -p calternal-notes pass, 567 passed, 6 ignored
cargo test --locked -j4 -p calternal-tasks pass, 68
cargo test --locked -j4 -p calternal-search pass, 30 passed, 2 ignored (incl. note_visibility)
cargo test --locked -j4 -p calternal-core -p calternal-engine -p calternal pass, 102 passed, 5 ignored
cargo test --locked -j4 -p calternal contract_matches_the_server pass
bun install --frozen-lockfile && bun run --cwd apps/web test pass, 100 (incl. 4 new notes-visibility tests)
node ci/check-case-paths.mjs, node ci/check-no-evidence.mjs pass
cd apps/web && bunx svelte-check --tsconfig ./tsconfig.json 2 errors, 15 warnings: same as the exact baseline (e2e review files)

Perf was not run here (separate #887 perf job). Decisions are in the PR body under "review fixes (76637), decisions". P2-4 (Archive for shared notes) and the hidden: key name are untouched, as asked: both are with kayg.

## review 76637 fixes, gates at `404cbff7` (dev-01, nix, `~/bin/slot`) | commit | fixes | |---|---| | `dd7bd0e8` | **P1** never destroy a non-flag value; **P2-2** one grammar for reader and writer; **P2-3** per-note results, re-plan on concurrent edits | | `e9bd21c7` | **P3-9** undo refusals → `BlockChanged` (`Forbidden` for another author); **P3-11** mid-action edit (device + file sync), undo after rename/trash, no-frontmatter tests | | `a5e31014` | **P3-7** search pause scoped to the pending notes; **P3-8** Archived place driven by `visibility_archived` (query-plan test); **P3-10** upgrade drops pre-archived notes from search | | `cf64f552`, `aaa80da6` | **P2-5** Tasks, backlinks and unlinked mentions leave out hidden/archived notes; **P2-6** web store follows `notes.saved` visibility | | `64e4141c` | test fixture fixes; a no-op execution reports `changed: false` per note | | `78add232`, `1ca15b9a`, `404cbff7` | rustfmt (the `1ca15b9a` one formats the perf job's `3d544cb9` so the fmt gate passes) | | command | result | |---|---| | `cargo fmt --all -- --check` | pass | | `cargo clippy --locked -j4 -p calternal-core -p calternal-notes -p calternal-engine -p calternal-search -p calternal-tasks -p calternal --all-targets -- -D warnings` | pass | | `cargo test --locked -j4 -p calternal-notes visibility` | pass, 85 | | `cargo test --locked -j4 -p calternal-notes` | pass, 567 passed, 6 ignored | | `cargo test --locked -j4 -p calternal-tasks` | pass, 68 | | `cargo test --locked -j4 -p calternal-search` | pass, 30 passed, 2 ignored (incl. `note_visibility`) | | `cargo test --locked -j4 -p calternal-core -p calternal-engine -p calternal` | pass, 102 passed, 5 ignored | | `cargo test --locked -j4 -p calternal contract_matches_the_server` | pass | | `bun install --frozen-lockfile && bun run --cwd apps/web test` | pass, 100 (incl. 4 new `notes-visibility` tests) | | `node ci/check-case-paths.mjs`, `node ci/check-no-evidence.mjs` | pass | | `cd apps/web && bunx svelte-check --tsconfig ./tsconfig.json` | 2 errors, 15 warnings: same as the exact baseline (e2e review files) | Perf was not run here (separate #887 perf job). Decisions are in the PR body under "review fixes (76637), decisions". P2-4 (Archive for shared notes) and the `hidden:` key name are untouched, as asked: both are with kayg.
kayg commented 2026-11-03 09:53:16 +00:00
Author
Owner
Copy link

#887 perf gate: dev 41f2efa3 vs head 404cbff7 (dev is already an ancestor of the head, so head = dev + #887)

Verdict: PASS against the gate's fixed p95 budgets (perf/budgets.json, unchanged; the >25 % noise rerun rule only applies to a breach and there was none). One real cost to know about: on a flagged Home the ordinary list p95 rises by about 5 ms (see below). Nothing is near a budget.

Setup. Both VMs (claude@calternal-perf-0{1,2}-backup01, 4c/7.7G) under ~/bin/perf-lock; disk 48 G / 30 G free before. Release binaries built on dev-01 through ~/bin/slot in nix, relayed through the control host, sha256sum -c OK on both VMs:

14b3c4bc…a7ea7f  base/calternal    c5224402…06bf2e  head/calternal
21490018…1b31   calternal-perf (head harness, used for both)
5e3a704a…077e   base/tasks-bench   31a39cff…fefb102 head/tasks-bench

Fixture: 10,000 notes, seed 1, 20 % archived (2,000) + 5 % hidden (500) via calternal-perf --archived-percent 20 --hidden-percent 5 (commit 3d544cb9, reused as is; it is in the PR head). Visible Home on head = 7,500, Archived place = 2,000. Sessions alternate base/head order. Numbers: p95 ms, median of 3 sessions [min–max] (harness runs 5 × samples per session).

Harness, flagged Home (perf-01 | perf-02)

metric base head budget
home.first_page 5.86 [5.15–25.5] | 2.82 [2.63–3.34] 10.73 [8.58–13.7] | 12.18 [7.46–12.8] ≤ 300
home.page 1.21 [1.09–1.75] | 1.41 [1.30–2.90] 6.12 [5.55–6.64] | 6.96 [5.48–7.51] ≤ 50
home.page_title 0.92 | 0.85 7.41 [6.97–7.52] | 6.72 [6.61–6.83] ≤ 50
home.page_tag 7.06 | 6.21 11.66 | 11.01 ≤ 50
home.page_next 1.15 | 1.14 6.05 | 5.68 ≤ 50
search.query 3.99 [3.80–4.11] | 3.86 [3.77–4.17] 3.92 [3.89–4.06] | 3.94 [3.91–4.16] ≤ 100
search.prefix 1.77 | 1.86 1.97 | 1.95 (≤ 100 search)
search.typo 3.02 | 2.99 3.05 [2.88–3.11] | 2.88 [2.81–2.91] (≤ 100 search)
open.note / sync.save_ack 1.33 / 5.16 | 1.43 / 6.30 1.41 / 6.07 | 1.44 / 6.49 ≤ 15 / ≤ 40
startup.healthz 9.43 | 7.20 6.53 | 8.68 ≤ 300

Zero scenario failures in all 12 reports. Noise on first_page/search.first_query is 40 % to 1,300 % (one sample per run); the ordinary pages are 6–47 %.

Attribution of the list delta. Control, same commands without the two flags, 3 sessions per VM: home.page base 1.77 | 2.60 vs head 1.79 | 1.58; home.first_page 3.52 | 7.78 vs 3.43 | 5.54; home.page_tag 6.64 | 7.70 vs 7.50 | 7.52; search.query 3.92 | 4.00 vs 4.11 | 3.87; search.typo 2.99 | 3.04 vs 3.32 | 3.00. So an unflagged Home shows no regression. The +5 ms appears only with flags set (the base lists all 10,000 including archived; head filters through note_visibility). Worst delta: perf-02 home.first_page +9.35 ms (2.82 → 12.18, single sample per run, 90 % noise); for the ordinary page home.page_title +5.9 to +6.5 ms.

Archived view and lists (probe, 5 restarts × 20 calls per session, 3 sessions)

probe base (perf-01 | 02) head (perf-01 | 02)
Archived place, first page HTTP 400 (place missing on dev) 13.24 [10.9–18.3] | 12.20 [9.6–16.2] (first call after start 13.2 | 14.0)
Archived + show_hidden 400 10.67 | 8.61
Tasks list today 0.54 | 0.48 0.52 | 0.54
Tasks list anytime 0.45 | 0.44 0.48 | 0.45
Tasks counts 0.40 | 0.37 0.46 | 0.39

tasks-bench --notes 10000 --runs 3 (2 sessions each) head vs base: tasks.update warm p95 14.9–20.4 vs 21.7–39.4 ms, first-touch 47–105 vs 59–100 ms; its own gates tasks.update and cold.index growth ≤ 10 % fail on both base and head on these VMs (same as before; not a regression).

Append/save

200 sequential acknowledged notes.append on a 1,000-line note, 20 alternating pairs per VM, content checked exactly once per marker (CALTERNAL_NOTES_SAVE_MS=15, release build): 200-append time median base 1.359 s [1.23–2.13] vs head 1.311 s [1.27–2.07] (perf-01), 1.341 vs 1.326 s (perf-02). Per-append p95 median 12.7 vs 12.1 ms and 12.2 vs 11.5 ms. No regression. (sync.save_ack p95 in the table above agrees.)

Gaps, stated plainly

  • Browser first paint is not measured. The perf VMs have no Node/Bun/Chromium and the brief forbids building there. home.first_page above is server time to the first list page, not paint. perf/web.sh needs a browser host.
  • 100k notes were not run (not asked).
  • No page-cache drop (no root); perf/run-vm.sh still names the retired host so I ran calternal-perf directly.
  • Bug on the head: crates/perf/src/bin/tasks-bench.rs:291 does not compile at 404cbff7 (missing hidden and archived in meta::Entry). I added hidden: false, archived: false locally only, reverted, nothing committed. cargo build --workspace / cargo clippy --all-targets would hit it if they build that bin.

Commands

Build (dev-01, each worktree, nix, TMPDIR on disk because /tmp had hit its inode limit):
~/bin/slot nix develop -c bash -c 'cargo build --locked --release -j4 -p calternal --bin calternal -p calternal-perf --bin calternal-perf --bin tasks-bench'
Harness (per session, per version v in base head, order alternating; under ~/bin/perf-lock):
./calternal-perf --bin ./$v/calternal --work $W --sizes 10k --runs 5 --cold-runs 0 --idle-secs 0 --storm-edits 1 --archived-percent 20 --hidden-percent 5 --seed 1 --budgets budgets.json --json … --md … (control: same without the two percent flags).
Views/append: throwaway probe.py (not committed): serve --create --dev-user 0192f0a0-0000-7000-8000-00000000f1f0 on a copy of the seeded fixture (bootstrapped with user create perf-bootstrap + storage re-pair), notes.list {"place":"archived","limit":50}, tasks.list, tasks.counts; append: notes.append ×200 per run, 20 pairs. tasks-bench --data D --state S --notes 10000 --runs 3.

Cleanup: seeded Homes, work dirs, binaries and dev-01 target dirs deleted; only ~1.5 MB of reports remain on each perf VM (~/perf/pr887/out). 3d544cb9 left no lock, units, processes or seeded Homes on the perf VMs. Nothing committed, nothing merged.

## #887 perf gate: dev `41f2efa3` vs head `404cbff7` (dev is already an ancestor of the head, so head = dev + #887) Verdict: **PASS** against the gate's fixed p95 budgets (`perf/budgets.json`, unchanged; the >25 % noise rerun rule only applies to a breach and there was none). One real cost to know about: on a flagged Home the ordinary list p95 rises by about 5 ms (see below). Nothing is near a budget. **Setup.** Both VMs (`claude@calternal-perf-0{1,2}-backup01`, 4c/7.7G) under `~/bin/perf-lock`; disk 48 G / 30 G free before. Release binaries built on dev-01 through `~/bin/slot` in nix, relayed through the control host, `sha256sum -c` OK on both VMs: ``` 14b3c4bc…a7ea7f base/calternal c5224402…06bf2e head/calternal 21490018…1b31 calternal-perf (head harness, used for both) 5e3a704a…077e base/tasks-bench 31a39cff…fefb102 head/tasks-bench ``` Fixture: 10,000 notes, seed 1, **20 % archived (2,000) + 5 % hidden (500)** via `calternal-perf --archived-percent 20 --hidden-percent 5` (commit `3d544cb9`, reused as is; it is in the PR head). Visible Home on head = 7,500, Archived place = 2,000. Sessions alternate base/head order. Numbers: p95 ms, median of 3 sessions [min–max] (harness runs 5 × samples per session). ### Harness, flagged Home (perf-01 | perf-02) | metric | base | head | budget | |---|---|---|---| | home.first_page | 5.86 [5.15–25.5] \| 2.82 [2.63–3.34] | 10.73 [8.58–13.7] \| 12.18 [7.46–12.8] | ≤ 300 | | home.page | 1.21 [1.09–1.75] \| 1.41 [1.30–2.90] | 6.12 [5.55–6.64] \| 6.96 [5.48–7.51] | ≤ 50 | | home.page_title | 0.92 \| 0.85 | 7.41 [6.97–7.52] \| 6.72 [6.61–6.83] | ≤ 50 | | home.page_tag | 7.06 \| 6.21 | 11.66 \| 11.01 | ≤ 50 | | home.page_next | 1.15 \| 1.14 | 6.05 \| 5.68 | ≤ 50 | | search.query | 3.99 [3.80–4.11] \| 3.86 [3.77–4.17] | 3.92 [3.89–4.06] \| 3.94 [3.91–4.16] | ≤ 100 | | search.prefix | 1.77 \| 1.86 | 1.97 \| 1.95 | (≤ 100 search) | | search.typo | 3.02 \| 2.99 | 3.05 [2.88–3.11] \| 2.88 [2.81–2.91] | (≤ 100 search) | | open.note / sync.save_ack | 1.33 / 5.16 \| 1.43 / 6.30 | 1.41 / 6.07 \| 1.44 / 6.49 | ≤ 15 / ≤ 40 | | startup.healthz | 9.43 \| 7.20 | 6.53 \| 8.68 | ≤ 300 | Zero scenario failures in all 12 reports. Noise on first_page/search.first_query is 40 % to 1,300 % (one sample per run); the ordinary pages are 6–47 %. **Attribution of the list delta.** Control, same commands without the two flags, 3 sessions per VM: `home.page` base 1.77 \| 2.60 vs head 1.79 \| 1.58; `home.first_page` 3.52 \| 7.78 vs 3.43 \| 5.54; `home.page_tag` 6.64 \| 7.70 vs 7.50 \| 7.52; `search.query` 3.92 \| 4.00 vs 4.11 \| 3.87; `search.typo` 2.99 \| 3.04 vs 3.32 \| 3.00. So an unflagged Home shows no regression. The +5 ms appears only with flags set (the base lists all 10,000 including archived; head filters through `note_visibility`). Worst delta: **perf-02 `home.first_page` +9.35 ms (2.82 → 12.18, single sample per run, 90 % noise)**; for the ordinary page `home.page_title` +5.9 to +6.5 ms. ### Archived view and lists (probe, 5 restarts × 20 calls per session, 3 sessions) | probe | base (perf-01 \| 02) | head (perf-01 \| 02) | |---|---|---| | Archived place, first page | HTTP 400 (place missing on dev) | 13.24 [10.9–18.3] \| 12.20 [9.6–16.2] (first call after start 13.2 \| 14.0) | | Archived + show_hidden | 400 | 10.67 \| 8.61 | | Tasks list today | 0.54 \| 0.48 | 0.52 \| 0.54 | | Tasks list anytime | 0.45 \| 0.44 | 0.48 \| 0.45 | | Tasks counts | 0.40 \| 0.37 | 0.46 \| 0.39 | `tasks-bench --notes 10000 --runs 3` (2 sessions each) head vs base: `tasks.update` warm p95 14.9–20.4 vs 21.7–39.4 ms, first-touch 47–105 vs 59–100 ms; its own gates `tasks.update` and `cold.index growth ≤ 10 %` fail on **both** base and head on these VMs (same as before; not a regression). ### Append/save 200 sequential acknowledged `notes.append` on a 1,000-line note, 20 alternating pairs per VM, content checked exactly once per marker (`CALTERNAL_NOTES_SAVE_MS=15`, release build): 200-append time median base 1.359 s [1.23–2.13] vs head 1.311 s [1.27–2.07] (perf-01), 1.341 vs 1.326 s (perf-02). Per-append p95 median 12.7 vs 12.1 ms and 12.2 vs 11.5 ms. No regression. (`sync.save_ack` p95 in the table above agrees.) ### Gaps, stated plainly - **Browser first paint is not measured.** The perf VMs have no Node/Bun/Chromium and the brief forbids building there. `home.first_page` above is server time to the first list page, not paint. `perf/web.sh` needs a browser host. - 100k notes were not run (not asked). - No page-cache drop (no root); `perf/run-vm.sh` still names the retired host so I ran `calternal-perf` directly. - **Bug on the head:** `crates/perf/src/bin/tasks-bench.rs:291` does not compile at `404cbff7` (missing `hidden` and `archived` in `meta::Entry`). I added `hidden: false, archived: false` locally only, reverted, nothing committed. `cargo build --workspace` / `cargo clippy --all-targets` would hit it if they build that bin. ### Commands Build (dev-01, each worktree, nix, `TMPDIR` on disk because /tmp had hit its inode limit): `~/bin/slot nix develop -c bash -c 'cargo build --locked --release -j4 -p calternal --bin calternal -p calternal-perf --bin calternal-perf --bin tasks-bench'` Harness (per session, per version `v` in `base head`, order alternating; under `~/bin/perf-lock`): `./calternal-perf --bin ./$v/calternal --work $W --sizes 10k --runs 5 --cold-runs 0 --idle-secs 0 --storm-edits 1 --archived-percent 20 --hidden-percent 5 --seed 1 --budgets budgets.json --json … --md …` (control: same without the two percent flags). Views/append: throwaway `probe.py` (not committed): `serve --create --dev-user 0192f0a0-0000-7000-8000-00000000f1f0` on a copy of the seeded fixture (bootstrapped with `user create perf-bootstrap` + `storage re-pair`), `notes.list {"place":"archived","limit":50}`, `tasks.list`, `tasks.counts`; append: `notes.append` ×200 per run, 20 pairs. `tasks-bench --data D --state S --notes 10000 --runs 3`. Cleanup: seeded Homes, work dirs, binaries and dev-01 target dirs deleted; only ~1.5 MB of reports remain on each perf VM (`~/perf/pr887/out`). `3d544cb9` left no lock, units, processes or seeded Homes on the perf VMs. Nothing committed, nothing merged.
kayg commented 2026-11-03 11:26:42 +00:00
Author
Owner
Copy link

#887 list-path regression: cause, fix and re-measure (head b1826c33)

Cause. The list path reads no files: MetaDb::page is never handed a path, and flags are already stored in note_visibility at index time. The cost was SQL. Each page ran n.id NOT IN (SELECT id FROM note_visibility WHERE archived = 1 OR hidden = 1), which builds a b-tree of every flagged id (about 2,500 on this Home) on every call, and then ran a multi-index-OR count over the same set for the total. Offline on the seeded 10k Home (same SQLite file, p50): that page query 2.78 ms, the count 0.99 ms, the unflagged page 0.17 ms.

Fix (8a31566d, new file crates/notes/src/meta/visible.rs; strict reader and write side untouched, so nothing can be archived but impossible to unarchive):

  • The Notes place anti-joins the sparse note_visibility by primary key for the rows a page returns: LEFT JOIN note_visibility v ON v.id = n.id … WHERE v.id IS NULL. No id set, no file read. With show_hidden the filter is COALESCE(v.archived,0)=0.
  • The Archived place needs no subquery: v.archived = 1 AND v.hidden = 0.
  • Totals come from counts: stats minus COUNT(*) of the flagged rows (a covering-index scan of the sparse table), or the tag's count minus its flagged notes. When task notes exist the count uses inclusion and exclusion over (tasks) and the flagged rows, so a note in both is counted once. Everything else falls back to the filtered count, as before.
  • Tests: tests/visibility_page.rs checks the totals for each flag, place, order and tag combination, and runs EXPLAIN QUERY PLAN on the SQL that pages actually ran (captured under cfg(test)): the list is SEARCH v USING PRIMARY KEY, with no SCAN v, LIST SUBQUERY, MATERIALIZE or TEMP B-TREE, and totals never scan notes. A tasks-crate test checks that Notes and Tasks pages total the rows they list when task notes and flagged notes overlap.
  • tasks-bench.rs:291 now sets hidden and archived (3a43357e). cargo build --all-targets --workspace builds.

Setup. claude@calternal-perf-0{1,2}-backup01, under ~/bin/perf-lock. Both binaries were built on dev-01 through ~/bin/slot (nix, release) and relayed through the control host; sha256sum matches on all three hosts. before = 404cbff7 (calternal 83bc03a4…), after = 1814475c (calternal 012f5eee…; b1826c33 differs by one test-only line). The harness binary is identical for both (calternal-perf b8519f97…). Same command as comment 76728: ./calternal-perf --bin ./$v/calternal --work $W --sizes 10k --runs 5 --cold-runs 0 --idle-secs 0 --storm-edits 1 [--archived-percent 20 --hidden-percent 5] --seed 1 --budgets budgets.json …. Flagged Home: 3 sessions per VM; unflagged control: 2 sessions per VM; order alternates by session. Numbers are p95 in ms, median of the sessions [min–max]. 20 reports, 0 scenario failures.

scenario failures in all reports: 0 reports: 20

flag

metric before 01 after 01 before 02 after 02
home.first_page 11.90 [11.48–13.15] 6.41 [3.83–6.45] 10.71 [8.93–12.26] 4.06 [3.28–5.11]
home.page 6.77 [6.65–6.90] 1.87 [1.40–4.32] 6.19 [6.14–8.22] 2.26 [1.68–2.61]
home.page_title 7.12 [7.02–7.82] 2.41 [2.28–2.45] 6.96 [6.34–7.00] 2.24 [1.93–2.83]
home.page_tag 11.62 [11.51–11.69] 8.27 [8.01–8.78] 11.27 [11.10–11.33] 8.83 [7.93–9.24]
home.page_next 6.59 [6.22–6.96] 2.00 [1.82–2.10] 5.69 [5.63–5.78] 1.83 [1.71–1.87]
search.query 4.32 [4.06–5.12] 4.29 [4.18–4.35] 3.87 [3.82–4.23] 4.13 [3.99–4.71]
search.typo 3.41 [2.88–3.49] 3.52 [3.30–3.56] 2.95 [2.86–3.48] 3.20 [2.88–3.58]
open.note 1.88 [1.48–1.91] 1.53 [1.49–1.82] 1.40 [1.40–1.52] 1.40 [1.33–1.84]
sync.save_ack 5.54 [4.09–7.85] 6.39 [4.41–7.14] 6.55 [5.86–7.61] 7.68 [4.66–7.93]

ctrl

metric before 01 after 01 before 02 after 02
home.first_page 5.42 [4.35–6.50] 4.49 [2.90–6.09] 7.28 [5.55–9.01] 4.11 [3.79–4.43]
home.page 2.04 [1.88–2.21] 2.07 [2.05–2.08] 1.79 [1.76–1.83] 1.77 [1.70–1.84]
home.page_title 1.23 [1.21–1.25] 1.10 [1.07–1.13] 1.20 [1.11–1.29] 1.04 [1.02–1.06]
home.page_tag 7.56 [7.48–7.65] 7.68 [7.25–8.11] 7.70 [7.50–7.90] 7.86 [7.31–8.42]
home.page_next 1.57 [1.55–1.59] 1.39 [1.31–1.47] 1.57 [1.49–1.65] 1.44 [1.30–1.58]
search.query 4.29 [4.29–4.30] 3.90 [3.89–3.92] 4.25 [4.16–4.33] 4.11 [4.08–4.14]
search.typo 3.24 [3.20–3.28] 2.99 [2.92–3.06] 3.02 [2.96–3.07] 3.07 [2.98–3.15]
open.note 1.72 [1.57–1.88] 1.63 [1.63–1.63] 1.61 [1.54–1.68] 1.57 [1.53–1.61]
sync.save_ack 4.94 [4.04–5.84] 5.10 [4.99–5.20] 5.52 [4.53–6.50] 5.61 [5.34–5.88]

Reading it.

  • The ordinary page (home.page) on the flagged Home: 6.77 → 1.87 (perf-01) and 6.19 → 2.26 (perf-02). The control is 2.07 / 1.77 (after) and 2.04 / 1.79 (before). So after-flagged minus control is −0.20 ms on perf-01 and +0.49 ms on perf-02, at the ≤ +0.5 ms target, and the head's 5× is gone. home.first_page falls from about 11 to 4–6 ms. Search, open and save are unchanged. The unflagged control did not move (before vs after within noise), so the unflagged path pays nothing.
  • Not fully at the target: home.page_title is still about +1.2 ms over the control (2.41 / 2.24 vs 1.10 / 1.04). This fixture flags the first 25 % of notes in path order, which is also title order, so the first title page skips about 2,500 flagged rows to find 51. The same query shape costs 0.12 ms plain and 0.94 ms flagged in SQLite. home.page_tag is +0.6 to +1.0 ms (8.27 / 8.83 vs 7.68 / 7.86): the tag total subtracts the tag's flagged notes (about 1.3 ms offline), and its page walk is unchanged. home.page_next is +0.4 to +0.6 ms. All are far below the 50 ms budgets.
  • Making every order O(page) whatever the flag distribution would need a vis column on notes with partial indexes WHERE vis = 0. That is a schema bump and touches the INSERT INTO notes sites and tests, so I did not include it; say if you want it.

Gates (dev-01, nix, ~/bin/slot).

  • cargo fmt --all -- --check: OK at b1826c33.
  • cargo clippy --all-targets --workspace --locked -- -D warnings: OK at b1826c33.
  • cargo build --all-targets --workspace --locked: OK at 1814475c.
  • cargo test -p calternal-notes -p calternal-tasks -p calternal-search -p calternal-core -p calternal-engine -p calternal: notes lib 567 passed, 2 failed (72578df8). One was my new plan test (unbound ?, fixed in 383fd20e, now passes). The other is tests::anchors::an_edited_block_is_refused_never_the_wrong_one: a title: Day line appeared after the refused move, which looks like a title-sync race in the full run under load. It passed 6 of 6 alone at 383fd20e and touches no list code, but I have not run it on dev, so a train should recheck it. Cargo stops after the first failing package, so I reran the rest with --no-fail-fast (tasks, search, core, engine, calternal, notes doctests): all passed. The tasks overlap test and hidden_and_archived_notes_leave_tasks_views pass at 1814475c.
  • Web: bunx vitest run 100 passed. bun run check reports 2 errors, both in e2e/review4-signout.spec.ts:225-226, a file this PR doesn't touch.

Cleanup: perf Homes, work dirs, binaries on both perf VMs, and the seeded fixtures, explain copies and release binaries on dev-01 are removed.

## #887 list-path regression: cause, fix and re-measure (head `b1826c33`) **Cause.** The list path reads no files: `MetaDb::page` is never handed a path, and flags are already stored in `note_visibility` at index time. The cost was SQL. Each page ran `n.id NOT IN (SELECT id FROM note_visibility WHERE archived = 1 OR hidden = 1)`, which builds a b-tree of every flagged id (about 2,500 on this Home) on every call, and then ran a multi-index-OR count over the same set for the total. Offline on the seeded 10k Home (same SQLite file, p50): that page query 2.78 ms, the count 0.99 ms, the unflagged page 0.17 ms. **Fix** (`8a31566d`, new file `crates/notes/src/meta/visible.rs`; strict reader and write side untouched, so nothing can be archived but impossible to unarchive): - The Notes place anti-joins the sparse `note_visibility` by primary key for the rows a page returns: `LEFT JOIN note_visibility v ON v.id = n.id … WHERE v.id IS NULL`. No id set, no file read. With `show_hidden` the filter is `COALESCE(v.archived,0)=0`. - The Archived place needs no subquery: `v.archived = 1 AND v.hidden = 0`. - Totals come from counts: stats minus `COUNT(*)` of the flagged rows (a covering-index scan of the sparse table), or the tag's count minus its flagged notes. When task notes exist the count uses inclusion and exclusion over `(tasks)` and the flagged rows, so a note in both is counted once. Everything else falls back to the filtered count, as before. - Tests: `tests/visibility_page.rs` checks the totals for each flag, place, order and tag combination, and runs `EXPLAIN QUERY PLAN` on the SQL that pages actually ran (captured under `cfg(test)`): the list is `SEARCH v USING PRIMARY KEY`, with no `SCAN v`, `LIST SUBQUERY`, `MATERIALIZE` or `TEMP B-TREE`, and totals never scan `notes`. A tasks-crate test checks that Notes and Tasks pages total the rows they list when task notes and flagged notes overlap. - `tasks-bench.rs:291` now sets `hidden` and `archived` (`3a43357e`). `cargo build --all-targets --workspace` builds. **Setup.** `claude@calternal-perf-0{1,2}-backup01`, under `~/bin/perf-lock`. Both binaries were built on dev-01 through `~/bin/slot` (nix, release) and relayed through the control host; `sha256sum` matches on all three hosts. before = `404cbff7` (`calternal` 83bc03a4…), after = `1814475c` (`calternal` 012f5eee…; `b1826c33` differs by one test-only line). The harness binary is identical for both (`calternal-perf` b8519f97…). Same command as comment 76728: `./calternal-perf --bin ./$v/calternal --work $W --sizes 10k --runs 5 --cold-runs 0 --idle-secs 0 --storm-edits 1 [--archived-percent 20 --hidden-percent 5] --seed 1 --budgets budgets.json …`. Flagged Home: 3 sessions per VM; unflagged control: 2 sessions per VM; order alternates by session. Numbers are p95 in ms, median of the sessions [min–max]. 20 reports, 0 scenario failures. scenario failures in all reports: 0 reports: 20 ## flag | metric | before 01 | after 01 | before 02 | after 02 | |---|---|---|---|---| | home.first_page | 11.90 [11.48–13.15] | 6.41 [3.83–6.45] | 10.71 [8.93–12.26] | 4.06 [3.28–5.11] | | home.page | 6.77 [6.65–6.90] | 1.87 [1.40–4.32] | 6.19 [6.14–8.22] | 2.26 [1.68–2.61] | | home.page_title | 7.12 [7.02–7.82] | 2.41 [2.28–2.45] | 6.96 [6.34–7.00] | 2.24 [1.93–2.83] | | home.page_tag | 11.62 [11.51–11.69] | 8.27 [8.01–8.78] | 11.27 [11.10–11.33] | 8.83 [7.93–9.24] | | home.page_next | 6.59 [6.22–6.96] | 2.00 [1.82–2.10] | 5.69 [5.63–5.78] | 1.83 [1.71–1.87] | | search.query | 4.32 [4.06–5.12] | 4.29 [4.18–4.35] | 3.87 [3.82–4.23] | 4.13 [3.99–4.71] | | search.typo | 3.41 [2.88–3.49] | 3.52 [3.30–3.56] | 2.95 [2.86–3.48] | 3.20 [2.88–3.58] | | open.note | 1.88 [1.48–1.91] | 1.53 [1.49–1.82] | 1.40 [1.40–1.52] | 1.40 [1.33–1.84] | | sync.save_ack | 5.54 [4.09–7.85] | 6.39 [4.41–7.14] | 6.55 [5.86–7.61] | 7.68 [4.66–7.93] | ## ctrl | metric | before 01 | after 01 | before 02 | after 02 | |---|---|---|---|---| | home.first_page | 5.42 [4.35–6.50] | 4.49 [2.90–6.09] | 7.28 [5.55–9.01] | 4.11 [3.79–4.43] | | home.page | 2.04 [1.88–2.21] | 2.07 [2.05–2.08] | 1.79 [1.76–1.83] | 1.77 [1.70–1.84] | | home.page_title | 1.23 [1.21–1.25] | 1.10 [1.07–1.13] | 1.20 [1.11–1.29] | 1.04 [1.02–1.06] | | home.page_tag | 7.56 [7.48–7.65] | 7.68 [7.25–8.11] | 7.70 [7.50–7.90] | 7.86 [7.31–8.42] | | home.page_next | 1.57 [1.55–1.59] | 1.39 [1.31–1.47] | 1.57 [1.49–1.65] | 1.44 [1.30–1.58] | | search.query | 4.29 [4.29–4.30] | 3.90 [3.89–3.92] | 4.25 [4.16–4.33] | 4.11 [4.08–4.14] | | search.typo | 3.24 [3.20–3.28] | 2.99 [2.92–3.06] | 3.02 [2.96–3.07] | 3.07 [2.98–3.15] | | open.note | 1.72 [1.57–1.88] | 1.63 [1.63–1.63] | 1.61 [1.54–1.68] | 1.57 [1.53–1.61] | | sync.save_ack | 4.94 [4.04–5.84] | 5.10 [4.99–5.20] | 5.52 [4.53–6.50] | 5.61 [5.34–5.88] | **Reading it.** - The ordinary page (`home.page`) on the flagged Home: 6.77 → 1.87 (perf-01) and 6.19 → 2.26 (perf-02). The control is 2.07 / 1.77 (after) and 2.04 / 1.79 (before). So after-flagged minus control is −0.20 ms on perf-01 and +0.49 ms on perf-02, at the ≤ +0.5 ms target, and the head's 5× is gone. `home.first_page` falls from about 11 to 4–6 ms. Search, open and save are unchanged. The unflagged control did not move (before vs after within noise), so the unflagged path pays nothing. - **Not fully at the target:** `home.page_title` is still about +1.2 ms over the control (2.41 / 2.24 vs 1.10 / 1.04). This fixture flags the first 25 % of notes in path order, which is also title order, so the first title page skips about 2,500 flagged rows to find 51. The same query shape costs 0.12 ms plain and 0.94 ms flagged in SQLite. `home.page_tag` is +0.6 to +1.0 ms (8.27 / 8.83 vs 7.68 / 7.86): the tag total subtracts the tag's flagged notes (about 1.3 ms offline), and its page walk is unchanged. `home.page_next` is +0.4 to +0.6 ms. All are far below the 50 ms budgets. - Making every order O(page) whatever the flag distribution would need a `vis` column on `notes` with partial indexes `WHERE vis = 0`. That is a schema bump and touches the `INSERT INTO notes` sites and tests, so I did not include it; say if you want it. **Gates (dev-01, nix, `~/bin/slot`).** - `cargo fmt --all -- --check`: OK at `b1826c33`. - `cargo clippy --all-targets --workspace --locked -- -D warnings`: OK at `b1826c33`. - `cargo build --all-targets --workspace --locked`: OK at `1814475c`. - `cargo test -p calternal-notes -p calternal-tasks -p calternal-search -p calternal-core -p calternal-engine -p calternal`: notes lib 567 passed, 2 failed (`72578df8`). One was my new plan test (unbound `?`, fixed in `383fd20e`, now passes). The other is `tests::anchors::an_edited_block_is_refused_never_the_wrong_one`: a `title: Day` line appeared after the refused move, which looks like a title-sync race in the full run under load. It passed 6 of 6 alone at `383fd20e` and touches no list code, but I have not run it on `dev`, so a train should recheck it. Cargo stops after the first failing package, so I reran the rest with `--no-fail-fast` (tasks, search, core, engine, calternal, notes doctests): all passed. The tasks overlap test and `hidden_and_archived_notes_leave_tasks_views` pass at `1814475c`. - Web: `bunx vitest run` 100 passed. `bun run check` reports 2 errors, both in `e2e/review4-signout.spec.ts:225-226`, a file this PR doesn't touch. Cleanup: perf Homes, work dirs, binaries on both perf VMs, and the seeded fixtures, explain copies and release binaries on dev-01 are removed.
kayg commented 2026-11-03 11:42:43 +00:00
Author
Owner
Copy link

H1 gates (head 396f878e, dev-01, nix, ~/bin/slot)

gate command result
fmt cargo fmt --all --check pass
clippy cargo clippy -j4 --all-targets -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine -- -D warnings pass
tests cargo test -j4 -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine pass (notes 579 passed, 0 failed; search, core, engine all ok)
web bun test in packages/text 3 pass (no web code changed)

New tests (byte-exact, with undo round trip): block_visibility::tests (old list, new list, both, flag plus list, move on edit, non-list values refused), visibility::tests::an_old_hidden_list_moves_when_the_note_flag_is_set, tests::visibility::an_old_hidden_block_list_moves_when_the_note_is_hidden (action level), search::tests::search_skips_hidden_blocks_in_both_list_forms.

Commits: f9748d7f feature, e95401a6 tests/fmt, 396f878e ADR 0056.

## H1 gates (head `396f878e`, dev-01, nix, `~/bin/slot`) | gate | command | result | |---|---|---| | fmt | `cargo fmt --all --check` | pass | | clippy | `cargo clippy -j4 --all-targets -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine -- -D warnings` | pass | | tests | `cargo test -j4 -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine` | pass (notes 579 passed, 0 failed; search, core, engine all ok) | | web | `bun test` in `packages/text` | 3 pass (no web code changed) | New tests (byte-exact, with undo round trip): `block_visibility::tests` (old list, new list, both, flag plus list, move on edit, non-list values refused), `visibility::tests::an_old_hidden_list_moves_when_the_note_flag_is_set`, `tests::visibility::an_old_hidden_block_list_moves_when_the_note_is_hidden` (action level), `search::tests::search_skips_hidden_blocks_in_both_list_forms`. Commits: `f9748d7f` feature, `e95401a6` tests/fmt, `396f878e` ADR 0056.
kayg commented 2026-11-03 11:51:02 +00:00
Author
Owner
Copy link

H1 follow-up: bare keys (head 6b8d385a, dev-01, nix, ~/bin/slot)

gate command result
fmt cargo fmt --all --check pass
clippy cargo clippy -j4 --all-targets -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine -p calternal-tasks -- -D warnings pass
tests cargo test -j4 -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine pass (notes 582 passed, 0 failed)
tasks (shares the writer) cargo test -j4 -p calternal-tasks 69 passed

Commits: 9f352dfc writer prints plain-word keys bare, 44f15ee9 + ff172a35 downstream expectations/fmt, 14507725 quoted-key block tests, 6b8d385a tasks expectations.

New byte-exact tests: strict_frontmatter::tests::spaced_and_plain_keys_are_written_bare, a_quoted_spaced_key_is_read_kept_and_normalised_when_rewritten, block_visibility::tests::a_quoted_key_is_read_kept_and_normalised_on_rewrite.

Note: the list items stay quoted ( - "^aa1"), because the strict grammar only reads ^-led text as a quoted string. The item style is the same one the writer uses for tags: today.

## H1 follow-up: bare keys (head `6b8d385a`, dev-01, nix, `~/bin/slot`) | gate | command | result | |---|---|---| | fmt | `cargo fmt --all --check` | pass | | clippy | `cargo clippy -j4 --all-targets -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine -p calternal-tasks -- -D warnings` | pass | | tests | `cargo test -j4 -p calternal-notes -p calternal-search -p calternal-core -p calternal-engine` | pass (notes 582 passed, 0 failed) | | tasks (shares the writer) | `cargo test -j4 -p calternal-tasks` | 69 passed | Commits: `9f352dfc` writer prints plain-word keys bare, `44f15ee9` + `ff172a35` downstream expectations/fmt, `14507725` quoted-key block tests, `6b8d385a` tasks expectations. New byte-exact tests: `strict_frontmatter::tests::spaced_and_plain_keys_are_written_bare`, `a_quoted_spaced_key_is_read_kept_and_normalised_when_rewritten`, `block_visibility::tests::a_quoted_key_is_read_kept_and_normalised_on_rewrite`. Note: the list items stay quoted (` - "^aa1"`), because the strict grammar only reads `^`-led text as a quoted string. The item style is the same one the writer uses for `tags:` today.
kayg commented 2026-11-03 11:58:56 +00:00
Author
Owner
Copy link

independent re-review at 6b8d385a (fixes for 76637, list perf, H1, bare-key writer)

i only read the code and built nothing. i diffed against origin/dev 41f2efa3, which is an ancestor of the head.

1. the 76637 findings

# status evidence
P1-1 overwrite of non-flag values fixed visibility.rs:262-287 decide only changes a key that is absent or a plain true/false. Lists, yes, True, strings and maps are skipped with a reason. Covered by visibility/tests.rs:12 (table incl. True, yes, []) and tests/visibility.rs:430 (byte-for-byte).
P2-2 one grammar fixed strict_frontmatter.rs:177 current uses the planner's own parse. visibility.rs:353-401: read/flag take the plain bit (no BOM/CRLF), and every reader passes it (meta.rs:1172, search.rs:152, room/mod.rs:556, meta.rs:1825). home.rs:624 reads the raw post bytes, which gives the same answer.
P2-3 per-note results fixed visibility.rs:148-201 gives a per-note done/skipped. dependencies: Vec::new() leaves notes that are already in place unguarded, and a refusal re-plans up to 3 times (:117).
P2-4 Archive on shared notes held with kayg (H2) not reviewed
P2-5 tasks/backlinks/tags fixed tasks/index.rs:260, backlinks.rs (NOT IN note_visibility + veiled), ops.rs unlinked mentions. Test: tasks/tests.rs:1241, tests/visibility.rs:817.
P2-6 other devices fixed (store only, UI in #877) notes.svelte.ts:244-259, notes-visibility.ts. See new P2-2 below.
P3-7 Home-wide search stall fixed indexer.rs refresh checks each reader's own items with validate_index(Some(..)). A reconcile still checks the whole provider (None), which is correct. Test tests/visibility.rs:708.
P3-8 Archived place O(N) fixed meta.rs ARCHIVED_FROM drives from visibility_archived. Test tests/visibility.rs:729.
P3-9 undo errors fixed home.rs:180,193,238 → BlockChanged, :188 → Forbidden.
P3-10 upgrade drops archived from search fixed generation.rs searchable column + list() filter, restart.rs SCHEMA_REFRESH revalidation. Test tests/visibility.rs:772.
P3-11 missing failure tests fixed tests/visibility.rs:546 (rename/trash), :635 (mid-action device and file edits), visibility/tests.rs:100 (no frontmatter).

2. the bare-key writer: no findings

  • No key changes quoting unless its own value is rewritten. plan splices only the target entry's span (strict_frontmatter.rs:96). audit::verify independently checks that every other line is byte-identical. The inverse is the recorded original bytes, so undo restores "status": exactly. Nested changes never re-render the parent key line. a_quoted_spaced_key_is_read_kept_and_normalised_when_rewritten covers this.
  • No bare key can be mis-read. plain_key requires word_key: ASCII lowercase words separated by single spaces, each word starting with a letter, and only [a-z0-9-] inside. So : , #, unicode, uppercase, leading or trailing spaces, double spaces, -/?/. starts, numbers, timestamps and .inf/.nan can't be printed bare. In fact none of them are valid keys at all, so the planner refuses the change. The YAML 1.1 value words (true false null yes no on off y n) stay quoted, and they're lowercase-only because uppercase keys are invalid anyway. The yaml_rust2 oracle re-decodes every output. Obsidian and js-yaml read hidden blocks: / last edited: the same way they already read last edited:. Small wording nit: letter-led 16-char ids (e7k3f9x2m8q1w4zr) print bare. Only digit-led ids stay quoted. That's harmless.
  • Callers. strict_frontmatter::plan is used only by notes visibility.rs, block_visibility.rs, write_executor/plan.rs:192 (a no-change validation) and tasks/status.rs. Contacts, money and calendar don't use it. The one test that looks at key bytes outside these (tasks/status_action.rs:335) already accepts both forms.

3. hidden-block migration: byte-exact and nothing is lost, with gaps below

The flag, old, new and both cases check out by reading and by block_visibility/tests.rs. ["hidden"] and ["hidden blocks"] don't trip the overlap check, because path elements compare whole. Unhide on an old list is a no-op. When Hide moves the list, it keeps the union. Undo restores the old hidden: list exactly. Any list it can't read leaves the note skipped, not rewritten.

findings

P2-1. existing notes never drop their hidden blocks from search.
search.rs:210 is the first time search filters hidden blocks (dev had no such filter). But the search stamp is the source hash (generation.rs:32, accepted-v1:<blake3 of text/path/doc>), and the indexer only re-reads items whose stamp differs (indexer.rs:238). The schema bump to 6 rebuilds source_versions with the same hashes.
Scenario: a note indexed on dev with hidden: ["^k2m8q1w4zre7k3f9"] (the ADR's own example) or hidden blocks: is upgraded. Searching for the hidden block's words still finds it until someone edits that note. search_skips_hidden_blocks_in_both_list_forms calls describe directly, so it can't see this.
Fix: bump the stamp to accepted-v2 (one line; costs one full re-index) and add an upgrade test like :772.

P2-2. bulk Unarchive/Unhide (or Undo of a bulk Archive/Hide) starts a burst of full list reads on every device.
notes.svelte.ts:254-258: every notes.saved whose visibility is unflagged, for a note that isn't loaded, sets two timers (400 ms and 2.5 s). Each timer calls loadNotes(), and that reads every page of the whole list (:141-171). Undo keeps ExactUndo (home.rs:197), so its events carry visibility too.
Scenario: unarchive 100 notes, or undo archiving 100. That's 100 events, so about 100 loadNotes() start together at 400 ms, before any of them has put the notes back in index. On a 10k Home that's about 2,000 notes.list calls on each open device. Unhiding a task note, which is never in the Notes list, always fires both reloads for nothing.
Fix: coalesce the "return" case into one pending reload, e.g. a timer plus a set of ids like refreshTimer, or an in-flight guard on loadNotes. Add a unit test that 100 return events give one read.

P3-1. an old hidden: list with bare items isn't moved, and a later block unhide will say "done" without unhiding.
The strict grammar parses hidden:\n - ^aa1 as Opaque, because ^ only reads as a quoted string. That bare form is the one people type and the one Obsidian writes. old_list (block_visibility.rs:133-139) treats Other as "no old list". So changes(raw, ["aa1"], false) returns an empty change set, but ids() still hides ^aa1 in search.
The ADR's own example form, hidden: ["^…"], is a flow list, which refuses the whole envelope. Such a note can't be hidden, archived or status-changed until it's edited in source. That's safe, but it doesn't match H1's "moves on the next edit".
No backend caller exists yet, so this is P3. Fix before the block action ships:

  • refuse with a reason when either key holds an opaque list;
  • change the ADR 0056 examples to the block-list form the writer actually prints.

P3-2. hidden blocks show up outside search.

  • meta.rs:1192 snippet doesn't skip hidden lines. If the first body line is a hidden block, its words become the list row preview.
  • line_is_hidden works per line (search.rs:210-216), so a hidden multi-line paragraph or a list item's children stay searchable. Only the line with the ^id marker is dropped.

P3-3. Hide on hidden: [] plus hidden blocks: [] also deletes hidden blocks: [].
block_visibility.rs:108-110. This loses no data, but it's one line more than the flag change. Skip the removal when the move added nothing.

P3-4. totals: correct by reading, thin by test.
I walked every visible::kept arm against its page filter:

  • plain: N − |F|
  • tag: n_tag − |F ∩ tag|
  • tasks: N − |T| − |F∖T|, and Tasks |T∖F|
  • show_hidden: F = archived only
  • Archived place: direct count over ARCHIVED_FROM
  • tag + tasks and every other filter: fall back to count_sql, which includes the join

All of them match, and only_tag correctly stops applying once the visibility filter is pushed. visibility_page.rs:52 doesn't cover Archived + tag, a common tag (≥ RARE_TAG, the EXISTS + INDEXED BY + join form), a kind filter or a cursor page. Add those rows.

counts: P1 0, P2 2, P3 4.

verdict

Not ready to train yet. Every 76637 finding is fixed, the writer change is safe, and the perf totals are right. Both P2s are small, contained fixes: a stamp bump with an upgrade test, and one coalesced reload. P2-2 only shows up once #877's bulk controls exist, which is exactly the pair being trained. Once those two land with their tests, it's ready to train with #877 without another full review round. The P3s can follow, and P3-1 must be fixed before the block hide action ships.

## independent re-review at `6b8d385a` (fixes for 76637, list perf, H1, bare-key writer) i only read the code and built nothing. i diffed against `origin/dev` `41f2efa3`, which is an ancestor of the head. ### 1. the 76637 findings | # | status | evidence | |---|---|---| | P1-1 overwrite of non-flag values | **fixed** | `visibility.rs:262-287` `decide` only changes a key that is absent or a plain `true`/`false`. Lists, `yes`, `True`, strings and maps are skipped with a reason. Covered by `visibility/tests.rs:12` (table incl. `True`, `yes`, `[]`) and `tests/visibility.rs:430` (byte-for-byte). | | P2-2 one grammar | **fixed** | `strict_frontmatter.rs:177` `current` uses the planner's own parse. `visibility.rs:353-401`: `read`/`flag` take the `plain` bit (no BOM/CRLF), and every reader passes it (`meta.rs:1172`, `search.rs:152`, `room/mod.rs:556`, `meta.rs:1825`). `home.rs:624` reads the raw post bytes, which gives the same answer. | | P2-3 per-note results | **fixed** | `visibility.rs:148-201` gives a per-note `done`/`skipped`. `dependencies: Vec::new()` leaves notes that are already in place unguarded, and a refusal re-plans up to 3 times (`:117`). | | P2-4 Archive on shared notes | held with kayg (H2) | not reviewed | | P2-5 tasks/backlinks/tags | **fixed** | `tasks/index.rs:260`, `backlinks.rs` (`NOT IN note_visibility` + `veiled`), `ops.rs` unlinked mentions. Test: `tasks/tests.rs:1241`, `tests/visibility.rs:817`. | | P2-6 other devices | **fixed** (store only, UI in #877) | `notes.svelte.ts:244-259`, `notes-visibility.ts`. See new P2-2 below. | | P3-7 Home-wide search stall | **fixed** | `indexer.rs` `refresh` checks each reader's own items with `validate_index(Some(..))`. A reconcile still checks the whole provider (`None`), which is correct. Test `tests/visibility.rs:708`. | | P3-8 Archived place O(N) | **fixed** | `meta.rs` `ARCHIVED_FROM` drives from `visibility_archived`. Test `tests/visibility.rs:729`. | | P3-9 undo errors | **fixed** | `home.rs:180,193,238` → `BlockChanged`, `:188` → `Forbidden`. | | P3-10 upgrade drops archived from search | **fixed** | `generation.rs` `searchable` column + `list()` filter, `restart.rs` `SCHEMA_REFRESH` revalidation. Test `tests/visibility.rs:772`. | | P3-11 missing failure tests | **fixed** | `tests/visibility.rs:546` (rename/trash), `:635` (mid-action device and file edits), `visibility/tests.rs:100` (no frontmatter). | ### 2. the bare-key writer: no findings - **No key changes quoting unless its own value is rewritten.** `plan` splices only the target entry's span (`strict_frontmatter.rs:96`). `audit::verify` independently checks that every other line is byte-identical. The inverse is the recorded original bytes, so undo restores `"status":` exactly. Nested changes never re-render the parent key line. `a_quoted_spaced_key_is_read_kept_and_normalised_when_rewritten` covers this. - **No bare key can be mis-read.** `plain_key` requires `word_key`: ASCII lowercase words separated by single spaces, each word starting with a letter, and only `[a-z0-9-]` inside. So `: `, `#`, unicode, uppercase, leading or trailing spaces, double spaces, `-`/`?`/`.` starts, numbers, timestamps and `.inf`/`.nan` can't be printed bare. In fact none of them are valid keys at all, so the planner refuses the change. The YAML 1.1 value words (`true false null yes no on off y n`) stay quoted, and they're lowercase-only because uppercase keys are invalid anyway. The yaml_rust2 oracle re-decodes every output. Obsidian and js-yaml read `hidden blocks:` / `last edited:` the same way they already read `last edited:`. Small wording nit: letter-led 16-char ids (`e7k3f9x2m8q1w4zr`) print bare. Only digit-led ids stay quoted. That's harmless. - **Callers.** `strict_frontmatter::plan` is used only by notes `visibility.rs`, `block_visibility.rs`, `write_executor/plan.rs:192` (a no-change validation) and `tasks/status.rs`. Contacts, money and calendar don't use it. The one test that looks at key bytes outside these (`tasks/status_action.rs:335`) already accepts both forms. ### 3. hidden-block migration: byte-exact and nothing is lost, with gaps below The flag, old, new and both cases check out by reading and by `block_visibility/tests.rs`. `["hidden"]` and `["hidden blocks"]` don't trip the overlap check, because path elements compare whole. Unhide on an old list is a no-op. When Hide moves the list, it keeps the union. Undo restores the old `hidden:` list exactly. Any list it can't read leaves the note skipped, not rewritten. ### findings **P2-1. existing notes never drop their hidden blocks from search.** `search.rs:210` is the first time search filters hidden blocks (dev had no such filter). But the search stamp is the source hash (`generation.rs:32`, `accepted-v1:<blake3 of text/path/doc>`), and the indexer only re-reads items whose stamp differs (`indexer.rs:238`). The schema bump to 6 rebuilds `source_versions` with the same hashes. Scenario: a note indexed on dev with `hidden: ["^k2m8q1w4zre7k3f9"]` (the ADR's own example) or `hidden blocks:` is upgraded. Searching for the hidden block's words still finds it until someone edits that note. `search_skips_hidden_blocks_in_both_list_forms` calls `describe` directly, so it can't see this. Fix: bump the stamp to `accepted-v2` (one line; costs one full re-index) and add an upgrade test like `:772`. **P2-2. bulk Unarchive/Unhide (or Undo of a bulk Archive/Hide) starts a burst of full list reads on every device.** `notes.svelte.ts:254-258`: every `notes.saved` whose `visibility` is unflagged, for a note that isn't loaded, sets two timers (400 ms and 2.5 s). Each timer calls `loadNotes()`, and that reads every page of the whole list (`:141-171`). Undo keeps `ExactUndo` (`home.rs:197`), so its events carry `visibility` too. Scenario: unarchive 100 notes, or undo archiving 100. That's 100 events, so about 100 `loadNotes()` start together at 400 ms, before any of them has put the notes back in `index`. On a 10k Home that's about 2,000 `notes.list` calls on each open device. Unhiding a task note, which is never in the Notes list, always fires both reloads for nothing. Fix: coalesce the "return" case into one pending reload, e.g. a timer plus a set of ids like `refreshTimer`, or an in-flight guard on `loadNotes`. Add a unit test that 100 return events give one read. **P3-1. an old `hidden:` list with bare items isn't moved, and a later block unhide will say "done" without unhiding.** The strict grammar parses `hidden:\n - ^aa1` as `Opaque`, because `^` only reads as a quoted string. That bare form is the one people type and the one Obsidian writes. `old_list` (`block_visibility.rs:133-139`) treats `Other` as "no old list". So `changes(raw, ["aa1"], false)` returns an empty change set, but `ids()` still hides `^aa1` in search. The ADR's own example form, `hidden: ["^…"]`, is a flow list, which refuses the whole envelope. Such a note can't be hidden, archived or status-changed until it's edited in source. That's safe, but it doesn't match H1's "moves on the next edit". No backend caller exists yet, so this is P3. Fix before the block action ships: - refuse with a reason when either key holds an opaque list; - change the ADR 0056 examples to the block-list form the writer actually prints. **P3-2. hidden blocks show up outside search.** - `meta.rs:1192` `snippet` doesn't skip hidden lines. If the first body line is a hidden block, its words become the list row preview. - `line_is_hidden` works per line (`search.rs:210-216`), so a hidden multi-line paragraph or a list item's children stay searchable. Only the line with the `^id` marker is dropped. **P3-3. Hide on `hidden: []` plus `hidden blocks: []` also deletes `hidden blocks: []`.** `block_visibility.rs:108-110`. This loses no data, but it's one line more than the flag change. Skip the removal when the move added nothing. **P3-4. totals: correct by reading, thin by test.** I walked every `visible::kept` arm against its page filter: - plain: N − |F| - tag: n_tag − |F ∩ tag| - tasks: N − |T| − |F∖T|, and Tasks |T∖F| - `show_hidden`: F = archived only - Archived place: direct count over `ARCHIVED_FROM` - tag + tasks and every other filter: fall back to `count_sql`, which includes the join All of them match, and `only_tag` correctly stops applying once the visibility filter is pushed. `visibility_page.rs:52` doesn't cover Archived + tag, a common tag (≥ `RARE_TAG`, the `EXISTS` + `INDEXED BY` + join form), a kind filter or a cursor page. Add those rows. counts: **P1 0, P2 2, P3 4.** ### verdict **Not ready to train yet.** Every 76637 finding is fixed, the writer change is safe, and the perf totals are right. Both P2s are small, contained fixes: a stamp bump with an upgrade test, and one coalesced reload. P2-2 only shows up once #877's bulk controls exist, which is exactly the pair being trained. Once those two land with their tests, it's ready to train with #877 without another full review round. The P3s can follow, and P3-1 must be fixed before the block hide action ships.
kayg commented 2026-11-03 12:30:14 +00:00
Author
Owner
Copy link

fixes for the re-review (76871), head 387e957e

finding commit what changed
P2-1 stamp 954bc6ce accepted-v1: → accepted-v2: (generation.rs STAMP_PREFIX). Upgrade test crates/search/tests/hidden_blocks_upgrade.rs: an index built the old way (v1 stamps, hidden words in the body) over three notes (hidden: list, hidden blocks: list, bare - ^id list) is reconciled by the new provider with no note edited. Before: 3 hits for the hidden word; after: 0; the visible line still finds all 3.
P2-1 perf 9e6fb181, 443e12ee see numbers below. The re-index of an index that already answers searches runs in paced slices (batches of 8, then rests 4x the batch time). A first build (empty index) is unchanged.
P2-2 bulk reload 9ffe4a05 pendingReload (notes-visibility.ts): one pending read for any number of returning notes (quiet 400 ms, at most 2 s from the first, one retry look at 2.5 s only if a note is still missing). Used for unarchive/unhide/undo and for restored notes. Tests count reads: 100 notes → 1 read; a lagging list → 2; a caught-up list → no retry.
P3-1 old lists 09806fb9 the planner grammar reads bare - ^id items and a one-line ["^id", ^id] list under hidden / hidden blocks only (every other key stays strict). So Hide/Archive/Unhide/block edits plan on such notes instead of refusing. Byte-exact tests in block_visibility/tests.rs (move on block hide, unhide clears, note Hide moves the list), grammar test in strict_frontmatter/tests.rs (other keys still refuse). A multi-line flow list still refuses (safe, noted).
P3-3 09806fb9 Hide on hidden: [] + hidden blocks: [] keeps the empty line (byte-exact test); Unhide with nothing to remove leaves it too.
P3-4 totals f9194dee visibility_page.rs: Archived × tag, kind filter, show-hidden, later pages (totals on every page, pages add up) against a model of the flags; common tag (2,100 notes, EXISTS form asserted) × place × show-hidden.
P3-2 #896 preview line + multi-line block search, left out on purpose.

gates (compute02, ~/bin/slot, Nix)

  • cargo fmt --all -- --check: 0
  • cargo clippy --all-targets -j4 -- -D warnings: 0
  • cargo test -j4 -p calternal-notes -p calternal-search -p calternal-tasks -p calternal-engine: 0 (all test result: ok; new tests ran)
  • bunx vitest run (apps/web): 9 files, 103 tests pass
  • svelte-check: 2 errors, both in e2e/review4-signout.spec.ts (not touched by this PR)

one-time re-index at 10k notes (perf-02, under perf-lock, release build, 3 reps each)

Throwaway in-process bench (not committed): 10,000 notes (~2.2 KB, 10% with a hidden block), index built with v1 stamps, then reconciled by the new provider while a loop runs Index::search and notes.page(limit 50) next to it. 2 tokio workers, 4 vCPU. "idle" = same loop with no re-index.

re-index peak RSS (start → peak) search p95 idle → during list p95 idle → during
unpaced (6b8d385a + stamp bump) 4.3–5.2 s 48 → 61–63 MB 2.5–3.3 → 8.6–14.1 ms 0.37–0.52 → 1.65–1.91 ms
batches of 16, rest 2x 8.2–8.3 s 45 → 55–57 MB 2.6 → 3.7–3.8 ms 0.40 → 0.50–0.57 ms
final: batches of 8, rest 4x 12.9–13.1 s 47 → 55–59 MB 2.51–2.57 → 2.85–2.91 ms 0.38 → 0.37–0.40 ms

Final run detail: search p50 2.04–2.06 → 2.09–2.10 ms, max during 3.7–3.8 ms (idle max 3.3–5.4); list p50 0.22–0.24 → 0.24 ms, max during 0.9–1.8 ms. Peak RSS is +8–12 MB over start (VmHWM 87–89 MB total, including the fixture build).

Honest read: list is flat. Search p95 is still ~+0.3 ms (+12%) for the ~13 s the one-time re-index runs; p50 +0.05 ms; no outlier above idle's own max. Outside that window nothing changes (no hot path touched: stamp string, indexer pacing only when the index is non-empty). If kayg wants literally zero during the window, the next step is to run it only when the search queue is empty; say so and I'll do it.

Not measured: the same job on the 100k Home (scales linearly: ~130 s paced).

## fixes for the re-review (76871), head `387e957e` | finding | commit | what changed | |---|---|---| | P2-1 stamp | `954bc6ce` | `accepted-v1:` → `accepted-v2:` (`generation.rs` `STAMP_PREFIX`). Upgrade test `crates/search/tests/hidden_blocks_upgrade.rs`: an index built the old way (v1 stamps, hidden words in the body) over three notes (`hidden:` list, `hidden blocks:` list, bare `- ^id` list) is reconciled by the new provider with no note edited. Before: 3 hits for the hidden word; after: 0; the visible line still finds all 3. | | P2-1 perf | `9e6fb181`, `443e12ee` | see numbers below. The re-index of an index that already answers searches runs in paced slices (batches of 8, then rests 4x the batch time). A first build (empty index) is unchanged. | | P2-2 bulk reload | `9ffe4a05` | `pendingReload` (`notes-visibility.ts`): one pending read for any number of returning notes (quiet 400 ms, at most 2 s from the first, one retry look at 2.5 s only if a note is still missing). Used for unarchive/unhide/undo and for restored notes. Tests count reads: 100 notes → 1 read; a lagging list → 2; a caught-up list → no retry. | | P3-1 old lists | `09806fb9` | the planner grammar reads bare `- ^id` items and a one-line `["^id", ^id]` list under `hidden` / `hidden blocks` only (every other key stays strict). So Hide/Archive/Unhide/block edits plan on such notes instead of refusing. Byte-exact tests in `block_visibility/tests.rs` (move on block hide, unhide clears, note Hide moves the list), grammar test in `strict_frontmatter/tests.rs` (other keys still refuse). A multi-line flow list still refuses (safe, noted). | | P3-3 | `09806fb9` | Hide on `hidden: []` + `hidden blocks: []` keeps the empty line (byte-exact test); Unhide with nothing to remove leaves it too. | | P3-4 totals | `f9194dee` | `visibility_page.rs`: Archived × tag, kind filter, show-hidden, later pages (totals on every page, pages add up) against a model of the flags; common tag (2,100 notes, `EXISTS` form asserted) × place × show-hidden. | | P3-2 | #896 | preview line + multi-line block search, left out on purpose. | ### gates (compute02, `~/bin/slot`, Nix) - `cargo fmt --all -- --check`: 0 - `cargo clippy --all-targets -j4 -- -D warnings`: 0 - `cargo test -j4 -p calternal-notes -p calternal-search -p calternal-tasks -p calternal-engine`: 0 (all `test result: ok`; new tests ran) - `bunx vitest run` (apps/web): 9 files, 103 tests pass - `svelte-check`: 2 errors, both in `e2e/review4-signout.spec.ts` (not touched by this PR) ### one-time re-index at 10k notes (perf-02, under `perf-lock`, release build, 3 reps each) Throwaway in-process bench (not committed): 10,000 notes (~2.2 KB, 10% with a hidden block), index built with v1 stamps, then reconciled by the new provider while a loop runs `Index::search` and `notes.page(limit 50)` next to it. 2 tokio workers, 4 vCPU. "idle" = same loop with no re-index. | | re-index | peak RSS (start → peak) | search p95 idle → during | list p95 idle → during | |---|---|---|---|---| | unpaced (`6b8d385a` + stamp bump) | 4.3–5.2 s | 48 → 61–63 MB | 2.5–3.3 → **8.6–14.1 ms** | 0.37–0.52 → **1.65–1.91 ms** | | batches of 16, rest 2x | 8.2–8.3 s | 45 → 55–57 MB | 2.6 → 3.7–3.8 ms | 0.40 → 0.50–0.57 ms | | **final: batches of 8, rest 4x** | **12.9–13.1 s** | 47 → 55–59 MB | 2.51–2.57 → **2.85–2.91 ms** | 0.38 → **0.37–0.40 ms** | Final run detail: search p50 2.04–2.06 → 2.09–2.10 ms, max during 3.7–3.8 ms (idle max 3.3–5.4); list p50 0.22–0.24 → 0.24 ms, max during 0.9–1.8 ms. Peak RSS is +8–12 MB over start (VmHWM 87–89 MB total, including the fixture build). **Honest read:** list is flat. Search p95 is still ~+0.3 ms (+12%) for the ~13 s the one-time re-index runs; p50 +0.05 ms; no outlier above idle's own max. Outside that window nothing changes (no hot path touched: stamp string, indexer pacing only when the index is non-empty). If kayg wants literally zero during the window, the next step is to run it only when the search queue is empty; say so and I'll do it. Not measured: the same job on the 100k Home (scales linearly: ~130 s paced).
kayg commented 2026-11-03 12:43:37 +00:00
Author
Owner
Copy link

re-index now runs only in the gaps, head c8afa867

Commits: 775bb7f6 (core foreground gauge: searches and list reads hold a guard; tests), 4e98f596 (the re-index of an index that already serves searches starts each batch of 8 only when no search/list read runs and none ended in the last 2 ms; gives up waiting after 500 ms under steady load so it always finishes; the rest-time heuristic is gone), c8afa867 (fmt). A first build (empty index) is unchanged.

Gates (compute02): fmt 0, clippy --all-targets -D warnings 0, core/notes/search/tasks/engine tests all ok.

perf-02, perf-lock, release build, same throwaway bench (idle = same loop, no re-index; "during" = loop running next to the re-index)

Home run re-index RSS start → peak search p95 idle → during list p95 idle → during
10k 1 7.5 s 54 → 62 MB 3.29 → 3.69 ms 0.51 → 0.49 ms
10k 2 6.3 s 48 → 55 MB 3.17 → 3.21 ms 0.47 → 0.38 ms
10k 3 6.2 s 50 → 60 MB 2.83 → 3.05 ms 0.40 → 0.38 ms
100k 1 138.8 s 129 → 182 MB (VmHWM 226) 5.78 → 5.42 ms 0.77 → 0.48 ms

Read: list p95 is flat or lower in every run. Search p95 at 10k: +0.40, +0.04, +0.22 ms, against an idle p95 that itself moves 2.83–3.29 ms between runs (2.5–2.6 ms in the earlier session), so it's inside run-to-run noise; search p50 +0.05–0.2 ms, max during (4.0–6.0 ms) at or below idle max (5.6–7.1 ms). At 100k search and list p95 during are at or below idle, with 14,646 samples during. For comparison, the previous unpaced code was 8.6–14.1 ms search / 1.65–1.91 ms list p95 at 10k.

Caveats: the bench sleeps 5 ms between requests, so it leaves gaps; under truly back-to-back load the re-index waits up to 500 ms per batch (it still finishes, at a slower rate) and one batch then runs. Single 100k run (idle control 400 samples). Bench not committed.

## re-index now runs only in the gaps, head `c8afa867` Commits: `775bb7f6` (core `foreground` gauge: searches and list reads hold a guard; tests), `4e98f596` (the re-index of an index that already serves searches starts each batch of 8 only when no search/list read runs and none ended in the last 2 ms; gives up waiting after 500 ms under steady load so it always finishes; the rest-time heuristic is gone), `c8afa867` (fmt). A first build (empty index) is unchanged. Gates (compute02): fmt 0, `clippy --all-targets -D warnings` 0, core/notes/search/tasks/engine tests all ok. ### perf-02, `perf-lock`, release build, same throwaway bench (idle = same loop, no re-index; "during" = loop running next to the re-index) | Home | run | re-index | RSS start → peak | search p95 idle → during | list p95 idle → during | |---|---|---|---|---|---| | 10k | 1 | 7.5 s | 54 → 62 MB | 3.29 → 3.69 ms | 0.51 → 0.49 ms | | 10k | 2 | 6.3 s | 48 → 55 MB | 3.17 → 3.21 ms | 0.47 → 0.38 ms | | 10k | 3 | 6.2 s | 50 → 60 MB | 2.83 → 3.05 ms | 0.40 → 0.38 ms | | **100k** | 1 | **138.8 s** | **129 → 182 MB** (VmHWM 226) | **5.78 → 5.42 ms** | **0.77 → 0.48 ms** | Read: list p95 is flat or lower in every run. Search p95 at 10k: +0.40, +0.04, +0.22 ms, against an idle p95 that itself moves 2.83–3.29 ms between runs (2.5–2.6 ms in the earlier session), so it's inside run-to-run noise; search p50 +0.05–0.2 ms, max during (4.0–6.0 ms) at or below idle max (5.6–7.1 ms). At 100k search and list p95 during are at or below idle, with 14,646 samples during. For comparison, the previous unpaced code was 8.6–14.1 ms search / 1.65–1.91 ms list p95 at 10k. Caveats: the bench sleeps 5 ms between requests, so it leaves gaps; under truly back-to-back load the re-index waits up to 500 ms per batch (it still finishes, at a slower rate) and one batch then runs. Single 100k run (idle control 400 samples). Bench not committed.
kayg commented 2026-11-04 06:26:36 +00:00
Author
Owner
Copy link

Fix for "can't archive or hide a note whose other keys use flow style" (head 59f7f0eb on notes-archive-hide)

Cause: the strict planner judged every line of the metadata, so tags: [a, b] (what the app writes) refused the whole note.

Change:

  • New key-scoped planner (strict_frontmatter/scoped.rs): the requested keys' own lines go through the same strict grammar and planner. Every other line is never read or changed. The metadata must still load as one YAML map, and after the edit the map must equal the old one with exactly the requested change.
  • Visibility reads (current_key), plans (plan_keys) and the executor's metadata check (ScopedMetadata { keys }) use it. The whole-file plan stays for Tasks.
  • Consistent CRLF metadata is now readable and writable. A mix of line ends or a lone CR still refuses. A comment on the key's own line, yes, True, quoted, list or map values still refuse that note.
  • Old tests that encoded "other keys' formatting skips the note" were changed to the new rule. Skips now come only from the changed key's own form.

Tests (byte-exact, with undo): archive, unarchive, hide, unhide on tags: [a, b], aliases: [x], flow+block mix, comments between keys, unindented lists, quoted values, multi-line flow, nested map, LF and CRLF; the old hidden: block list moving next to flow lists; end to end: tag a note through a room client (tags: [a, b]), then archive, unarchive, hide, unhide.

Gates on dev-01 (~/bin/slot, -j4, after train BM14d finished):

  • cargo fmt --all --check: clean (after style: fmt).
  • cargo clippy --workspace --all-targets --locked -- -D warnings: clean.
  • cargo test --locked -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine: all pass (notes lib 592 passed, 6 ignored; tasks 69; 0 failures). Tasks strict-writer users unchanged and passing.

Perf spot check (archive one / archive 100, flat against c8afa867): pending, see the next comment. The perf VMs are held by other jobs.

**Fix for "can't archive or hide a note whose other keys use flow style" (head 59f7f0eb on `notes-archive-hide`)** Cause: the strict planner judged every line of the metadata, so `tags: [a, b]` (what the app writes) refused the whole note. Change: - New key-scoped planner (`strict_frontmatter/scoped.rs`): the requested keys' own lines go through the same strict grammar and planner. Every other line is never read or changed. The metadata must still load as one YAML map, and after the edit the map must equal the old one with exactly the requested change. - Visibility reads (`current_key`), plans (`plan_keys`) and the executor's metadata check (`ScopedMetadata { keys }`) use it. The whole-file `plan` stays for Tasks. - Consistent CRLF metadata is now readable and writable. A mix of line ends or a lone CR still refuses. A comment on the key's own line, `yes`, `True`, quoted, list or map values still refuse that note. - Old tests that encoded "other keys' formatting skips the note" were changed to the new rule. Skips now come only from the changed key's own form. Tests (byte-exact, with undo): archive, unarchive, hide, unhide on `tags: [a, b]`, `aliases: [x]`, flow+block mix, comments between keys, unindented lists, quoted values, multi-line flow, nested map, LF and CRLF; the old `hidden:` block list moving next to flow lists; end to end: tag a note through a room client (`tags: [a, b]`), then archive, unarchive, hide, unhide. Gates on dev-01 (`~/bin/slot`, `-j4`, after train BM14d finished): - `cargo fmt --all --check`: clean (after `style: fmt`). - `cargo clippy --workspace --all-targets --locked -- -D warnings`: clean. - `cargo test --locked -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine`: all pass (notes lib 592 passed, 6 ignored; tasks 69; 0 failures). Tasks strict-writer users unchanged and passing. Perf spot check (archive one / archive 100, flat against c8afa867): pending, see the next comment. The perf VMs are held by other jobs.
kayg commented 2026-11-04 06:56:20 +00:00
Author
Owner
Copy link

independent review: key-scoped planner (head 59f7f0eb, commits 358767d1, 387a9452, 51b5b237)

i only read the code and built nothing. i diffed against origin/dev 41f2efa3. i looked up yaml-rust2 0.10.4's source on dev-01: duplicate keys are a load error, and True/TRUE load as booleans.

Counts: P1 1, P2 2, P3 4. The splice itself is sound. Every forward splice sits on the cut lines of a requested key, and the undo is checked byte for byte (scoped.rs:89). The real problems are next to it: one reader that disagrees with the writer, one way to write mixed line ends, and one check that only the executor runs.

P1

P1-1. A CRLF note whose body has any LF-only line reads as not hidden or archived in the list and search index, but the writer reads it as hidden or archived. visibility.rs:406-409: plain() looks for b"\n---\n" to find the end of the metadata. A CRLF closer is \r\n---\r\n, so it is never found, and end becomes the whole file. Then any LF-only line or lone CR in the body makes plain false, and flag() returns false (meta.rs:1172 via search.rs:152).

  • Input: ---\r\ntags: [a, b]\r\n---\r\n# Note\nbody\r\n. This is exactly style_tests.rs:30 wrap(.., "\r\n"), but those tests only assert read(raw), never flag(text, plain(raw)).
  • Effect: Hide reports done, changed, but the file-scan row still says not hidden, so the note stays in search and lists. A second Hide answers "already done". With a room open, room/mod.rs:556 (plain_style = no BOM → true) sets the row to hidden, and the next file rescan clears it again. So the flag flips depending on whether the note is open.
  • Fix: find the end with the same closer View::parse uses (---{eol} as a whole line, with eol taken from the first line), and count CRs only inside the metadata. Add a flag(decoded, plain(raw)) == read(raw) assertion to the style matrix.

P2

P2-1. On a CRLF note with no metadata, Archive or Hide writes an LF envelope above a CRLF body, so the file has mixed line ends. scoped.rs:67-70 passes a note with no metadata to the whole-file plan, and that writes b"---\n" … b"---\n" (strict_frontmatter.rs:119,123). grammar.rs:80-88 never looks at the body's line ends.

  • Input: # Title\r\nbody\r\n, then Archive → ---\narchived: true\n---\n# Title\r\nbody\r\n.
  • Effect: undo is still exact, but this breaks "write back exactly CRLF, never mixed". text.rs:45 (majority style) then makes the next room save rewrite the envelope as CRLF. After that, the recorded inverse no longer matches, so Undo turns into a conflict.
  • Fix: in the scoped path, choose the envelope's line end from the body: CRLF when every line end in the body is CRLF, LF when every one is LF, and refuse a mix. Add a test.

P2-2. Hide plans each note with key hidden only, but the executor checks hidden and hidden blocks. One note with an odd hidden blocks: line fails the whole batch of up to 100 notes as a "changed" conflict. visibility.rs:143-145 puts ScopedMetadata { keys: [hidden, hidden blocks] } on the batch. decide/plan_keys (:172, :183) only parse hidden. plan.rs:204-219 runs check_keys on every image with both keys, refuses, the loop re-plans 3 times (visibility.rs:119), and then returns BlockChanged "unsupported metadata: edit in source mode" for every note.

  • Inputs: hidden blocks: [] # keep, a multi-line flow hidden blocks: [^aa,\n ^bb], or hidden blocks: yes.
  • Fix: in plan_visibility, run fm::check_keys(&pre, &scoped_keys(key)) for each note and skip that note with UNSUPPORTED, so a skip happens at planning and never fails the whole batch. Add a 2-note test: one plain note, one with hidden blocks: [] # x.

P3

P3-1. The read() fast path stops at the first \n---, not at the closer line (visibility.rs:374-377). Input: ---\na: 1\n---x: 2\narchived: true\n---\n. ---x is a valid plain key, and View reads archived: true, but the fast path cuts before it and returns false. So Archive answers "already done" while the note still shows. This is rare. Use the same whole-line closer as in P1-1.

P3-2. Notes that are refused but would be safe to edit (over-refusal, no data at risk):

  • a final --- with no line end (a note that is only frontmatter) or a ... closer (scoped.rs:146,166);
  • any line that starts with something other than ASCII alphanumeric/_/", and contains the key text anywhere, refuses the whole note (scoped.rs:210). Examples: 'title': things I archived, état: archived, 'notes': hidden gems.

P3-3. Merge keys. yaml-rust2 keeps << as an ordinary key, and the mini scan skips it. So for <<: {archived: true}, Calternal reads "not archived", while js-yaml-style YAML 1.1 readers see it as archived. Calternal stays consistent with itself, but it differs from other tools. Leave it as is, and add it to the only_the_changed_keys_form_refuses notes.

P3-4. Perf not shown yet. current_key now loads the whole metadata as YAML on top of the grammar parse. read() calls it for both flags on every indexed or live-saved note whose metadata contains the key bytes (meta.rs:1172-1173, room/mod.rs:556, home.rs:624). The archive one / archive 100 numbers in 77415 are still pending. Index rebuild and live-save p95 also need a dev control, because of the no-regression rule.

Checked and fine

  • Line spans: a key inside a flow map or multi-line flow at column 0 (tags: [\narchived: true\n]): the mini scan picks the line, the YAML has no top-level key, and current refuses (scoped.rs:38-46). Indented block-scalar (|, >) content and nested-map lines start with a space, so they are never cut. archived_at, archived2 and hidden blocks vs hidden are exact-name matches. Duplicate requested keys refuse (:204). Other duplicate keys make the YAML load fail. A BOM refuses on both reader and writer. A --- inside a value must be indented, or it is the closer for both.
  • Quoted keys: a key the scan can't see, like "\x61rchived": true (JSON fails, YAML gives archived), gives (None, Some) and refuses. An insert would also make a duplicate key and fail the reload.
  • YAML equality: non-target bytes are never spliced, so equality only has to catch lines the scan put under the wrong key. A splice inside a folded quoted or plain scalar changes that scalar's value, and same_map refuses it. Yaml::Real compares as text. Anchors, aliases and tags on a requested line refuse in the grammar (style_tests.rs:107). Elsewhere they load the same before and after.
  • CRLF: View refuses a lone CR, an LF in a CRLF block, and a last line with no line end. Replacements are mapped \n→eol (:79). Expected bytes are the real bytes, so the result is never mixed except in P2-1.
  • Mapping: the planner only merges empty inserts at the same point (strict_frontmatter.rs:126-139), so no splice can cover two separate spans. An offset that isn't a line start refuses (StaleSplice).
  • Undo: exact by construction. plan re-applies the inverse (:89), and Home undo checks recorded expected bytes and room versions.
  • Room edits: leases compare pre with the room. A refusal happens before the journal records anything, then the action re-plans from fresh bytes. I didn't re-review the lease code (covered in 76637/76871).
  • Tasks: unchanged. It still uses the whole-file plan.

Verdict

Not yet safe to train with #877. No input I found corrupts or loses note bytes, and undo is exact. But P1-1 makes Hide fail without telling the person, on a shape that is in the PR's own test fixture. P2-1 writes mixed line ends, which the brief rules out. P2-2 makes one odd note fail a whole bulk Hide. All three are small, local fixes with one test each. Train after those fixes, plus the P3-4 perf numbers.

## independent review: key-scoped planner (head `59f7f0eb`, commits `358767d1`, `387a9452`, `51b5b237`) i only read the code and built nothing. i diffed against `origin/dev` `41f2efa3`. i looked up `yaml-rust2` 0.10.4's source on dev-01: duplicate keys are a load error, and `True`/`TRUE` load as booleans. **Counts: P1 1, P2 2, P3 4.** The splice itself is sound. Every forward splice sits on the cut lines of a requested key, and the undo is checked byte for byte (`scoped.rs:89`). The real problems are next to it: one reader that disagrees with the writer, one way to write mixed line ends, and one check that only the executor runs. ### P1 **P1-1. A CRLF note whose body has any LF-only line reads as not hidden or archived in the list and search index, but the writer reads it as hidden or archived.** `visibility.rs:406-409`: `plain()` looks for `b"\n---\n"` to find the end of the metadata. A CRLF closer is `\r\n---\r\n`, so it is never found, and `end` becomes the whole file. Then any LF-only line or lone CR in the *body* makes `plain` false, and `flag()` returns false (`meta.rs:1172` via `search.rs:152`). - Input: `---\r\ntags: [a, b]\r\n---\r\n# Note\nbody\r\n`. This is exactly `style_tests.rs:30` `wrap(.., "\r\n")`, but those tests only assert `read(raw)`, never `flag(text, plain(raw))`. - Effect: Hide reports `done, changed`, but the file-scan row still says not hidden, so the note stays in search and lists. A second Hide answers "already done". With a room open, `room/mod.rs:556` (`plain_style` = no BOM → true) sets the row to hidden, and the next file rescan clears it again. So the flag flips depending on whether the note is open. - Fix: find the end with the same closer `View::parse` uses (`---{eol}` as a whole line, with `eol` taken from the first line), and count CRs only inside the metadata. Add a `flag(decoded, plain(raw))` == `read(raw)` assertion to the style matrix. ### P2 **P2-1. On a CRLF note with no metadata, Archive or Hide writes an LF envelope above a CRLF body, so the file has mixed line ends.** `scoped.rs:67-70` passes a note with no metadata to the whole-file `plan`, and that writes `b"---\n"` … `b"---\n"` (`strict_frontmatter.rs:119,123`). `grammar.rs:80-88` never looks at the body's line ends. - Input: `# Title\r\nbody\r\n`, then Archive → `---\narchived: true\n---\n# Title\r\nbody\r\n`. - Effect: undo is still exact, but this breaks "write back exactly CRLF, never mixed". `text.rs:45` (majority style) then makes the next room save rewrite the envelope as CRLF. After that, the recorded inverse no longer matches, so Undo turns into a conflict. - Fix: in the scoped path, choose the envelope's line end from the body: CRLF when every line end in the body is CRLF, LF when every one is LF, and refuse a mix. Add a test. **P2-2. Hide plans each note with key `hidden` only, but the executor checks `hidden` and `hidden blocks`. One note with an odd `hidden blocks:` line fails the whole batch of up to 100 notes as a "changed" conflict.** `visibility.rs:143-145` puts `ScopedMetadata { keys: [hidden, hidden blocks] }` on the batch. `decide`/`plan_keys` (`:172`, `:183`) only parse `hidden`. `plan.rs:204-219` runs `check_keys` on every image with both keys, refuses, the loop re-plans 3 times (`visibility.rs:119`), and then returns `BlockChanged` "unsupported metadata: edit in source mode" for every note. - Inputs: `hidden blocks: [] # keep`, a multi-line flow `hidden blocks: [^aa,\n ^bb]`, or `hidden blocks: yes`. - Fix: in `plan_visibility`, run `fm::check_keys(&pre, &scoped_keys(key))` for each note and skip that note with `UNSUPPORTED`, so a skip happens at planning and never fails the whole batch. Add a 2-note test: one plain note, one with `hidden blocks: [] # x`. ### P3 **P3-1. The `read()` fast path stops at the first `\n---`, not at the closer line** (`visibility.rs:374-377`). Input: `---\na: 1\n---x: 2\narchived: true\n---\n`. `---x` is a valid plain key, and `View` reads `archived: true`, but the fast path cuts before it and returns false. So Archive answers "already done" while the note still shows. This is rare. Use the same whole-line closer as in P1-1. **P3-2. Notes that are refused but would be safe to edit** (over-refusal, no data at risk): - a final `---` with no line end (a note that is only frontmatter) or a `...` closer (`scoped.rs:146,166`); - any line that starts with something other than ASCII alphanumeric/`_`/`"`, and contains the key text anywhere, refuses the whole note (`scoped.rs:210`). Examples: `'title': things I archived`, `état: archived`, `'notes': hidden gems`. **P3-3. Merge keys.** `yaml-rust2` keeps `<<` as an ordinary key, and the mini scan skips it. So for `<<: {archived: true}`, Calternal reads "not archived", while js-yaml-style YAML 1.1 readers see it as archived. Calternal stays consistent with itself, but it differs from other tools. Leave it as is, and add it to the `only_the_changed_keys_form_refuses` notes. **P3-4. Perf not shown yet.** `current_key` now loads the whole metadata as YAML on top of the grammar parse. `read()` calls it for both flags on every indexed or live-saved note whose metadata contains the key bytes (`meta.rs:1172-1173`, `room/mod.rs:556`, `home.rs:624`). The archive one / archive 100 numbers in 77415 are still pending. Index rebuild and live-save p95 also need a dev control, because of the no-regression rule. ### Checked and fine - **Line spans:** a key inside a flow map or multi-line flow at column 0 (`tags: [\narchived: true\n]`): the mini scan picks the line, the YAML has no top-level key, and `current` refuses (`scoped.rs:38-46`). Indented block-scalar (`|`, `>`) content and nested-map lines start with a space, so they are never cut. `archived_at`, `archived2` and `hidden blocks` vs `hidden` are exact-name matches. Duplicate requested keys refuse (`:204`). Other duplicate keys make the YAML load fail. A BOM refuses on both reader and writer. A `---` inside a value must be indented, or it is the closer for both. - **Quoted keys:** a key the scan can't see, like `"\x61rchived": true` (JSON fails, YAML gives `archived`), gives `(None, Some)` and refuses. An insert would also make a duplicate key and fail the reload. - **YAML equality:** non-target bytes are never spliced, so equality only has to catch lines the scan put under the wrong key. A splice inside a folded quoted or plain scalar changes that scalar's value, and `same_map` refuses it. `Yaml::Real` compares as text. Anchors, aliases and tags on a requested line refuse in the grammar (`style_tests.rs:107`). Elsewhere they load the same before and after. - **CRLF:** `View` refuses a lone CR, an LF in a CRLF block, and a last line with no line end. Replacements are mapped `\n`→`eol` (`:79`). Expected bytes are the real bytes, so the result is never mixed except in P2-1. - **Mapping:** the planner only merges empty inserts at the same point (`strict_frontmatter.rs:126-139`), so no splice can cover two separate spans. An offset that isn't a line start refuses (`StaleSplice`). - **Undo:** exact by construction. `plan` re-applies the inverse (`:89`), and Home undo checks recorded expected bytes and room versions. - **Room edits:** leases compare `pre` with the room. A refusal happens before the journal records anything, then the action re-plans from fresh bytes. I didn't re-review the lease code (covered in 76637/76871). - **Tasks:** unchanged. It still uses the whole-file `plan`. ### Verdict **Not yet safe to train with #877.** No input I found corrupts or loses note bytes, and undo is exact. But P1-1 makes Hide fail without telling the person, on a shape that is in the PR's own test fixture. P2-1 writes mixed line ends, which the brief rules out. P2-2 makes one odd note fail a whole bulk Hide. All three are small, local fixes with one test each. Train after those fixes, plus the P3-4 perf numbers.
kayg commented 2026-11-04 07:48:48 +00:00
Author
Owner
Copy link

simplified: no CRLF flag writes (head 2d51718a)

The round-3 P1 and P2-1 both came from CRLF support, so it is gone instead of patched.

Changes (693ce8ec, then test/fmt follow-ups):

  • Windows line ends are refused. A note with any \r, in metadata or body, is skipped by Archive, Hide and their undo: "this note uses Windows line endings; open it once in Calternal to fix, then try again". Never written, never mixed. The CRLF write paths and their tests are removed.
  • Reader and writer agree. plain() is false for any \r, and plain_style is false for a CRLF room, so such a note is never counted hidden or archived (list, search, live save).
  • P1 test. The review's exact shape ---\r\ntags: [a, b]\r\n---\r\n# Note\nbody\r\n, plus the no-metadata case # Title\r\nbody\r\n, run through read, flag(decoded, plain(raw)), decide_all, plan_keys and check_keys.
  • P2-2. The executor's ScopedMetadata now has only the changed key (same scope as planning), so hidden blocks: [] # keep skips nothing and fails nothing. Test: a batch of a plain note, an odd-neighbour note and a CRLF note; the first two are done, the CRLF one is skipped with its reason, the batch is not failed, undo is byte exact.
  • P3-1. The read() quick check ends at the first whole --- line, so ---x: 2 is no longer taken for a closer. Test added.
  • Left as is, on purpose: P3-2 (over-refusal: final --- without a line end, ... closer, odd key starts), P3-3 (merge keys differ from YAML 1.1 readers), P3-4 (perf numbers: coming in the next comment).

Gates (dev-01, ~/bin/slot, -j4, head 2d51718a):

  • cargo fmt --all --check: 0
  • cargo clippy --workspace --all-targets --locked -j4 -- -D warnings: 0
  • cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine: exit 0, 702 passed, 0 failed.

Perf spot check (dev 41f2efa3 vs c8afa867 vs 2d51718a, ABAB on perf-01 with both perf locks) is queued behind other jobs' locks. Release binaries are built and checksummed on both perf VMs.

## simplified: no CRLF flag writes (head `2d51718a`) The round-3 P1 and P2-1 both came from CRLF support, so it is gone instead of patched. **Changes** (`693ce8ec`, then test/fmt follow-ups): - **Windows line ends are refused.** A note with any `\r`, in metadata or body, is skipped by Archive, Hide and their undo: "this note uses Windows line endings; open it once in Calternal to fix, then try again". Never written, never mixed. The CRLF write paths and their tests are removed. - **Reader and writer agree.** `plain()` is false for any `\r`, and `plain_style` is false for a CRLF room, so such a note is never counted hidden or archived (list, search, live save). - **P1 test.** The review's exact shape `---\r\ntags: [a, b]\r\n---\r\n# Note\nbody\r\n`, plus the no-metadata case `# Title\r\nbody\r\n`, run through `read`, `flag(decoded, plain(raw))`, `decide_all`, `plan_keys` and `check_keys`. - **P2-2.** The executor's `ScopedMetadata` now has only the changed key (same scope as planning), so `hidden blocks: [] # keep` skips nothing and fails nothing. Test: a batch of a plain note, an odd-neighbour note and a CRLF note; the first two are done, the CRLF one is skipped with its reason, the batch is not failed, undo is byte exact. - **P3-1.** The `read()` quick check ends at the first whole `---` line, so `---x: 2` is no longer taken for a closer. Test added. - **Left as is, on purpose:** P3-2 (over-refusal: final `---` without a line end, `...` closer, odd key starts), P3-3 (merge keys differ from YAML 1.1 readers), P3-4 (perf numbers: coming in the next comment). **Gates** (dev-01, `~/bin/slot`, `-j4`, head `2d51718a`): - `cargo fmt --all --check`: 0 - `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings`: 0 - `cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine`: exit 0, 702 passed, 0 failed. Perf spot check (dev `41f2efa3` vs `c8afa867` vs `2d51718a`, ABAB on perf-01 with both perf locks) is queued behind other jobs' locks. Release binaries are built and checksummed on both perf VMs.
kayg commented 2026-11-04 08:30:59 +00:00
Author
Owner
Copy link

perf spot check: simplified head 2d51718a vs c8afa867 vs dev 41f2efa3

Setup: perf-01, perf-lock held on perf-01 and perf-02 for the whole run (both holders stopped afterwards). Release builds, sha256 checked on both VMs. Throwaway probe, not committed: 4,000-note Home (25% archived, 10% hidden, tags in block style), 6 rounds, each round runs the three binaries in alternating order (ABC, CBA, ...). Per round: first start with indexing, a restart, 60 warm requests each of notes.list, notes.list place=archived, search.query; then 30 single archive + unarchive, and 5 batches of 100 archive + unarchive. Values are the median over 6 rounds in ms (min–max in brackets). Dev has no Archive action, so those cells are n/a.

metric dev 41f2efa3 old c8afa867 new 2d51718a
first start + index 1523.9 (1357–1628) 1296.0 (937–1337) 1253.6 (1139–1294)
restart ready 62.4 64.7 61.8
list first page p50 / p95 0.74 / 0.88 0.89 / 1.04 0.92 / 1.10
list archived p50 / p95 n/a 3.58 / 4.81 3.67 / 4.79
search p50 / p95 0.87 / 1.15 0.89 / 1.05 0.90 / 1.12
archive one p50 / p95 n/a 27.3 / 47.6 25.1 / 43.0
unarchive one p50 n/a 28.0 25.1
archive 100 p50 / max n/a 808.7 / 1017.6 776.6 / 957.0
unarchive 100 p50 n/a 817.6 773.4

Read: new vs c8afa867 is flat or better on every Archive path (archive one −8% p50, archive 100 −4% p50, ranges barely overlap), and flat on the archived list and search. List first page moves +0.03 ms p50 / +0.06 ms p95 against c8afa867 and the ranges overlap (0.81–1.00 vs 0.72–0.90), so the new head is inside run-to-run spread, but it is the only number that points the wrong way, so it is worth a look in the final review. Both flagged builds sit +0.15–0.2 ms above dev on this call; that gap was already in c8afa867, not added by this change. Archive 100 is ~800 ms on this slow shared disk (about 8 ms per note), unchanged in shape. Index rebuild and live-save p95 against a dev control (P3-4) are not part of this spot check.

## perf spot check: simplified head `2d51718a` vs `c8afa867` vs dev `41f2efa3` Setup: perf-01, `perf-lock` held on perf-01 and perf-02 for the whole run (both holders stopped afterwards). Release builds, sha256 checked on both VMs. Throwaway probe, not committed: 4,000-note Home (25% archived, 10% hidden, tags in block style), 6 rounds, each round runs the three binaries in alternating order (ABC, CBA, ...). Per round: first start with indexing, a restart, 60 warm requests each of `notes.list`, `notes.list place=archived`, `search.query`; then 30 single archive + unarchive, and 5 batches of 100 archive + unarchive. Values are the median over 6 rounds in ms (min–max in brackets). Dev has no Archive action, so those cells are n/a. | metric | dev 41f2efa3 | old c8afa867 | new 2d51718a | |---|---|---|---| | first start + index | 1523.9 (1357–1628) | 1296.0 (937–1337) | 1253.6 (1139–1294) | | restart ready | 62.4 | 64.7 | 61.8 | | list first page p50 / p95 | 0.74 / 0.88 | 0.89 / 1.04 | 0.92 / 1.10 | | list archived p50 / p95 | n/a | 3.58 / 4.81 | 3.67 / 4.79 | | search p50 / p95 | 0.87 / 1.15 | 0.89 / 1.05 | 0.90 / 1.12 | | archive one p50 / p95 | n/a | 27.3 / 47.6 | 25.1 / 43.0 | | unarchive one p50 | n/a | 28.0 | 25.1 | | archive 100 p50 / max | n/a | 808.7 / 1017.6 | 776.6 / 957.0 | | unarchive 100 p50 | n/a | 817.6 | 773.4 | Read: new vs c8afa867 is flat or better on every Archive path (archive one −8% p50, archive 100 −4% p50, ranges barely overlap), and flat on the archived list and search. List first page moves +0.03 ms p50 / +0.06 ms p95 against c8afa867 and the ranges overlap (0.81–1.00 vs 0.72–0.90), so the new head is inside run-to-run spread, but it is the only number that points the wrong way, so it is worth a look in the final review. Both flagged builds sit +0.15–0.2 ms above dev on this call; that gap was already in c8afa867, not added by this change. Archive 100 is ~800 ms on this slow shared disk (about 8 ms per note), unchanged in shape. Index rebuild and live-save p95 against a dev control (P3-4) are not part of this spot check.
kayg commented 2026-11-04 09:57:13 +00:00
Author
Owner
Copy link

perf-VM check: archive/hide (#887)

The staged ~/perf/fix887/{base,new} could not be measured: both are Rust test binaries (libtest options, no user create / server CLI), built 06:18 CET, before head 59f7f0eb and 2d51718a. I measured the server binaries I found staged in ~/perf/b887/{dev,new} on perf-02 instead (built 08:27 CET = 07:27Z, before the latest head 2d51718a at 07:48Z, so not the current head). Used the existing throwaway probe ~/perf/p887/run887.py, 4000 notes, 5 rounds with alternating order, three variants: dev (base), new, and a byte copy of dev (control). perf-01 + perf-02 locks held; run on perf-02. Cells: median [min-max], n=5, ms.

metric dev (base) new dev copy (control) new vs base
first_start.index_ms 1149 [1046-1398] 929 [908-1240] 1328 [1073-1582] -19% (control spread is bigger than the gap)
restart.ready_ms 60.7 [60.3-63.1] 63.4 [61.3-64.5] 62.4 [61.0-68.8] +4% (within control)
list.first_page p50 0.61 [0.57-0.70] 0.78 [0.75-0.87] 0.64 [0.62-0.73] +28%
list.first_page p95 0.67 [0.63-0.78] 0.90 [0.80-1.01] 0.75 [0.65-0.83] +34%
search.query p50 0.78 [0.73-0.89] 0.84 [0.73-0.89] 0.81 [0.77-0.90] +8% (within control)
search.query p95 0.82 [0.76-1.00] 0.94 [0.77-1.00] 0.93 [0.82-1.04] +15% (within control)
list.archived p50 (new only) n/a 3.15 [3.08-3.65] n/a -
archive 1 note p50 / p95 n/a 22.3 / 39.0 n/a -
unarchive 1 note p50 n/a 22.7 n/a -
archive 100 notes p50 / max n/a 725 / 886 n/a -
unarchive 100 notes p50 n/a 721 n/a -

(base has no archive action; list.archived returns 400 there, so those rows have no base.)

Verdict: regressed on list.first_page. About +0.17 ms (+28% p50, +34% p95), outside the control's spread (control vs dev +5%), and in all 5 rounds new is above every dev run. Search and restart are flat; first index looks faster but the control is noisier than the gap. Archive-one at ~22 ms p50 and archive-100 at ~725 ms (7 ms/note) are new-feature costs for you to judge. This is a stale-ish build (before 2d51718a): re-measure on the current head with a server binary before relying on it. Per the no-perf-regression rule, the list.first_page delta needs an explanation or fix.

## perf-VM check: archive/hide (#887) **The staged `~/perf/fix887/{base,new}` could not be measured:** both are Rust test binaries (libtest options, no `user create` / server CLI), built 06:18 CET, before head `59f7f0eb` and `2d51718a`. I measured the server binaries I found staged in `~/perf/b887/{dev,new}` on perf-02 instead (built 08:27 CET = 07:27Z, before the latest head `2d51718a` at 07:48Z, so **not the current head**). Used the existing throwaway probe `~/perf/p887/run887.py`, 4000 notes, 5 rounds with alternating order, three variants: `dev` (base), `new`, and a byte copy of `dev` (control). perf-01 + perf-02 locks held; run on perf-02. Cells: median [min-max], n=5, ms. | metric | dev (base) | new | dev copy (control) | new vs base | |---|---|---|---|---| | first_start.index_ms | 1149 [1046-1398] | 929 [908-1240] | 1328 [1073-1582] | -19% (control spread is bigger than the gap) | | restart.ready_ms | 60.7 [60.3-63.1] | 63.4 [61.3-64.5] | 62.4 [61.0-68.8] | +4% (within control) | | **list.first_page p50** | **0.61 [0.57-0.70]** | **0.78 [0.75-0.87]** | 0.64 [0.62-0.73] | **+28%** | | **list.first_page p95** | **0.67 [0.63-0.78]** | **0.90 [0.80-1.01]** | 0.75 [0.65-0.83] | **+34%** | | search.query p50 | 0.78 [0.73-0.89] | 0.84 [0.73-0.89] | 0.81 [0.77-0.90] | +8% (within control) | | search.query p95 | 0.82 [0.76-1.00] | 0.94 [0.77-1.00] | 0.93 [0.82-1.04] | +15% (within control) | | list.archived p50 (new only) | n/a | 3.15 [3.08-3.65] | n/a | - | | archive 1 note p50 / p95 | n/a | 22.3 / 39.0 | n/a | - | | unarchive 1 note p50 | n/a | 22.7 | n/a | - | | archive 100 notes p50 / max | n/a | 725 / 886 | n/a | - | | unarchive 100 notes p50 | n/a | 721 | n/a | - | (base has no archive action; `list.archived` returns 400 there, so those rows have no base.) **Verdict: regressed on `list.first_page`.** About +0.17 ms (+28% p50, +34% p95), outside the control's spread (control vs dev +5%), and in all 5 rounds new is above every dev run. Search and restart are flat; first index looks faster but the control is noisier than the gap. Archive-one at ~22 ms p50 and archive-100 at ~725 ms (7 ms/note) are new-feature costs for you to judge. This is a stale-ish build (before `2d51718a`): re-measure on the current head with a server binary before relying on it. Per the no-perf-regression rule, the `list.first_page` delta needs an explanation or fix.
kayg commented 2026-11-04 13:35:23 +00:00
Author
Owner
Copy link

list first page: profile finding + fix (67fa2521; perf table follows when the perf locks are free)

Finding (EXPLAIN QUERY PLAN + a timed loop over the same SQLite file, 4,000-note store, 30% flagged):

page SQL plan µs / query
dev SCAN n USING INDEX notes_updated 82–87
#887 2d51718a SCAN n USING INDEX notes_updated + SEARCH v USING PRIMARY KEY (id=?) LEFT-JOIN 107

The extra cost is the per-row LEFT JOIN note_visibility probe: about 50 B-tree lookups on a second table for every page (+20–25 µs in SQLite, the rest is the request path). The old code also ran SELECT EXISTS(...) (3.7 µs) and a stats count minus COUNT(*) over the flagged rows (17.8 µs) on every call. Nothing reads files or decodes notes on this path (checked: page is handed no path), so the gap was join + two extra queries, not flag decoding.

Fix

  • notes.flagged INTEGER NOT NULL DEFAULT 0, written at save/index time from the same flags (hidden || archived), plus a partial index notes_open ON notes (updated DESC, id) WHERE flagged = 0. The default page is WHERE n.flagged = 0 ORDER BY updated DESC, id LIMIT 51: the plan is SCAN n USING INDEX notes_open, no join. In SQLite it measures the same as dev (86–94 µs vs 86–88 µs).
  • "Any flagged?" and the page total read two stats rows (flagged, flagged_archived) that triggers on note_visibility keep up to date, instead of EXISTS and COUNT(*) per request.
  • Show hidden still joins note_visibility (it needs the faded flags); the Archived place is unchanged.
  • Upgrade: a v6 list store keeps its rows; the column, the partial index and both counts are filled from note_visibility when it opens (test: a_store_without_the_flagged_column_keeps_its_flags). SCHEMA_VERSION 6 → 7.
  • Plan test now asserts the default page has no note_visibility reference, uses n.flagged = 0 and notes_open, and has no scan of v, sort or id set.

Gates (dev-01, ~/bin/slot, -j4, head 67fa2521): cargo fmt --all --check 0; cargo clippy --workspace --all-targets --locked -j4 -- -D warnings clean; cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf: 742 passed, 0 failed. (An earlier run on bcc68363 failed 5 restart tests: they used a positional INSERT INTO notes VALUES, and the schema-6 upgrade lacked the column. Both fixed in 67fa2521.)

## list first page: profile finding + fix (`67fa2521`; perf table follows when the perf locks are free) **Finding** (EXPLAIN QUERY PLAN + a timed loop over the same SQLite file, 4,000-note store, 30% flagged): | page SQL | plan | µs / query | |---|---|---| | dev | `SCAN n USING INDEX notes_updated` | 82–87 | | #887 `2d51718a` | `SCAN n USING INDEX notes_updated` + `SEARCH v USING PRIMARY KEY (id=?) LEFT-JOIN` | 107 | The extra cost is the per-row `LEFT JOIN note_visibility` probe: about 50 B-tree lookups on a second table for every page (+20–25 µs in SQLite, the rest is the request path). The old code also ran `SELECT EXISTS(...)` (3.7 µs) and a stats count minus `COUNT(*)` over the flagged rows (17.8 µs) on every call. Nothing reads files or decodes notes on this path (checked: `page` is handed no path), so the gap was join + two extra queries, not flag decoding. **Fix** - `notes.flagged INTEGER NOT NULL DEFAULT 0`, written at save/index time from the same flags (`hidden || archived`), plus a partial index `notes_open ON notes (updated DESC, id) WHERE flagged = 0`. The default page is `WHERE n.flagged = 0 ORDER BY updated DESC, id LIMIT 51`: the plan is `SCAN n USING INDEX notes_open`, no join. In SQLite it measures the same as dev (86–94 µs vs 86–88 µs). - "Any flagged?" and the page total read two `stats` rows (`flagged`, `flagged_archived`) that triggers on `note_visibility` keep up to date, instead of `EXISTS` and `COUNT(*)` per request. - Show hidden still joins `note_visibility` (it needs the faded flags); the Archived place is unchanged. - Upgrade: a v6 list store keeps its rows; the column, the partial index and both counts are filled from `note_visibility` when it opens (test: `a_store_without_the_flagged_column_keeps_its_flags`). `SCHEMA_VERSION` 6 → 7. - Plan test now asserts the default page has no `note_visibility` reference, uses `n.flagged = 0` and `notes_open`, and has no scan of `v`, sort or id set. **Gates** (dev-01, `~/bin/slot`, `-j4`, head `67fa2521`): `cargo fmt --all --check` 0; `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` clean; `cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf`: 742 passed, 0 failed. (An earlier run on `bcc68363` failed 5 restart tests: they used a positional `INSERT INTO notes VALUES`, and the schema-6 upgrade lacked the column. Both fixed in `67fa2521`.)
kayg commented 2026-11-04 16:02:53 +00:00
Author
Owner
Copy link

perf: list first page is flat vs dev (head 9c8789e7)

Where the time went (profile by stage logs on dev-01 + SQLite plan/timing on the same store; perf is not allowed on these VMs, kernel perf_event_paranoid=3). The join was only part of it. In the server's own notes.page.candidates stage, query_us was ~+80-120 µs on 2d51718a:

  1. the per-row LEFT JOIN note_visibility probe (+20 µs);
  2. the total with task notes: COUNT(*) FROM note_visibility v WHERE v.id NOT IN (SELECT id FROM tasks) ran a subquery per flagged row: 70 µs per page (SQLite: 70.5 µs vs 1.8 µs for dev's count);
  3. SELECT EXISTS(...) + COUNT(*) of flagged rows on every page, and count queries re-compiled on every call (query_row is not cached);
  4. two extra JSON fields (hidden, archived) on every row of every page.
    File reads/decoding were not involved (page is handed no path).

Fix (commits bcc68363 column + partial index, 67fa2521 upgrade path, 21ef14e3 totals, 6412d060 no flag bytes on plain rows, 1f222a24 cached counts, 9c8789e7 cached per-note statements): flag stored as notes.flagged at index/save time + partial index notes_open (updated DESC, id) WHERE flagged = 0 that the default page names; flagged counts are two stats rows kept by triggers; totals with tasks/tags walk the shorter side; a plain row is written without hidden/archived (clients already read a missing flag as false); count and visibility statements are compiled once. Show hidden, Archived and the other orders keep their previous SQL. Existing list stores upgrade in place (test a_store_without_the_flagged_column_keeps_its_flags). Plan test: the default page has no note_visibility in its SQL, uses n.flagged = 0 / notes_open, no scan of v, no sort, no id set; page still takes no path, so it cannot read a note.

Perf, perf-01, both perf locks held for the whole run (stopped afterwards), release builds sha256-checked on both VMs, throwaway probe (not committed), 6 ABAB-alternating rounds, dev = 41f2efa3, dev2 = the same dev binary again (control), new4 = 9c8789e7, old = 2d51718a. Each round: first start with indexing, restart, 200 warm requests per list/search call, 30 single + 5 batches of 100 archive/unarchive. Median over rounds, ms (min-max).

4,000 notes, 25% archived, 10% hidden

metric dev dev2 new4 old
list.first_page.p50 0.606 (0.59-0.63) 0.606 (0.60-0.62) 0.604 (0.57-0.64) 0.751 (0.73-0.76)
list.first_page.p95 0.656 (0.64-0.80) 0.679 (0.66-0.70) 0.674 (0.61-0.72) 0.801 (0.79-0.84)
list.archived.p50 n/a n/a 3.164 (3.12-3.28) 3.171 (3.15-3.18)
search.query.p50 0.769 (0.74-0.78) 0.767 (0.75-0.77) 0.771 (0.76-0.81) 0.768 (0.75-0.78)
search.query.p95 0.912 (0.84-0.95) 0.946 (0.88-0.98) 0.828 (0.81-0.97) 0.809 (0.80-0.96)
archive1.p50 n/a n/a 94.516 (83.33-112.98) 86.225 (84.72-108.10)
archive100.p50 n/a n/a 2638.972 (2581.81-3338.41) 2676.991 (2655.21-3518.24)
unarchive100.p50 n/a n/a 2431.168 (2389.93-2486.25) 2461.478 (2392.29-3221.44)
first_start.index_ms 1350.997 (1267.37-1436.84) 1349.798 (1263.70-1534.71) 1290.924 (1239.18-1480.56) 1112.765 (1099.62-1207.01)
restart.ready_ms 87.934 (60.21-114.62) 112.640 (60.37-116.70) 89.396 (60.55-113.32) 87.447 (60.59-113.17)

paired rounds where A is faster than B (lower is better):

  • new4 vs dev: first_page.p50 4/6, first_page.p95 3/6, archived.p50 0/0, search.query.p50 2/6, search.query.p95 5/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 5/6
  • dev2 vs dev: first_page.p50 3/6, first_page.p95 1/6, archived.p50 0/0, search.query.p50 3/6, search.query.p95 1/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 3/6
  • new4 vs dev2: first_page.p50 3/6, first_page.p95 3/6, archived.p50 0/0, search.query.p50 1/6, search.query.p95 5/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 3/6
  • new4 vs old: first_page.p50 6/6, first_page.p95 6/6, archived.p50 3/6, search.query.p50 2/6, search.query.p95 2/6, archive1.p50 2/6, archive100.p50 5/6, unarchive100.p50 5/6, first_start.index_ms 0/6

paired median delta of list.first_page.p50 (A - B, ms and %):

  • new4 - dev: -0.003 ms (-0.5%), range -0.048..+0.027
  • dev2 - dev: -0.001 ms (-0.2%), range -0.015..+0.013
  • new4 - dev2: -0.002 ms (-0.3%), range -0.043..+0.033
  • new4 - old: -0.154 ms (-20.3%), range -0.164..-0.106

10,000 notes, 2% archived, 4% hidden (sparse)

metric dev dev2 new4
list.first_page.p50 0.633 (0.61-0.64) 0.609 (0.59-0.65) 0.599 (0.54-0.63)
list.first_page.p95 0.720 (0.69-0.77) 0.681 (0.63-0.76) 0.662 (0.61-0.77)
list.archived.p50 n/a n/a 0.871 (0.86-0.95)
search.query.p50 0.792 (0.78-0.81) 0.771 (0.76-0.80) 0.787 (0.76-0.81)
search.query.p95 0.848 (0.83-0.99) 0.842 (0.81-0.88) 0.863 (0.81-0.97)
archive1.p50 n/a n/a 84.507 (83.51-103.10)
archive100.p50 n/a n/a 2623.191 (2576.58-2705.75)
unarchive100.p50 n/a n/a 2435.951 (2398.27-2562.32)
first_start.index_ms 2799.408 (2667.79-2942.91) 2810.021 (2699.67-2912.33) 3288.683 (3220.75-3697.59)
restart.ready_ms 157.675 (152.10-165.35) 157.664 (156.14-162.40) 156.012 (114.87-166.95)

paired rounds where A is faster than B (lower is better):

  • new4 vs dev: first_page.p50 6/6, first_page.p95 5/6, archived.p50 0/0, search.query.p50 4/6, search.query.p95 3/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 0/6
  • dev2 vs dev: first_page.p50 5/6, first_page.p95 4/6, archived.p50 0/0, search.query.p50 5/6, search.query.p95 3/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 3/6
  • new4 vs dev2: first_page.p50 5/6, first_page.p95 3/6, archived.p50 0/0, search.query.p50 2/6, search.query.p95 3/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 0/6

paired median delta of list.first_page.p50 (A - B, ms and %):

  • new4 - dev: -0.030 ms (-4.9%), range -0.093..-0.010
  • dev2 - dev: -0.021 ms (-3.4%), range -0.038..+0.011
  • new4 - dev2: -0.012 ms (-2.1%), range -0.069..+0.012

Earlier heads, each measured against dev in its own ABAB session (list first page p50, new - dev): 67fa2521 +0.12 ms (+20%) in the 4,000-note store and +0.063 ms in the sparse one; 21ef14e3 +0.052 ms (4,000); 6412d060 +0.006 ms (4,000, flat) but +0.043 ms (+6.9%, 0/6 paired wins) in the sparse 10,000 store. The last two commits (cached count statements, no flag bytes) closed that.

Verdict on list first page: flat. 4,000 notes: new4 - dev = -0.003 ms (-0.5%, paired range -0.048..+0.027), control dev2 - dev = -0.001 ms. 10,000 sparse: new4 - dev = -0.030 ms (-4.9%, 6/6 faster), new4 - dev2 = -0.012 ms (-2.1%, 5/6); the control itself is -3.4% from dev there, so read it as flat-to-faster, not as a win. Archive one / archive 100 / unarchive 100, the archived list and search are flat against 2d51718a (see the tables; new4 list first page is 6/6 faster than 2d51718a).

Still open, not fixed here: first start + index at 10,000 notes is slower than dev (3,289 ms vs 2,799 ms, +17%, 0/6 paired wins; at 4,000 notes, dense, it is flat or faster because archived notes are not search-indexed). Debug stage timing on perf-01 (4 rounds, dev vs new4, 10,000 sparse): list ready 623 -> 654 ms, meta.apply 247 -> 260 ms (that part was +130 ms before 9c8789e7 compiled the per-note visibility statements once), meta.indexes 51 -> 55 ms; the rest (~+420 ms) is in the search indexing phase of the first build (per note: block_visibility::ids + hidden-line filter in describe_with_body, searchable_source, searchable column on source_versions). The search crate logs no stage timings, and perf counters are off, so I could not split it further in the time the perf locks were mine. It needs its own fix before #887 can merge under the no-regression rule.

Gates (dev-01, ~/bin/slot, -j4, head 9c8789e7): cargo fmt --all --check 0; cargo clippy --workspace --all-targets --locked -j4 -- -D warnings 0; cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf -p calternal: exit 0, 809 passed, 0 failed, 13 ignored. Release binary of 9c8789e7: sha256 e2aa89ee...a00.

## perf: list first page is flat vs dev (head `9c8789e7`) **Where the time went** (profile by stage logs on dev-01 + SQLite plan/timing on the same store; `perf` is not allowed on these VMs, kernel `perf_event_paranoid=3`). The join was only part of it. In the server's own `notes.page.candidates` stage, `query_us` was ~+80-120 µs on `2d51718a`: 1. the per-row `LEFT JOIN note_visibility` probe (+20 µs); 2. the total with task notes: `COUNT(*) FROM note_visibility v WHERE v.id NOT IN (SELECT id FROM tasks)` ran a subquery per flagged row: **70 µs** per page (SQLite: 70.5 µs vs 1.8 µs for dev's count); 3. `SELECT EXISTS(...)` + `COUNT(*)` of flagged rows on every page, and count queries re-compiled on every call (`query_row` is not cached); 4. two extra JSON fields (`hidden`, `archived`) on every row of every page. File reads/decoding were not involved (`page` is handed no path). **Fix** (commits `bcc68363` column + partial index, `67fa2521` upgrade path, `21ef14e3` totals, `6412d060` no flag bytes on plain rows, `1f222a24` cached counts, `9c8789e7` cached per-note statements): flag stored as `notes.flagged` at index/save time + partial index `notes_open (updated DESC, id) WHERE flagged = 0` that the default page names; flagged counts are two `stats` rows kept by triggers; totals with tasks/tags walk the shorter side; a plain row is written without `hidden`/`archived` (clients already read a missing flag as false); count and visibility statements are compiled once. Show hidden, Archived and the other orders keep their previous SQL. Existing list stores upgrade in place (test `a_store_without_the_flagged_column_keeps_its_flags`). Plan test: the default page has no `note_visibility` in its SQL, uses `n.flagged = 0` / `notes_open`, no scan of `v`, no sort, no id set; `page` still takes no path, so it cannot read a note. **Perf**, perf-01, both perf locks held for the whole run (stopped afterwards), release builds sha256-checked on both VMs, throwaway probe (not committed), 6 ABAB-alternating rounds, `dev` = `41f2efa3`, `dev2` = the same dev binary again (control), `new4` = `9c8789e7`, `old` = `2d51718a`. Each round: first start with indexing, restart, 200 warm requests per list/search call, 30 single + 5 batches of 100 archive/unarchive. Median over rounds, ms (min-max). ### 4,000 notes, 25% archived, 10% hidden | metric | dev | dev2 | new4 | old | |---|---|---|---|---| | list.first_page.p50 | 0.606 (0.59-0.63) | 0.606 (0.60-0.62) | 0.604 (0.57-0.64) | 0.751 (0.73-0.76) | | list.first_page.p95 | 0.656 (0.64-0.80) | 0.679 (0.66-0.70) | 0.674 (0.61-0.72) | 0.801 (0.79-0.84) | | list.archived.p50 | n/a | n/a | 3.164 (3.12-3.28) | 3.171 (3.15-3.18) | | search.query.p50 | 0.769 (0.74-0.78) | 0.767 (0.75-0.77) | 0.771 (0.76-0.81) | 0.768 (0.75-0.78) | | search.query.p95 | 0.912 (0.84-0.95) | 0.946 (0.88-0.98) | 0.828 (0.81-0.97) | 0.809 (0.80-0.96) | | archive1.p50 | n/a | n/a | 94.516 (83.33-112.98) | 86.225 (84.72-108.10) | | archive100.p50 | n/a | n/a | 2638.972 (2581.81-3338.41) | 2676.991 (2655.21-3518.24) | | unarchive100.p50 | n/a | n/a | 2431.168 (2389.93-2486.25) | 2461.478 (2392.29-3221.44) | | first_start.index_ms | 1350.997 (1267.37-1436.84) | 1349.798 (1263.70-1534.71) | 1290.924 (1239.18-1480.56) | 1112.765 (1099.62-1207.01) | | restart.ready_ms | 87.934 (60.21-114.62) | 112.640 (60.37-116.70) | 89.396 (60.55-113.32) | 87.447 (60.59-113.17) | paired rounds where A is faster than B (lower is better): - new4 vs dev: first_page.p50 4/6, first_page.p95 3/6, archived.p50 0/0, search.query.p50 2/6, search.query.p95 5/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 5/6 - dev2 vs dev: first_page.p50 3/6, first_page.p95 1/6, archived.p50 0/0, search.query.p50 3/6, search.query.p95 1/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 3/6 - new4 vs dev2: first_page.p50 3/6, first_page.p95 3/6, archived.p50 0/0, search.query.p50 1/6, search.query.p95 5/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 3/6 - new4 vs old: first_page.p50 6/6, first_page.p95 6/6, archived.p50 3/6, search.query.p50 2/6, search.query.p95 2/6, archive1.p50 2/6, archive100.p50 5/6, unarchive100.p50 5/6, first_start.index_ms 0/6 paired median delta of list.first_page.p50 (A - B, ms and %): - new4 - dev: -0.003 ms (-0.5%), range -0.048..+0.027 - dev2 - dev: -0.001 ms (-0.2%), range -0.015..+0.013 - new4 - dev2: -0.002 ms (-0.3%), range -0.043..+0.033 - new4 - old: -0.154 ms (-20.3%), range -0.164..-0.106 ### 10,000 notes, 2% archived, 4% hidden (sparse) | metric | dev | dev2 | new4 | |---|---|---|---| | list.first_page.p50 | 0.633 (0.61-0.64) | 0.609 (0.59-0.65) | 0.599 (0.54-0.63) | | list.first_page.p95 | 0.720 (0.69-0.77) | 0.681 (0.63-0.76) | 0.662 (0.61-0.77) | | list.archived.p50 | n/a | n/a | 0.871 (0.86-0.95) | | search.query.p50 | 0.792 (0.78-0.81) | 0.771 (0.76-0.80) | 0.787 (0.76-0.81) | | search.query.p95 | 0.848 (0.83-0.99) | 0.842 (0.81-0.88) | 0.863 (0.81-0.97) | | archive1.p50 | n/a | n/a | 84.507 (83.51-103.10) | | archive100.p50 | n/a | n/a | 2623.191 (2576.58-2705.75) | | unarchive100.p50 | n/a | n/a | 2435.951 (2398.27-2562.32) | | first_start.index_ms | 2799.408 (2667.79-2942.91) | 2810.021 (2699.67-2912.33) | 3288.683 (3220.75-3697.59) | | restart.ready_ms | 157.675 (152.10-165.35) | 157.664 (156.14-162.40) | 156.012 (114.87-166.95) | paired rounds where A is faster than B (lower is better): - new4 vs dev: first_page.p50 6/6, first_page.p95 5/6, archived.p50 0/0, search.query.p50 4/6, search.query.p95 3/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 0/6 - dev2 vs dev: first_page.p50 5/6, first_page.p95 4/6, archived.p50 0/0, search.query.p50 5/6, search.query.p95 3/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 3/6 - new4 vs dev2: first_page.p50 5/6, first_page.p95 3/6, archived.p50 0/0, search.query.p50 2/6, search.query.p95 3/6, archive1.p50 0/0, archive100.p50 0/0, unarchive100.p50 0/0, first_start.index_ms 0/6 paired median delta of list.first_page.p50 (A - B, ms and %): - new4 - dev: -0.030 ms (-4.9%), range -0.093..-0.010 - dev2 - dev: -0.021 ms (-3.4%), range -0.038..+0.011 - new4 - dev2: -0.012 ms (-2.1%), range -0.069..+0.012 Earlier heads, each measured against dev in its own ABAB session (list first page p50, new - dev): `67fa2521` +0.12 ms (+20%) in the 4,000-note store and +0.063 ms in the sparse one; `21ef14e3` +0.052 ms (4,000); `6412d060` +0.006 ms (4,000, flat) but **+0.043 ms (+6.9%, 0/6 paired wins) in the sparse 10,000 store**. The last two commits (cached count statements, no flag bytes) closed that. **Verdict on list first page: flat.** 4,000 notes: new4 - dev = -0.003 ms (-0.5%, paired range -0.048..+0.027), control dev2 - dev = -0.001 ms. 10,000 sparse: new4 - dev = -0.030 ms (-4.9%, 6/6 faster), new4 - dev2 = -0.012 ms (-2.1%, 5/6); the control itself is -3.4% from dev there, so read it as flat-to-faster, not as a win. Archive one / archive 100 / unarchive 100, the archived list and search are flat against `2d51718a` (see the tables; new4 list first page is 6/6 faster than `2d51718a`). **Still open, not fixed here: first start + index at 10,000 notes is slower than dev** (3,289 ms vs 2,799 ms, +17%, 0/6 paired wins; at 4,000 notes, dense, it is flat or faster because archived notes are not search-indexed). Debug stage timing on perf-01 (4 rounds, dev vs new4, 10,000 sparse): list ready 623 -> 654 ms, `meta.apply` 247 -> 260 ms (that part was +130 ms before `9c8789e7` compiled the per-note visibility statements once), `meta.indexes` 51 -> 55 ms; the rest (~+420 ms) is in the search indexing phase of the first build (per note: `block_visibility::ids` + hidden-line filter in `describe_with_body`, `searchable_source`, `searchable` column on `source_versions`). The search crate logs no stage timings, and perf counters are off, so I could not split it further in the time the perf locks were mine. It needs its own fix before #887 can merge under the no-regression rule. **Gates** (dev-01, `~/bin/slot`, `-j4`, head `9c8789e7`): `cargo fmt --all --check` 0; `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` 0; `cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf -p calternal`: exit 0, 809 passed, 0 failed, 13 ignored. Release binary of `9c8789e7`: sha256 `e2aa89ee...a00`.
kayg commented 2026-11-04 16:24:25 +00:00
Author
Owner
Copy link

first-build indexing: root cause and implementation plan

SQLite timing on perf-01 with both perf locks held, six alternating rounds, 1,000 primary-key source fetches per case from disposable 10,000-note stores: dev's first query on each fresh connection takes median 229.7 ms total; head 9c8789e7 takes 271.6 ms. The PR database has 40 schema entries vs dev's 33. Removing just visibility_added, visibility_removed, visibility_changed, visibility_hidden, visibility_archived, and notes_open from a disposable copy reduces that stage to 237.3 ms, without changing the sources or the searchable column/filter. JSON decode is ~7–9 ms and connection open ~50–55 ms per 1,000. This identifies repeated schema parsing on a fresh per-note connection, rather than per-note file reads, trigger execution during search, or the searchable predicate itself. At 10,000 notes the extra first-query cost is ~420 ms, matching the earlier profile. Both stores contain 10,000 source rows. The query is the existing primary-key SELECT source ... WHERE id = ? AND born <= epoch AND (ended IS NULL OR ended > epoch); PR adds AND searchable = 1.

Plan: add an optional bounded batch document read to SearchProvider; providers without it keep the existing parallel single-item fallback. Notes reads a batch through one transient connection and prepared statement, applies the same archived predicate, and derives the same search bodies. The indexer preserves input order, removals, atomic publication and validation. No schema/trigger/counter changes, no limits changed, no permanent connection cache.

Sizes/complexity: current cold batches stay at 64, paced batches at 8; O(batch) source/document memory and O(notes) total work. Connections/schema parses fall from O(notes) to O(batches). Source reads hold neither a writer lock nor a transaction across awaits.

failure effect detection recovery test
expired generation batch refused existing before/after token checks retry from new snapshot retained-generation and batch snapshot tests
archived/missing item no document same searchable predicate stage removal batch parity test
provider returns incomplete batch refuse publication length validation provider retry indexer batch test
batch read error/cancellation old index retained staged transaction/publish boundary retry existing atomic publication tests + batch failure test

Gates: dev-01 via ~/bin/slot, fmt, workspace clippy all-targets -D warnings, notes/tasks/core/engine/search/perf/calternal tests. Perf-01 with both locks, ≥6 alternating rounds and dev-vs-dev control at 4,000 dense / 10,000 sparse, first build, list, search and archive paths. Artifacts remain in PR comments. No merge.

## first-build indexing: root cause and implementation plan SQLite timing on perf-01 with both perf locks held, six alternating rounds, 1,000 primary-key source fetches per case from disposable 10,000-note stores: dev's first query on each fresh connection takes median 229.7 ms total; head `9c8789e7` takes 271.6 ms. The PR database has 40 schema entries vs dev's 33. Removing just `visibility_added`, `visibility_removed`, `visibility_changed`, `visibility_hidden`, `visibility_archived`, and `notes_open` from a disposable copy reduces that stage to 237.3 ms, without changing the sources or the `searchable` column/filter. JSON decode is ~7–9 ms and connection open ~50–55 ms per 1,000. This identifies repeated schema parsing on a fresh per-note connection, rather than per-note file reads, trigger execution during search, or the searchable predicate itself. At 10,000 notes the extra first-query cost is ~420 ms, matching the earlier profile. Both stores contain 10,000 source rows. The query is the existing primary-key `SELECT source ... WHERE id = ? AND born <= epoch AND (ended IS NULL OR ended > epoch)`; PR adds `AND searchable = 1`. Plan: add an optional bounded batch document read to SearchProvider; providers without it keep the existing parallel single-item fallback. Notes reads a batch through one transient connection and prepared statement, applies the same archived predicate, and derives the same search bodies. The indexer preserves input order, removals, atomic publication and validation. No schema/trigger/counter changes, no limits changed, no permanent connection cache. Sizes/complexity: current cold batches stay at 64, paced batches at 8; O(batch) source/document memory and O(notes) total work. Connections/schema parses fall from O(notes) to O(batches). Source reads hold neither a writer lock nor a transaction across awaits. | failure | effect | detection | recovery | test | |---|---|---|---|---| | expired generation | batch refused | existing before/after token checks | retry from new snapshot | retained-generation and batch snapshot tests | | archived/missing item | no document | same searchable predicate | stage removal | batch parity test | | provider returns incomplete batch | refuse publication | length validation | provider retry | indexer batch test | | batch read error/cancellation | old index retained | staged transaction/publish boundary | retry | existing atomic publication tests + batch failure test | Gates: dev-01 via `~/bin/slot`, fmt, workspace clippy all-targets `-D warnings`, notes/tasks/core/engine/search/perf/calternal tests. Perf-01 with both locks, ≥6 alternating rounds and dev-vs-dev control at 4,000 dense / 10,000 sparse, first build, list, search and archive paths. Artifacts remain in PR comments. No merge.
kayg commented 2026-11-04 16:33:35 +00:00
Author
Owner
Copy link

fix pushed: b4e87385bf0d1bb8db3985737ec88b82a864068d

perf(search): fetch accepted note sources in bounded batches adds the optional provider batch read, uses one transient source connection/prepared statement per Notes batch, and leaves providers without a batch reader on parallel single-item reads. Cold batches remain 64 and paced batches 8. Archived sources remain available outside search; missing/archived items retain None slots; order, duplicates, snapshot expiry, staged removals and atomic publication are preserved. Source/body parsing runs on blocking workers. No schema changes, trigger skipping, counter changes, raised limits, or temporary timing code.

The focused red run failed on the missing batch API, as expected; the green run passes both new tests. Regression coverage checks single/batch parity, archived and missing entries, duplicate IDs, retained generations, expiry, removal of a missing document, and refusal of an incomplete provider result before publication. Existing failed/cancelled publication tests also pass.

dev-01 gates, Nix dev shell, all heavy commands via ~/bin/slot, head b4e87385:

command result
cargo fmt --all --check exit 0
cargo clippy --workspace --all-targets --locked -j4 -- -D warnings exit 0
cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf -p calternal exit 0, 811 passed, 0 failed, 13 ignored

Additional SQL control on the same PR database: 1,000 source primary-key lookups with a shared connection take 4.44–5.13 ms over six rounds (median 4.53 ms), versus ~272 ms with per-item connection/schema setup. This is diagnostic SQL timing, not a substitute for the release end-to-end matrix. Release build and perf matrix are next; no first-build verdict claimed yet.

## fix pushed: `b4e87385bf0d1bb8db3985737ec88b82a864068d` `perf(search): fetch accepted note sources in bounded batches` adds the optional provider batch read, uses one transient source connection/prepared statement per Notes batch, and leaves providers without a batch reader on parallel single-item reads. Cold batches remain 64 and paced batches 8. Archived sources remain available outside search; missing/archived items retain `None` slots; order, duplicates, snapshot expiry, staged removals and atomic publication are preserved. Source/body parsing runs on blocking workers. No schema changes, trigger skipping, counter changes, raised limits, or temporary timing code. The focused red run failed on the missing batch API, as expected; the green run passes both new tests. Regression coverage checks single/batch parity, archived and missing entries, duplicate IDs, retained generations, expiry, removal of a missing document, and refusal of an incomplete provider result before publication. Existing failed/cancelled publication tests also pass. **dev-01 gates**, Nix dev shell, all heavy commands via `~/bin/slot`, head `b4e87385`: | command | result | |---|---| | `cargo fmt --all --check` | exit 0 | | `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` | exit 0 | | `cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf -p calternal` | exit 0, 811 passed, 0 failed, 13 ignored | Additional SQL control on the same PR database: 1,000 source primary-key lookups with a shared connection take 4.44–5.13 ms over six rounds (median 4.53 ms), versus ~272 ms with per-item connection/schema setup. This is diagnostic SQL timing, not a substitute for the release end-to-end matrix. Release build and perf matrix are next; no first-build verdict claimed yet.
kayg commented 2026-11-04 16:58:03 +00:00
Author
Owner
Copy link

first-build regression fixed: b4e87385bf0d1bb8db3985737ec88b82a864068d

Finding. The dominant added search-indexing cost was parsing the larger meta.db schema for every note. Snapshot::read opened a fresh connection per source, whose first SELECT source parsed all schema objects: 40 on #887 versus 33 on dev. Six-round SQLite controls measured ~272 ms per 1,000 first queries on #887 versus ~230 ms on dev. Removing only the added visibility triggers/indexes from a disposable copy reduced this to ~237 ms without changing sources or searchable. These are read-only queries: the triggers are parsed here, not executed. This explains roughly the earlier +420 ms at 10,000 notes. The flags were already computed in the accepted metadata pass, and archived rows already filtered by their stored searchable bit; there was no extra note-file read in this phase. Hidden-block body filtering remains unchanged. The notes_open partial index is already dropped during a fresh 10,000-note fill and rebuilt afterward, so maintaining it row by row was not the missing search-phase cost.

Fix. The indexer asks Notes for its existing bounded batch through an optional SearchProvider::docs method. One transient connection and prepared statement serve the batch; archived/missing rows keep aligned None entries. Existing parallel single-document reads remain the fallback for other providers. Cold batches remain 64, paced batches 8; validation, generation retention, removals, ordering and atomic publication remain intact. Body generation runs on blocking workers. The schema, counters, triggers, visibility grammar and accepted-v2 stamp are unchanged. Diagnostic same-connection timing is 4.44–5.13 ms per 1,000 source lookups, median 4.53 ms.

Gates. Exact commands and results: dev-01, Nix, ~/bin/slot, fmt check and workspace all-targets clippy -D warnings both exit 0; notes/tasks/core/engine/search/perf/calternal tests exit 0, 811 passed, 0 failed, 13 ignored. Release build: ~/bin/slot cargo build --release --locked -j4 -p calternal, exit 0. Both new tests first failed on the absent batch API, then passed with the implementation. No timing assertions added to functional tests.

Perf method. perf-01 only, both perf VM locks held together throughout SQL profiling, the main matrix and the tail control. Six paired alternating rounds per fixture: [dev, dev2, before, new], reversed on the next round, fresh Home/index for every main run. dev2 is the identical dev binary (control). Same fixture bytes across cases. Exact populations: 4,000 = 1,000 archived + 400 hidden + 2,600 plain; 10,000 = 200 archived + 400 hidden + 9,400 plain. Every case has all N accepted source rows; dev indexes all N search documents, #887 indexes N minus archived, as required. Each round includes first start/index, restart, warm list/search requests, 30 single archive/unarchive pairs and five 100-note archive/unarchive pairs where supported. Dev has no Archive actions or Archived place, so those cells are n/a; before is the behavior-equivalent comparison for those paths. Figures are median (min–max); paired tables use the median of per-round differences and percentages, with wins out of six.

Binary provenance (release SHA-256 verified through control):

  • dev / dev2: 41f2efa387b74175971a838f602535089d0d5fb4, ef22c21cabe7711730ce68c4a16fc473ec55c042aa9b4bb9a64211335d3780a7 (matches the dev-01 pr887-perf-base build).
  • before: 9c8789e725451ca0e2b50c5cb1211a15ebd590ca, e2aa89eed0439ab68e87aa554ce294313d2b32afb8eda01a5dab82208463ca00.
  • new: b4e87385bf0d1bb8db3985737ec88b82a864068d, 05707b663717a077655ce6160e9e937b19a95f324685e26fb34ca8cf2a192c52 (matches dev-01, control and perf-01).

First-build verdict: faster than dev, 6/6 paired wins at both sizes. 4,000: 1,338.420 → 561.205 ms, paired −58.1%. 10,000: 2,785.148 → 1,065.816 ms, paired −61.7%. The previous #887 head still reproduces the sparse regression here (3,320.306 ms), so the fixture continues to detect it.

Other paths: flat within the controls, or faster. Sparse first-page p95's paired +0.007 ms (+1.0%) is smaller than the same-binary dev control's +0.015 ms (+2.1%); restart +1.7% is likewise below its +2.4% control. Search and bulk archive/unarchive show no slowdown. Archived-list p95 in the main matrix was +2.5%/+4.1% against before; the larger query-only control below did not reproduce that: −1.1%/+0.3%, 3/6 wins at both sizes. The independent same-binary archived control itself differs by −4.6% at 4,000 notes. Against that faster duplicate control the new head is +4.4%, 0/6 wins; this is disclosed below rather than choosing only the favorable baseline. The variation also exists between identical old binaries, so this does not isolate a code regression. Archived-list p50 is +0.4% (0.012/0.004 ms) against the original baseline. All deltas are retained below.

Both holder units were stopped and verified inactive after all measurements. All 48 main rows and 36 tail-control rows were checked; both jobs exited 0. Seeded Homes and the copied candidate binary were removed; only small probe/report files remain outside the repository. No merge.

4,000 notes, 25% archived / 10% hidden

metric (ms) dev dev2 control before new
first_start.index_ms 1338.420 (1261.81–1451.13) 1328.403 (1301.31–1409.51) 1361.888 (1269.03–1575.08) 561.205 (553.79–562.37)
restart.ready_ms 60.820 (60.54–114.08) 87.546 (60.56–115.17) 88.246 (63.58–114.09) 61.240 (60.28–112.25)
list.first_page.p50 0.620 (0.58–0.63) 0.600 (0.59–0.63) 0.598 (0.59–0.61) 0.596 (0.58–0.61)
list.first_page.p95 0.697 (0.63–0.73) 0.664 (0.63–0.74) 0.668 (0.61–0.74) 0.656 (0.62–0.68)
list.archived.p50 n/a n/a 3.041 (3.02–3.10) 3.045 (3.00–3.08)
list.archived.p95 n/a n/a 3.437 (3.21–3.64) 3.473 (3.19–3.72)
search.query.p50 0.773 (0.76–0.82) 0.763 (0.75–0.79) 0.759 (0.75–0.80) 0.753 (0.74–0.77)
search.query.p95 0.966 (0.93–1.18) 0.908 (0.88–0.95) 0.814 (0.80–1.18) 0.805 (0.78–1.02)
archive1.p50 n/a n/a 87.230 (82.33–112.10) 87.703 (86.90–90.27)
archive1.p95 n/a n/a 103.120 (101.60–138.63) 104.596 (99.01–119.85)
unarchive1.p50 n/a n/a 86.371 (84.64–111.75) 86.375 (84.93–90.20)
archive100.p50 n/a n/a 2674.181 (2643.68–3466.61) 2596.798 (2571.30–2646.03)
archive100.max n/a n/a 2961.878 (2795.09–3635.71) 2794.999 (2769.72–3555.31)
unarchive100.p50 n/a n/a 2455.188 (2438.46–3100.25) 2433.288 (2408.73–2644.80)

Paired median deltas and wins (lower is better):

metric new − dev dev2 − dev control new − before
first_start.index_ms -777.388 ms / -58.1%; 6/6 +0.485 ms / +0.0%; 3/6 -804.358 ms / -59.1%; 6/6
restart.ready_ms -0.162 ms / -0.3%; 4/6 +1.282 ms / +1.9%; 1/6 -2.545 ms / -4.0%; 5/6
list.first_page.p50 -0.021 ms / -3.4%; 5/6 -0.007 ms / -1.1%; 3/6 -0.002 ms / -0.3%; 3/6
list.first_page.p95 -0.035 ms / -5.1%; 6/6 -0.005 ms / -0.7%; 3/6 -0.016 ms / -2.5%; 5/6
list.archived.p50 n/a n/a -0.013 ms / -0.4%; 4/6
list.archived.p95 n/a n/a +0.081 ms / +2.5%; 2/6
search.query.p50 -0.021 ms / -2.8%; 6/6 -0.009 ms / -1.1%; 5/6 -0.010 ms / -1.3%; 6/6
search.query.p95 -0.146 ms / -15.6%; 5/6 -0.058 ms / -6.0%; 4/6 -0.008 ms / -1.0%; 5/6
archive1.p50 n/a n/a +0.359 ms / +0.4%; 3/6
archive1.p95 n/a n/a +0.335 ms / +0.4%; 3/6
unarchive1.p50 n/a n/a -0.799 ms / -0.9%; 5/6
archive100.p50 n/a n/a -66.466 ms / -2.5%; 6/6
archive100.max n/a n/a -64.270 ms / -2.0%; 5/6
unarchive100.p50 n/a n/a -24.358 ms / -1.0%; 6/6

10,000 notes, 2% archived / 4% hidden

metric (ms) dev dev2 control before new
first_start.index_ms 2785.148 (2691.60–2939.76) 2821.547 (2787.97–2922.19) 3320.306 (3162.41–3542.53) 1065.816 (1062.94–1074.20)
restart.ready_ms 154.504 (152.28–162.17) 158.685 (154.70–168.48) 158.279 (156.11–165.53) 156.947 (151.33–159.72)
list.first_page.p50 0.621 (0.60–0.63) 0.618 (0.59–0.64) 0.601 (0.58–0.62) 0.602 (0.59–0.65)
list.first_page.p95 0.690 (0.65–0.83) 0.716 (0.66–0.83) 0.663 (0.63–0.73) 0.684 (0.65–0.76)
list.archived.p50 n/a n/a 1.147 (1.13–1.18) 1.154 (1.15–1.18)
list.archived.p95 n/a n/a 1.293 (1.23–1.39) 1.323 (1.26–1.48)
search.query.p50 0.790 (0.75–0.83) 0.781 (0.77–0.84) 0.786 (0.76–0.80) 0.778 (0.77–0.78)
search.query.p95 0.868 (0.83–0.91) 0.860 (0.82–0.97) 0.858 (0.83–0.94) 0.829 (0.81–0.85)
archive1.p50 n/a n/a 86.676 (82.89–100.93) 85.081 (84.08–87.07)
archive1.p95 n/a n/a 117.131 (101.41–125.42) 101.043 (99.54–103.81)
unarchive1.p50 n/a n/a 86.804 (83.67–98.91) 86.378 (85.00–87.81)
archive100.p50 n/a n/a 2632.628 (2606.37–2701.45) 2610.460 (2558.89–2654.74)
archive100.max n/a n/a 2930.050 (2853.80–3256.42) 2803.243 (2728.24–2857.15)
unarchive100.p50 n/a n/a 2466.086 (2373.61–2485.54) 2362.177 (2316.55–2381.13)

Paired median deltas and wins (lower is better):

metric new − dev dev2 − dev control new − before
first_start.index_ms -1719.821 ms / -61.7%; 6/6 +28.815 ms / +1.1%; 3/6 -2251.735 ms / -67.8%; 6/6
restart.ready_ms +2.662 ms / +1.7%; 1/6 +3.869 ms / +2.4%; 1/6 -0.302 ms / -0.2%; 4/6
list.first_page.p50 -0.016 ms / -2.6%; 5/6 -0.003 ms / -0.5%; 4/6 +0.006 ms / +1.1%; 2/6
list.first_page.p95 +0.007 ms / +1.0%; 2/6 +0.015 ms / +2.1%; 1/6 +0.022 ms / +3.2%; 1/6
list.archived.p50 n/a n/a +0.005 ms / +0.4%; 2/6
list.archived.p95 n/a n/a +0.051 ms / +4.1%; 2/6
search.query.p50 -0.014 ms / -1.7%; 4/6 -0.002 ms / -0.3%; 4/6 -0.005 ms / -0.7%; 3/6
search.query.p95 -0.037 ms / -4.3%; 5/6 -0.004 ms / -0.4%; 4/6 -0.027 ms / -3.1%; 5/6
archive1.p50 n/a n/a -2.002 ms / -2.3%; 4/6
archive1.p95 n/a n/a -14.745 ms / -12.6%; 6/6
unarchive1.p50 n/a n/a -0.583 ms / -0.7%; 4/6
archive100.p50 n/a n/a -38.483 ms / -1.4%; 5/6
archive100.max n/a n/a -83.959 ms / -2.9%; 6/6
unarchive100.p50 n/a n/a -95.850 ms / -3.9%; 6/6

archived-list tail control

Six alternating rounds, 60 warmups then 1,000 requests per case. before and before2 use the identical 9c8789e7 binary on independently seeded but logically identical stores. No archive writes during these query-only runs. All times are ms.

notes / metric before before2 control new new − before (paired) before2 − before control new − before2
4,000 / p50 3.023 (3.00–3.03) 3.016 (3.01–3.02) 3.036 (2.99–3.13) +0.012 / +0.4%; 2/6 wins -0.006 / -0.2%; 5/6 wins +0.021 / +0.7%; 2/6 wins
4,000 / p95 3.381 (3.26–3.48) 3.206 (3.17–3.25) 3.353 (3.26–3.62) -0.037 / -1.1%; 3/6 wins -0.157 / -4.6%; 6/6 wins +0.140 / +4.4%; 0/6 wins
10,000 / p50 1.138 (1.12–1.17) 1.134 (1.13–1.14) 1.143 (1.13–1.17) +0.004 / +0.4%; 3/6 wins -0.008 / -0.7%; 4/6 wins +0.015 / +1.3%; 2/6 wins
10,000 / p95 1.264 (1.22–1.50) 1.262 (1.20–1.31) 1.254 (1.22–1.50) +0.003 / +0.3%; 3/6 wins -0.002 / -0.2%; 3/6 wins +0.020 / +1.7%; 1/6 wins
## first-build regression fixed: `b4e87385bf0d1bb8db3985737ec88b82a864068d` **Finding.** The dominant added search-indexing cost was parsing the larger `meta.db` schema for every note. `Snapshot::read` opened a fresh connection per source, whose first `SELECT source` parsed all schema objects: 40 on #887 versus 33 on dev. Six-round SQLite controls measured ~272 ms per 1,000 first queries on #887 versus ~230 ms on dev. Removing only the added visibility triggers/indexes from a disposable copy reduced this to ~237 ms without changing sources or `searchable`. These are read-only queries: the triggers are parsed here, not executed. This explains roughly the earlier +420 ms at 10,000 notes. The flags were already computed in the accepted metadata pass, and archived rows already filtered by their stored `searchable` bit; there was no extra note-file read in this phase. Hidden-block body filtering remains unchanged. The `notes_open` partial index is already dropped during a fresh 10,000-note fill and rebuilt afterward, so maintaining it row by row was not the missing search-phase cost. **Fix.** The indexer asks Notes for its existing bounded batch through an optional `SearchProvider::docs` method. One transient connection and prepared statement serve the batch; archived/missing rows keep aligned `None` entries. Existing parallel single-document reads remain the fallback for other providers. Cold batches remain 64, paced batches 8; validation, generation retention, removals, ordering and atomic publication remain intact. Body generation runs on blocking workers. The schema, counters, triggers, visibility grammar and accepted-v2 stamp are unchanged. Diagnostic same-connection timing is 4.44–5.13 ms per 1,000 source lookups, median 4.53 ms. **Gates.** [Exact commands and results](https://git.kayg.org/kayg/calternal/pulls/887#issuecomment-78404): dev-01, Nix, `~/bin/slot`, fmt check and workspace all-targets clippy `-D warnings` both exit 0; notes/tasks/core/engine/search/perf/calternal tests exit 0, **811 passed, 0 failed, 13 ignored**. Release build: `~/bin/slot cargo build --release --locked -j4 -p calternal`, exit 0. Both new tests first failed on the absent batch API, then passed with the implementation. No timing assertions added to functional tests. **Perf method.** perf-01 only, both perf VM locks held together throughout SQL profiling, the main matrix and the tail control. Six paired alternating rounds per fixture: `[dev, dev2, before, new]`, reversed on the next round, fresh Home/index for every main run. `dev2` is the identical dev binary (control). Same fixture bytes across cases. Exact populations: 4,000 = 1,000 archived + 400 hidden + 2,600 plain; 10,000 = 200 archived + 400 hidden + 9,400 plain. Every case has all N accepted source rows; dev indexes all N search documents, #887 indexes N minus archived, as required. Each round includes first start/index, restart, warm list/search requests, 30 single archive/unarchive pairs and five 100-note archive/unarchive pairs where supported. Dev has no Archive actions or Archived place, so those cells are n/a; `before` is the behavior-equivalent comparison for those paths. Figures are median (min–max); paired tables use the median of per-round differences and percentages, with wins out of six. Binary provenance (release SHA-256 verified through control): - `dev` / `dev2`: `41f2efa387b74175971a838f602535089d0d5fb4`, `ef22c21cabe7711730ce68c4a16fc473ec55c042aa9b4bb9a64211335d3780a7` (matches the dev-01 `pr887-perf-base` build). - `before`: `9c8789e725451ca0e2b50c5cb1211a15ebd590ca`, `e2aa89eed0439ab68e87aa554ce294313d2b32afb8eda01a5dab82208463ca00`. - `new`: `b4e87385bf0d1bb8db3985737ec88b82a864068d`, `05707b663717a077655ce6160e9e937b19a95f324685e26fb34ca8cf2a192c52` (matches dev-01, control and perf-01). **First-build verdict: faster than dev, 6/6 paired wins at both sizes.** 4,000: 1,338.420 → 561.205 ms, paired −58.1%. 10,000: 2,785.148 → 1,065.816 ms, paired −61.7%. The previous #887 head still reproduces the sparse regression here (3,320.306 ms), so the fixture continues to detect it. **Other paths: flat within the controls, or faster.** Sparse first-page p95's paired +0.007 ms (+1.0%) is smaller than the same-binary dev control's +0.015 ms (+2.1%); restart +1.7% is likewise below its +2.4% control. Search and bulk archive/unarchive show no slowdown. Archived-list p95 in the main matrix was +2.5%/+4.1% against `before`; the larger query-only control below did not reproduce that: −1.1%/+0.3%, 3/6 wins at both sizes. The independent same-binary archived control itself differs by −4.6% at 4,000 notes. Against that faster duplicate control the new head is +4.4%, 0/6 wins; this is disclosed below rather than choosing only the favorable baseline. The variation also exists between identical old binaries, so this does not isolate a code regression. Archived-list p50 is +0.4% (0.012/0.004 ms) against the original baseline. All deltas are retained below. Both holder units were stopped and verified inactive after all measurements. All 48 main rows and 36 tail-control rows were checked; both jobs exited 0. Seeded Homes and the copied candidate binary were removed; only small probe/report files remain outside the repository. No merge. ### 4,000 notes, 25% archived / 10% hidden | metric (ms) | dev | dev2 control | before | new | |---|---|---|---|---| | first_start.index_ms | 1338.420 (1261.81–1451.13) | 1328.403 (1301.31–1409.51) | 1361.888 (1269.03–1575.08) | 561.205 (553.79–562.37) | | restart.ready_ms | 60.820 (60.54–114.08) | 87.546 (60.56–115.17) | 88.246 (63.58–114.09) | 61.240 (60.28–112.25) | | list.first_page.p50 | 0.620 (0.58–0.63) | 0.600 (0.59–0.63) | 0.598 (0.59–0.61) | 0.596 (0.58–0.61) | | list.first_page.p95 | 0.697 (0.63–0.73) | 0.664 (0.63–0.74) | 0.668 (0.61–0.74) | 0.656 (0.62–0.68) | | list.archived.p50 | n/a | n/a | 3.041 (3.02–3.10) | 3.045 (3.00–3.08) | | list.archived.p95 | n/a | n/a | 3.437 (3.21–3.64) | 3.473 (3.19–3.72) | | search.query.p50 | 0.773 (0.76–0.82) | 0.763 (0.75–0.79) | 0.759 (0.75–0.80) | 0.753 (0.74–0.77) | | search.query.p95 | 0.966 (0.93–1.18) | 0.908 (0.88–0.95) | 0.814 (0.80–1.18) | 0.805 (0.78–1.02) | | archive1.p50 | n/a | n/a | 87.230 (82.33–112.10) | 87.703 (86.90–90.27) | | archive1.p95 | n/a | n/a | 103.120 (101.60–138.63) | 104.596 (99.01–119.85) | | unarchive1.p50 | n/a | n/a | 86.371 (84.64–111.75) | 86.375 (84.93–90.20) | | archive100.p50 | n/a | n/a | 2674.181 (2643.68–3466.61) | 2596.798 (2571.30–2646.03) | | archive100.max | n/a | n/a | 2961.878 (2795.09–3635.71) | 2794.999 (2769.72–3555.31) | | unarchive100.p50 | n/a | n/a | 2455.188 (2438.46–3100.25) | 2433.288 (2408.73–2644.80) | Paired median deltas and wins (lower is better): | metric | new − dev | dev2 − dev control | new − before | |---|---|---|---| | first_start.index_ms | -777.388 ms / -58.1%; 6/6 | +0.485 ms / +0.0%; 3/6 | -804.358 ms / -59.1%; 6/6 | | restart.ready_ms | -0.162 ms / -0.3%; 4/6 | +1.282 ms / +1.9%; 1/6 | -2.545 ms / -4.0%; 5/6 | | list.first_page.p50 | -0.021 ms / -3.4%; 5/6 | -0.007 ms / -1.1%; 3/6 | -0.002 ms / -0.3%; 3/6 | | list.first_page.p95 | -0.035 ms / -5.1%; 6/6 | -0.005 ms / -0.7%; 3/6 | -0.016 ms / -2.5%; 5/6 | | list.archived.p50 | n/a | n/a | -0.013 ms / -0.4%; 4/6 | | list.archived.p95 | n/a | n/a | +0.081 ms / +2.5%; 2/6 | | search.query.p50 | -0.021 ms / -2.8%; 6/6 | -0.009 ms / -1.1%; 5/6 | -0.010 ms / -1.3%; 6/6 | | search.query.p95 | -0.146 ms / -15.6%; 5/6 | -0.058 ms / -6.0%; 4/6 | -0.008 ms / -1.0%; 5/6 | | archive1.p50 | n/a | n/a | +0.359 ms / +0.4%; 3/6 | | archive1.p95 | n/a | n/a | +0.335 ms / +0.4%; 3/6 | | unarchive1.p50 | n/a | n/a | -0.799 ms / -0.9%; 5/6 | | archive100.p50 | n/a | n/a | -66.466 ms / -2.5%; 6/6 | | archive100.max | n/a | n/a | -64.270 ms / -2.0%; 5/6 | | unarchive100.p50 | n/a | n/a | -24.358 ms / -1.0%; 6/6 | ### 10,000 notes, 2% archived / 4% hidden | metric (ms) | dev | dev2 control | before | new | |---|---|---|---|---| | first_start.index_ms | 2785.148 (2691.60–2939.76) | 2821.547 (2787.97–2922.19) | 3320.306 (3162.41–3542.53) | 1065.816 (1062.94–1074.20) | | restart.ready_ms | 154.504 (152.28–162.17) | 158.685 (154.70–168.48) | 158.279 (156.11–165.53) | 156.947 (151.33–159.72) | | list.first_page.p50 | 0.621 (0.60–0.63) | 0.618 (0.59–0.64) | 0.601 (0.58–0.62) | 0.602 (0.59–0.65) | | list.first_page.p95 | 0.690 (0.65–0.83) | 0.716 (0.66–0.83) | 0.663 (0.63–0.73) | 0.684 (0.65–0.76) | | list.archived.p50 | n/a | n/a | 1.147 (1.13–1.18) | 1.154 (1.15–1.18) | | list.archived.p95 | n/a | n/a | 1.293 (1.23–1.39) | 1.323 (1.26–1.48) | | search.query.p50 | 0.790 (0.75–0.83) | 0.781 (0.77–0.84) | 0.786 (0.76–0.80) | 0.778 (0.77–0.78) | | search.query.p95 | 0.868 (0.83–0.91) | 0.860 (0.82–0.97) | 0.858 (0.83–0.94) | 0.829 (0.81–0.85) | | archive1.p50 | n/a | n/a | 86.676 (82.89–100.93) | 85.081 (84.08–87.07) | | archive1.p95 | n/a | n/a | 117.131 (101.41–125.42) | 101.043 (99.54–103.81) | | unarchive1.p50 | n/a | n/a | 86.804 (83.67–98.91) | 86.378 (85.00–87.81) | | archive100.p50 | n/a | n/a | 2632.628 (2606.37–2701.45) | 2610.460 (2558.89–2654.74) | | archive100.max | n/a | n/a | 2930.050 (2853.80–3256.42) | 2803.243 (2728.24–2857.15) | | unarchive100.p50 | n/a | n/a | 2466.086 (2373.61–2485.54) | 2362.177 (2316.55–2381.13) | Paired median deltas and wins (lower is better): | metric | new − dev | dev2 − dev control | new − before | |---|---|---|---| | first_start.index_ms | -1719.821 ms / -61.7%; 6/6 | +28.815 ms / +1.1%; 3/6 | -2251.735 ms / -67.8%; 6/6 | | restart.ready_ms | +2.662 ms / +1.7%; 1/6 | +3.869 ms / +2.4%; 1/6 | -0.302 ms / -0.2%; 4/6 | | list.first_page.p50 | -0.016 ms / -2.6%; 5/6 | -0.003 ms / -0.5%; 4/6 | +0.006 ms / +1.1%; 2/6 | | list.first_page.p95 | +0.007 ms / +1.0%; 2/6 | +0.015 ms / +2.1%; 1/6 | +0.022 ms / +3.2%; 1/6 | | list.archived.p50 | n/a | n/a | +0.005 ms / +0.4%; 2/6 | | list.archived.p95 | n/a | n/a | +0.051 ms / +4.1%; 2/6 | | search.query.p50 | -0.014 ms / -1.7%; 4/6 | -0.002 ms / -0.3%; 4/6 | -0.005 ms / -0.7%; 3/6 | | search.query.p95 | -0.037 ms / -4.3%; 5/6 | -0.004 ms / -0.4%; 4/6 | -0.027 ms / -3.1%; 5/6 | | archive1.p50 | n/a | n/a | -2.002 ms / -2.3%; 4/6 | | archive1.p95 | n/a | n/a | -14.745 ms / -12.6%; 6/6 | | unarchive1.p50 | n/a | n/a | -0.583 ms / -0.7%; 4/6 | | archive100.p50 | n/a | n/a | -38.483 ms / -1.4%; 5/6 | | archive100.max | n/a | n/a | -83.959 ms / -2.9%; 6/6 | | unarchive100.p50 | n/a | n/a | -95.850 ms / -3.9%; 6/6 | ### archived-list tail control Six alternating rounds, 60 warmups then 1,000 requests per case. `before` and `before2` use the identical `9c8789e7` binary on independently seeded but logically identical stores. No archive writes during these query-only runs. All times are ms. | notes / metric | before | before2 control | new | new − before (paired) | before2 − before control | new − before2 | |---|---|---|---|---|---|---| | 4,000 / p50 | 3.023 (3.00–3.03) | 3.016 (3.01–3.02) | 3.036 (2.99–3.13) | +0.012 / +0.4%; 2/6 wins | -0.006 / -0.2%; 5/6 wins | +0.021 / +0.7%; 2/6 wins | | 4,000 / p95 | 3.381 (3.26–3.48) | 3.206 (3.17–3.25) | 3.353 (3.26–3.62) | -0.037 / -1.1%; 3/6 wins | -0.157 / -4.6%; 6/6 wins | +0.140 / +4.4%; 0/6 wins | | 10,000 / p50 | 1.138 (1.12–1.17) | 1.134 (1.13–1.14) | 1.143 (1.13–1.17) | +0.004 / +0.4%; 3/6 wins | -0.008 / -0.7%; 4/6 wins | +0.015 / +1.3%; 2/6 wins | | 10,000 / p95 | 1.264 (1.22–1.50) | 1.262 (1.20–1.31) | 1.254 (1.22–1.50) | +0.003 / +0.3%; 3/6 wins | -0.002 / -0.2%; 3/6 wins | +0.020 / +1.7%; 1/6 wins |
kayg commented 2026-11-04 17:28:24 +00:00
Author
Owner
Copy link

P2 — Compatible upgrades can leave tasks from already hidden or archived notes visible.

Scenario: an existing list store contains a task note whose file already has archived: true (or hidden: true). open_conn preserves compatible rows and prefixes their stamps, but open_describers is called with discarded_rows and TasksDescriber::VERSION remains 8. On revalidation, same_core ignores the visibility flags, so the unchanged-file fast path updates the note flag and skips the describer store. The Notes page and search then treat the note as archived/hidden while its existing task rows remain in tasks.list and tasks.counts.

Fix: bump the Tasks describer version or otherwise invalidate its snapshot when visibility semantics are introduced, and cover an upgrade from an existing task snapshot with a flagged note.

P1: 0 · P2: 1 · P3: 0

**P2 — Compatible upgrades can leave tasks from already hidden or archived notes visible.** Scenario: an existing list store contains a task note whose file already has `archived: true` (or `hidden: true`). `open_conn` preserves compatible rows and prefixes their stamps, but `open_describers` is called with `discarded_rows` and `TasksDescriber::VERSION` remains `8`. On revalidation, `same_core` ignores the visibility flags, so the unchanged-file fast path updates the note flag and skips the describer store. The Notes page and search then treat the note as archived/hidden while its existing task rows remain in `tasks.list` and `tasks.counts`. Fix: bump the Tasks describer version or otherwise invalidate its snapshot when visibility semantics are introduced, and cover an upgrade from an existing task snapshot with a flagged note. P1: 0 · P2: 1 · P3: 0
kayg commented 2026-11-04 17:38:01 +00:00
Author
Owner
Copy link

follow-up to review 78562: the compatible-upgrade fast path keeps version-8 task rows because visibility is absent from its core-row equality check. i am adding a regression that restores dev 41f2efa3's list schema (v5) and Tasks snapshot v8 over unchanged files already carrying hidden: true or archived: true, then checks all Tasks lists/groupings, retained closed rows, tag badges and Untracked. visible task/checkbox controls must survive. the fix will bump the Tasks describer version and use the existing atomic drop/rebuild path; no limits change.

validation: dev-01 fmt, workspace all-targets clippy -D warnings, notes/tasks/core/engine/search/perf/calternal tests; fresh first start + index at 4k/10k against dev 41f2efa3, six alternating rounds with a same-binary dev control, holding both perf locks. dev-01 currently has an active BM14e train, so the focused failing run is waiting through ~/bin/slot.

follow-up to review 78562: the compatible-upgrade fast path keeps version-8 task rows because visibility is absent from its core-row equality check. i am adding a regression that restores dev 41f2efa3's list schema (v5) and Tasks snapshot v8 over unchanged files already carrying `hidden: true` or `archived: true`, then checks all Tasks lists/groupings, retained closed rows, tag badges and Untracked. visible task/checkbox controls must survive. the fix will bump the Tasks describer version and use the existing atomic drop/rebuild path; no limits change. validation: dev-01 fmt, workspace all-targets clippy `-D warnings`, notes/tasks/core/engine/search/perf/calternal tests; fresh first start + index at 4k/10k against dev 41f2efa3, six alternating rounds with a same-binary dev control, holding both perf locks. dev-01 currently has an active BM14e train, so the focused failing run is waiting through `~/bin/slot`.
kayg commented 2026-11-04 17:48:37 +00:00
Author
Owner
Copy link

fix pushed: 451ea80b (fix(tasks): rebuild snapshots for note visibility upgrades), following regression coverage in 77e11f3a.

Tasks describer version is now 9. Opening a version-8 snapshot atomically drops the derived task tables and clears the note stamps through the existing describer-version migration, so unchanged flagged files cannot skip rebuilding task rows. The added documentation explains why this version is required. No query filters, schema SQL, or limits changed.

An independent release-binary reproduction used the actual dev 41f2efa3 binary to create a schema-5 / Tasks-v8 store with two unchanged notes already carrying hidden: true and archived: true. After opening it with b4e87385, discovery completed but tasks.counts still returned anytime: 2 and tags: {sample: 2}. The assertion that both must be zero failed as expected. Both perf locks were held for this reproduction; both holder units are now stopped and verified inactive.

The committed Rust regression and full gates are still queued behind the dev-01 BM14e train through ~/bin/slot. No passing-gates or final perf claim yet.

fix pushed: `451ea80b` (`fix(tasks): rebuild snapshots for note visibility upgrades`), following regression coverage in `77e11f3a`. Tasks describer version is now 9. Opening a version-8 snapshot atomically drops the derived task tables and clears the note stamps through the existing describer-version migration, so unchanged flagged files cannot skip rebuilding task rows. The added documentation explains why this version is required. No query filters, schema SQL, or limits changed. An independent release-binary reproduction used the actual dev `41f2efa3` binary to create a schema-5 / Tasks-v8 store with two unchanged notes already carrying `hidden: true` and `archived: true`. After opening it with `b4e87385`, discovery completed but `tasks.counts` still returned `anytime: 2` and `tags: {sample: 2}`. The assertion that both must be zero failed as expected. Both perf locks were held for this reproduction; both holder units are now stopped and verified inactive. The committed Rust regression and full gates are still queued behind the dev-01 BM14e train through `~/bin/slot`. No passing-gates or final perf claim yet.
kayg commented 2026-11-04 19:45:29 +00:00
Author
Owner
Copy link

upgrade fix: dev-01 gates passed on 451ea80bee4cac69f9c7f20a83d5bd52c118eddb

Nix dev shell in ~/Developer/Worktrees/notes-archive-hide; heavy commands through ~/bin/slot:

command result
cargo fmt --all --check exit 0
~/bin/slot cargo clippy --workspace --all-targets --locked -j4 -- -D warnings exit 0
~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf -p calternal exit 0; 812 passed, 0 failed, 13 ignored

The committed regression, index::upgrade_tests::compatible_upgrade_removes_preexisting_hidden_and_archived_tasks, first failed on version 8 after discovery completed: Today 2, Upcoming 2, Anytime 3, Someday 2, Untracked 5, and both private / private/child tag badges 8. The fixture retains the original flagged files and their stamps, restores dev 41f2efa3's schema-5 / Tasks-v8 snapshot, and includes visible task/checkbox controls. On version 9 it passes: only the visible task and checkbox remain, with no flagged task in any of the five lists across seven grouping modes, even when their ids are retained for the visit. Tag badges are empty. All 70 Tasks tests pass. No speed assertions were added to functional tests.

The release build is still running. The actual dev-binary store upgrade check and the 4k/10k first-start/index ABAB comparison with a same-binary dev control follow it. No final perf verdict yet.

## upgrade fix: dev-01 gates passed on `451ea80bee4cac69f9c7f20a83d5bd52c118eddb` Nix dev shell in `~/Developer/Worktrees/notes-archive-hide`; heavy commands through `~/bin/slot`: | command | result | |---|---| | `cargo fmt --all --check` | exit 0 | | `~/bin/slot cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` | exit 0 | | `~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-core -p calternal-engine -p calternal-search -p calternal-perf -p calternal` | exit 0; 812 passed, 0 failed, 13 ignored | The committed regression, `index::upgrade_tests::compatible_upgrade_removes_preexisting_hidden_and_archived_tasks`, first failed on version 8 after discovery completed: Today 2, Upcoming 2, Anytime 3, Someday 2, Untracked 5, and both `private` / `private/child` tag badges 8. The fixture retains the original flagged files and their stamps, restores dev 41f2efa3's schema-5 / Tasks-v8 snapshot, and includes visible task/checkbox controls. On version 9 it passes: only the visible task and checkbox remain, with no flagged task in any of the five lists across seven grouping modes, even when their ids are retained for the visit. Tag badges are empty. All 70 Tasks tests pass. No speed assertions were added to functional tests. The release build is still running. The actual dev-binary store upgrade check and the 4k/10k first-start/index ABAB comparison with a same-binary dev control follow it. No final perf verdict yet.
kayg commented 2026-11-04 19:55:14 +00:00
Author
Owner
Copy link

review 78562 fixed and validated: 451ea80bee4cac69f9c7f20a83d5bd52c118eddb

Tasks describer version 9 invalidates the version-8 snapshot through the existing atomic drop/rebuild mechanism. Unchanged files already marked hidden or archived are described again, so the compatible core-upgrade fast path cannot retain their task rows. The code comment explains the version change. No limits or production SQL changed.

dev-01 gates: pass. Fmt, workspace all-targets clippy -D warnings, and notes/tasks/core/engine/search/perf/calternal tests all exit 0: 812 passed, 0 failed, 13 ignored. Exact commands and regression red/green evidence. Release build also exits 0: ~/bin/slot cargo build --release --locked -j4 -p calternal, in the Nix dev shell. Both build jobs are finished and the worktree is clean at this head.

Actual dev-store upgrade: pass. The dev 41f2efa3 release binary created the schema-5 / Tasks-v8 store from unchanged files with hidden: true and archived: true. It initially reports Anytime 2 and tag sample 2. Opening that same store with the fixed release binary completes discovery and reports all task counts zero and an empty tag map. The pre-fix b4e87385 binary retained both tasks in this reproduction. The committed regression additionally checks all five Tasks lists across seven grouping modes, retained done/cancelled ids, parent/child tag badges, linked and plain Untracked checkboxes, and visible task/checkbox controls.

First start + complete index: still faster than dev at both sizes, 6/6 paired wins each.

Measurement ran on perf-01 with the shared locks acquired on perf-01 first, then perf-02, held together throughout the real-store upgrade check and all measurements. At each size: six dev → new pairs (ABAB), with an independent fresh-Home run of the identical dev binary after every pair as the control. Every run copies the same fixture bytes and starts with no index. Timing covers process start until search reports indexing complete, using the same 50 ms polling method as the previous first-build comparison.

Fixtures retain the previous populations: 4,000 notes = 1,000 archived + 400 hidden + 2,600 plain; 10,000 = 200 archived + 400 hidden + 9,400 plain. Every run verifies all N accepted source rows; the new binary also verifies Tasks version 9 and 3,000 / 9,800 searchable source rows respectively. Limits are unchanged.

Times are milliseconds, median (min–max), six runs per cell:

notes dev 41f2efa3 identical dev control fixed 451ea80b
4,000 1335.596 (1325.620–1522.088) 1317.120 (1309.400–1428.602) 560.796 (547.191–603.116)
10,000 2843.795 (2687.671–3107.327) 2805.514 (2670.703–2888.557) 1076.366 (1051.362–1119.455)

Paired medians (computed per round before taking the median):

notes fixed − dev dev control − dev fixed wins
4,000 −779.409 ms / −58.54% −13.956 ms / −1.05% 6/6
10,000 −1,764.537 ms / −62.05% −78.341 ms / −2.78% 6/6

Binary provenance, SHA-256 verified through control:

  • dev / control: 41f2efa387b74175971a838f602535089d0d5fb4, ef22c21cabe7711730ce68c4a16fc473ec55c042aa9b4bb9a64211335d3780a7.
  • fixed: 451ea80bee4cac69f9c7f20a83d5bd52c118eddb, 972b96adda318cde2bf2351381e252bd5e1c734cd1053785f701fb1fb266e18a, identical on dev-01, control, and perf-01.

Perf command, outside the repository:

python3 ~/perf/pr887-upgrade/first_start.py ~/perf/pr887-upgrade/first-start.json \
  dev="$HOME/perf/k887/bin/dev" new="$HOME/perf/pr887-upgrade/candidate" dev2="$HOME/perf/k887/bin/dev"

The matrix exits 0; all 36 rows were checked. Both perf holder units were stopped and verified inactive immediately after the measurements. The perf job and both build jobs are inactive. Seeded Homes and relayed temporary binaries were removed; only small scripts/logs/JSON reports remain outside the repository. No merge.

all 36 first-start/index measurements (JSON lines) 
{"n":4000,"round":0,"label":"dev","first_start.index_ms":1336.2298929714598,"source_rows":4000}
{"n":4000,"round":0,"label":"new","first_start.index_ms":547.1909500192851,"source_rows":4000}
{"n":4000,"round":0,"label":"dev2","first_start.index_ms":1317.586746008601,"source_rows":4000}
{"n":4000,"round":1,"label":"dev","first_start.index_ms":1438.908189011272,"source_rows":4000}
{"n":4000,"round":1,"label":"new","first_start.index_ms":551.1595969437622,"source_rows":4000}
{"n":4000,"round":1,"label":"dev2","first_start.index_ms":1309.3999510165304,"source_rows":4000}
{"n":4000,"round":2,"label":"dev","first_start.index_ms":1522.0879960106686,"source_rows":4000}
{"n":4000,"round":2,"label":"new","first_start.index_ms":603.1158480327576,"source_rows":4000}
{"n":4000,"round":2,"label":"dev2","first_start.index_ms":1310.1074260193855,"source_rows":4000}
{"n":4000,"round":3,"label":"dev","first_start.index_ms":1325.6203189957887,"source_rows":4000}
{"n":4000,"round":3,"label":"new","first_start.index_ms":556.4083730569109,"source_rows":4000}
{"n":4000,"round":3,"label":"dev2","first_start.index_ms":1345.9341079578735,"source_rows":4000}
{"n":4000,"round":4,"label":"dev","first_start.index_ms":1334.9615240003914,"source_rows":4000}
{"n":4000,"round":4,"label":"new","first_start.index_ms":565.1828869595192,"source_rows":4000}
{"n":4000,"round":4,"label":"dev2","first_start.index_ms":1428.6024239845574,"source_rows":4000}
{"n":4000,"round":5,"label":"dev","first_start.index_ms":1325.9220319450833,"source_rows":4000}
{"n":4000,"round":5,"label":"new","first_start.index_ms":565.9189359867014,"source_rows":4000}
{"n":4000,"round":5,"label":"dev2","first_start.index_ms":1316.6531420429237,"source_rows":4000}
{"n":10000,"round":0,"label":"dev","first_start.index_ms":3107.326956000179,"source_rows":10000}
{"n":10000,"round":0,"label":"new","first_start.index_ms":1051.361825026106,"source_rows":10000}
{"n":10000,"round":0,"label":"dev2","first_start.index_ms":2888.5565429809503,"source_rows":10000}
{"n":10000,"round":1,"label":"dev","first_start.index_ms":2777.955155004747,"source_rows":10000}
{"n":10000,"round":1,"label":"new","first_start.index_ms":1119.454824947752,"source_rows":10000}
{"n":10000,"round":1,"label":"dev2","first_start.index_ms":2701.213534048293,"source_rows":10000}
{"n":10000,"round":2,"label":"dev","first_start.index_ms":2847.459157987032,"source_rows":10000}
{"n":10000,"round":2,"label":"new","first_start.index_ms":1084.0549829881638,"source_rows":10000}
{"n":10000,"round":2,"label":"dev2","first_start.index_ms":2767.5195849733427,"source_rows":10000}
{"n":10000,"round":3,"label":"dev","first_start.index_ms":2840.1312429923564,"source_rows":10000}
{"n":10000,"round":3,"label":"new","first_start.index_ms":1074.4622339843772,"source_rows":10000}
{"n":10000,"round":3,"label":"dev2","first_start.index_ms":2886.371561035048,"source_rows":10000}
{"n":10000,"round":4,"label":"dev","first_start.index_ms":2687.670619052369,"source_rows":10000}
{"n":10000,"round":4,"label":"new","first_start.index_ms":1070.7542049931362,"source_rows":10000}
{"n":10000,"round":4,"label":"dev2","first_start.index_ms":2843.509231985081,"source_rows":10000}
{"n":10000,"round":5,"label":"dev","first_start.index_ms":2868.0820309673436,"source_rows":10000}
{"n":10000,"round":5,"label":"new","first_start.index_ms":1078.270419035107,"source_rows":10000}
{"n":10000,"round":5,"label":"dev2","first_start.index_ms":2670.703004929237,"source_rows":10000}
## review 78562 fixed and validated: `451ea80bee4cac69f9c7f20a83d5bd52c118eddb` Tasks describer version **9** invalidates the version-8 snapshot through the existing atomic drop/rebuild mechanism. Unchanged files already marked hidden or archived are described again, so the compatible core-upgrade fast path cannot retain their task rows. The code comment explains the version change. No limits or production SQL changed. **dev-01 gates: pass.** Fmt, workspace all-targets clippy `-D warnings`, and notes/tasks/core/engine/search/perf/calternal tests all exit 0: **812 passed, 0 failed, 13 ignored**. [Exact commands and regression red/green evidence](https://git.kayg.org/kayg/calternal/pulls/887#issuecomment-78880). Release build also exits 0: `~/bin/slot cargo build --release --locked -j4 -p calternal`, in the Nix dev shell. Both build jobs are finished and the worktree is clean at this head. **Actual dev-store upgrade: pass.** The dev `41f2efa3` release binary created the schema-5 / Tasks-v8 store from unchanged files with `hidden: true` and `archived: true`. It initially reports Anytime 2 and tag `sample` 2. Opening that same store with the fixed release binary completes discovery and reports all task counts zero and an empty tag map. The pre-fix `b4e87385` binary retained both tasks in this reproduction. The committed regression additionally checks all five Tasks lists across seven grouping modes, retained done/cancelled ids, parent/child tag badges, linked and plain Untracked checkboxes, and visible task/checkbox controls. **First start + complete index: still faster than dev at both sizes, 6/6 paired wins each.** Measurement ran on perf-01 with the shared locks acquired on **perf-01 first, then perf-02**, held together throughout the real-store upgrade check and all measurements. At each size: six `dev → new` pairs (ABAB), with an independent fresh-Home run of the **identical dev binary** after every pair as the control. Every run copies the same fixture bytes and starts with no index. Timing covers process start until search reports indexing complete, using the same 50 ms polling method as the previous first-build comparison. Fixtures retain the previous populations: 4,000 notes = 1,000 archived + 400 hidden + 2,600 plain; 10,000 = 200 archived + 400 hidden + 9,400 plain. Every run verifies all N accepted source rows; the new binary also verifies Tasks version 9 and 3,000 / 9,800 searchable source rows respectively. Limits are unchanged. Times are milliseconds, median (min–max), six runs per cell: | notes | dev `41f2efa3` | identical dev control | fixed `451ea80b` | |---|---|---|---| | 4,000 | 1335.596 (1325.620–1522.088) | 1317.120 (1309.400–1428.602) | 560.796 (547.191–603.116) | | 10,000 | 2843.795 (2687.671–3107.327) | 2805.514 (2670.703–2888.557) | 1076.366 (1051.362–1119.455) | Paired medians (computed per round before taking the median): | notes | fixed − dev | dev control − dev | fixed wins | |---|---|---|---| | 4,000 | −779.409 ms / −58.54% | −13.956 ms / −1.05% | 6/6 | | 10,000 | −1,764.537 ms / −62.05% | −78.341 ms / −2.78% | 6/6 | Binary provenance, SHA-256 verified through control: - dev / control: `41f2efa387b74175971a838f602535089d0d5fb4`, `ef22c21cabe7711730ce68c4a16fc473ec55c042aa9b4bb9a64211335d3780a7`. - fixed: `451ea80bee4cac69f9c7f20a83d5bd52c118eddb`, `972b96adda318cde2bf2351381e252bd5e1c734cd1053785f701fb1fb266e18a`, identical on dev-01, control, and perf-01. Perf command, outside the repository: ```sh python3 ~/perf/pr887-upgrade/first_start.py ~/perf/pr887-upgrade/first-start.json \ dev="$HOME/perf/k887/bin/dev" new="$HOME/perf/pr887-upgrade/candidate" dev2="$HOME/perf/k887/bin/dev" ``` The matrix exits 0; all 36 rows were checked. Both perf holder units were stopped and verified inactive immediately after the measurements. The perf job and both build jobs are inactive. Seeded Homes and relayed temporary binaries were removed; only small scripts/logs/JSON reports remain outside the repository. No merge. <details> <summary>all 36 first-start/index measurements (JSON lines)</summary> ```json {"n":4000,"round":0,"label":"dev","first_start.index_ms":1336.2298929714598,"source_rows":4000} {"n":4000,"round":0,"label":"new","first_start.index_ms":547.1909500192851,"source_rows":4000} {"n":4000,"round":0,"label":"dev2","first_start.index_ms":1317.586746008601,"source_rows":4000} {"n":4000,"round":1,"label":"dev","first_start.index_ms":1438.908189011272,"source_rows":4000} {"n":4000,"round":1,"label":"new","first_start.index_ms":551.1595969437622,"source_rows":4000} {"n":4000,"round":1,"label":"dev2","first_start.index_ms":1309.3999510165304,"source_rows":4000} {"n":4000,"round":2,"label":"dev","first_start.index_ms":1522.0879960106686,"source_rows":4000} {"n":4000,"round":2,"label":"new","first_start.index_ms":603.1158480327576,"source_rows":4000} {"n":4000,"round":2,"label":"dev2","first_start.index_ms":1310.1074260193855,"source_rows":4000} {"n":4000,"round":3,"label":"dev","first_start.index_ms":1325.6203189957887,"source_rows":4000} {"n":4000,"round":3,"label":"new","first_start.index_ms":556.4083730569109,"source_rows":4000} {"n":4000,"round":3,"label":"dev2","first_start.index_ms":1345.9341079578735,"source_rows":4000} {"n":4000,"round":4,"label":"dev","first_start.index_ms":1334.9615240003914,"source_rows":4000} {"n":4000,"round":4,"label":"new","first_start.index_ms":565.1828869595192,"source_rows":4000} {"n":4000,"round":4,"label":"dev2","first_start.index_ms":1428.6024239845574,"source_rows":4000} {"n":4000,"round":5,"label":"dev","first_start.index_ms":1325.9220319450833,"source_rows":4000} {"n":4000,"round":5,"label":"new","first_start.index_ms":565.9189359867014,"source_rows":4000} {"n":4000,"round":5,"label":"dev2","first_start.index_ms":1316.6531420429237,"source_rows":4000} {"n":10000,"round":0,"label":"dev","first_start.index_ms":3107.326956000179,"source_rows":10000} {"n":10000,"round":0,"label":"new","first_start.index_ms":1051.361825026106,"source_rows":10000} {"n":10000,"round":0,"label":"dev2","first_start.index_ms":2888.5565429809503,"source_rows":10000} {"n":10000,"round":1,"label":"dev","first_start.index_ms":2777.955155004747,"source_rows":10000} {"n":10000,"round":1,"label":"new","first_start.index_ms":1119.454824947752,"source_rows":10000} {"n":10000,"round":1,"label":"dev2","first_start.index_ms":2701.213534048293,"source_rows":10000} {"n":10000,"round":2,"label":"dev","first_start.index_ms":2847.459157987032,"source_rows":10000} {"n":10000,"round":2,"label":"new","first_start.index_ms":1084.0549829881638,"source_rows":10000} {"n":10000,"round":2,"label":"dev2","first_start.index_ms":2767.5195849733427,"source_rows":10000} {"n":10000,"round":3,"label":"dev","first_start.index_ms":2840.1312429923564,"source_rows":10000} {"n":10000,"round":3,"label":"new","first_start.index_ms":1074.4622339843772,"source_rows":10000} {"n":10000,"round":3,"label":"dev2","first_start.index_ms":2886.371561035048,"source_rows":10000} {"n":10000,"round":4,"label":"dev","first_start.index_ms":2687.670619052369,"source_rows":10000} {"n":10000,"round":4,"label":"new","first_start.index_ms":1070.7542049931362,"source_rows":10000} {"n":10000,"round":4,"label":"dev2","first_start.index_ms":2843.509231985081,"source_rows":10000} {"n":10000,"round":5,"label":"dev","first_start.index_ms":2868.0820309673436,"source_rows":10000} {"n":10000,"round":5,"label":"new","first_start.index_ms":1078.270419035107,"source_rows":10000} {"n":10000,"round":5,"label":"dev2","first_start.index_ms":2670.703004929237,"source_rows":10000} ``` </details>
kayg commented 2026-11-05 04:08:38 +00:00
Author
Owner
Copy link

First-paint diet (BM15, #947). Head 47457c52.

First-paint bytes are brotli q11 from apps/web/scripts/bundle-size.mjs (JS of the first-paint closure = the index chunk; CSS of the first stylesheet). Dev baseline 41f2efa3: 238,797 B JS, 19,250 B CSS. A second figure, "hash-normalised", is the same chunk with the 8-character chunk-file hashes replaced by a fixed string: the hashes are random, and they move the real brotli number by ±300 B between two builds of identical code, so a few-hundred-byte claim is only meaningful in the normalised figure (dev: 238,804). Chunk budgets: every new lazy chunk's line is taken from the first-script line or the idle-screens line so the total stays 404.2; perf/web.sh --bundle-only --gate passes on this head.

first JS first CSS lazy JS
dev 41f2efa3 238,797 19,250 122,936
this PR 239,130 (+333; normalised 238,653 = −151) 19,250 (0) 123,424 (+488)

What moved out of the first script: the hide/archive/restore rules (listMove, visibilityOf, the batched list read) now load on the first such event from another device (notes-visibility-apply.ts, 0.5 KB; a failed fetch reads the list again). applyEntry keeps one moveNote (shared by deletes and by "leave") and a four-line check. The hidden/archived wire fields no longer default to false. The real +333 is hash luck on top of an expected −100 to −150; the remaining cost is the eight action names in actions.gen.ts (generated, about 80 raw bytes).

No web spec belongs to this PR. Lint ok, svelte-check = dev baseline (2 errors in review4-signout.spec.ts), web unit tests 55 passed.

**First-paint diet (BM15, #947).** Head `47457c52`. First-paint bytes are brotli q11 from `apps/web/scripts/bundle-size.mjs` (JS of the first-paint closure = the `index` chunk; CSS of the first stylesheet). Dev baseline `41f2efa3`: **238,797 B JS, 19,250 B CSS**. A second figure, "hash-normalised", is the same chunk with the 8-character chunk-file hashes replaced by a fixed string: the hashes are random, and they move the real brotli number by ±300 B between two builds of identical code, so a few-hundred-byte claim is only meaningful in the normalised figure (dev: 238,804). Chunk budgets: every new lazy chunk's line is taken from the first-script line or the idle-screens line so the total stays 404.2; `perf/web.sh --bundle-only --gate` passes on this head. | | first JS | first CSS | lazy JS | |---|---|---|---| | dev `41f2efa3` | 238,797 | 19,250 | 122,936 | | this PR | 239,130 (**+333**; normalised 238,653 = **−151**) | 19,250 (0) | 123,424 (+488) | What moved out of the first script: the hide/archive/restore rules (`listMove`, `visibilityOf`, the batched list read) now load on the first such event from another device (`notes-visibility-apply.ts`, 0.5 KB; a failed fetch reads the list again). `applyEntry` keeps one `moveNote` (shared by deletes and by "leave") and a four-line check. The `hidden`/`archived` wire fields no longer default to `false`. The real +333 is hash luck on top of an expected −100 to −150; the remaining cost is the eight action names in `actions.gen.ts` (generated, about 80 raw bytes). No web spec belongs to this PR. Lint ok, `svelte-check` = dev baseline (2 errors in `review4-signout.spec.ts`), web unit tests 55 passed.
kayg commented 2026-11-06 07:45:11 +00:00
Author
Owner
Copy link

WebKit specs: pending: std VMs paused. No WebKit jobs are starting while the orchestrator has paused the std VMs.

WebKit specs: pending: std VMs paused. No WebKit jobs are starting while the orchestrator has paused the std VMs.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Merge conflict checking is in progress. Try again in few moments.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin notes-archive-hide:notes-archive-hide
git switch notes-archive-hide

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff notes-archive-hide
git switch notes-archive-hide
git rebase dev
git switch dev
git merge --ff-only notes-archive-hide
git switch notes-archive-hide
git rebase dev
git switch dev
git merge --no-ff notes-archive-hide
git switch dev
git merge --squash notes-archive-hide
git switch dev
git merge --ff-only notes-archive-hide
git switch dev
git merge notes-archive-hide
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!887
No description provided.