Search foundation: provider trait, per-user FTS5 index, live from the journal

kayg commented

2026-10-13 19:18:42 +00:00

Owner

Part of #10: the search foundation (ADR 0031, decisions S1-S10). API only; the palette UI gets its own mockup.

What it does

One provider interface (calternal_core::SearchProvider). Plugins describe their own things and search never reads their data (ADR 0010). Notes is the first provider (NotesSearch): title, tags (frontmatter and inline #tags), dates, and the body with markdown marks removed, keeping file line numbers.
One index per person: users/<id>/search/search.db. Derived data, rebuildable, kept apart from the journal's write lock.
- FTS5 unicode61 with prefix indexes
- bigrams for scripts without spaces (Chinese, Japanese, Thai)
- title trigrams
- tags table
- frecency
Live: follows the change journal, reconciles provider stamps when it starts (heals whatever it missed), and never blocks writes or startup. Answers say indexing: true while it catches up.
Ranking: exact title, then titles with every word, then frecency and relevance.
Forgiving matching: every word is a prefix, typos are corrected from the index's own vocabulary when exact hits are few, and letters inside a title still find it.
Results: snippets with UTF-16 highlight ranges, title highlights, and the line / block / link to jump to (/note/<id>#^block or #L<line>).
Filters: tag: / #tag (nested tags count), kind:, before: / after: (year, month or day, in the caller's timezone). They also come as structured fields for chips, the CLI and MCP.
Actions:
- search.query (read)
- search.visit (an item was opened, so frecency rises)
- An empty query returns recent and frecent items.

Speed (interim: idle staging VM, 4 vCPU; perf VM was busy with #14)

Generated Homes (cargo run --release -p calternal-search --example bench -- N): Zipf-distributed words, daily logs, tags, a little Chinese and Hindi.

	10k notes (19 MB text)	100k notes (194 MB text)
all queries p50 / p95	6.8 / 8.2 ms	20.3 / 31.7 ms
worst query p95 (common word / two words)	8.5 ms	38.6 ms
typo query p95	8.8 ms	25.7 ms
Chinese / Hindi p95	5.5 / 8.4 ms	10.8 / 15.3 ms
empty search (recent) p95	0.9 ms	6.6 ms
re-index one edit, database work p50 / p95	0.6 / 13 ms	0.5 / 15 ms
full first build	3.3 s	41 s (in the background; searches answer meanwhile)
index on disk	71 MB	681 MB

The budget is p95 under 50 ms at 10k, met with room to spare, and 100k is also under 50 ms.

Tests

11 search behaviour tests over a fake provider:
- ranking, prefixes, typos (meetng, dlhi, jorunal)
- 7 scripts (Chinese, Japanese, Arabic, Devanagari, Thai, Russian, Latin with accents)
- live updates and deletes, restart healing
- per-user isolation
- tag / kind / date filters
- snippets with line and block
- odd FTS syntax in input
- answering during a first build
11 unit tests. Notes provider tests.
An end-to-end test through the real binary. Notes made through the API, appended through the API, written by another program into Notes/, and deleted outside all show up within a moment, at the right file line.

Follow-ups

Run the bench on the perf VM once #14 frees it, and wire it into the #44 harness as a gate.
Index size at 100k (681 MB) is mostly prefix indexes and stored bodies; measure without the 2-letter prefix index.
A note that was only ever renamed needs a notes.renamed event (#9) for a live title update; reconcile catches it on restart today.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Part of #10: the search foundation (ADR 0031, decisions S1-S10). API only; the palette UI gets its own mockup. ## What it does - **One provider interface** (`calternal_core::SearchProvider`). Plugins describe their own things and search never reads their data (ADR 0010). Notes is the first provider (`NotesSearch`): title, tags (frontmatter and inline `#tags`), dates, and the body with markdown marks removed, keeping file line numbers. - **One index per person**: `users/<id>/search/search.db`. Derived data, rebuildable, kept apart from the journal's write lock. - FTS5 `unicode61` with prefix indexes - bigrams for scripts without spaces (Chinese, Japanese, Thai) - title trigrams - tags table - frecency - **Live**: follows the change journal, reconciles provider stamps when it starts (heals whatever it missed), and never blocks writes or startup. Answers say `indexing: true` while it catches up. - **Ranking**: exact title, then titles with every word, then frecency and relevance. - **Forgiving matching**: every word is a prefix, typos are corrected from the index's own vocabulary when exact hits are few, and letters inside a title still find it. - **Results**: snippets with UTF-16 highlight ranges, title highlights, and the `line` / `block` / `link` to jump to (`/note/<id>#^block` or `#L<line>`). - **Filters**: `tag:` / `#tag` (nested tags count), `kind:`, `before:` / `after:` (year, month or day, in the caller's timezone). They also come as structured fields for chips, the CLI and MCP. - **Actions**: - `search.query` (read) - `search.visit` (an item was opened, so frecency rises) - An empty query returns recent and frecent items. ## Speed (interim: idle staging VM, 4 vCPU; perf VM was busy with #14) Generated Homes (`cargo run --release -p calternal-search --example bench -- N`): Zipf-distributed words, daily logs, tags, a little Chinese and Hindi. | | 10k notes (19 MB text) | 100k notes (194 MB text) | |---|---|---| | all queries p50 / p95 | 6.8 / 8.2 ms | 20.3 / 31.7 ms | | worst query p95 (common word / two words) | 8.5 ms | 38.6 ms | | typo query p95 | 8.8 ms | 25.7 ms | | Chinese / Hindi p95 | 5.5 / 8.4 ms | 10.8 / 15.3 ms | | empty search (recent) p95 | 0.9 ms | 6.6 ms | | re-index one edit, database work p50 / p95 | 0.6 / 13 ms | 0.5 / 15 ms | | full first build | 3.3 s | 41 s (in the background; searches answer meanwhile) | | index on disk | 71 MB | 681 MB | The budget is p95 under 50 ms at 10k, met with room to spare, and 100k is also under 50 ms. ## Tests - 11 search behaviour tests over a fake provider: - ranking, prefixes, typos (`meetng`, `dlhi`, `jorunal`) - 7 scripts (Chinese, Japanese, Arabic, Devanagari, Thai, Russian, Latin with accents) - live updates and deletes, restart healing - per-user isolation - tag / kind / date filters - snippets with line and block - odd FTS syntax in input - answering during a first build - 11 unit tests. Notes provider tests. - An end-to-end test through the real binary. Notes made through the API, appended through the API, written by another program into `Notes/`, and deleted outside all show up within a moment, at the right file line. ## Follow-ups - Run the bench on the perf VM once #14 frees it, and wire it into the #44 harness as a gate. - Index size at 100k (681 MB) is mostly prefix indexes and stored bodies; measure without the 2-letter prefix index. - A note that was only ever renamed needs a `notes.renamed` event (#9) for a live title update; reconcile catches it on restart today. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 2 commits

2026-10-13 19:18:43 +00:00

Search foundation: provider trait, per-user FTS5 index, live from the journal 7073ec94dc

- calternal-core: SearchProvider + SearchDoc, so every plugin can offer
  things to find without search reading its data (ADR 0010).
- calternal-search: one index per person (users/<id>/search/search.db,
  derived and rebuildable, apart from the journal's write lock):
  FTS5 unicode61 with prefix indexes, bigrams for scripts without spaces
  (Chinese, Japanese, Thai), title trigrams, tags, frecency.
  Live: follows the change journal, reconciles provider stamps on start
  (heals anything missed), never blocks writes or startup, reports
  `indexing` while catching up.
  Ranking: exact title, titles with every word, then frecency and
  relevance. Prefixes, typo correction from the index's own vocabulary
  when exact hits are few, letters inside titles. Snippets with UTF-16
  marks, line and block deep links. Filters: tag (nested), kind,
  before/after, also as structured fields for chips, CLI and MCP.
  Cost stays flat as Homes grow: title + newest + frecent candidates,
  BM25 only within a small budget.
- Actions: search.query (read), search.visit (frecency).
- calternal-notes: NotesSearch provider (title, tags, dates, cleaned
  body with file line numbers).
- Tests: ranking, typos, prefixes, 7 scripts, live updates, restart
  healing, per-user isolation, filters, snippets, odd input, and an
  end-to-end test through the real server incl. outside file writes.
- examples/bench.rs: generated Homes, build, query p50/p95, re-index.

Part of #10

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: flat cost at 100k notes, WAL kept small, spaced-edit bench

CI / rust (pull_request) Successful in 1m16s

Details

CI / web (pull_request) Successful in 1m16s

Details

511d8274ea

- Rank fully only when matches are few; with many matches (or filters)
  ranking gets a 4 ms budget and titles, newest and frecent matches
  carry the result. Snippets are cut in Rust from the shown bodies
  instead of a second full-text pass (was 12 ms of every search).
- Changed items get a new row id, so "newest matches" is a cheap
  backwards walk; a first build indexes oldest first.
- The writer checkpoints the WAL itself: every 64 writes in a busy
  stretch, and in quiet moments, with an 8 MiB size cap. A 100k-note
  index went from 4 GB (unbounded WAL) to 681 MB.
- Bench: edits spaced like typing, and the database share of their cost.

Part of #10

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-13 19:18:54 +00:00

Search foundation: provider trait and FTS5 #10

kayg commented

2026-10-13 19:20:59 +00:00

Author

Owner

Reviewed by Claude: provider trait in core, per-user search.db (separate file so indexing never takes the journal lock), journal-driven live updates with startup healing, typo/prefix/CJK handling, per-user isolation test. 10k p95 8.2 ms, 100k p95 31.7 ms on staging; perf-VM run follows in #44. CI green. Merging. Follow-ups: 681 MB index at 100k (try dropping 2-char prefix index), rename event from #9.