Search foundation: provider trait and FTS5 #10

New issue

Closed

opened 2026-10-13 15:01:15 +00:00 by kayg · 2 comments

kayg commented

2026-10-13 15:01:15 +00:00

Owner

Search foundation (ADR 0031).

SearchProvider trait, FTS5 (unicode61 + trigram) per user, providers per plugin, permission-filtered on the server.
Ranking (title, frecency, body), typo/prefix tolerance, snippets with block deep links.
Benchmark on the perf VM: first results < 50 ms on the full v0 data copy, as a gate. Palette UI gets its own mockup later.

Search foundation (ADR 0031). - SearchProvider trait, FTS5 (unicode61 + trigram) per user, providers per plugin, permission-filtered on the server. - Ranking (title, frecency, body), typo/prefix tolerance, snippets with block deep links. - Benchmark on the perf VM: first results < 50 ms on the full v0 data copy, as a gate. Palette UI gets its own mockup later.

kayg added the

backend

perf

labels

2026-10-13 15:01:15 +00:00

kayg referenced this issue

2026-10-13 19:18:43 +00:00

Search foundation: provider trait, per-user FTS5 index, live from the journal #53

kayg commented

2026-10-13 19:18:54 +00:00

Author

Owner

Progress: the search foundation is up for review in PR #53 (API only; the palette waits for its own mockup).

Interim speed, idle staging VM, 4 vCPU (the perf VM was busy with #14):

	10k notes	100k notes
all queries p50 / p95	6.8 / 8.2 ms	20.3 / 31.7 ms
worst query p95	8.5 ms	38.6 ms
re-index one edit (database work) p50	0.6 ms	0.5 ms
first build (background)	3.3 s	41 s
index size	71 MB	681 MB

Budget: under 50 ms p95 at 10k. Met, and 100k stays under it too.

What changed while measuring:

Ranking cost. Full ranking of every match (BM25) is now used only when matches are few. With many matches, cost stays flat by gathering title matches, the newest matches (row ids follow when things changed) and frecent ones.
Snippets are now cut in Rust. They were 12 ms of every search.
The write-ahead log grew without limit during a first build: 4 GB at 100k. The writer now checkpoints it in quiet moments and caps it at 8 MiB.

Perf VM rerun and a #44 gate are follow-ups.

Progress: the search foundation is up for review in **PR #53** (API only; the palette waits for its own mockup). Interim speed, idle staging VM, 4 vCPU (the perf VM was busy with #14): | | 10k notes | 100k notes | |---|---|---| | all queries p50 / p95 | 6.8 / 8.2 ms | 20.3 / 31.7 ms | | worst query p95 | 8.5 ms | 38.6 ms | | re-index one edit (database work) p50 | 0.6 ms | 0.5 ms | | first build (background) | 3.3 s | 41 s | | index size | 71 MB | 681 MB | Budget: under 50 ms p95 at 10k. Met, and 100k stays under it too. What changed while measuring: - **Ranking cost.** Full ranking of every match (BM25) is now used only when matches are few. With many matches, cost stays flat by gathering title matches, the newest matches (row ids follow when things changed) and frecent ones. - **Snippets** are now cut in Rust. They were 12 ms of every search. - **The write-ahead log** grew without limit during a first build: 4 GB at 100k. The writer now checkpoints it in quiet moments and caps it at 8 MiB. Perf VM rerun and a #44 gate are follow-ups.

kayg referenced this issue

2026-10-14 12:49:34 +00:00

Idea (later): OCR and automatic tagging for documents (Documents plugin) #273

kayg referenced this issue

2026-10-14 12:52:23 +00:00

Search surface: results left with a preview right, filters at the bottom (mockup first) #275

kayg referenced this issue

2026-10-14 15:16:56 +00:00

Search surface: results left with a preview right, filters at the bottom (mockup first) #275

kayg referenced this issue

2026-10-15 09:47:15 +00:00

Saved searches as Views in the sidebar (smart folders, very easy to create) #312