kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Search: titles get an index of their own #339

Closed
kayg wants to merge 0 commits from issue/44-search-titles into dev
pull from: issue/44-search-titles
merge into: kayg:dev
Conversation 1 Commits - Files changed -
kayg commented 2026-10-14 18:49:47 +00:00
Owner
Copy link

Part of #44 (search p95 under 50 ms at 100k). On top of #315: merge that first; this PR's own change is the last commit (2c07d11). It is independent of #338 and of the whole-words PR.

A title match asked the full index for the words in the title column. That walks every note holding the words in any column: 3 to 11 ms at 100,000 notes, and 8 ms for the palette's first letter, which can only match titles.

The new titles table is an FTS5 index of the title words alone. It is contentless (no text kept) and takes about 3 MB at 100,000 notes. Puts and removes keep it in step, like tri. An index made before it fills it on open from docs, which takes a quarter second at 100k. There is no version bump and no rebuild.

Measured, FTS5 alone on the 100k harness index:

title lookup full index titles
morning* walk* 6.0 ms 0.49 ms
garden* plan* 13.9 0.43
coffee* 3.4 0.21
g* (first keystroke) ~8 0.68

With #338 and the whole-words PR, every harness query takes 0.7–3.9 ms of search time (it was 9–34). Perf-VM numbers for the three together will be posted on #44.

New test: an_older_index_gets_its_titles_filled. cargo test -p calternal-search and clippy are clean.

🤖 Generated with Claude Code

Part of #44 (search p95 under 50 ms at 100k). **On top of #315**: merge that first; this PR's own change is the last commit (2c07d11). It is independent of #338 and of the whole-words PR. A title match asked the full index for the words in the title column. That walks every note holding the words in any column: 3 to 11 ms at 100,000 notes, and 8 ms for the palette's first letter, which can only match titles. The new `titles` table is an FTS5 index of the title words alone. It is contentless (no text kept) and takes about 3 MB at 100,000 notes. Puts and removes keep it in step, like `tri`. An index made before it fills it on open from `docs`, which takes a quarter second at 100k. There is no version bump and no rebuild. **Measured**, FTS5 alone on the 100k harness index: | title lookup | full index | `titles` | |---|---|---| | `morning* walk*` | 6.0 ms | 0.49 ms | | `garden* plan*` | 13.9 | 0.43 | | `coffee*` | 3.4 | 0.21 | | `g*` (first keystroke) | ~8 | 0.68 | With #338 and the whole-words PR, every harness query takes 0.7–3.9 ms of search time (it was 9–34). Perf-VM numbers for the three together will be posted on #44. New test: `an_older_index_gets_its_titles_filled`. `cargo test -p calternal-search` and clippy are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
Profiling search on the 100,000-note fixture (perf VM p95 ~460 ms)
found three costs, none of them the index itself:

- Typo vocabulary: it was rebuilt on the query path whenever the index
  had changed, and building it reads the whole index (1.6 s at 100k).
  It is now asked for only when exact matches are few, and a big one is
  rebuilt on a thread of its own while searches use the one they have; a
  small one (quick to build, or under 5,000 items) is still rebuilt in
  place.
- Ranking with many matches: the 4 ms budget couldn't stop it, because
  FTS5 merges a common word's whole posting list in one step that never
  reaches the progress handler: 0.3 to 0.8 s. With many matches, titles,
  recency and frecency already cover the candidates, so it is skipped.
- Frecency: `MATCH ... AND rowid IN (200 ids)` runs the whole match
  again for every listed row (0.6 to 1.6 s for 30 rows). The recently
  used items are now checked on their text, stopping at the first words
  that satisfy the query.

The search tests' settle helper waits 250 ms (was 30) for the indexer to
pick up an emitted change; on a loaded CI host 30 ms was sometimes too
short (`the_index_follows_changes_live`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Phase timings of steady-state searches at 100,000 notes on the perf VM:
checking the most-used items took 25 to 45 ms (it reads their text, long
notes included), and ranking title matches about 30 ms (FTS5 scores every
note holding the words, in any column, before keeping titles).

- The most-used items are checked only when there are many matches:
  with few, the newest matches already hold all of them. And only the
  64 most recent of them.
- Title matches come newest first and the ranking that orders all
  candidates orders them too.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Search: one letter looks at titles; short words aren't typo-fixed
Some checks failed
CI / server (pull_request) Successful in 23s
Details
CI / rust (pull_request) Successful in 1m59s
Details
CI / web (pull_request) Failing after 2m48s
Details
29ef5cf68a 
More phase timings at 100,000 notes on the perf VM:
- A one-letter query (the palette's first keystroke) took 75-100 ms: a
  one-letter prefix isn't in FTS5's prefix lists, so the body search
  merged every word with that letter. One letter now looks at titles.
- One or two letters with few matches went through typo correction,
  which has nothing to fix in them: now only words of three letters or
  more are corrected.
- Checking the most-used items read their whole text, 50,000-word notes
  included; now their first 16 KB, where what they are about usually is.

Steady state at 100k (index caught up), 110 queries: p50 58-62 ms, p95
79-85 ms (dev: p50 85-165, p95 135-560).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Search: titles get an index of their own
Some checks failed
CI / server (pull_request) Successful in 27s
Details
CI / rust (pull_request) Successful in 2m8s
Details
CI / web (pull_request) Failing after 2m44s
Details
2c07d114a5 
A title match asked the full index for words in the title column, which
walks every note holding the words in any column: 3 to 11 ms at 100,000
notes, and 8 ms for the palette's first letter. `titles` holds the words
of titles alone (no text kept, ~3 MB at 100,000 notes): the same lookups
take under 1 ms. An index made before it fills it on open from what it
has, in a quarter second at 100,000 notes, rather than rebuilding. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-15 02:22:57 +00:00
Author
Owner
Copy link

Merged into dev via train j (#391).

Merged into dev via train j (#391).
kayg closed this pull request 2026-10-15 02:22:58 +00:00
Some checks failed
CI / server (pull_request) Successful in 27s
Details
CI / rust (pull_request) Successful in 2m8s
Details
CI / web (pull_request) Failing after 2m44s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!339
No description provided.