kayg/calternal

Watch

Fork 0

Search: ranking with per-user frecency #62

New issue

Closed

opened 2026-09-24 15:53:03 +00:00 by kayg · 4 comments

kayg commented

2026-09-24 15:53:03 +00:00

Owner

Owner decision S6 (DESIGN §32): start from calternal.js's scheme (bm25 with title ≫ tags > headings > body, gentle recency) and add per-user frecency of opened results (stored in the index, decays over time). Exact title matches always rank first. No global popularity, no cross-user learning. Add a small offline evaluation set from the owner's-style queries to prevent ranking regressions.

Context for the owning job

Repo: kayg/calternal (~/Developer/calternal). Read CLAUDE.md, CONTEXT.md and docs/DESIGN.md (§15, §18 budgets, §31, §32) first. Prior art: calternal.js docs/search.md (read-only at /home/kayg/Developer/calternal.js) — its palette UX, > command mode, ranking weights, a11y combobox pattern and recents carry over.
Existing code: crates/calternal-search (Tantivy index, watcher + reconcile, providers via calternal-plugin fan-out), the ⌘K registry in apps/web.
Owner rules: search must be ultra fast (⌘K results < 50 ms p95 at 100k items; first keystroke to first results < 16 ms for client providers); file over app (indexes are derived and rebuildable); performance first but never at the cost of finesse; calternal.js design system (Claude reviews screenshots; floating window over a dimmed + blurred background, spring motion, reduced-motion respected); never ship sample data; atomic commits (commit every 30–45 min); adversarial testing after API work.
Comment on this issue when you start, on findings, when blocked, and when finished. Never close it.

Owner decision S6 (DESIGN §32): start from calternal.js's scheme (bm25 with title ≫ tags > headings > body, gentle recency) and add **per-user frecency** of opened results (stored in the index, decays over time). Exact title matches always rank first. No global popularity, no cross-user learning. Add a small offline evaluation set from the owner's-style queries to prevent ranking regressions. ## Context for the owning job - Repo: kayg/calternal (~/Developer/calternal). Read CLAUDE.md, CONTEXT.md and docs/DESIGN.md (§15, §18 budgets, §31, §32) first. Prior art: calternal.js `docs/search.md` (read-only at /home/kayg/Developer/calternal.js) — its palette UX, `>` command mode, ranking weights, a11y combobox pattern and recents carry over. - Existing code: `crates/calternal-search` (Tantivy index, watcher + reconcile, providers via calternal-plugin fan-out), the ⌘K registry in apps/web. - Owner rules: search must be **ultra fast** (⌘K results < 50 ms p95 at 100k items; first keystroke to first results < 16 ms for client providers); file over app (indexes are derived and rebuildable); performance first but never at the cost of finesse; calternal.js design system (Claude reviews screenshots; floating window over a dimmed + blurred background, spring motion, reduced-motion respected); never ship sample data; atomic commits (commit every 30–45 min); adversarial testing after API work. - Comment on this issue when you start, on findings, when blocked, and when finished. Never close it.

kayg referenced this issue

2026-09-24 15:54:11 +00:00

Search: sources — PDF text extraction, events, bookmarks, tasks, tags in the index #58

kayg referenced this issue

2026-09-24 15:54:12 +00:00

Search: operators as autocompleted pills and natural dates #61

kayg commented

2026-09-24 15:54:13 +00:00

Author

Owner

Starting the search backend job for issues #58, #61, and #62. Branch: job/search-backend; base SHA: 55a2752a4c4c7a087fce55868871a8576fea31a7. I am reading the current index, plugin interfaces, and API contract before implementation.

Starting the search backend job for issues #58, #61, and #62. Branch: `job/search-backend`; base SHA: `55a2752a4c4c7a087fce55868871a8576fea31a7`. I am reading the current index, plugin interfaces, and API contract before implementation.

kayg commented

2026-09-24 16:54:54 +00:00

Author

Owner

The existing query code added an exact-title boost to each SearchHit score after Tantivy selected TopDocs, but it kept the group ordered by raw BM25. That meant the exact-title boost did not change the returned order. The query now sorts by BM25 plus exact-title and recency boosts; the indexer applies a per-User open-history boost and re-sorts before selecting results. The offline ranking vector covers exact titles, headings, tags, and type filters.

kayg commented

2026-09-24 17:45:10 +00:00

Author

Owner

Finished on branch job/search-backend at head 9e3cb6ed6b677a0af990955821f4aa96b24b3866.

Implemented weighted title/tag/heading/body ranking, exact-title priority, gentle 30-day-half-life recency, and per-user open-result frecency. Frecency is capped at 32 opens per result, expires after 90 days, and prunes to 2,048 rows per plugin. It does not use global popularity or cross-user activity. Added an offline ranking corpus and queries.

At 100k documents, 200 search queries measured 100k search query latency: p50=6369 us p95=13817 us (200 queries), under the 50 ms p95 target.

Gates: cargo fmt --check passed (no output); cargo clippy --all-targets -- -D warnings passed (Finished dev profile [unoptimized + debuginfo] target(s) in 6.29s); cargo test --workspace passed; bash packages/api-client/check-generated.sh passed; bash tests/adversarial/run.sh passed with ==== FINDINGS 0.

Finished on branch `job/search-backend` at head `9e3cb6ed6b677a0af990955821f4aa96b24b3866`. Implemented weighted title/tag/heading/body ranking, exact-title priority, gentle 30-day-half-life recency, and per-user open-result frecency. Frecency is capped at 32 opens per result, expires after 90 days, and prunes to 2,048 rows per plugin. It does not use global popularity or cross-user activity. Added an offline ranking corpus and queries. At 100k documents, 200 search queries measured `100k search query latency: p50=6369 us p95=13817 us (200 queries)`, under the 50 ms p95 target. Gates: `cargo fmt --check` passed (no output); `cargo clippy --all-targets -- -D warnings` passed (`Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.29s`); `cargo test --workspace` passed; `bash packages/api-client/check-generated.sh` passed; `bash tests/adversarial/run.sh` passed with `==== FINDINGS 0`.

kayg referenced this issue from a commit

2026-09-24 18:46:07 +00:00

Merge job/search-backend: indexed search filters, operators, frecency ranking (#58, #61, #62)

kayg commented

2026-10-01 05:08:56 +00:00

Author

Owner

Completed on dev in cf996cb5ec (Merge job/search-backend: indexed search filters, operators, frecency ranking (#58, #61, #62)).

Completed on dev in cf996cb5ecb742560df81dadeb3b83bdbae817d3 (Merge job/search-backend: indexed search filters, operators, frecency ranking (#58, #61, #62)).

kayg closed this issue

2026-10-01 05:08:56 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#62

No description provided.

Rows
Columns