Search: Tantivy index, watcher + reconcile, content search provider

kayg commented

2026-09-24 09:40:13 +00:00

Owner

Codex job search.

Base: 05adaa01c7

Codex job `search`. Base: 05adaa01c7c88d0d6c7daaf7690d15919e94e8ed

kayg commented

2026-09-24 09:41:27 +00:00

Author

Owner

Starting search index work on branch job/search, based on 05adaa01c7c88d0d6c7daaf7690d15919e94e8ed.

Starting search index work on branch `job/search`, based on `05adaa01c7c88d0d6c7daaf7690d15919e94e8ed`.

kayg commented

2026-09-24 09:46:36 +00:00

Author

Owner

Finding: calternal-server already owns /api/v1/search fan-out through SearchProvider, but SearchContext has no allowed roots. Auth currently exposes only ResourceScope::HomeAndShared { user_id }; there is no share table or share-root resolver in this checkout. I will add the permitted explicit root allow-list to the plugin search context and have the server supply the authenticated user's Home. Shared roots cannot be supplied until an auth/files share resolver exists. The existing SearchResponse is flat, so the index can group results internally but the current API cannot expose group headings without an out-of-scope calternal-api change.

Finding: `calternal-server` already owns `/api/v1/search` fan-out through `SearchProvider`, but `SearchContext` has no allowed roots. Auth currently exposes only `ResourceScope::HomeAndShared { user_id }`; there is no share table or share-root resolver in this checkout. I will add the permitted explicit root allow-list to the plugin search context and have the server supply the authenticated user's Home. Shared roots cannot be supplied until an auth/files share resolver exists. The existing `SearchResponse` is flat, so the index can group results internally but the current API cannot expose group headings without an out-of-scope `calternal-api` change.

kayg commented

2026-09-24 10:52:39 +00:00

Author

Owner

Finding: cargo check -p calternal-server reports that #[derive(RustEmbed)] cannot find apps/web/build/ in this checkout. The compiler then reports Frontend::get missing because the derive did not run. This directory is outside the owned files for issue #13; I will not generate or edit it. I will validate the new search crate directly and keep the server integration within its owned files. The same check also found an OpenAPI helper call that needs an Indexer argument; I am fixing that in crates/calternal-server/src/main.rs.

Finding: `cargo check -p calternal-server` reports that `#[derive(RustEmbed)]` cannot find `apps/web/build/` in this checkout. The compiler then reports `Frontend::get` missing because the derive did not run. This directory is outside the owned files for issue #13; I will not generate or edit it. I will validate the new search crate directly and keep the server integration within its owned files. The same check also found an OpenAPI helper call that needs an Indexer argument; I am fixing that in `crates/calternal-server/src/main.rs`.

kayg commented

2026-09-24 11:00:39 +00:00

Author

Owner

Decision: reconcile hashes every indexable text file within the 1 MiB text cap, even when size and mtime are unchanged. It compares size and mtime for binary or larger files, and hashes those files when watcher events report changes. This avoids re-reading large media on every five-minute scan. A missed event plus an out-of-band same-size, same-mtime rewrite of a large or binary file will not be detected by that scan; the server-writer rule and watcher cover supported writes. The design does not set a full-hash policy for large files.

kayg commented

2026-09-24 11:07:15 +00:00

Author

Owner

Benchmark finding: the initial 100k synthetic-document run measured p50=42,175 us and p95=80,400 us. Every exact query was also expanding fuzzy terms. Fuzzy matching now runs only after an exact/prefix query returns no hits; the rerun measured p50=7,029 us and p95=13,958 us over 200 queries. Added a typo-transposition assertion to retain fallback coverage.

kayg commented

2026-09-24 11:19:07 +00:00

Author

Owner

Finished job/search at head dc36c34db922795fd42d9073c6f271d226fc736f (base 05adaa01c7c88d0d6c7daaf7690d15919e94e8ed). Two atomic commits: 2320006 adds the index/reconciler; dc36c34 wires it into the server and adds reindex.

Implemented the Tantivy index, SQLite manifest, watcher/debounce/reconcile actor, rebuild command, caller-root filtering, typo fallback, filename prefixes, Markdown metadata/snippets, and SearchPlugin registration. Search crate integration tests and plugin tests pass. Working tree is clean; cargo clean removed 4.4 GiB.

Gate output:

cargo fmt --check: exit 0, no output.

cargo clippy --all-targets -- -D warnings: exit 101. Verbatim failure excerpt:

error: #[derive(RustEmbed)] folder '/home/kayg/Developer/calternal-wt/search/crates/calternal-server/../../apps/web/build/' does not exist. cwd: '/home/kayg/Developer/calternal-wt/search'
  --> crates/calternal-server/src/main.rs:41:1
error: could not compile `calternal-server` (bin "calternal-server") due to 3 previous errors

cargo test: exit 101, same missing-frontend-build blocker. Verbatim ending:

error: could not compile `calternal-server` (bin "calternal-server" test) due to 3 previous errors
warning: build failed, waiting for other jobs to finish...

Search crate test output:

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.73s

Benchmark output:

100k search query latency: p50=7029 us p95=13958 us (200 queries)
test query::tests::benchmark_search_latency_at_one_hundred_thousand_files ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.32s

Known gaps/design choices: server auth currently supplies only the caller's Home; the existing auth context does not supply share roots. SearchResponse is flat, so type groups are formed internally then flattened by the provider. Reconcile hashes text files up to 1 MiB; larger and non-text files use size+mtime, so missed watcher events cannot detect same-size/same-mtime edits to those files. Result links use /notes?path=<encoded-path> and /files?path=<encoded-path> because the design does not specify those routes.

Finished `job/search` at head `dc36c34db922795fd42d9073c6f271d226fc736f` (base `05adaa01c7c88d0d6c7daaf7690d15919e94e8ed`). Two atomic commits: `2320006` adds the index/reconciler; `dc36c34` wires it into the server and adds `reindex`. Implemented the Tantivy index, SQLite manifest, watcher/debounce/reconcile actor, rebuild command, caller-root filtering, typo fallback, filename prefixes, Markdown metadata/snippets, and SearchPlugin registration. Search crate integration tests and plugin tests pass. Working tree is clean; `cargo clean` removed 4.4 GiB. Gate output: `cargo fmt --check`: exit 0, no output. `cargo clippy --all-targets -- -D warnings`: exit 101. Verbatim failure excerpt: ``` error: #[derive(RustEmbed)] folder '/home/kayg/Developer/calternal-wt/search/crates/calternal-server/../../apps/web/build/' does not exist. cwd: '/home/kayg/Developer/calternal-wt/search' --> crates/calternal-server/src/main.rs:41:1 error: could not compile `calternal-server` (bin "calternal-server") due to 3 previous errors ``` `cargo test`: exit 101, same missing-frontend-build blocker. Verbatim ending: ``` error: could not compile `calternal-server` (bin "calternal-server" test) due to 3 previous errors warning: build failed, waiting for other jobs to finish... ``` Search crate test output: ``` test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.73s ``` Benchmark output: ``` 100k search query latency: p50=7029 us p95=13958 us (200 queries) test query::tests::benchmark_search_latency_at_one_hundred_thousand_files ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.32s ``` Known gaps/design choices: server auth currently supplies only the caller's Home; the existing auth context does not supply share roots. `SearchResponse` is flat, so type groups are formed internally then flattened by the provider. Reconcile hashes text files up to 1 MiB; larger and non-text files use size+mtime, so missed watcher events cannot detect same-size/same-mtime edits to those files. Result links use `/notes?path=<encoded-path>` and `/files?path=<encoded-path>` because the design does not specify those routes.

kayg commented

2026-09-24 11:19:55 +00:00

Author

Owner

Follow-up integration gap from the final registry review: this checkout has no Files core plugin or filename-only search provider. rg -n 'distributed_slice\\(calternal_plugin::CORE_PLUGINS\\)|fn search_providers' crates finds only the System plugin registration, the plugin registry hook, the server test provider, and the new Search provider. The Search provider therefore currently returns filename and content matches together. It cannot delegate names to the Files plugin described in the job until that provider exists. The existing public SearchResponse is also flat, so result type groups are flattened at the provider boundary.

Follow-up integration gap from the final registry review: this checkout has no Files core plugin or filename-only search provider. `rg -n 'distributed_slice\\(calternal_plugin::CORE_PLUGINS\\)|fn search_providers' crates` finds only the System plugin registration, the plugin registry hook, the server test provider, and the new Search provider. The Search provider therefore currently returns filename and content matches together. It cannot delegate names to the Files plugin described in the job until that provider exists. The existing public SearchResponse is also flat, so result type groups are flattened at the provider boundary.

kayg referenced this issue from a commit

2026-09-24 11:27:20 +00:00

Merge job/search: Tantivy index, watcher and reconcile, content search (#13)

kayg commented

2026-10-01 05:08:27 +00:00

Author

Owner

Completed on dev in 49fb86f1c3 (Merge job/search: Tantivy index, watcher and reconcile, content search (#13)).

Completed on dev in 49fb86f1c36af264b9c22031669c8dd26897ae38 (Merge job/search: Tantivy index, watcher and reconcile, content search (#13)).

kayg closed this issue

2026-10-01 05:08:28 +00:00

kayg referenced this issue