Decide whether local search indexes compaction-preserved rows #34

New issue

Open

opened 2026-08-22 12:55:20 +00:00 by kayg · 0 comments

kayg commented

2026-08-22 12:55:20 +00:00

Owner

Decision needed

Decide whether local SQLite FTS5 search should index compaction-preserved rows that the transcript does not display. No decision has been made. This is the remaining compaction policy question after today's history investigation; it is distinct from the REST retrieval cap tracked in #28.

What was measured

Local search's corpus is exactly what the client downloads. The client currently requests GET /api/sessions/{id}/messages?limit=500; the gateway hard-clamps limit to 500. An explicit limit implies order=oldest, so the newest messages are the ones missing. offset is honored, while cursor is silently ignored. Because the response has no total or has-more flag, termination is returned < limit. The cap is being fixed separately by paging with offset, active rows only; #28 tracks that work.

Measured row counts on the two truncated sessions (verbatim):

20260819_103056_d3d63d   active=556   include_compacted=4743 -> no, 556
20260816_100129_456b45d6 active=896   include_compacted=4743

Precisely, session 20260819_103056_d3d63d returns 556 either way. Session 20260816_100129_456b45d6 returns 896 active rows and 4,743 with include_compacted=true, so 3,847 rows exist that the transcript does not show.

Source evidence

hermes_cli/web_routers/sessions.py:601-648 — the dashboard GET /api/sessions/{session_id}/messages route declares include_compacted and passes it to SessionDB.get_messages.
hermes_cli/web_routers/sessions.py:722-757 — the /api/sessions/{session_id}/export handler calls get_messages at :754-757 with keyset pagination (limit/after_id) but does not set include_compacted; export therefore omits compacted rows too.
The alternate adapter has the same message endpoint/cap contract at gateway/platforms/api_server.py:3626-3683.

Tradeoff

Indexing compacted rows would make those 3,847 messages searchable, but they are pre-compaction originals that the transcript deliberately does not render. A hit on one would be a search result that cannot scroll to its own message because that message is not in the displayed transcript. This is a real tradeoff between corpus completeness and a coherent result-to-transcript relationship, not an obvious win.

The transcript display must remain active-rows-only either way. This issue is only deciding the search corpus and result behavior; it must not make hidden compacted rows appear in the transcript.

Concrete options

Option A: Index compacted rows and mark them archived/unscrollable

Index active and compaction-preserved rows, and carry provenance in each search result so a compacted hit is visibly marked Archived — not in transcript (or equivalent). Selecting it must not pretend to scroll to a displayed message; it may offer the result snippet/metadata as context. Consequence: corpus completeness improves, but users must understand that some hits cannot navigate to a transcript row.

Option B: Exclude compacted rows from local search

Index active rows only, matching the displayed transcript. Consequence: every result retains a coherent result-to-transcript relationship, but compaction-preserved history is intentionally unsearchable. This is the simpler and less surprising behavior.

No decision has been made between these options.

Acceptance criteria for the eventual choice

The chosen policy is explicit in the search/index contract: active-only (Option B) or active-plus-compacted with provenance (Option A); no implicit dependence on a fetch default.
A fixture containing both active and compaction-preserved rows verifies the chosen inclusion/exclusion policy and remains deterministic across index rebuild and incremental reconciliation.
The transcript UI/API remains active-rows-only in both policies; compacted rows never become displayed transcript messages as a side effect of search indexing.
If Option A is chosen, every compacted hit is visibly labeled archived/unscrollable, cannot claim a scroll target, and active hits still navigate to their displayed message. If Option B is chosen, a term present only in compacted rows produces no local-search hit while a term in an active row still produces a navigable hit.
Search result identity and account/session scoping remain stable, and the decision does not reintroduce the 500-message retrieval problem addressed separately by #28.

Duplicate check

I searched the open issue list (24 open issues) and read the bodies of #8, #11, #23, #28, #29, #32, and #33. #8 covers the local BM25 search feature; #11 covers Spotlight indexing; #23 covers find within the current displayed conversation; #28 covers the REST history cap and is related but distinct; #29 covers Modules; #32 covers accounts/profiles; and #33 covers sidebar account display. None decides whether local search should index compaction-preserved, transcript-hidden rows. This issue is not a duplicate.

## Decision needed Decide whether local SQLite FTS5 search should index compaction-preserved rows that the transcript does not display. **No decision has been made.** This is the remaining compaction policy question after today's history investigation; it is distinct from the REST retrieval cap tracked in [#28](../issues/28). ## What was measured Local search's corpus is exactly what the client downloads. The client currently requests `GET /api/sessions/{id}/messages?limit=500`; the gateway hard-clamps `limit` to 500. An explicit `limit` implies `order=oldest`, so the newest messages are the ones missing. `offset` is honored, while `cursor` is silently ignored. Because the response has no total or has-more flag, termination is `returned < limit`. The cap is being fixed separately by paging with `offset`, active rows only; #28 tracks that work. Measured row counts on the two truncated sessions (verbatim): ``` 20260819_103056_d3d63d active=556 include_compacted=4743 -> no, 556 20260816_100129_456b45d6 active=896 include_compacted=4743 ``` Precisely, session `20260819_103056_d3d63d` returns 556 either way. Session `20260816_100129_456b45d6` returns 896 active rows and 4,743 with `include_compacted=true`, so 3,847 rows exist that the transcript does not show. ## Source evidence - `hermes_cli/web_routers/sessions.py:601-648` — the dashboard `GET /api/sessions/{session_id}/messages` route declares `include_compacted` and passes it to `SessionDB.get_messages`. - `hermes_cli/web_routers/sessions.py:722-757` — the `/api/sessions/{session_id}/export` handler calls `get_messages` at `:754-757` with keyset pagination (`limit`/`after_id`) but does not set `include_compacted`; export therefore omits compacted rows too. - The alternate adapter has the same message endpoint/cap contract at `gateway/platforms/api_server.py:3626-3683`. ## Tradeoff Indexing compacted rows would make those 3,847 messages searchable, but they are pre-compaction originals that the transcript deliberately does not render. A hit on one would be a search result that cannot scroll to its own message because that message is not in the displayed transcript. This is a real tradeoff between corpus completeness and a coherent result-to-transcript relationship, not an obvious win. The transcript display must remain **active-rows-only either way**. This issue is only deciding the search corpus and result behavior; it must not make hidden compacted rows appear in the transcript. ## Concrete options ### Option A: Index compacted rows and mark them archived/unscrollable Index active and compaction-preserved rows, and carry provenance in each search result so a compacted hit is visibly marked `Archived — not in transcript` (or equivalent). Selecting it must not pretend to scroll to a displayed message; it may offer the result snippet/metadata as context. Consequence: corpus completeness improves, but users must understand that some hits cannot navigate to a transcript row. ### Option B: Exclude compacted rows from local search Index active rows only, matching the displayed transcript. Consequence: every result retains a coherent result-to-transcript relationship, but compaction-preserved history is intentionally unsearchable. This is the simpler and less surprising behavior. No decision has been made between these options. ## Acceptance criteria for the eventual choice - [ ] The chosen policy is explicit in the search/index contract: active-only (Option B) or active-plus-compacted with provenance (Option A); no implicit dependence on a fetch default. - [ ] A fixture containing both active and compaction-preserved rows verifies the chosen inclusion/exclusion policy and remains deterministic across index rebuild and incremental reconciliation. - [ ] The transcript UI/API remains active-rows-only in both policies; compacted rows never become displayed transcript messages as a side effect of search indexing. - [ ] If Option A is chosen, every compacted hit is visibly labeled archived/unscrollable, cannot claim a scroll target, and active hits still navigate to their displayed message. If Option B is chosen, a term present only in compacted rows produces no local-search hit while a term in an active row still produces a navigable hit. - [ ] Search result identity and account/session scoping remain stable, and the decision does not reintroduce the 500-message retrieval problem addressed separately by #28. ## Duplicate check I searched the open issue list (24 open issues) and read the bodies of #8, #11, #23, #28, #29, #32, and #33. #8 covers the local BM25 search feature; #11 covers Spotlight indexing; #23 covers find within the current displayed conversation; #28 covers the REST history cap and is related but distinct; #29 covers Modules; #32 covers accounts/profiles; and #33 covers sidebar account display. None decides whether local search should index compaction-preserved, transcript-hidden rows. This issue is not a duplicate.

kayg referenced this issue from a commit

2026-08-22 13:02:51 +00:00

fix(history): page REST history so search covers every message

kayg referenced this issue

2026-08-22 18:27:02 +00:00

Tomorrow: sidebar, search, and Settings UI plan #42