Search rebuild temporarily hides committed results #362

Closed
opened 2026-09-28 16:36:10 +00:00 by kayg · 4 comments
Owner

The one-time adversarial round on a real local server found that Search temporarily loses a committed hit during a full staged rebuild.

tests/adversarial/search_chaos.py uploaded SearchChaos/café-nfc.txt containing unicodenfcsentinel and waited until Search returned the marker before starting POST /api/v1/admin/search/rebuild. During that rebuild, all eight concurrent Search requests returned HTTP 200 but omitted the marker. Two further samples while the rebuild was in progress also returned HTTP 200 without it. The probe reported eight concurrent Search during full rebuild ... lost the committed hit findings and two old Index stopped returning a live hit during staged rebuild findings.

Please investigate how the active Index remains visible while the replacement is staged. The search code is outside the mobile-focus job scope, so this issue records the finding for the owning job. The overall adversarial script was time-boxed at 15 minutes and exited 124; this probe itself completed and reported these failures.

The one-time adversarial round on a real local server found that Search temporarily loses a committed hit during a full staged rebuild. `tests/adversarial/search_chaos.py` uploaded `SearchChaos/café-nfc.txt` containing `unicodenfcsentinel` and waited until Search returned the marker before starting `POST /api/v1/admin/search/rebuild`. During that rebuild, all eight concurrent Search requests returned HTTP 200 but omitted the marker. Two further samples while the rebuild was in progress also returned HTTP 200 without it. The probe reported eight `concurrent Search during full rebuild ... lost the committed hit` findings and two `old Index stopped returning a live hit during staged rebuild` findings. Please investigate how the active Index remains visible while the replacement is staged. The search code is outside the mobile-focus job scope, so this issue records the finding for the owning job. The overall adversarial script was time-boxed at 15 minutes and exited 124; this probe itself completed and reported these failures.
Author
Owner

Reproduced during the single split-data adversarial round for #291 on 2026-09-28. The probe again reported that a 200 response omitted the committed unicodenfcsentinel during full/staged rebuild. Server logs showed repeated indexer queue-full warnings and one Tantivy segment merge cancelled because a term file was missing from the staged rebuild directory. This run also included the 20,000-write watcher-overflow fixture. It did not establish cross-User result exposure; see the related availability tracking in #345.

Reproduced during the single split-data adversarial round for #291 on 2026-09-28. The probe again reported that a 200 response omitted the committed `unicodenfcsentinel` during full/staged rebuild. Server logs showed repeated indexer queue-full warnings and one Tantivy segment merge cancelled because a term file was missing from the staged rebuild directory. This run also included the 20,000-write watcher-overflow fixture. It did not establish cross-User result exposure; see the related availability tracking in #345.
Author
Owner

On this run at HEAD f1f7e0ab, the real-server search chaos probe observed GET /api/v1/search?q=unicodenfcsentinel return HTTP 200 without the committed marker while POST /api/v1/admin/search/rebuild was running. It emitted FAIL search chaos: old Index stopped returning a live hit during staged rebuild. The server log from the same run also recorded a Tantivy segment merge cancelled because a .term file under .system/index/tantivy-rebuild/ was missing, plus watcher-queue-full warnings during overflow. The index implementation is outside this UI job's scope; this is evidence for the existing search rebuild issue.

On this run at HEAD f1f7e0ab, the real-server search chaos probe observed `GET /api/v1/search?q=unicodenfcsentinel` return HTTP 200 without the committed marker while `POST /api/v1/admin/search/rebuild` was running. It emitted `FAIL search chaos: old Index stopped returning a live hit during staged rebuild`. The server log from the same run also recorded a Tantivy segment merge cancelled because a `.term` file under `.system/index/tantivy-rebuild/` was missing, plus watcher-queue-full warnings during overflow. The index implementation is outside this UI job's scope; this is evidence for the existing search rebuild issue.
Author
Owner

Reproduced during the single adversarial run for #188 at branch HEAD 9bd81553. After the 20,000-write watcher-overflow fixture, the Search chaos probe ran concurrent queries while a staged Search rebuild was active. It reported HTTP 200 responses without the committed unicodenfcsentinel hit, including repeated old Index stopped returning a live hit during staged rebuild findings. This is the existing rebuild-integrity failure tracked here; this run did not establish cross-User result exposure. The Photos two-User isolation section is still running.

Reproduced during the single adversarial run for #188 at branch HEAD 9bd81553. After the 20,000-write watcher-overflow fixture, the Search chaos probe ran concurrent queries while a staged Search rebuild was active. It reported HTTP 200 responses without the committed `unicodenfcsentinel` hit, including repeated `old Index stopped returning a live hit during staged rebuild` findings. This is the existing rebuild-integrity failure tracked here; this run did not establish cross-User result exposure. The Photos two-User isolation section is still running.
Author
Owner

Fixed by job/search-rebuild (publication drains readers before the directory exchange), merged into dev by Claude. Closing.

Fixed by job/search-rebuild (publication drains readers before the directory exchange), merged into dev by Claude. Closing.
kayg closed this issue 2026-09-28 22:48:32 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#362
No description provided.