Search index rebuild can drop existing hits during watcher load #336

Closed
opened 2026-09-28 12:44:10 +00:00 by kayg · 2 comments
Owner

The local adversarial pass at c86efc6f reported three missing-hit checks during a staged Search index rebuild with a 20,000-item watcher-overflow fixture: old Index stopped returning a live hit during staged rebuild (sentinel unicodenfcsentinel). The probe then timed out waiting for its restart-ready-4 marker after 1800 polls at 0.1 seconds. The harness exited and cleaned up the local server; later requests returned 502 local adversarial server is unavailable, so those 502s were not independently validated and may be a cleanup effect. Other adversarial jobs were active on the shared host. Please check whether the old Index remains queryable during a staged rebuild under watcher load and add a regression probe if this is reproducible. No root cause is established.

The local adversarial pass at `c86efc6f` reported three missing-hit checks during a staged Search index rebuild with a 20,000-item watcher-overflow fixture: `old Index stopped returning a live hit during staged rebuild` (sentinel `unicodenfcsentinel`). The probe then timed out waiting for its `restart-ready-4` marker after 1800 polls at 0.1 seconds. The harness exited and cleaned up the local server; later requests returned 502 `local adversarial server is unavailable`, so those 502s were not independently validated and may be a cleanup effect. Other adversarial jobs were active on the shared host. Please check whether the old Index remains queryable during a staged rebuild under watcher load and add a regression probe if this is reproducible. No root cause is established.
Author
Owner

A second local adversarial run reproduced the same finding on merged dev (6c18d807, with the preview-attach fix at 8d9f86b5): tests/adversarial/run.sh with ADVERSARIAL_ATTACK2_ONLY=1 and the 20,000-item watcher-overflow fixture recorded 1,231 checks where Search returned 200 but omitted the committed unicodenfcsentinel hit during a staged rebuild. The one time-boxed round is continuing through the remaining API probes; I will add its final status after it exits. No root cause has been established or changed in this UI job.

A second local adversarial run reproduced the same finding on merged `dev` (`6c18d807`, with the preview-attach fix at `8d9f86b5`): `tests/adversarial/run.sh` with `ADVERSARIAL_ATTACK2_ONLY=1` and the 20,000-item watcher-overflow fixture recorded 1,231 checks where Search returned `200` but omitted the committed `unicodenfcsentinel` hit during a staged rebuild. The one time-boxed round is continuing through the remaining API probes; I will add its final status after it exits. No root cause has been established or changed in this UI job.
Author
Owner

Fixed by job/search-rebuild (publication drains readers before the directory exchange), merged into dev by Claude. Closing.

Fixed by job/search-rebuild (publication drains readers before the directory exchange), merged into dev by Claude. Closing.
kayg closed this issue 2026-09-28 22:48:30 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#336
No description provided.