Search: a second User's committed hit disappears during rebuild/overflow recovery

kayg commented

2026-10-04 17:18:03 +00:00

Owner

From the orchestrator's full adversarial run on 7b (1b08bac4d): FAIL search chaos: second User lost its committed Search hit during rebuild or overflow recovery. Reproduce with the search chaos section (two Users, concurrent writes during an index rebuild / channel overflow recovery). A committed document of the second User must stay searchable through rebuild and recovery. Find out whether it is pre-existing on production (dev) or a 7b regression.

From the orchestrator's full adversarial run on 7b (1b08bac4d): `FAIL search chaos: second User lost its committed Search hit during rebuild or overflow recovery`. Reproduce with the search chaos section (two Users, concurrent writes during an index rebuild / channel overflow recovery). A committed document of the second User must stay searchable through rebuild and recovery. Find out whether it is pre-existing on production (dev) or a 7b regression.

kayg commented

2026-10-04 17:19:57 +00:00

Author

Owner

Starting repair on job/searchlost-1066, based at dcad855ee063927c5d95c0a539559377c3db1129 (production). I am tracing the two-User rebuild and event-channel overflow cases from tests/adversarial/search_chaos.py on this branch and comparing them with job/7b-reconcile. I will report the reproduction classification and evidence before the fix.

Starting repair on `job/searchlost-1066`, based at `dcad855ee063927c5d95c0a539559377c3db1129` (production). I am tracing the two-User rebuild and event-channel overflow cases from `tests/adversarial/search_chaos.py` on this branch and comparing them with `job/7b-reconcile`. I will report the reproduction classification and evidence before the fix.

kayg commented

2026-10-04 17:43:37 +00:00

Author

Owner

Evidence update: the complete ADVERSARIAL_SEARCH_ONLY=1 Search chaos probe passed on production-based dcad855ee063927c5d95c0a539559377c3db1129. It provisioned two Users, observed User B's committed Note during the staged rebuild, completed the 20,000-file overflow recovery with all three markers searchable, and passed the planned SIGKILL/startup repair. No loss reproduced on production in this run. I am comparing the same probe on 7b revision 1b08bac4d next.

Evidence update: the complete `ADVERSARIAL_SEARCH_ONLY=1` Search chaos probe passed on production-based `dcad855ee063927c5d95c0a539559377c3db1129`. It provisioned two Users, observed User B's committed Note during the staged rebuild, completed the 20,000-file overflow recovery with all three markers searchable, and passed the planned SIGKILL/startup repair. No loss reproduced on production in this run. I am comparing the same probe on 7b revision `1b08bac4d` next.

kayg commented

2026-10-04 17:48:33 +00:00

Author

Owner

The 7b comparison did not reach the probe. Its local server build stopped while compiling llama-cpp-sys-2; sccache reported path must be shorter than SUN_LEN and failed to launch the compiler. This is a build-environment failure before server startup, so it gives no Search result. I am retrying the same comparison with sccache disabled for this local build.

The 7b comparison did not reach the probe. Its local server build stopped while compiling `llama-cpp-sys-2`; `sccache` reported `path must be shorter than SUN_LEN` and failed to launch the compiler. This is a build-environment failure before server startup, so it gives no Search result. I am retrying the same comparison with sccache disabled for this local build.

kayg commented

2026-10-04 19:05:16 +00:00

Author

Owner

Reproduction update (2026-10-04): ran ADVERSARIAL_SEARCH_ONLY=1 bash tests/adversarial/run.sh on the exact job/7b-reconcile head 9fb9a4bfb2488152c83d50c55441ff5f43b572b2. The 7b run exited 1 on two early 5-second Unicode marker waits (unicodenfcsentinel, unicodenfdsentinel); it did not report the #1066 second-User miss, a watcher-overflow marker miss, or an unhealthy integrity result. The server logged expected queue-full warnings during the deliberate 20,000-write burst, then reached the planned SIGKILL restart. The same Search chaos section passed on production base dcad855ee063927c5d95c0a539559377c3db1129.

Added a deterministic calternal-search integration race test. It verifies the filesystem scan order, gates after Bob's Notes, writes Bob's committed Note after the staged pass, and writes Alice's Note after the first overflow scan; it also polls Bob's original committed hit during the rebuild and two full overflow recoveries. The focused test passes in 7.99 s on this branch. Current evidence does not reproduce the reported second-User loss; I am treating the two 5-second Unicode waits as SLOW-only pending their final gate report.

Reproduction update (2026-10-04): ran `ADVERSARIAL_SEARCH_ONLY=1 bash tests/adversarial/run.sh` on the exact `job/7b-reconcile` head `9fb9a4bfb2488152c83d50c55441ff5f43b572b2`. The 7b run exited 1 on two early 5-second Unicode marker waits (`unicodenfcsentinel`, `unicodenfdsentinel`); it did not report the #1066 second-User miss, a watcher-overflow marker miss, or an unhealthy integrity result. The server logged expected queue-full warnings during the deliberate 20,000-write burst, then reached the planned SIGKILL restart. The same Search chaos section passed on production base `dcad855ee063927c5d95c0a539559377c3db1129`. Added a deterministic `calternal-search` integration race test. It verifies the filesystem scan order, gates after Bob's Notes, writes Bob's committed Note after the staged pass, and writes Alice's Note after the first overflow scan; it also polls Bob's original committed hit during the rebuild and two full overflow recoveries. The focused test passes in 7.99 s on this branch. Current evidence does not reproduce the reported second-User loss; I am treating the two 5-second Unicode waits as SLOW-only pending their final gate report.

kayg commented

2026-10-04 19:47:11 +00:00

Author

Owner

#1066 final report

Built: Added a deterministic Search Index integration test. It checks the real filesystem scan order, holds the staged rebuild and overflow scans after Bob's Notes, writes one Bob Note and then an Alice Note only after their Homes have been scanned, forces two full overflow recoveries, and polls Bob's committed hit across each scan and publication.

Diagnosis: The Search chaos section passed on production base dcad855ee063927c5d95c0a539559377c3db1129. On the exact job/7b-reconcile head 9fb9a4bfb2488152c83d50c55441ff5f43b572b2, the real-server round reached the concurrent writes, overflow recovery, and planned SIGKILL restart. It did not report a second-User hit loss, a recovery-marker miss, or an unhealthy integrity result. That run exited 1 only because two early Unicode marker searches missed their five-second waits. The wait is load-sensitive; persistence is not established. The source comparison found no change to shared rebuild or overflow-recovery logic in the 7b delta. The controlled regression test passes on the merged branch. The historical miss is not reproducible, so I made no speculative Search behavior change.

Files changed by this job:

crates/calternal-search/tests/indexer.rs — deterministic two-User rebuild and repeated overflow recovery test.
crates/plugins/notes/src/lib.rs — rustfmt-only cleanup required because the merged origin/dev failed the workspace format gate.

Commits: 7c49c4028 formats the imported Notes test; 2d9b5ea18 adds the Search race test. Final head: 2d9b5ea182229f54d7e70f293b40f8c95673f75b.

Gates (output excerpts verbatim):

cargo fmt --check
[no output; exit 0]

cargo clippy -p calternal-search --all-targets -- -D warnings
    Checking calternal-search v0.0.1 (...)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 16s

cargo test -p calternal-search
    test result: ok. 53 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 24.12s
    test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.55s
    test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s
    test result: ok. 26 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 593.77s
    test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
    test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
    test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 4.33s
    test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
    test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s

cargo test -p calternal-server
    test result: ok. 210 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 84.38s
    test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 26.74s
    test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.33s

Web build completed before the Rust gates. cargo clean output: Removed 22317 files, 19.6GiB total.

Known gap: The original reported second-User miss has no confirmed root cause because it did not reproduce on either tested revision. The two 5-second Unicode waits from the 7b run remain unclassified beyond being load-sensitive; they are outside the reported second-User failure.

Decision: Keep this change test-only. It encodes the required ordering and overflow recovery without changing product behavior in the absence of a reproducible bug.

READY FOR MERGE: yes (regression guard; no behavior change).

#1066 final report **Built:** Added a deterministic Search Index integration test. It checks the real filesystem scan order, holds the staged rebuild and overflow scans after Bob's Notes, writes one Bob Note and then an Alice Note only after their Homes have been scanned, forces two full overflow recoveries, and polls Bob's committed hit across each scan and publication. **Diagnosis:** The Search chaos section passed on production base `dcad855ee063927c5d95c0a539559377c3db1129`. On the exact `job/7b-reconcile` head `9fb9a4bfb2488152c83d50c55441ff5f43b572b2`, the real-server round reached the concurrent writes, overflow recovery, and planned SIGKILL restart. It did not report a second-User hit loss, a recovery-marker miss, or an unhealthy integrity result. That run exited 1 only because two early Unicode marker searches missed their five-second waits. The wait is load-sensitive; persistence is not established. The source comparison found no change to shared rebuild or overflow-recovery logic in the 7b delta. The controlled regression test passes on the merged branch. The historical miss is not reproducible, so I made no speculative Search behavior change. **Files changed by this job:** - `crates/calternal-search/tests/indexer.rs` — deterministic two-User rebuild and repeated overflow recovery test. - `crates/plugins/notes/src/lib.rs` — rustfmt-only cleanup required because the merged `origin/dev` failed the workspace format gate. **Commits:** `7c49c4028` formats the imported Notes test; `2d9b5ea18` adds the Search race test. Final head: `2d9b5ea182229f54d7e70f293b40f8c95673f75b`. **Gates (output excerpts verbatim):** ```text cargo fmt --check [no output; exit 0] cargo clippy -p calternal-search --all-targets -- -D warnings Checking calternal-search v0.0.1 (...) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 16s cargo test -p calternal-search test result: ok. 53 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 24.12s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.55s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s test result: ok. 26 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 593.77s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 4.33s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s cargo test -p calternal-server test result: ok. 210 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 84.38s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 26.74s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.33s ``` Web build completed before the Rust gates. `cargo clean` output: `Removed 22317 files, 19.6GiB total`. **Known gap:** The original reported second-User miss has no confirmed root cause because it did not reproduce on either tested revision. The two 5-second Unicode waits from the 7b run remain unclassified beyond being load-sensitive; they are outside the reported second-User failure. **Decision:** Keep this change test-only. It encodes the required ordering and overflow recovery without changing product behavior in the absence of a reproducible bug. **READY FOR MERGE: yes** (regression guard; no behavior change).

kayg commented

2026-10-04 19:48:03 +00:00

Author

Owner

Correction to the final report's Search clippy excerpt: the exact output tail was:

    Checking calternal-search v0.0.1 (/home/kayg/Developer/calternal-wt/searchlost-1066/crates/calternal-search)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 16s

Correction to the final report's Search clippy excerpt: the exact output tail was: ```text Checking calternal-search v0.0.1 (/home/kayg/Developer/calternal-wt/searchlost-1066/crates/calternal-search) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 16s ```

kayg commented

2026-10-04 19:50:04 +00:00

Author

Owner

Orchestrator: the job could not reproduce the loss on production code or on 7b; it added a regression guard (tests only). Keeping this open until a clean low-load adversarial run shows the search chaos section green.

kayg commented

2026-10-06 05:12:45 +00:00

Author

Owner

Starting #1066 on job/search-1066, based on origin/dev at d0061ec3df127d81c86729d07d899b0bf2b6de91. I have read the binding search decisions (§32 and §48); the issue requires per-User index isolation through rebuild and overflow recovery. I will merge the current origin/dev before reproduction, then add the deterministic regression and interleaving property test.

Starting #1066 on `job/search-1066`, based on `origin/dev` at `d0061ec3df127d81c86729d07d899b0bf2b6de91`. I have read the binding search decisions (§32 and §48); the issue requires per-User index isolation through rebuild and overflow recovery. I will merge the current `origin/dev` before reproduction, then add the deterministic regression and interleaving property test.

kayg commented

2026-10-06 05:56:48 +00:00

Author

Owner

Finding on refreshed origin/dev (d0061ec3): the #956 fix that writes staged shared rebuilds into ready private Indexes is an ancestor of both origin/dev and the reported 7b commit 1b08bac4d (git merge-base --is-ancestor 4a26749f3 <ref> exits 0 for each). The current Search code uses a separate private directory per User and has no shared per-User generation counter. I could not confirm a production data-loss path from a shared cursor or cross-User delete.

I added a deterministic test that holds publication after the scan, commits Alice and Bob files concurrently, fills the actor queue, exceeds the targeted-rescan cap to force a full scan, then checks Bob's earlier hit and each User's recovered hits. It passes on current origin/dev. I also added a proptest over three Users that interleaves commits with a global rebuild, a private rebuild, and integrity recovery, then checks every committed marker against every User. That passes too. The issue does not reproduce on this branch; the exact cause of the 7b report remains unconfirmed. I am continuing with the requested gates.

Finding on refreshed `origin/dev` (`d0061ec3`): the #956 fix that writes staged shared rebuilds into ready private Indexes is an ancestor of both `origin/dev` and the reported 7b commit `1b08bac4d` (`git merge-base --is-ancestor 4a26749f3 <ref>` exits 0 for each). The current Search code uses a separate private directory per User and has no shared per-User generation counter. I could not confirm a production data-loss path from a shared cursor or cross-User delete. I added a deterministic test that holds publication after the scan, commits Alice and Bob files concurrently, fills the actor queue, exceeds the targeted-rescan cap to force a full scan, then checks Bob's earlier hit and each User's recovered hits. It passes on current `origin/dev`. I also added a proptest over three Users that interleaves commits with a global rebuild, a private rebuild, and integrity recovery, then checks every committed marker against every User. That passes too. The issue does not reproduce on this branch; the exact cause of the 7b report remains unconfirmed. I am continuing with the requested gates.

kayg commented

2026-10-06 06:10:01 +00:00

Author

Owner

Gate finding: cargo test -p calternal-search -- --test-threads=4 ran all unit tests and the integration binary. Both new #1066 tests passed. The integration binary finished 26 passed; 1 failed; the sole failure is the unchanged search_returns_committed_hits_while_the_frecency_pool_is_busy fixture. Its CREATE TABLE search_manifest omits mime, and reconcile_at_start() fails with Background("error returned from database: (code: 1) no such column: mime") at crates/calternal-search/tests/indexer.rs:1213. I did not alter the old fixture because the owner rule says not to change existing fixtures to make a test pass. The existing overflow recovery test passed.

Gate finding: `cargo test -p calternal-search -- --test-threads=4` ran all unit tests and the integration binary. Both new #1066 tests passed. The integration binary finished `26 passed; 1 failed`; the sole failure is the unchanged `search_returns_committed_hits_while_the_frecency_pool_is_busy` fixture. Its `CREATE TABLE search_manifest` omits `mime`, and `reconcile_at_start()` fails with `Background("error returned from database: (code: 1) no such column: mime")` at `crates/calternal-search/tests/indexer.rs:1213`. I did not alter the old fixture because the owner rule says not to change existing fixtures to make a test pass. The existing overflow recovery test passed.

kayg referenced this issue from a commit

2026-10-06 06:21:09 +00:00

test(search): cover per-user rebuild overflow recovery (#1066)

kayg commented

2026-10-06 06:22:06 +00:00

Author

Owner

Completed on job/search-1066.

Head SHA: de39537e7a14714e11da8c2d25f324a3732d0a1b

Finding and decision: the reported disappearance did not reproduce on the merged origin/dev. The staged rebuild/private-upsert fix (4a26749f3, #956) is an ancestor of both the reported 7b commit 1b08bac4d and current dev. I found no shared per-User generation counter or cross-User delete in the current path. I added regression coverage without changing production behavior: a deterministic two-User concurrent commit + staged rebuild + queue overflow test, and a generated three-User commit/rebuild/integrity-recovery test that checks exact per-User hit membership and isolation. This keeps the current behavior because the requested failure was not reproducible; no additional design decision was needed.

Files changed:

crates/calternal-search/src/indexer.rs
crates/calternal-search/tests/indexer.rs

Gates:

cargo fmt --check

(exit 0; no output)

cargo clippy -p calternal-search --all-targets -- -D warnings

Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.64s

Focused deterministic regression test:

test indexer::tests::private_hits_survive_concurrent_commits_and_overflow_during_rebuild ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 56 filtered out; finished in 3.33s

Focused property test:

test interleaved_user_commits_and_rebuilds_keep_search_isolated ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 26 filtered out; finished in 13.57s

cargo test -p calternal-search -- --test-threads=4:

test result: ok. 56 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 22.22s
test result: FAILED. 26 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 315.46s
error: test failed, to rerun pass `-p calternal-search --test indexer`

The sole failure is the existing unchanged search_returns_committed_hits_while_the_frecency_pool_is_busy test. Its manual database fixture lacks the mime column required by reconcile_at_start(); it fails with Background("error returned from database: (code: 1) no such column: mime") at crates/calternal-search/tests/indexer.rs:1213. I did not alter its fixture or expectation.

bun run check:

PASS production dependency licences: 759 locked package releases across apps/web, apps/docs and packages
perf-lint: PASS; 0 violations; 22104 scoped exceptions
svelte-check found 0 errors and 4 warnings in 3 files

bun run test:

Test Files  266 passed (266)
Tests  1830 passed (1830)
Duration  320.32s

Known gap: the existing Search integration fixture failure above remains for the merge round to resolve. The web check reports four existing CSS warnings and zero errors. No user-facing UI or API behavior changed, so no screenshots, server adversarial run, or feature benchmark applies.

Completed on `job/search-1066`. Head SHA: `de39537e7a14714e11da8c2d25f324a3732d0a1b` Finding and decision: the reported disappearance did not reproduce on the merged `origin/dev`. The staged rebuild/private-upsert fix (`4a26749f3`, #956) is an ancestor of both the reported 7b commit `1b08bac4d` and current dev. I found no shared per-User generation counter or cross-User delete in the current path. I added regression coverage without changing production behavior: a deterministic two-User concurrent commit + staged rebuild + queue overflow test, and a generated three-User commit/rebuild/integrity-recovery test that checks exact per-User hit membership and isolation. This keeps the current behavior because the requested failure was not reproducible; no additional design decision was needed. Files changed: - `crates/calternal-search/src/indexer.rs` - `crates/calternal-search/tests/indexer.rs` Gates: `cargo fmt --check` ```text (exit 0; no output) ``` `cargo clippy -p calternal-search --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.64s ``` Focused deterministic regression test: ```text test indexer::tests::private_hits_survive_concurrent_commits_and_overflow_during_rebuild ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 56 filtered out; finished in 3.33s ``` Focused property test: ```text test interleaved_user_commits_and_rebuilds_keep_search_isolated ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 26 filtered out; finished in 13.57s ``` `cargo test -p calternal-search -- --test-threads=4`: ```text test result: ok. 56 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 22.22s test result: FAILED. 26 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 315.46s error: test failed, to rerun pass `-p calternal-search --test indexer` ``` The sole failure is the existing unchanged `search_returns_committed_hits_while_the_frecency_pool_is_busy` test. Its manual database fixture lacks the `mime` column required by `reconcile_at_start()`; it fails with `Background("error returned from database: (code: 1) no such column: mime")` at `crates/calternal-search/tests/indexer.rs:1213`. I did not alter its fixture or expectation. `bun run check`: ```text PASS production dependency licences: 759 locked package releases across apps/web, apps/docs and packages perf-lint: PASS; 0 violations; 22104 scoped exceptions svelte-check found 0 errors and 4 warnings in 3 files ``` `bun run test`: ```text Test Files 266 passed (266) Tests 1830 passed (1830) Duration 320.32s ``` Known gap: the existing Search integration fixture failure above remains for the merge round to resolve. The web check reports four existing CSS warnings and zero errors. No user-facing UI or API behavior changed, so no screenshots, server adversarial run, or feature benchmark applies.

Rows
Columns

Search: a second User's committed hit disappears during rebuild/overflow recovery #1066