Notes projection rebuild fails alone with Index busy in 7b

kayg commented

2026-10-03 16:16:59 +00:00

Owner

Found in merge round #867 round 2, branch job/merge-round-7b2. The focused command preserves the original 650 Daily Log fixture:

CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=3 cargo test -p calternal-plugin-notes daily_log_projection_rebuild_resumes_without_markdown_writes -- --test-threads=4

Verbatim output:

called `Result::unwrap()` on an `Err` value: JobError { message: "Index is busy; retry shortly" }
failures:
    tests::daily_log_projection_rebuild_resumes_without_markdown_writes
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 256 filtered out; finished in 178.37s

The full Notes run also reports this failure, and tests::unicode_titles::random_unicode_task_titles_survive_create_read_rename_and_caldav receives 503 instead of its unchanged 201 expectation. The Unicode test is queued alone; no standalone result is claimed yet. Do not weaken either expectation or lower the fixture size to pass the gate. The host is shared, but a returned storage error is not classified as a SLOW-only result.

Three other Notes full-run failures have focused fixes: both migration tests now pass alone, and the saved Task view contract passes after checking every derived field while permitting publication metadata (#746). The nested Daily Log case for #1005 passes in the full crate run. It is not being claimed as a known baseline failure.

Evidence: artifacts/round2-notes-rebuild-alone.log and artifacts/round2-rust-gates/calternal-plugin-notes.test.log in the worktree. Keep the merge blocked until the queue/Index interaction and full crate gate are resolved. No origin/dev baseline result is claimed.

Notes projection rebuild fails alone with Index busy in 7b Found in merge round #867 round 2, branch job/merge-round-7b2. The focused command preserves the original 650 Daily Log fixture: ``` CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=3 cargo test -p calternal-plugin-notes daily_log_projection_rebuild_resumes_without_markdown_writes -- --test-threads=4 ``` Verbatim output: ``` called `Result::unwrap()` on an `Err` value: JobError { message: "Index is busy; retry shortly" } failures: tests::daily_log_projection_rebuild_resumes_without_markdown_writes test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 256 filtered out; finished in 178.37s ``` The full Notes run also reports this failure, and `tests::unicode_titles::random_unicode_task_titles_survive_create_read_rename_and_caldav` receives 503 instead of its unchanged 201 expectation. The Unicode test is queued alone; no standalone result is claimed yet. Do not weaken either expectation or lower the fixture size to pass the gate. The host is shared, but a returned storage error is not classified as a SLOW-only result. Three other Notes full-run failures have focused fixes: both migration tests now pass alone, and the saved Task view contract passes after checking every derived field while permitting publication metadata (#746). The nested Daily Log case for #1005 passes in the full crate run. It is not being claimed as a known baseline failure. Evidence: artifacts/round2-notes-rebuild-alone.log and artifacts/round2-rust-gates/calternal-plugin-notes.test.log in the worktree. Keep the merge blocked until the queue/Index interaction and full crate gate are resolved. No origin/dev baseline result is claimed.

kayg referenced this issue

2026-10-03 16:34:53 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-03 17:05:26 +00:00

Author

Owner

Fresh standalone reproduction on 527c7f74: the unchanged 650 Daily note projection rebuild fails at the awaited rebuild with . The rebuild calls Notes and Task projection write transactions that map transient SQLx failures to 503 but do not retry the full transaction. I am applying the shared bounded retry policy at those idempotent transaction boundaries and keeping the fixture and assertions unchanged.

Fresh standalone reproduction on 527c7f74: the unchanged 650 Daily note projection rebuild fails at the awaited rebuild with . The rebuild calls Notes and Task projection write transactions that map transient SQLx failures to 503 but do not retry the full transaction. I am applying the shared bounded retry policy at those idempotent transaction boundaries and keeping the fixture and assertions unchanged.

kayg commented

2026-10-03 17:06:13 +00:00

Author

Owner

Fresh standalone reproduction on 527c7f74: the unchanged 650 Daily note projection rebuild fails after 50.59 seconds with JobError message Index is busy; retry shortly. The rebuild calls Notes and Task projection writes that can map transient database locks to 503 but do not retry their whole transactions. I will use the existing bounded backoff at those idempotent transaction boundaries. The fixture and assertions stay unchanged.

Fresh standalone reproduction on 527c7f74: the unchanged 650 Daily note projection rebuild fails after 50.59 seconds with JobError message Index is busy; retry shortly. The rebuild calls Notes and Task projection writes that can map transient database locks to 503 but do not retry their whole transactions. I will use the existing bounded backoff at those idempotent transaction boundaries. The fixture and assertions stay unchanged.

kayg commented

2026-10-03 17:50:41 +00:00

Author

Owner

Fix and solo regression result: store::index and the Notes-only projection helper were writing Task rows twice. Removing that duplicate transaction cut rebuild work, but the unchanged 650-fixture test still reproduced SQLite contention during its seeded half. Task and Note projections now retry only SQLite BUSY/LOCKED by starting a fresh complete transaction; other 503 classes are not retried. The original test passes alone after the change:

test tests::daily_log_projection_rebuild_resumes_without_markdown_writes ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 257 filtered out; finished in 130.51s

The filesystem fixtures, 650-note count and assertions are unchanged. The Task-first order now also matches the batch Calendar attachment invariant in DESIGN §29.

Fix and solo regression result: `store::index` and the Notes-only projection helper were writing Task rows twice. Removing that duplicate transaction cut rebuild work, but the unchanged 650-fixture test still reproduced SQLite contention during its seeded half. Task and Note projections now retry only SQLite BUSY/LOCKED by starting a fresh complete transaction; other 503 classes are not retried. The original test passes alone after the change: ``` test tests::daily_log_projection_rebuild_resumes_without_markdown_writes ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 257 filtered out; finished in 130.51s ``` The filesystem fixtures, 650-note count and assertions are unchanged. The Task-first order now also matches the batch Calendar attachment invariant in DESIGN §29.

kayg referenced this issue

2026-10-03 20:34:41 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 08:22:07 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 17:42:32 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-04 17:42:33 +00:00

Author

Owner

Round 7b7 reproduces #1022 in the full Notes gate on the assembled branch. The original fixture and assertions are unchanged. Command: cargo test -p calternal-plugin-notes -- --test-threads=4.

---- tests::daily_log_projection_rebuild_resumes_without_markdown_writes stdout ----
thread 'tests::daily_log_projection_rebuild_resumes_without_markdown_writes' (2027553) panicked at crates/plugins/notes/src/lib.rs:13590:10:
called `Result::unwrap()` on an `Err` value: JobError { message: "Index is busy; retry shortly" }
failures:
    tests::daily_log_projection_rebuild_resumes_without_markdown_writes
test result: FAILED. 263 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 273.80s

The standalone #1062 regression passed. The new Cross-User route regression also passed in this full gate. Notes clippy passed. The focused Daily Log rebuild will run after the remaining requested crate gates. This storage result is not classified as SLOW. Staging remains blocked unless the gate is resolved.

Round 7b7 reproduces #1022 in the full Notes gate on the assembled branch. The original fixture and assertions are unchanged. Command: `cargo test -p calternal-plugin-notes -- --test-threads=4`. ```text ---- tests::daily_log_projection_rebuild_resumes_without_markdown_writes stdout ---- thread 'tests::daily_log_projection_rebuild_resumes_without_markdown_writes' (2027553) panicked at crates/plugins/notes/src/lib.rs:13590:10: called `Result::unwrap()` on an `Err` value: JobError { message: "Index is busy; retry shortly" } failures: tests::daily_log_projection_rebuild_resumes_without_markdown_writes test result: FAILED. 263 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 273.80s ``` The standalone #1062 regression passed. The new Cross-User route regression also passed in this full gate. Notes clippy passed. The focused Daily Log rebuild will run after the remaining requested crate gates. This storage result is not classified as SLOW. Staging remains blocked unless the gate is resolved.

kayg referenced this issue from a commit

2026-10-04 17:44:32 +00:00

Make Files-origin Note misses indistinguishable across Users

kayg referenced this issue

2026-10-04 18:00:40 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 18:27:22 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 18:27:23 +00:00

INCIDENT: Notes change events loop on production (~2,500/min), Log entry writes time out #1062

kayg commented

2026-10-04 18:27:23 +00:00

Author

Owner

Round 7b7 final: the original 650-file rebuild test passed alone. No fixture or expectation changed.

Command: cargo test -p calternal-plugin-notes daily_log_projection_rebuild_resumes_without_markdown_writes -- --test-threads=1.

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 265 filtered out; finished in 57.06s

The full Notes gate still has the storage failure reported above (263 passed, 1 failed, 2 ignored). It was not run again, and it is not classified as SLOW. All other requested crate and web gates passed. The branch stays not ready for staging. Final head: 21420777f3. Complete report is on #867 and #1062, and in docs/audits/merge-round-7b7.md.

Round 7b7 final: the original 650-file rebuild test passed alone. No fixture or expectation changed. Command: `cargo test -p calternal-plugin-notes daily_log_projection_rebuild_resumes_without_markdown_writes -- --test-threads=1`. ```text test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 265 filtered out; finished in 57.06s ``` The full Notes gate still has the storage failure reported above (263 passed, 1 failed, 2 ignored). It was not run again, and it is not classified as SLOW. All other requested crate and web gates passed. The branch stays not ready for staging. Final head: 21420777f39e31e230db85151c9d5ac474295df2. Complete report is on #867 and #1062, and in docs/audits/merge-round-7b7.md.

Rows
Columns

Notes projection rebuild fails alone with Index busy in 7b #1022