Notes test: 700-Note reconcile regression flakes on the shared event bus #1065

New issue

Closed

opened 2026-10-04 17:15:07 +00:00 by kayg · 3 comments

kayg commented

2026-10-04 17:15:07 +00:00

Owner

The #1062 regression seven_hundred_notes_reconcile_without_feedback (crates/plugins/notes/src/lib.rs) subscribes to the process-global calternal_plugin::event_bus(). In the full cargo test -p calternal-plugin-notes run, other tests publish on the same bus at the same time, so the subscriber lags ("one repair must not overflow its subscriber") or counts foreign events. It passes alone (3/3). Isolate it: filter by a unique user id AND tolerate unrelated traffic without the bounded channel overflowing (e.g. a test-local bus handle, or count from the Files events table for this user), so the full suite is deterministic.

The #1062 regression `seven_hundred_notes_reconcile_without_feedback` (crates/plugins/notes/src/lib.rs) subscribes to the process-global `calternal_plugin::event_bus()`. In the full `cargo test -p calternal-plugin-notes` run, other tests publish on the same bus at the same time, so the subscriber lags ("one repair must not overflow its subscriber") or counts foreign events. It passes alone (3/3). Isolate it: filter by a unique user id AND tolerate unrelated traffic without the bounded channel overflowing (e.g. a test-local bus handle, or count from the Files events table for this user), so the full suite is deterministic.

kayg commented

2026-10-04 17:17:15 +00:00

Author

Owner

Starting #1065 on branch job/flake-1065, based on dcad855ee063927c5d95c0a539559377c3db1129 (dev). The test subscribes to the process-global 1,024-slot broadcast bus before a 700-Note reconcile, then fails on any lag. It also uses a unique per-test User ID, but drains only after reconciliation, so unrelated package-test traffic can fill the receiver while it is idle. I am investigating a concurrent filtered drain that keeps the event proof intact.

Starting #1065 on branch `job/flake-1065`, based on `dcad855ee063927c5d95c0a539559377c3db1129` (`dev`). The test subscribes to the process-global 1,024-slot broadcast bus before a 700-Note reconcile, then fails on any lag. It also uses a unique per-test User ID, but drains only after reconciliation, so unrelated package-test traffic can fill the receiver while it is idle. I am investigating a concurrent filtered drain that keeps the event proof intact.

kayg referenced this issue

2026-10-04 17:22:09 +00:00

INCIDENT: Notes change events loop on production (~2,500/min), Log entry writes time out #1062

kayg commented

2026-10-04 17:38:24 +00:00

Author

Owner

Baseline evidence: the original full Notes package suite passed once (193 passed, 1 ignored), so the reported failure is intermittent. The test waited until all 700 reconciled sources had been published before reading its subscriber. I changed the test to drain the process-global bus concurrently, filter by the test's unique User ID, and take a synchronized snapshot after each operation. The focused regression passed (1 passed). It now checks 700 indexed notices across exactly the 700 source paths, zero User events on an unchanged reconcile, and only one indexed plus one removed notice for duplicate index/removal calls.

Baseline evidence: the original full Notes package suite passed once (193 passed, 1 ignored), so the reported failure is intermittent. The test waited until all 700 reconciled sources had been published before reading its subscriber. I changed the test to drain the process-global bus concurrently, filter by the test's unique User ID, and take a synchronized snapshot after each operation. The focused regression passed (1 passed). It now checks 700 `indexed` notices across exactly the 700 source paths, zero User events on an unchanged reconcile, and only one `indexed` plus one `removed` notice for duplicate index/removal calls.

kayg commented

2026-10-04 18:11:45 +00:00

Author

Owner

Finished #1065.

Built: The 700-Note regression test now drains the process-global event bus during reconcile, captures all events for its unique User, and takes synchronized snapshots. It asserts exactly one indexed event for each of the 700 source paths, zero events on an unchanged second reconcile, and only one indexed plus one removed event after duplicate index/removal calls. The production event bus and bridge behavior are unchanged.

Files: crates/plugins/notes/src/lib.rs

Branch/head: job/flake-1065 / 87e1cdee4e279c68f49c86195c4316e53c01b432

Gate output:

cargo fmt --check: no output; exit status 0.
cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings:
Finished \dev` profile [unoptimized + debuginfo] target(s) in 2m 45s`
Five consecutive cargo test -p calternal-plugin-notes runs. Each run also passed the Apple replay integration test and doc tests:
1. test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 179.63s
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
  test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
2. test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 218.90s
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.60s
  test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
3. test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 285.25s
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.61s
  test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
4. test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 602.52s
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.85s
  test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
5. test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 273.34s
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.90s
  test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Cleanup: Removed 7052 files, 2.6GiB total

Known gaps: The original intermittent failure did not reproduce in the single pre-change full-suite baseline run. The updated regression passed five consecutive full-suite runs. No production behavior changed.

Decisions not covered by DESIGN: DESIGN does not specify test synchronization. I kept the production shared bus in the test and added a concurrent User-scoped observer with snapshot barriers instead of adding a test-only event bus.

READY FOR MERGE: yes

Finished #1065. **Built:** The 700-Note regression test now drains the process-global event bus during reconcile, captures all events for its unique User, and takes synchronized snapshots. It asserts exactly one `indexed` event for each of the 700 source paths, zero events on an unchanged second reconcile, and only one `indexed` plus one `removed` event after duplicate index/removal calls. The production event bus and bridge behavior are unchanged. **Files:** `crates/plugins/notes/src/lib.rs` **Branch/head:** `job/flake-1065` / `87e1cdee4e279c68f49c86195c4316e53c01b432` **Gate output:** - `cargo fmt --check`: no output; exit status 0. - `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`: `Finished \`dev\` profile [unoptimized + debuginfo] target(s) in 2m 45s` - Five consecutive `cargo test -p calternal-plugin-notes` runs. Each run also passed the Apple replay integration test and doc tests: 1. `test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 179.63s` `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s` `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s` 2. `test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 218.90s` `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.60s` `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s` 3. `test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 285.25s` `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.61s` `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s` 4. `test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 602.52s` `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.85s` `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s` 5. `test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 273.34s` `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.90s` `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s` - Cleanup: `Removed 7052 files, 2.6GiB total` **Known gaps:** The original intermittent failure did not reproduce in the single pre-change full-suite baseline run. The updated regression passed five consecutive full-suite runs. No production behavior changed. **Decisions not covered by DESIGN:** DESIGN does not specify test synchronization. I kept the production shared bus in the test and added a concurrent User-scoped observer with snapshot barriers instead of adding a test-only event bus. **READY FOR MERGE: yes**

kayg referenced this issue

2026-10-04 18:55:26 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 18:55:27 +00:00

SHARING: one Share dialog everywhere + share Notes, Canvases and folders #1034

kayg referenced this issue

2026-10-04 18:55:28 +00:00

SHARING: invite links for people without an account #1035

kayg referenced this issue

2026-10-04 22:08:32 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038