INCIDENT: Notes change events loop on production (~2,500/min), Log entry writes time out #1062

Closed
opened 2026-10-04 15:22:55 +00:00 by kayg · 12 comments
Owner

Production incident (2026-10-04, c39ffe5d9, ongoing)

Since 15:26:54 CEST the server publishes a Files change event for every Note (~688 distinct Notes/ paths, each ~35× per 10 min, ≈2,500 events/min). The owner's Log entry "Send all" requests time out after the 20 s client timeout.

Evidence (read-only):

  • files_events: 23,871 rows in 10 min; 23,801 under Notes/, 688 distinct paths, uniform ~35 each.
  • Log: only calternal_webdav::profile lines stage="change_event_commit" and stage="sse_wakeup_signal" (crates/plugins/files/src/lib.rs publish_inner, called via publish_home_change from bridge_note_changes (Notes plugin events) and watch_home_changes in crates/calternal-server/src/wire.rs). No other stages, so these are not real uploads.
  • Note files on disk are NOT modified (no .md changed in 2 min). inotify on Notes/ for 15 s: only 688 OPEN events (every Note read once per ~17 s pass), no MODIFY/ATTRIB/CLOSE_WRITE.
  • Jobs table: no job kind is looping (thumbnails, notifications, one mail.sync).
  • A production restart at 17:19 did not stop it: the rate climbed back to ~2,400/min within a minute. Something persistent drives it: a reconnecting client (web/PWA SSE subscriber whose refetch triggers server-side Notes events → bridge → Files change → SSE → refetch …), or a server-side periodic pass.
  • The sqlx writer thread is at ~50 % CPU.
  • No app password was used in that window except CalDAV ("16pm caldav", ~25 min before 17:14); web sessions active.

Wanted (blocking, hotfix)

  1. Root cause: which code path reads every Note and emits a Notes plugin event (or publish_home_change) per Note. Candidates: a GET/projection endpoint that calls changed(); a client-driven refetch loop through SSE; CalDAV REPORT handling; a periodic projection refresh (#643/#647 refetch-without-Markdown-change).
  2. Fix so read paths never emit change events, and an unchanged Note never produces a change event (compare content hash before publishing). Add a loop guard: the bridge must not re-publish events that originate from its own publications.
  3. Regression test: an idle server with 700 Notes and an open SSE client produces ~0 change events per minute; a single Log entry write produces O(1) events.
  4. Add an idle steady-state probe to bench/ and tests/adversarial/ (change events per minute on an idle instance with real-size data) so this class is caught before production.
## Production incident (2026-10-04, c39ffe5d9, ongoing) Since **15:26:54 CEST** the server publishes a Files change event for every Note (~688 distinct Notes/ paths, each ~35× per 10 min, ≈2,500 events/min). The owner's Log entry "Send all" requests time out after the 20 s client timeout. Evidence (read-only): - `files_events`: 23,871 rows in 10 min; 23,801 under `Notes/`, 688 distinct paths, uniform ~35 each. - Log: only `calternal_webdav::profile` lines `stage="change_event_commit"` and `stage="sse_wakeup_signal"` (crates/plugins/files/src/lib.rs `publish_inner`, called via `publish_home_change` from `bridge_note_changes` (Notes plugin events) and `watch_home_changes` in crates/calternal-server/src/wire.rs). No other stages, so these are not real uploads. - Note files on disk are NOT modified (no .md changed in 2 min). inotify on Notes/ for 15 s: only 688 OPEN events (every Note read once per ~17 s pass), no MODIFY/ATTRIB/CLOSE_WRITE. - Jobs table: no job kind is looping (thumbnails, notifications, one mail.sync). - A production restart at 17:19 did not stop it: the rate climbed back to ~2,400/min within a minute. Something persistent drives it: a reconnecting client (web/PWA SSE subscriber whose refetch triggers server-side Notes events → bridge → Files change → SSE → refetch …), or a server-side periodic pass. - The sqlx writer thread is at ~50 % CPU. - No app password was used in that window except CalDAV ("16pm caldav", ~25 min before 17:14); web sessions active. ## Wanted (blocking, hotfix) 1. Root cause: which code path reads every Note and emits a Notes plugin event (or publish_home_change) per Note. Candidates: a GET/projection endpoint that calls `changed()`; a client-driven refetch loop through SSE; CalDAV REPORT handling; a periodic projection refresh (#643/#647 refetch-without-Markdown-change). 2. Fix so read paths never emit change events, and an unchanged Note never produces a change event (compare content hash before publishing). Add a loop guard: the bridge must not re-publish events that originate from its own publications. 3. Regression test: an idle server with 700 Notes and an open SSE client produces ~0 change events per minute; a single Log entry write produces O(1) events. 4. Add an idle steady-state probe to bench/ and tests/adversarial/ (change events per minute on an idle instance with real-size data) so this class is caught before production.
Author
Owner

Starting #1062 on branch job/loop-1062, base c39ffe5d9 (production). Production stays read-only. Tracing unchanged Note indexing and shared event-bus lag recovery; will reproduce with 700 Notes, a browser SSE subscriber, and CalDAV polling before reporting a hotfix.

Starting #1062 on branch job/loop-1062, base c39ffe5d9 (production). Production stays read-only. Tracing unchanged Note indexing and shared event-bus lag recovery; will reproduce with 700 Notes, a browser SSE subscriber, and CalDAV polling before reporting a hotfix.
Author
Owner

Owner closed calternal on every device (app + all browser tabs) at ~17:24 CEST. Change-event rate afterwards per 30 s: 1205, 1225, 1175, 1170, 1209, 1155, so unchanged. The loop is not driven by an open web/PWA client. Remaining candidates: a server-side periodic pass, or a background protocol client that keeps running when the app is closed (the iPhone/Mac Calendar or Reminders CalDAV account "16pm caldav", Files/WebDAV clients, the Notes IMAP edge). Start with server-side timers and CalDAV REPORT/PROPFIND handling.

Owner closed calternal on every device (app + all browser tabs) at ~17:24 CEST. Change-event rate afterwards per 30 s: 1205, 1225, 1175, 1170, 1209, 1155, so unchanged. **The loop is not driven by an open web/PWA client.** Remaining candidates: a server-side periodic pass, or a background protocol client that keeps running when the app is closed (the iPhone/Mac Calendar or Reminders CalDAV account "16pm caldav", Files/WebDAV clients, the Notes IMAP edge). Start with server-side timers and CalDAV REPORT/PROPFIND handling.
Author
Owner

Root cause confirmed in code at production base c39ffe5d9: Notes reconcile_user calls tasks_store::reconcile_user, which calls store::index for every Note, then store::reconcile_user, which indexes every Note again. index_note_projection unconditionally emits notes/indexed, even though imap::record_change already detects an unchanged content hash. For 688 Notes one pass emits at least 1,376 events into the 1,024-slot shared bus. The Notes subscriber is busy running that same reconciliation; its next receive is Lagged, which runs full reconciliation again. Each pass reads every Note without modifying Markdown and bridge_note_changes turns every indexed notification into a durable files_events row. This explains the uniform per-path rate, read-only inotify evidence, persistence across restart, and writer contention without a looping durable job or a client.

Fix in progress: reuse the existing transactional IMAP content-hash comparison to emit indexed only for a changed source, preserve dependency projection rebuilds, and restrict the Notes-to-Files bridge to durable file mutation kinds. A projection notification must not become a second file mutation publication. Local 700-Note reproduction and steady-state probe follow.

Root cause confirmed in code at production base c39ffe5d9: Notes `reconcile_user` calls `tasks_store::reconcile_user`, which calls `store::index` for every Note, then `store::reconcile_user`, which indexes every Note again. `index_note_projection` unconditionally emits `notes/indexed`, even though `imap::record_change` already detects an unchanged content hash. For 688 Notes one pass emits at least 1,376 events into the 1,024-slot shared bus. The Notes subscriber is busy running that same reconciliation; its next receive is `Lagged`, which runs full reconciliation again. Each pass reads every Note without modifying Markdown and `bridge_note_changes` turns every `indexed` notification into a durable `files_events` row. This explains the uniform per-path rate, read-only inotify evidence, persistence across restart, and writer contention without a looping durable job or a client. Fix in progress: reuse the existing transactional IMAP content-hash comparison to emit `indexed` only for a changed source, preserve dependency projection rebuilds, and restrict the Notes-to-Files bridge to durable file mutation kinds. A projection notification must not become a second file mutation publication. Local 700-Note reproduction and steady-state probe follow.
Author
Owner

More evidence (17:28–17:31 CEST):

  • The owner turned the iPhone Calendar/Reminders CalDAV account off. The rate stayed the same: 1305, 1051, 1229, 1122, 1207 per 30 s. Not CalDAV from the phone.
  • ss on production shows no inbound connection to :8080 at all while the loop runs. Outbound: two :443 peers (provider sync or calendar subscription fetches). The loop is server-internal.
  • The log shows bursts of calternal_plugin_notes::tasks_store: Skipped a Task projection with an empty title (#623) every ~10–13 s, in step with the per-pass Note reads. The tasks projection is re-indexing on every pass. Look for a periodic pass, or a self-triggering Notes event → bridge → watcher/adopt → re-index → event cycle that persists across restarts (for example a persisted cursor or queue that never advances).
More evidence (17:28–17:31 CEST): - The owner turned the iPhone Calendar/Reminders CalDAV account off. The rate stayed the same: 1305, 1051, 1229, 1122, 1207 per 30 s. **Not CalDAV from the phone.** - `ss` on production shows **no inbound connection to :8080 at all** while the loop runs. Outbound: two :443 peers (provider sync or calendar subscription fetches). **The loop is server-internal.** - The log shows bursts of `calternal_plugin_notes::tasks_store: Skipped a Task projection with an empty title (#623)` every ~10–13 s, in step with the per-pass Note reads. The tasks projection is re-indexing on every pass. Look for a periodic pass, or a self-triggering Notes event → bridge → watcher/adopt → re-index → event cycle that persists across restarts (for example a persisted cursor or queue that never advances).
Author
Owner

Local production-base reproduction completed (c39ffe5d9, 700 Notes: 350 Daily notes with 24 timed Log bullets each, 350 ordinary Notes; 2,414,890 Markdown bytes). Real Chromium production app had an open Files SSE stream.

Minute samples: current web 976 Files events (975.92/min); current web with CalDAV area REPORT polling 1,621 (1,598.90/min); historical 2026-10-03 web app source 22f13995a with the same installed package dependencies 1,882 (1,881.97/min); idle after one Log write 1,364 (1,363.98/min). A single Log POST eventually returned 201 after 74,066.36 ms, beyond the production client's 20 s timeout. Its request plus the following 5 s had 2,170 Files events. This is a server-side feedback loop; neither a stale client nor DAV is required to drive it.

The Notes hash fix is committed as 8ebca1b755. The 700-Note focused regression and Notes clippy passed; full Notes tests passed: test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 135.84s plus Apple replay test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.39s. Fetched origin and merged origin/dev once: Already up to date. Files and server gates are running, followed by the same hotfix idle probe. No production changes.

Local production-base reproduction completed (c39ffe5d9, 700 Notes: 350 Daily notes with 24 timed Log bullets each, 350 ordinary Notes; 2,414,890 Markdown bytes). Real Chromium production app had an open Files SSE stream. Minute samples: current web 976 Files events (975.92/min); current web with CalDAV area REPORT polling 1,621 (1,598.90/min); historical 2026-10-03 web app source 22f13995a with the same installed package dependencies 1,882 (1,881.97/min); idle after one Log write 1,364 (1,363.98/min). A single Log POST eventually returned 201 after 74,066.36 ms, beyond the production client's 20 s timeout. Its request plus the following 5 s had 2,170 Files events. This is a server-side feedback loop; neither a stale client nor DAV is required to drive it. The Notes hash fix is committed as 8ebca1b755b316f0d022940e09b5a60598a7a5cd. The 700-Note focused regression and Notes clippy passed; full Notes tests passed: `test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 135.84s` plus Apple replay `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.39s`. Fetched origin and merged origin/dev once: `Already up to date.` Files and server gates are running, followed by the same hotfix idle probe. No production changes.
Author
Owner

READY FOR HOTFIX: yes, for the event-loop repair. Cold-start latency remains a SLOW limitation.

Branch: job/loop-1062. Base: c39ffe5d9. Head: c358622337. Six atomic commits; no push, merge into dev, deploy or production changes.

Built: stop the Notes reconciliation feedback loop. reconcile_user indexes every Note through Task and Notes passes. Unconditional notes/indexed publication produced more than 1,024 events for 688 Notes, overflowed its own subscriber, and triggered full reconciliation again. The Files bridge converted projection notifications into durable file events. No client is needed to sustain this loop.

The patch reuses the existing transactional IMAP hash/path/deletion comparison. Unchanged sources do not emit indexed/removed notices; dependency projections still rebuild. The Notes-to-Files bridge accepts durable file mutation kinds only. The filesystem watcher publishes after Notes adoption, so Calendar reads committed projections. Added a 700-Note bus-capacity/read/duplicate-write/removal regression and a bridge-kind regression. Existing test expectations are unchanged.

Files: crates/plugins/notes/src/{imap.rs,store.rs,lib.rs}; crates/calternal-server/src/wire.rs; tests/adversarial/notes_idle.mjs; bench/notes-idle.mjs.

Evidence: real local server, production browser build with macOS platform emulation, open Files SSE, 700 Notes (350 Daily notes with 24 timed Log bullets; 350 ordinary Notes), CalDAV REPORT polling every 15 s, and the fully historical 2026-10-03 app/UI/editor build 22f13995a. Production-base reproduction had 976–1,882 Files events per minute and a single Log POST took 74,066.36 ms. Baseline fixture also ran legitimate upgrade backfills; treat rates as reproduction evidence, not a calibrated throughput comparison. The patched full-minute probe had zero Files events and zero change_event_commit stages in all four phases, unchanged Markdown bytes and timestamps, and a Log POST returned 201 in 1,461.78 ms with four events. On the final watcher-ordering binary, the 700-Note run again had zero events and unchanged Markdown in all three pre-write phases (15 s each), but the Log POST hit TimeoutError after 20,221.56 ms; fresh initialization took 264,717.34 ms. This cold-start timing result is not hidden or classified as a pass. A final focused 24-Note run passed: all three idle phases were zero, Log returned 201 in 4,879.89 ms with four events, Send all returned 201 in 3,970.72 ms with three events, and its entry appeared in Calendar. The fully historical client was used in both 700-Note patched runs. Artifacts: hotfix-idle.json/log, final-idle.log, final-focused.json/log and the full gate logs under artifacts/loop-1062/.

Performance: local busy host, load average 21.21/22.81/20.83 in the full-minute run. CPU samples were 109.37%, 68.9%, 30.6%, 32.35%; RSS 581,554,176 through 719,163,392 bytes while initial background indexing settled. No matching idle-event metric exists in docs/perf/baseline.json. These are incident measurements, not isolated perf VM results or p50/p95 estimates. Profile command: CALTERNAL_SERVER_BIN= NOTES_IDLE_OLD_BUILD= node bench/notes-idle.mjs.

Gates (OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4; apps/web production build completed first):

cargo fmt --check
(exit 0; no output)
cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.39s
cargo test -p calternal-plugin-notes -- --test-threads=4
test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 135.84s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.39s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
cargo clippy -p calternal-plugin-files --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 26s
cargo test -p calternal-plugin-files -- --test-threads=4
test result: ok. 158 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 114.24s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
cargo clippy -p calternal-server --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 01s
cargo test -p calternal-server -- --test-threads=4
test result: ok. 164 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 54.11s

No web source changed, so bun run check is not required. Both new JavaScript entry points pass node --check. Fetched origin and merged origin/dev once before final gates: Already up to date.

Decisions: reuse the existing IMAP source hash rather than add another ledger or short-circuit dependency repairs; exclude derived Notes notifications from the Files mutation bridge; move the watcher hint after adoption to preserve Calendar freshness. Seed completed upgrade ledgers only while the fixture server is stopped, because production has already completed those upgrades; the probe performs real projection initialization before idle measurement. No schema, dependency or API contract changes.

UX gaps closed: The repeated repair storm stops; the successful 700-Note run shows bounded Log events. The final focused run verifies Send all projection freshness. UX gaps left / known gaps: cold Home initialization still takes a finite full scan (94,762.7 ms and 264,717.34 ms across the two 700-Note runs); the second migrated fixture still exceeded the 20 s Log timeout after setup. Exact remaining lock contention was not isolated. This is a recorded SLOW finding and does not invalidate the zero-event regression, but this hotfix does not promise a 20 s write bound on this busy host. No cold-start performance redesign in this hotfix. No UI code changed. Production remains untouched. The orchestrator must deploy and verify steady-state production counters and Log / Send all after deployment.

Cleanup completed: cargo clean reported Removed 17954 files, 10.1GiB total. Generated web builds, historical fixture and local test data were removed. Git status is clean. Re-read the doc comments in all six changed files.

READY FOR HOTFIX: yes, for the event-loop repair. Cold-start latency remains a SLOW limitation. Branch: job/loop-1062. Base: c39ffe5d9. Head: c3586223376945a85ca2b47946027627cd7f38c3. Six atomic commits; no push, merge into dev, deploy or production changes. Built: stop the Notes reconciliation feedback loop. `reconcile_user` indexes every Note through Task and Notes passes. Unconditional `notes/indexed` publication produced more than 1,024 events for 688 Notes, overflowed its own subscriber, and triggered full reconciliation again. The Files bridge converted projection notifications into durable file events. No client is needed to sustain this loop. The patch reuses the existing transactional IMAP hash/path/deletion comparison. Unchanged sources do not emit indexed/removed notices; dependency projections still rebuild. The Notes-to-Files bridge accepts durable file mutation kinds only. The filesystem watcher publishes after Notes adoption, so Calendar reads committed projections. Added a 700-Note bus-capacity/read/duplicate-write/removal regression and a bridge-kind regression. Existing test expectations are unchanged. Files: crates/plugins/notes/src/{imap.rs,store.rs,lib.rs}; crates/calternal-server/src/wire.rs; tests/adversarial/notes_idle.mjs; bench/notes-idle.mjs. Evidence: real local server, production browser build with macOS platform emulation, open Files SSE, 700 Notes (350 Daily notes with 24 timed Log bullets; 350 ordinary Notes), CalDAV REPORT polling every 15 s, and the fully historical 2026-10-03 app/UI/editor build 22f13995a. Production-base reproduction had 976–1,882 Files events per minute and a single Log POST took 74,066.36 ms. Baseline fixture also ran legitimate upgrade backfills; treat rates as reproduction evidence, not a calibrated throughput comparison. The patched full-minute probe had zero Files events and zero change_event_commit stages in all four phases, unchanged Markdown bytes and timestamps, and a Log POST returned 201 in 1,461.78 ms with four events. On the final watcher-ordering binary, the 700-Note run again had zero events and unchanged Markdown in all three pre-write phases (15 s each), but the Log POST hit TimeoutError after 20,221.56 ms; fresh initialization took 264,717.34 ms. This cold-start timing result is not hidden or classified as a pass. A final focused 24-Note run passed: all three idle phases were zero, Log returned 201 in 4,879.89 ms with four events, Send all returned 201 in 3,970.72 ms with three events, and its entry appeared in Calendar. The fully historical client was used in both 700-Note patched runs. Artifacts: hotfix-idle.json/log, final-idle.log, final-focused.json/log and the full gate logs under artifacts/loop-1062/. Performance: local busy host, load average 21.21/22.81/20.83 in the full-minute run. CPU samples were 109.37%, 68.9%, 30.6%, 32.35%; RSS 581,554,176 through 719,163,392 bytes while initial background indexing settled. No matching idle-event metric exists in docs/perf/baseline.json. These are incident measurements, not isolated perf VM results or p50/p95 estimates. Profile command: CALTERNAL_SERVER_BIN=<server> NOTES_IDLE_OLD_BUILD=<historical build> node bench/notes-idle.mjs. Gates (OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4; apps/web production build completed first): ``` cargo fmt --check (exit 0; no output) cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.39s cargo test -p calternal-plugin-notes -- --test-threads=4 test result: ok. 193 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 135.84s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.39s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-plugin-files --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 26s cargo test -p calternal-plugin-files -- --test-threads=4 test result: ok. 158 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 114.24s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 01s cargo test -p calternal-server -- --test-threads=4 test result: ok. 164 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 54.11s ``` No web source changed, so bun run check is not required. Both new JavaScript entry points pass node --check. Fetched origin and merged origin/dev once before final gates: Already up to date. Decisions: reuse the existing IMAP source hash rather than add another ledger or short-circuit dependency repairs; exclude derived Notes notifications from the Files mutation bridge; move the watcher hint after adoption to preserve Calendar freshness. Seed completed upgrade ledgers only while the fixture server is stopped, because production has already completed those upgrades; the probe performs real projection initialization before idle measurement. No schema, dependency or API contract changes. UX gaps closed: The repeated repair storm stops; the successful 700-Note run shows bounded Log events. The final focused run verifies Send all projection freshness. UX gaps left / known gaps: cold Home initialization still takes a finite full scan (94,762.7 ms and 264,717.34 ms across the two 700-Note runs); the second migrated fixture still exceeded the 20 s Log timeout after setup. Exact remaining lock contention was not isolated. This is a recorded SLOW finding and does not invalidate the zero-event regression, but this hotfix does not promise a 20 s write bound on this busy host. No cold-start performance redesign in this hotfix. No UI code changed. Production remains untouched. The orchestrator must deploy and verify steady-state production counters and Log / Send all after deployment. Cleanup completed: cargo clean reported `Removed 17954 files, 10.1GiB total`. Generated web builds, historical fixture and local test data were removed. Git status is clean. Re-read the doc comments in all six changed files.
Author
Owner

Hotfix deployed to production (2026-10-04 19:12 CEST, dcad855ee)

Cherry-picked 8ebca1b75 (publish projection changes only when the source hash changes) and dfa15d38b (keep projection notices out of the Notes→Files mutation bridge) onto production c39ffe5d9.

Production change-event rate per 30 s: before ~1,200; after the deploy 0, 0, 0, 0. Healthy in 12 s; no panics or errors.

Gates on the hotfix: cargo fmt --check clean; clippy clean (notes, files). Notes tests: 192 passed, 1 failed. The failure is the new seven_hundred_notes_reconcile_without_feedback, which overflows the process-global event bus when it runs in parallel with other tests. Alone it passes 3/3 (test result: ok. 1 passed). That is test isolation, filed separately. The idle-rate probe (6153d62d0) joins the next round.

## Hotfix deployed to production (2026-10-04 19:12 CEST, dcad855ee) Cherry-picked 8ebca1b75 (publish projection changes only when the source hash changes) and dfa15d38b (keep projection notices out of the Notes→Files mutation bridge) onto production c39ffe5d9. Production change-event rate per 30 s: before ~1,200; after the deploy **0, 0, 0, 0**. Healthy in 12 s; no panics or errors. Gates on the hotfix: `cargo fmt --check` clean; clippy clean (notes, files). Notes tests: 192 passed, 1 failed. The failure is the new `seven_hundred_notes_reconcile_without_feedback`, which overflows the process-global event bus when it runs in parallel with other tests. Alone it passes 3/3 (`test result: ok. 1 passed`). That is test isolation, filed separately. The idle-rate probe (6153d62d0) joins the next round.
kayg closed this issue 2026-10-04 17:15:06 +00:00
Author
Owner

Round 7b7 started on job/7b-reconcile at 1b08bac4d. The job brief replaces the original read-only scope of #867. Fetched origin/dev and started its merge. Three conflicts are in wire.rs, notes/imap.rs and notes/store.rs. I will preserve the #1062 hash-change and durable-mutation bridge guards, fix the Notes Files identity response profile, then run the requested regression, idle SSE check and gates. No push or deploy.

Round 7b7 started on job/7b-reconcile at 1b08bac4d. The job brief replaces the original read-only scope of #867. Fetched origin/dev and started its merge. Three conflicts are in wire.rs, notes/imap.rs and notes/store.rs. I will preserve the #1062 hash-change and durable-mutation bridge guards, fix the Notes Files identity response profile, then run the requested regression, idle SSE check and gates. No push or deploy.
Author
Owner

Remaining hotfix gates (dcad855ee): clippy clean for calternal-plugin-files and calternal-server; tests calternal-plugin-files passed=158 failed=0, calternal-server passed=164 failed=0. The only red item is the parallel-run flake tracked in #1065.

Remaining hotfix gates (dcad855ee): clippy clean for calternal-plugin-files and calternal-server; tests `calternal-plugin-files passed=158 failed=0`, `calternal-server passed=164 failed=0`. The only red item is the parallel-run flake tracked in #1065.
Author
Owner

The #1062 merge keeps 7b's transaction retries and User-derived IMAP UIDVALIDITY. Projection notices use the transaction's source-change result. The bridge accepts created, updated, moved, retitled, trashed and deleted only. Both branches' server tests remain.

The response leak was caused by checking the Markdown extension before existence. The Cross-User matrix replaces all path characters except slashes, so its missing control has no Markdown extension. The route now validates and resolves a path only inside the caller's Home before it checks the file type. Foreign Markdown and missing paths use the same Root metadata lookup and 404 envelope. Traversal still returns 400. A new route regression and a real HTTP probe cover the three denial cases and a positive own-Note read.

The first web check failed at a stale perf exception for record_change. The hotfix changes 206 function-bound fingerprints in imap.rs/store.rs; the route changes seven more. Exact unchanged calls are rebound. New Home-path operations are pure bindings. The source Option check is also pure. This removes one exception and allows the new bounded metadata lookup to stay explicit as IO debt owned by #702, without increasing the ratchet. No latency or projection-adoption result is invented.

The first Rust compile failed on byte-array fixture readers. The new fixtures now pass byte slices to Root::write. The first full web test gate passed: 222 files and 1510 tests.

The #1062 merge keeps 7b's transaction retries and User-derived IMAP UIDVALIDITY. Projection notices use the transaction's source-change result. The bridge accepts created, updated, moved, retitled, trashed and deleted only. Both branches' server tests remain. The response leak was caused by checking the Markdown extension before existence. The Cross-User matrix replaces all path characters except slashes, so its missing control has no Markdown extension. The route now validates and resolves a path only inside the caller's Home before it checks the file type. Foreign Markdown and missing paths use the same Root metadata lookup and 404 envelope. Traversal still returns 400. A new route regression and a real HTTP probe cover the three denial cases and a positive own-Note read. The first web check failed at a stale perf exception for record_change. The hotfix changes 206 function-bound fingerprints in imap.rs/store.rs; the route changes seven more. Exact unchanged calls are rebound. New Home-path operations are pure bindings. The source Option check is also pure. This removes one exception and allows the new bounded metadata lookup to stay explicit as IO debt owned by #702, without increasing the ratchet. No latency or projection-adoption result is invented. The first Rust compile failed on byte-array fixture readers. The new fixtures now pass byte slices to Root::write. The first full web test gate passed: 222 files and 1510 tests.
Author
Owner

The round 7b7 local correctness probe passed on a real branch server. Command:
python3 tests/adversarial/notes_idle.py --server "$CARGO_TARGET_DIR/debug/calternal-server" --data-root "$PWD/target/tmp" --json artifacts/7b7/idle.json.

The Instance had 700 Notes and an open Files SSE client. After startup repair, the 60-second idle window produced zero Files event rows and zero SSE change frames. One actual Note body edit produced four Files events. This proves the #1062 guard on the assembled branch.

The probe also alternated 20 requests per denial case: another User's existing Markdown path, the extensionless missing-path control, and a valid missing Markdown path. Every case returned 404 with the same 57-byte body. The runtime build identity and timings follow verbatim:
{
"success": true,
"load": [
22.5966796875,
25.18359375,
24.15771484375
],
"build": {
"source_commit": "5a10cbccce",
"binary_sha256": "c9d4ae0ea626c8f16757be17d1a732bef2fda6a72f2fb2d725658f58a26f6b65"
},
"notes": 700,
"idle_seconds": 60.002,
"files_events": 0,
"files_events_per_minute": 0.0,
"sse_change_frames": 0,
"response_profiles": {
"status": 404,
"body_bytes": 57,
"samples_per_case": 20,
"cases": [
{
"p50_ms": 4.569,
"p95_ms": 10.406
},
{
"p50_ms": 4.158,
"p95_ms": 8.32
},
{
"p50_ms": 4.663,
"p95_ms": 5.561
}
]
},
"single_edit_files_events": 4
}

The round 7b7 local correctness probe passed on a real branch server. Command: `python3 tests/adversarial/notes_idle.py --server "$CARGO_TARGET_DIR/debug/calternal-server" --data-root "$PWD/target/tmp" --json artifacts/7b7/idle.json`. The Instance had 700 Notes and an open Files SSE client. After startup repair, the 60-second idle window produced zero Files event rows and zero SSE change frames. One actual Note body edit produced four Files events. This proves the #1062 guard on the assembled branch. The probe also alternated 20 requests per denial case: another User's existing Markdown path, the extensionless missing-path control, and a valid missing Markdown path. Every case returned 404 with the same 57-byte body. The runtime build identity and timings follow verbatim: { "success": true, "load": [ 22.5966796875, 25.18359375, 24.15771484375 ], "build": { "source_commit": "5a10cbcccee8756baf39f48df0a77a7e23296e7b", "binary_sha256": "c9d4ae0ea626c8f16757be17d1a732bef2fda6a72f2fb2d725658f58a26f6b65" }, "notes": 700, "idle_seconds": 60.002, "files_events": 0, "files_events_per_minute": 0.0, "sse_change_frames": 0, "response_profiles": { "status": 404, "body_bytes": 57, "samples_per_case": 20, "cases": [ { "p50_ms": 4.569, "p95_ms": 10.406 }, { "p50_ms": 4.158, "p95_ms": 8.32 }, { "p50_ms": 4.663, "p95_ms": 5.561 } ] }, "single_edit_files_events": 4 }
Author
Owner

Round 7b7 finished. Final head: 21420777f39e31e230db85151c9d5ac474295df2. Branch: job/7b-reconcile.

Merge round 7b7 — #867 and #1062

Date: 2026-10-04. Branch: job/7b-reconcile.

Changes

  • Merge origin/dev at dcad855ee063927c5d95c0a539559377c3db1129 into the job base 1b08bac4d. Keep 7b startup tests, access tests, transaction retries and User-derived IMAP UIDVALIDITY. Keep the #1062 source-change result and durable-mutation bridge filter. Commit: 0eabbe53e.
  • Resolve Files-origin Note paths inside the caller's Home before the Markdown type check. A foreign path and a missing path return the same 404 body. Keep traversal validation before file access. Add a route regression. Commit: c59564131.
  • Add tests/adversarial/notes_idle.py. It uses the existing local-server fixture. It waits for startup repair, checks an open Files SSE connection for one minute, compares three HTTP denial cases and checks one real Note edit. Commit: 5a10cbccc.
  • Update exact performance source pins. Mark path operations and the source Option check as pure calls. Keep the new metadata lookup as explicit IO debt owned by #702. The exception count stays at 19,340.

Files

  • crates/calternal-server/src/wire.rs
  • crates/plugins/notes/src/imap.rs
  • crates/plugins/notes/src/store.rs
  • crates/plugins/notes/src/lib.rs
  • contracts/perf/registry.json
  • contracts/perf/exceptions.json
  • tests/adversarial/notes_idle.py
  • docs/audits/merge-round-7b7.md

Local proof

The standalone seven_hundred_notes_reconcile_without_feedback command passed. The bridge filter regression passed in the Server suite. The Files-origin route regression passed in the Notes suite.

The live probe passed with 700 Notes. The open Files SSE connection stayed open for 60.002 seconds. It received no change frames. The durable Files event count did not change. One real Note body edit produced four Files events.

Each denial case had 20 alternating requests: foreign Markdown, missing path with no Markdown extension, and missing Markdown. All cases returned 404 with the same 57-byte body. Their p50/p95 times were 4.569/10.406 ms, 4.158/8.320 ms and 4.663/5.561 ms. These are local correctness samples on a shared host, not performance budget samples.

Runtime source: 5a10cbcccee8756baf39f48df0a77a7e23296e7b.
Binary SHA-256: c9d4ae0ea626c8f16757be17d1a732bef2fda6a72f2fb2d725658f58a26f6b65.
Load at probe start: 22.5967, 25.1836, 24.1577.

Decisions

The design does not specify the order of file-type checks on this route. Use one Home-relative metadata lookup before the type check. This gives missing and foreign paths the same lookup and response. It adds one bounded metadata lookup to a successful open.

UX gaps closed

No UI changed. The API denial profile no longer exposes the file-type validation order through this Cross-User control.

UX gaps left

No new UI gap was found in this API-only work. This round does not repeat the earlier UI reviews.

Verification

All Cargo commands use CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and the worktree target/tmp. The preset Cargo target directory was kept. No workspace Rust gate ran.

cargo fmt --check: exit 0, no output.

calternal-plugin-notes

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 53s

cargo test -p calternal-plugin-notes -- --test-threads=4:

test result: FAILED. 263 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 273.80s

calternal-plugin-files

cargo clippy -p calternal-plugin-files --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 07s

cargo test -p calternal-plugin-files -- --test-threads=4:

test result: ok. 234 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 262.27s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server

The first clippy attempt failed because apps/web/build was absent. The real production web build then passed. The clippy retry passed:

cargo clippy -p calternal-server --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 28s

cargo test -p calternal-server -- --test-threads=4:

test result: ok. 210 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 158.84s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.34s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.64s

Web

bun run check first failed at stale performance source pins. The updated pins passed. bun run test --maxWorkers=2 passed. bun run build passed and supplied the real embedded assets for the Server gates.

perf-lint: PASS; 0 violations; 19340 scoped exceptions
svelte-check found 0 errors and 4 warnings in 3 files
 Test Files  222 passed (222)
      Tests  1510 passed (1510)

cargo clippy -p calternal-search --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 11s

cargo test -p calternal-search -- --test-threads=4:

test result: ok. 53 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 23.71s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.94s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.13s
test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 418.52s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 8.24s
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Focused regressions

cargo test -p calternal-plugin-notes seven_hundred_notes_reconcile_without_feedback -- --test-threads=1:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 265 filtered out; finished in 132.23s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

cargo test -p calternal-plugin-notes daily_log_projection_rebuild_resumes_without_markdown_writes -- --test-threads=1:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 265 filtered out; finished in 57.06s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

The standalone rebuild passed with its original 650 files and assertions. The earlier full-suite failure remains a failed gate. No full suite was run again. All requested scenarios finished; none are deferred.

Known gaps

The full Notes gate failed at daily_log_projection_rebuild_resumes_without_markdown_writes. Its original 650-file fixture and assertions were kept. It received JobError { message: "Index is busy; retry shortly" }. This result is recorded on #1022 and #867. It is not classified as SLOW.

Two Notes tests, three Files tests, three Search tests and nine Server tests are ignored by the existing suites. Four Svelte warnings remain in untouched source files. The first new test compile used byte arrays instead of byte slices; the fixture readers were fixed before the passing regression.

Module and function comments in the changed Rust files and the new probe were read again. The IMAP module comment was corrected to describe 7b's User-derived empty-mailbox epoch. This final change affects comments only. Format passed again with no output. Cargo cleanup removed 23,081 files and 20.2 GiB. The web build and .svelte-kit/output directories were removed. No push or deploy ran. No issue was closed. The final issue comment states the report head SHA.

READY FOR STAGING: no. The full Notes gate has a failure.

Round 7b7 finished. Final head: `21420777f39e31e230db85151c9d5ac474295df2`. Branch: `job/7b-reconcile`. # Merge round 7b7 — #867 and #1062 Date: 2026-10-04. Branch: `job/7b-reconcile`. ## Changes - Merge `origin/dev` at `dcad855ee063927c5d95c0a539559377c3db1129` into the job base `1b08bac4d`. Keep 7b startup tests, access tests, transaction retries and User-derived IMAP UIDVALIDITY. Keep the #1062 source-change result and durable-mutation bridge filter. Commit: `0eabbe53e`. - Resolve Files-origin Note paths inside the caller's Home before the Markdown type check. A foreign path and a missing path return the same 404 body. Keep traversal validation before file access. Add a route regression. Commit: `c59564131`. - Add `tests/adversarial/notes_idle.py`. It uses the existing local-server fixture. It waits for startup repair, checks an open Files SSE connection for one minute, compares three HTTP denial cases and checks one real Note edit. Commit: `5a10cbccc`. - Update exact performance source pins. Mark path operations and the source Option check as pure calls. Keep the new metadata lookup as explicit IO debt owned by #702. The exception count stays at 19,340. ## Files - `crates/calternal-server/src/wire.rs` - `crates/plugins/notes/src/imap.rs` - `crates/plugins/notes/src/store.rs` - `crates/plugins/notes/src/lib.rs` - `contracts/perf/registry.json` - `contracts/perf/exceptions.json` - `tests/adversarial/notes_idle.py` - `docs/audits/merge-round-7b7.md` ## Local proof The standalone `seven_hundred_notes_reconcile_without_feedback` command passed. The bridge filter regression passed in the Server suite. The Files-origin route regression passed in the Notes suite. The live probe passed with 700 Notes. The open Files SSE connection stayed open for 60.002 seconds. It received no change frames. The durable Files event count did not change. One real Note body edit produced four Files events. Each denial case had 20 alternating requests: foreign Markdown, missing path with no Markdown extension, and missing Markdown. All cases returned 404 with the same 57-byte body. Their p50/p95 times were 4.569/10.406 ms, 4.158/8.320 ms and 4.663/5.561 ms. These are local correctness samples on a shared host, not performance budget samples. Runtime source: `5a10cbcccee8756baf39f48df0a77a7e23296e7b`. Binary SHA-256: `c9d4ae0ea626c8f16757be17d1a732bef2fda6a72f2fb2d725658f58a26f6b65`. Load at probe start: 22.5967, 25.1836, 24.1577. ## Decisions The design does not specify the order of file-type checks on this route. Use one Home-relative metadata lookup before the type check. This gives missing and foreign paths the same lookup and response. It adds one bounded metadata lookup to a successful open. ## UX gaps closed No UI changed. The API denial profile no longer exposes the file-type validation order through this Cross-User control. ## UX gaps left No new UI gap was found in this API-only work. This round does not repeat the earlier UI reviews. ## Verification All Cargo commands use `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4` and the worktree `target/tmp`. The preset Cargo target directory was kept. No workspace Rust gate ran. `cargo fmt --check`: exit 0, no output. ### calternal-plugin-notes `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 53s ``` `cargo test -p calternal-plugin-notes -- --test-threads=4`: ```text test result: FAILED. 263 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 273.80s ``` ### calternal-plugin-files `cargo clippy -p calternal-plugin-files --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 07s ``` `cargo test -p calternal-plugin-files -- --test-threads=4`: ```text test result: ok. 234 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 262.27s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-server The first clippy attempt failed because `apps/web/build` was absent. The real production web build then passed. The clippy retry passed: `cargo clippy -p calternal-server --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 28s ``` `cargo test -p calternal-server -- --test-threads=4`: ```text test result: ok. 210 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 158.84s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.34s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.64s ``` ### Web `bun run check` first failed at stale performance source pins. The updated pins passed. `bun run test --maxWorkers=2` passed. `bun run build` passed and supplied the real embedded assets for the Server gates. ```text perf-lint: PASS; 0 violations; 19340 scoped exceptions svelte-check found 0 errors and 4 warnings in 3 files Test Files 222 passed (222) Tests 1510 passed (1510) ``` ### calternal-search `cargo clippy -p calternal-search --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 11s ``` `cargo test -p calternal-search -- --test-threads=4`: ```text test result: ok. 53 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 23.71s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.94s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.13s test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 418.52s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 8.24s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### Focused regressions `cargo test -p calternal-plugin-notes seven_hundred_notes_reconcile_without_feedback -- --test-threads=1`: ```text test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 265 filtered out; finished in 132.23s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s ``` `cargo test -p calternal-plugin-notes daily_log_projection_rebuild_resumes_without_markdown_writes -- --test-threads=1`: ```text test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 265 filtered out; finished in 57.06s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s ``` The standalone rebuild passed with its original 650 files and assertions. The earlier full-suite failure remains a failed gate. No full suite was run again. All requested scenarios finished; none are deferred. ## Known gaps The full Notes gate failed at `daily_log_projection_rebuild_resumes_without_markdown_writes`. Its original 650-file fixture and assertions were kept. It received `JobError { message: "Index is busy; retry shortly" }`. This result is recorded on #1022 and #867. It is not classified as SLOW. Two Notes tests, three Files tests, three Search tests and nine Server tests are ignored by the existing suites. Four Svelte warnings remain in untouched source files. The first new test compile used byte arrays instead of byte slices; the fixture readers were fixed before the passing regression. Module and function comments in the changed Rust files and the new probe were read again. The IMAP module comment was corrected to describe 7b's User-derived empty-mailbox epoch. This final change affects comments only. Format passed again with no output. Cargo cleanup removed 23,081 files and 20.2 GiB. The web build and `.svelte-kit/output` directories were removed. No push or deploy ran. No issue was closed. The final issue comment states the report head SHA. READY FOR STAGING: no. The full Notes gate has a failure.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#1062
No description provided.