PERF: separate Mail IDLE listeners from sync worker slots (#663)

kayg commented

2026-10-02 13:09:08 +00:00

Owner

Context: #663 sync architecture audit, DESIGN §45 and instant-interaction rule 8. Source evidence, not measured latency. Base c4a61e8cf; confirmed at job/merge-round-7a 2f4482ded. No product change in this audit.

Evidence at round-7a:

crates/plugins/mail/src/sync.rs:131 sets max_concurrency to 3 for mail.sync. The comment explicitly includes backfill, IDLE and retries in this cap.
:154 holds the per-account guard across sync_account.
:358 awaits wait_for_hint. :1213 waits in IDLE up to IDLE_WINDOW (300 s, :60).
:183 schedules enqueue_poll after every completed run; :237 sets the next run to now + 300 s. After one IDLE notification and its delta, the listener stops. Mail arriving in this scheduled gap has no listener from this job. The non-IDLE branch also sleeps 300 s inside a worker before the delayed next job.
calternal-db/src/worker.rs tracks in-flight jobs per kind and only offers a kind while it has available concurrency. Waiting jobs consume that budget.

Reasoned impact: three quiet completed Connected Accounts can occupy all Mail sync slots for five minutes. A fourth account waits although no history ingest is running. After an IDLE hint there is a five-minute listener gap. This contradicts new-mail-within-5-s (§53) even when the provider offers IDLE. The exact wait under real load is not measured.

Concrete fix: separate supervised IDLE connection slots from bounded ingest jobs. An IDLE hint enqueues a deduplicated delta and immediately resumes listening. Use a separate connection budget and fair scheduling across Users. Poll fallback must use queue run_at, not sleep in an ingest slot; transient provider failures need bounded exponential backoff with jitter. Preserve atomic UID/page commits and account serialization.

Tests: with a virtual clock and a deterministic provider, start four or more accounts; three idle listeners must not delay the fourth history page. Send two hints on one account less than five minutes apart; each must update the cache within 5 s. Check fallback polling, failure/reconnect and job deduplication. Extend bench/mail-sync.py for mixed quiet accounts and one busy account; measure listener count, queue wait, p50/p95, CPU/RSS.

Duplicate search: searched all issue titles for sync/IDLE; read #613, #614 titles and #668 scope. #613 covers first-sync state, #614 future JMAP, #668 app push. None owns this IMAP scheduler/listener lifecycle. This is a performance/availability finding; no crash or security hole was demonstrated.

Context: #663 sync architecture audit, DESIGN §45 and instant-interaction rule 8. Source evidence, not measured latency. Base c4a61e8cf; confirmed at job/merge-round-7a 2f4482ded. No product change in this audit. Evidence at round-7a: - crates/plugins/mail/src/sync.rs:131 sets max_concurrency to 3 for mail.sync. The comment explicitly includes backfill, IDLE and retries in this cap. - :154 holds the per-account guard across sync_account. - :358 awaits wait_for_hint. :1213 waits in IDLE up to IDLE_WINDOW (300 s, :60). - :183 schedules enqueue_poll after every completed run; :237 sets the next run to now + 300 s. After one IDLE notification and its delta, the listener stops. Mail arriving in this scheduled gap has no listener from this job. The non-IDLE branch also sleeps 300 s inside a worker before the delayed next job. - calternal-db/src/worker.rs tracks in-flight jobs per kind and only offers a kind while it has available concurrency. Waiting jobs consume that budget. Reasoned impact: three quiet completed Connected Accounts can occupy all Mail sync slots for five minutes. A fourth account waits although no history ingest is running. After an IDLE hint there is a five-minute listener gap. This contradicts new-mail-within-5-s (§53) even when the provider offers IDLE. The exact wait under real load is not measured. Concrete fix: separate supervised IDLE connection slots from bounded ingest jobs. An IDLE hint enqueues a deduplicated delta and immediately resumes listening. Use a separate connection budget and fair scheduling across Users. Poll fallback must use queue run_at, not sleep in an ingest slot; transient provider failures need bounded exponential backoff with jitter. Preserve atomic UID/page commits and account serialization. Tests: with a virtual clock and a deterministic provider, start four or more accounts; three idle listeners must not delay the fourth history page. Send two hints on one account less than five minutes apart; each must update the cache within 5 s. Check fallback polling, failure/reconnect and job deduplication. Extend bench/mail-sync.py for mixed quiet accounts and one busy account; measure listener count, queue wait, p50/p95, CPU/RSS. Duplicate search: searched all issue titles for sync/IDLE; read #613, #614 titles and #668 scope. #613 covers first-sync state, #614 future JMAP, #668 app push. None owns this IMAP scheduler/listener lifecycle. This is a performance/availability finding; no crash or security hole was demonstrated.

kayg referenced this issue

2026-10-02 13:11:34 +00:00

PERF: separate Mail IDLE waits from the three sync worker slots (#663) #763

kayg commented

2026-10-02 13:11:34 +00:00

Author

Owner

Concurrent sync audit finding: #753 was filed first for the same three-slot IDLE lifecycle; #763 appeared during this audit. Please keep one implementation owner for both. #753 also records the 300 s enqueue_poll gap after a single IDLE hint/delta, so new arrivals have no continuous listener. Scope this fix to both scheduler fairness and continuous IDLE; do not implement two listener managers. Source confirmed on round-7a 2f4482ded, sync.rs:131, :183, :237, :358 and :1213. No runtime latency measurement in this audit.

Concurrent sync audit finding: #753 was filed first for the same three-slot IDLE lifecycle; #763 appeared during this audit. Please keep one implementation owner for both. #753 also records the 300 s enqueue_poll gap after a single IDLE hint/delta, so new arrivals have no continuous listener. Scope this fix to both scheduler fairness and continuous IDLE; do not implement two listener managers. Source confirmed on round-7a 2f4482ded, sync.rs:131, :183, :237, :358 and :1213. No runtime latency measurement in this audit.

kayg referenced this issue

2026-10-02 13:16:50 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:19:27 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:27:15 +00:00

Author

Owner

Starting implementation on job/mailperf, based on job/merge-round-7a at 2f4482ded066d9c5d9c59130377907f7fd2916c9. The target areas are IDLE scheduling, bounded delta/expunge work, and unread-count invalidation. I will merge origin/dev and job/merge-round-7a before the final gates.

Starting implementation on `job/mailperf`, based on `job/merge-round-7a` at `2f4482ded066d9c5d9c59130377907f7fd2916c9`. The target areas are IDLE scheduling, bounded delta/expunge work, and unread-count invalidation. I will merge `origin/dev` and `job/merge-round-7a` before the final gates.

kayg referenced this issue

2026-10-02 18:07:21 +00:00

PERF: separate Mail IDLE waits from the three sync worker slots (#663) #763

kayg commented

2026-10-02 18:08:14 +00:00

Author

Owner

Finished on job/mailperf, head 2e724c0529ef38623fd86c239d638371be5d9702.

Built separate three-worker mail.idle and mail.sync pools, deduplicated immediate delta wakes, capped jittered retries, shared per-run delta/expunge budget, durable 1,024-UID scan ranges and 80-row prune pages. Mail mutations publish post-commit folder invalidations; a User-scoped SSE stream sends only generic refreshes, and the sidebar rereads live folder counts.

Commits: de814554f Bound Mail sync and expunge work; c6c4f7176 Stream Mail folder count invalidations. Required merges: 0c7f1a8b4 (origin/dev) and 2e724c052 (job/merge-round-7a). #763 is closed as duplicate.

Gate output:

Post-merge cargo fmt --check: exit 0, no output.
Pre-merge cargo test -p calternal-plugin-mail: test result: ok. 54 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 60.95s; doc-tests: test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s.
Focused sidebar test: Test Files 1 passed (1); Tests 2 passed (2).
Post-merge clippy started rebuilding dependencies and was interrupted at the four-hour cap (exit 130). Post-merge Mail tests, calternal-plugin and calternal-server gates, bun run check/test, the real-server adversarial run, production screenshots and perf VM measurements remain undone. The bench runner reports p50/p95, CPU and RSS, but does not yet isolate delta/expunge phases.
cargo clean: Removed 5719 files, 1.6GiB total. No web build output existed to remove.

Decisions not settled by DESIGN: the shared Plugin event uses virtual scope folders; SSE carries no item data and clients reread snapshots; scan pages span 1,024 UIDs; job concurrency is three per kind; retries use stable jitter capped at five minutes; browser changes coalesce for 100 ms.

UX gaps closed: unread badges update across mounted Mail views after provider sync and read-state changes. UX gaps left: macOS production screenshots and browser review were not completed.

Finished on `job/mailperf`, head `2e724c0529ef38623fd86c239d638371be5d9702`. Built separate three-worker `mail.idle` and `mail.sync` pools, deduplicated immediate delta wakes, capped jittered retries, shared per-run delta/expunge budget, durable 1,024-UID scan ranges and 80-row prune pages. Mail mutations publish post-commit folder invalidations; a User-scoped SSE stream sends only generic refreshes, and the sidebar rereads live folder counts. Commits: `de814554f` Bound Mail sync and expunge work; `c6c4f7176` Stream Mail folder count invalidations. Required merges: `0c7f1a8b4` (`origin/dev`) and `2e724c052` (`job/merge-round-7a`). #763 is closed as duplicate. Gate output: - Post-merge `cargo fmt --check`: exit 0, no output. - Pre-merge `cargo test -p calternal-plugin-mail`: `test result: ok. 54 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 60.95s`; doc-tests: `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s`. - Focused sidebar test: `Test Files 1 passed (1)`; `Tests 2 passed (2)`. - Post-merge clippy started rebuilding dependencies and was interrupted at the four-hour cap (exit 130). Post-merge Mail tests, `calternal-plugin` and `calternal-server` gates, `bun run check/test`, the real-server adversarial run, production screenshots and perf VM measurements remain undone. The bench runner reports p50/p95, CPU and RSS, but does not yet isolate delta/expunge phases. - `cargo clean`: `Removed 5719 files, 1.6GiB total`. No web build output existed to remove. Decisions not settled by DESIGN: the shared Plugin event uses virtual scope `folders`; SSE carries no item data and clients reread snapshots; scan pages span 1,024 UIDs; job concurrency is three per kind; retries use stable jitter capped at five minutes; browser changes coalesce for 100 ms. UX gaps closed: unread badges update across mounted Mail views after provider sync and read-state changes. UX gaps left: macOS production screenshots and browser review were not completed.

kayg commented

2026-10-02 20:04:11 +00:00

Author

Owner

Independent read-only review: job/mailperf at 2e724c0529ef38623fd86c239d638371be5d9702, against origin/dev at c4faf184df726a9375ae0c13bdfb6018ac2cf57e. Review branch job/rev2-mailperf starts from that origin/dev SHA. No build or test was run.

Two P2 findings remain in the #753 listener lifecycle. All-state IDLE search identifies #753 as the owner and #763 as its closed duplicate.

Quiet IDLE cycles never queue account reconciliation. crates/plugins/mail/src/sync.rs:1554 watches INBOX only. :1589–1594 returns false on normal timeout. :290–314 queues sync only for a hint, then creates another listener. Delivery to another folder, or delivery between listeners, can stay absent without limit while INBOX stays quiet. DESIGN §45 requires full-history delta reconciliation; §53 requires new mail within 5 seconds. Keep a durable periodic check independent of hints. A regression test must deliver to another folder with no INBOX hints, advance the clock through timeouts, and check cache convergence. Repeat between DONE and the next IDLE and with four Connected Accounts. Non-IDLE checks must use run_at rather than sleeping in a listener slot.
Manual sync can create two listeners for one Connected Account. sync.rs:95–106 always gives manual sync poll_slot a, so :183–207 queues listener b. After a listener b timeout, listener a can be active. Manual sync then starts b alongside it. The a/b dedup keys differ (:402–408); there is no listener guard. One account can use two of the three listener slots and queue two wake jobs for one provider change. The sync route reaches this path at crates/plugins/mail/src/routes.rs:1148. This breaks #753's bounded and fair connection lifecycle. Use one supervised listener per account. Test manual sync while listener a waits and assert one connection, including retry and reconnect transitions and progress for another User.

These are source findings. No runtime delay, crash or cross-User disclosure was measured. Full evidence and test ideas are in audit-findings.md and review-mailperf.md on the review branch. Keep both findings in #753; they need one listener/reconciliation lifecycle fix.

Independent read-only review: `job/mailperf` at `2e724c0529ef38623fd86c239d638371be5d9702`, against `origin/dev` at `c4faf184df726a9375ae0c13bdfb6018ac2cf57e`. Review branch `job/rev2-mailperf` starts from that origin/dev SHA. No build or test was run. Two P2 findings remain in the #753 listener lifecycle. All-state IDLE search identifies #753 as the owner and #763 as its closed duplicate. 1. **Quiet IDLE cycles never queue account reconciliation.** `crates/plugins/mail/src/sync.rs:1554` watches INBOX only. `:1589–1594` returns false on normal timeout. `:290–314` queues sync only for a hint, then creates another listener. Delivery to another folder, or delivery between listeners, can stay absent without limit while INBOX stays quiet. DESIGN §45 requires full-history delta reconciliation; §53 requires new mail within 5 seconds. Keep a durable periodic check independent of hints. A regression test must deliver to another folder with no INBOX hints, advance the clock through timeouts, and check cache convergence. Repeat between DONE and the next IDLE and with four Connected Accounts. Non-IDLE checks must use run_at rather than sleeping in a listener slot. 2. **Manual sync can create two listeners for one Connected Account.** `sync.rs:95–106` always gives manual sync poll_slot a, so `:183–207` queues listener b. After a listener b timeout, listener a can be active. Manual sync then starts b alongside it. The a/b dedup keys differ (`:402–408`); there is no listener guard. One account can use two of the three listener slots and queue two wake jobs for one provider change. The sync route reaches this path at `crates/plugins/mail/src/routes.rs:1148`. This breaks #753's bounded and fair connection lifecycle. Use one supervised listener per account. Test manual sync while listener a waits and assert one connection, including retry and reconnect transitions and progress for another User. These are source findings. No runtime delay, crash or cross-User disclosure was measured. Full evidence and test ideas are in `audit-findings.md` and `review-mailperf.md` on the review branch. Keep both findings in #753; they need one listener/reconciliation lifecycle fix.

kayg referenced this issue

2026-10-02 20:05:31 +00:00

Mail: read-state changes leave sidebar unread counts stale #771

kayg referenced this issue

2026-10-03 04:32:08 +00:00

Mail: read-state changes leave sidebar unread counts stale #771

kayg referenced this issue from a commit

2026-10-03 04:41:25 +00:00

Fix Mail perf review findings F1-F5 (#771, #753, #757)

kayg commented

2026-10-03 07:08:39 +00:00

Author

Owner

The independent review found F1 and F2 at the predecessor SHA 2e724c0529ef38623fd86c239d638371be5d9702. On current job/mailperf at 7a22fbadecfc273f7b128867e3fbeb0265f7a700, the listener chain keeps a durable delayed reconciliation poll: schedule_after_sync ensures it after a complete sync, and schedule_after_idle ensures it after quiet cycles. quiet_idle_cycles_keep_one_durable_reconciliation_poll and quiet_listener_starts_the_poll_for_older_accounts cover this. ensure_listener checks both alternating listener keys while account work is serialized; manual_sync_keeps_one_listener_per_account covers a manual sync during a leased listener and duplicate legacy listeners. Source/test review only so far; requested Mail gates are next.

The independent review found F1 and F2 at the predecessor SHA `2e724c0529ef38623fd86c239d638371be5d9702`. On current `job/mailperf` at `7a22fbadecfc273f7b128867e3fbeb0265f7a700`, the listener chain keeps a durable delayed reconciliation poll: `schedule_after_sync` ensures it after a complete sync, and `schedule_after_idle` ensures it after quiet cycles. `quiet_idle_cycles_keep_one_durable_reconciliation_poll` and `quiet_listener_starts_the_poll_for_older_accounts` cover this. `ensure_listener` checks both alternating listener keys while account work is serialized; `manual_sync_keeps_one_listener_per_account` covers a manual sync during a leased listener and duplicate legacy listeners. Source/test review only so far; requested Mail gates are next.

Rows
Columns

PERF: separate Mail IDLE listeners from sync worker slots (#663) #753