kayg/calternal

Watch

Fork 0

PERF: Notes IMAP SELECT and refresh retain full mailbox sources per connection (#663) #808

New issue

Open

opened 2026-10-02 13:12:48 +00:00 by kayg · 4 comments

kayg commented

2026-10-02 13:12:48 +00:00

Owner

Parent audit #663. Baseline c4a61e8cf0; checked against round 7a.

M4: Notes IMAP retains source text per connection and rebuilds it on refresh

Status: source-confirmed; audit finding.

Evidence: crates/calternal-imap/src/store.rs:17 stores source String in every
Message; :30 puts all messages in a connection Snapshot.
crates/plugins/notes/src/imap.rs:738 loads up to 4,097 live revisions before
mailbox filtering and rejects more than 4,096. It reads and hashes each Note
under the User writer lock. Selected source bytes are capped at 16 MiB (:814).
Session::refresh (calternal-imap/src/session.rs:609) constructs the next full
snapshot before it checks the revision clock. FETCH bases have a separate
8 MiB cap (session.rs:18). Default listener caps are 64 total and eight per
User (calternal-server/src/notes_imap.rs:64–70).
Round 7a keeps the full-source snapshot and both corpus limits (:761 in
its Notes provider). This is bounded, but the bound is expensive and makes
large valid Homes unusable through this surface.

Impact estimate: 64 selected connections can retain 1,024 MiB of source
text, plus up to 512 MiB in merge bases. During concurrent refreshes, another
1,024 MiB can be live, before rows, hashes, output, TLS and allocator costs.
Eight connections for one User can retain 192 MiB in sources/bases.
A Home with more than 4,096 live Notes fails even for a small Tag mailbox,
because the revision cap is tested before folder filtering. This concerns
Notes, not the future external Mail proxy.

Fix: keep a shared revision-keyed mailbox metadata view (UID, flags, dates,
identity and source revision), fetch source lazily, and store bounded shared
merge bases. Check the durable clock before reading bodies. Precompute
mailbox membership and projection text in the Index so refresh does not
read every file under the writer lock. Preserve IMAP sequence numbers and
UID/MODSEQ semantics; HTTP keyset pages cannot simply replace IMAP SELECT.

Test: a large synthetic Note corpus, a small Tag mailbox within it, multiple
selected connections and one edit. Check that per-connection source bytes
remain bounded independently of corpus size, unchanged refresh reads no
bodies, UID order stays stable, and Apple merge/EXPUNGE tests still pass.

Duplicate search: all-state IMAP/memory/unbounded. #550 concerns APPEND
latency; #644 concerns edit collisions. This issue concerns SELECT/refresh
residency and corpus limits.

Parent audit #663. Baseline c4a61e8cf090170f35b1bed3350d9de20c83ecd5; checked against round 7a. M4: Notes IMAP retains source text per connection and rebuilds it on refresh Status: source-confirmed; audit finding. Evidence: crates/calternal-imap/src/store.rs:17 stores source String in every Message; :30 puts all messages in a connection Snapshot. crates/plugins/notes/src/imap.rs:738 loads up to 4,097 live revisions before mailbox filtering and rejects more than 4,096. It reads and hashes each Note under the User writer lock. Selected source bytes are capped at 16 MiB (:814). Session::refresh (calternal-imap/src/session.rs:609) constructs the next full snapshot before it checks the revision clock. FETCH bases have a separate 8 MiB cap (session.rs:18). Default listener caps are 64 total and eight per User (calternal-server/src/notes_imap.rs:64–70). Round 7a keeps the full-source snapshot and both corpus limits (:761 in its Notes provider). This is bounded, but the bound is expensive and makes large valid Homes unusable through this surface. Impact estimate: 64 selected connections can retain 1,024 MiB of source text, plus up to 512 MiB in merge bases. During concurrent refreshes, another 1,024 MiB can be live, before rows, hashes, output, TLS and allocator costs. Eight connections for one User can retain 192 MiB in sources/bases. A Home with more than 4,096 live Notes fails even for a small Tag mailbox, because the revision cap is tested before folder filtering. This concerns Notes, not the future external Mail proxy. Fix: keep a shared revision-keyed mailbox metadata view (UID, flags, dates, identity and source revision), fetch source lazily, and store bounded shared merge bases. Check the durable clock before reading bodies. Precompute mailbox membership and projection text in the Index so refresh does not read every file under the writer lock. Preserve IMAP sequence numbers and UID/MODSEQ semantics; HTTP keyset pages cannot simply replace IMAP SELECT. Test: a large synthetic Note corpus, a small Tag mailbox within it, multiple selected connections and one edit. Check that per-connection source bytes remain bounded independently of corpus size, unchanged refresh reads no bodies, UID order stays stable, and Apple merge/EXPUNGE tests still pass. Duplicate search: all-state IMAP/memory/unbounded. #550 concerns APPEND latency; #644 concerns edit collisions. This issue concerns SELECT/refresh residency and corpus limits.

kayg referenced this issue

2026-10-02 13:13:24 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:22:10 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:25:41 +00:00

Author

Owner

Starting work on job/notesperf, based on job/merge-round-7a at 2f4482ded066d9c5d9c59130377907f7fd2916c9. I will address the issue with focused changes and regression coverage, then report the final head SHA and verbatim gate output here.

Starting work on `job/notesperf`, based on `job/merge-round-7a` at `2f4482ded066d9c5d9c59130377907f7fd2916c9`. I will address the issue with focused changes and regression coverage, then report the final head SHA and verbatim gate output here.

kayg commented

2026-10-02 17:23:59 +00:00

Author

Owner

Decision for #808: the #665 revision cache and #666 snapshot LRU are web client state helpers in apps/web/src/lib/api/revision-cache.ts and apps/web/src/lib/*; #667 mutation receipts and #668 change stream do not expose a Notes IMAP source projection in this branch. I kept the IMAP revision-source sharing local to the Notes IMAP adapter, keyed by User, Note, modseq and content hash, because that is the only layer that owns these immutable message sources. This avoids wiring IMAP to a client cache or changing another crate’s behavior.

kayg commented

2026-10-02 19:14:43 +00:00

Author

Owner

IMAP Message.source is now Arc<str>, and Notes reuses one allocation for the same User, Note identity, modseq and content hash across live sessions. Weak entries do not retain old bodies after snapshots end; changed revisions get a new allocation. This cache stays in the Notes IMAP adapter because the shared #665 revision cache and #666 snapshot helpers are web client state, while #667/#668 do not expose IMAP source projections here. Regression evidence: cargo test -p calternal-plugin-notes --lib home_revisions_survive_restart_and_isolate_users passed; it checks pointer sharing, revision replacement, UID stability and User isolation. Commit: 3537938f1. Performance numbers are pending.

IMAP `Message.source` is now `Arc<str>`, and Notes reuses one allocation for the same User, Note identity, modseq and content hash across live sessions. Weak entries do not retain old bodies after snapshots end; changed revisions get a new allocation. This cache stays in the Notes IMAP adapter because the shared #665 revision cache and #666 snapshot helpers are web client state, while #667/#668 do not expose IMAP source projections here. Regression evidence: `cargo test -p calternal-plugin-notes --lib home_revisions_survive_restart_and_isolate_users` passed; it checks pointer sharing, revision replacement, UID stability and User isolation. Commit: 3537938f1. Performance numbers are pending.

kayg referenced this issue from a commit

2026-10-02 20:25:54 +00:00

bench: select Notes IMAP refresh in adversarial fixture

kayg referenced this issue

2026-10-02 21:09:16 +00:00

PERF: mount Notes before optional syntax grammars load (#663) #747

kayg commented

2026-10-02 21:11:09 +00:00

Author

Owner

Implemented in 3537938f1: Arc-backed immutable sources and a weak revision cache share Notes content across live IMAP sessions. Added/extended the select profile, but the perf VM lock was busy and local server startup timed out; no before/after measurements are reported.

Implemented in 3537938f1: Arc-backed immutable sources and a weak revision cache share Notes content across live IMAP sessions. Added/extended the select profile, but the perf VM lock was busy and local server startup timed out; no before/after measurements are reported.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#808

No description provided.

Rows
Columns