PERF: Mail rule 1 — serve interactive reads from committed precomputed projections (#663) #684

New issue

Open

opened 2026-10-02 05:36:33 +00:00 by kayg · 1 comment

kayg commented

2026-10-02 05:36:33 +00:00

Owner

Context: #663 matrix and DESIGN §58 rule 1. Scope: Mail must serve interactive reads from committed precomputed projections. Shared client machinery belongs only to #665–#668.

Evidence at c4a61e8cf0:

Endpoint: GET /api/v1/mail/messages/{id}/attachments/{section_id}.
Source: crates/plugins/mail/src/routes.rs:1479. Attachment open connects to IMAP in the handler. message_detail (:1404) extracts links and can sanitize raw HTML on each read. Inbox headers already use the Index.
Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence.

Expected and regression tests:
Hold a plugin writer/indexer transaction during reads; GET returns the last complete committed projection through the WAL reader pool. Instrument external-call/file-read/parse counts and require zero on a warm UI read. Prove changed input is indexed before its new revision is published; queued provider side effects reconcile after reconnect/restart. Test a cold Index rebuild, concurrent source edits, deleted items and two-User/share isolation. Source files and authoritative security state keep their DESIGN ownership.

Performance test:
Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison.

Reuse/ownership:
Maintain existing remote-image consent and sanitized output. Precompute link lists/sanitized variants at ingest; enqueue attachment retrieval and serve cached/projection state while unavailable. This does not add sending or change #640 layouts. Reuse provider sync jobs and #626 UID generations.
Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code.

Acceptance:
Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input.

Representative measurement from the audit (not a full-view budget result):
/api/v1/mail/inbox/messages?limit=100, five serial requests after fixture readiness, all HTTP 200. Median/p95/max 41.1/48.0/48.0 ms; five-request burst median/p95/max 53.8/54.0/54.0 ms. Serial window server CPU 20 ms, RSS 444309504 bytes; no ETag on these sampled responses. Load inside lock 3.66/2.1/0.93.
Shared release server source cc25c441b7a974185622a1dee853cf38686d2b67, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9; embedded production SPA; Chromium browser on the build host through SSH/HTTPS. Server/Home/Index on perf VM HDD emulator: direct-I/O loop, 8 ms read/write dm-delay, 200 IOPS and 150 MiB/s caps. Every measured phase held flock -w 14400 /root/perf.lock. Qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty, Admin one User.
Structural source evidence above is the newer audit base, not the measured binary revision. No claim that these revisions are equivalent. The baseline in docs/perf/baseline.json uses another fixture/build/transport; no regression ratio is valid here. See #663 for matching baseline endpoint values and coverage gaps.

Context: #663 matrix and DESIGN §58 rule 1. Scope: Mail must serve interactive reads from committed precomputed projections. Shared client machinery belongs only to #665–#668. Evidence at c4a61e8cf090170f35b1bed3350d9de20c83ecd5: - Endpoint: `GET /api/v1/mail/messages/{id}/attachments/{section_id}`. - Source: `crates/plugins/mail/src/routes.rs:1479`. Attachment open connects to IMAP in the handler. message_detail (:1404) extracts links and can sanitize raw HTML on each read. Inbox headers already use the Index. - Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence. Expected and regression tests: Hold a plugin writer/indexer transaction during reads; GET returns the last complete committed projection through the WAL reader pool. Instrument external-call/file-read/parse counts and require zero on a warm UI read. Prove changed input is indexed before its new revision is published; queued provider side effects reconcile after reconnect/restart. Test a cold Index rebuild, concurrent source edits, deleted items and two-User/share isolation. Source files and authoritative security state keep their DESIGN ownership. Performance test: Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison. Reuse/ownership: Maintain existing remote-image consent and sanitized output. Precompute link lists/sanitized variants at ingest; enqueue attachment retrieval and serve cached/projection state while unavailable. This does not add sending or change #640 layouts. Reuse provider sync jobs and #626 UID generations. Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code. Acceptance: Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input. Representative measurement from the audit (not a full-view budget result): `/api/v1/mail/inbox/messages?limit=100`, five serial requests after fixture readiness, all HTTP 200. Median/p95/max 41.1/48.0/48.0 ms; five-request burst median/p95/max 53.8/54.0/54.0 ms. Serial window server CPU 20 ms, RSS 444309504 bytes; no ETag on these sampled responses. Load inside lock 3.66/2.1/0.93. Shared release server source `cc25c441b7a974185622a1dee853cf38686d2b67`, binary SHA-256 `2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9`; embedded production SPA; Chromium browser on the build host through SSH/HTTPS. Server/Home/Index on perf VM HDD emulator: direct-I/O loop, 8 ms read/write dm-delay, 200 IOPS and 150 MiB/s caps. Every measured phase held `flock -w 14400 /root/perf.lock`. Qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty, Admin one User. Structural source evidence above is the newer audit base, not the measured binary revision. No claim that these revisions are equivalent. The baseline in docs/perf/baseline.json uses another fixture/build/transport; no regression ratio is valid here. See #663 for matching baseline endpoint values and coverage gaps.

kayg referenced this issue

2026-10-02 05:46:53 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 17:39:09 +00:00

Author

Owner

P2: header and size FETCH requests download the full body (#684)

Evidence: crates/calternal-imap/src/session.rs:476 and :492 call
provider.render for RFC822.SIZE and header fields. For Mail, a cache miss
opens an upstream connection and fetches the entire MIME body
(crates/plugins/mail/src/proxy.rs:257, :291). The message loop awaits
each fetch in order (session.rs:452). Even a size or subject request can
need all attachment bytes. A message larger than 4 MiB fails the header
request because the full-body size check fails (proxy.rs:303).
Result: initial envelope fetches require one connection and full body per
uncached message. Large messages cannot supply even their headers. The
SELECT-only profile does not cover this client-visible path.
Rule: performance first; DESIGN §53 requires a usable 10,000-message Inbox
within 2 s and negligible local command time. Existing issue #684 owns Mail
reads from committed projections.
Fix: persist exact headers and RFC822.SIZE at sync time. Add provider
metadata hooks. Keep exact full MIME retrieval for actual body requests.
Serve cached metadata for large messages without applying the body limit.
Test idea: with no body cache, fetch only size and selected headers for a
message larger than 4 MiB. Assert correct metadata and no full-body download.
Measure the initial client envelope fetch, including a concurrent client.

## P2: header and size FETCH requests download the full body (#684) - Evidence: `crates/calternal-imap/src/session.rs:476` and `:492` call `provider.render` for RFC822.SIZE and header fields. For Mail, a cache miss opens an upstream connection and fetches the entire MIME body (`crates/plugins/mail/src/proxy.rs:257`, `:291`). The message loop awaits each fetch in order (`session.rs:452`). Even a size or subject request can need all attachment bytes. A message larger than 4 MiB fails the header request because the full-body size check fails (`proxy.rs:303`). - Result: initial envelope fetches require one connection and full body per uncached message. Large messages cannot supply even their headers. The SELECT-only profile does not cover this client-visible path. - Rule: performance first; DESIGN §53 requires a usable 10,000-message Inbox within 2 s and negligible local command time. Existing issue #684 owns Mail reads from committed projections. - Fix: persist exact headers and RFC822.SIZE at sync time. Add provider metadata hooks. Keep exact full MIME retrieval for actual body requests. Serve cached metadata for large messages without applying the body limit. - Test idea: with no body cache, fetch only size and selected headers for a message larger than 4 MiB. Assert correct metadata and no full-body download. Measure the initial client envelope fetch, including a concurrent client.

kayg referenced this issue

2026-10-02 17:41:43 +00:00

calternaldav: IMAP mail proxy, all Connected Accounts as one mail account (grill) #486

kayg referenced this issue

2026-10-02 19:50:40 +00:00

calternaldav: IMAP mail proxy, all Connected Accounts as one mail account (grill) #486

kayg referenced this issue

2026-10-02 20:13:08 +00:00

calternaldav: IMAP mail proxy, all Connected Accounts as one mail account (grill) #486

kayg referenced this issue

2026-10-02 23:44:50 +00:00

calternaldav: IMAP mail proxy, all Connected Accounts as one mail account (grill) #486