Mail: large HTML messages render as plain text (cached body limit)

kayg commented

2026-10-07 14:31:31 +00:00

Owner

Owner report 2026-10-07 on production 0419170fe: "are we still trying to show emails as plaintext? please render them as they are, they look horrible otherwise." Screenshot: a long marketing HTML mail ("Welcome to Uppbeat") shows "This message exceeds the cached body limit. The remaining content is unavailable here." and then the text/plain part with very long tracking URLs in parentheses.
Cause (apps/web/src/lib/mail/MailReaderContent.svelte:105 and the Mail cache store body_truncated): when a message body exceeds the cache limit, the HTML part is not rendered and the reader falls back to body_text.
Expected: every message with an HTML part renders as HTML (sanitised, in the existing sandboxed iframe with no scripts, no network except the image proxy policy already in place), regardless of size. If the cached copy is truncated, the reader fetches the full HTML body on demand from the provider (bounded size, e.g. up to 25 MB, streamed), caches it within the existing byte budget, and renders it; the plain-text fallback is only for messages without an HTML part or when the fetch fails (then say so plainly with Retry). Never show the "cached body limit" note when the full body can be fetched. The plain-text view also wraps long URLs instead of printing them in full.

Owner report 2026-10-07 on production 0419170fe: "are we still trying to show emails as plaintext? please render them as they are, they look horrible otherwise." Screenshot: a long marketing HTML mail ("Welcome to Uppbeat") shows "This message exceeds the cached body limit. The remaining content is unavailable here." and then the text/plain part with very long tracking URLs in parentheses. Cause (apps/web/src/lib/mail/MailReaderContent.svelte:105 and the Mail cache store body_truncated): when a message body exceeds the cache limit, the HTML part is not rendered and the reader falls back to body_text. Expected: every message with an HTML part renders as HTML (sanitised, in the existing sandboxed iframe with no scripts, no network except the image proxy policy already in place), regardless of size. If the cached copy is truncated, the reader fetches the full HTML body on demand from the provider (bounded size, e.g. up to 25 MB, streamed), caches it within the existing byte budget, and renders it; the plain-text fallback is only for messages without an HTML part or when the fetch fails (then say so plainly with Retry). Never show the "cached body limit" note when the full body can be fetched. The plain-text view also wraps long URLs instead of printing them in full.

mailhtml-1372-f319b5a43-after-large-1440-dark.png

471 KiB

mailhtml-1372-f319b5a43-after-large-1440-light.png

578 KiB

mailhtml-1372-f319b5a43-after-large-390-dark.png

118 KiB

mailhtml-1372-f319b5a43-after-large-390-light.png

134 KiB

mailhtml-1372-f319b5a43-after-large-820-dark.png

376 KiB

mailhtml-1372-f319b5a43-after-newsletter-1440-light.png

601 KiB

mailhtml-1372-f319b5a43-after-plain-quote-1440-dark.png

541 KiB

mailhtml-1372-f319b5a43-after-unstyled-390-dark.png

124 KiB

mailhtml-1372-f319b5a43-after-unstyled-390-light.png

140 KiB

mailhtml-1372-f319b5a43-after-unstyled-820-light.png

417 KiB

mailhtml-1372-f319b5a43-after-large-820-light.png

404 KiB

mailhtml-1372-f319b5a43-after-newsletter-1440-dark.png

522 KiB

mailhtml-1372-f319b5a43-after-newsletter-390-dark.png

88 KiB

mailhtml-1372-f319b5a43-after-newsletter-390-light.png

91 KiB

mailhtml-1372-f319b5a43-after-newsletter-820-dark.png

393 KiB

mailhtml-1372-f319b5a43-after-newsletter-820-light.png

410 KiB

mailhtml-1372-f319b5a43-after-plain-quote-1440-light.png

670 KiB

mailhtml-1372-f319b5a43-after-plain-quote-390-dark.png

140 KiB

mailhtml-1372-f319b5a43-after-plain-quote-390-light.png

158 KiB

mailhtml-1372-f319b5a43-after-plain-quote-820-dark.png

419 KiB

mailhtml-1372-f319b5a43-after-plain-quote-820-light.png

456 KiB

mailhtml-1372-f319b5a43-after-unstyled-1440-dark.png

488 KiB

mailhtml-1372-f319b5a43-after-unstyled-1440-light.png

600 KiB

mailhtml-1372-f319b5a43-after-unstyled-820-dark.png

387 KiB

mailhtml-1372-f319b5a43-after-plain-quote-1440-dark.png

mailhtml-1372-f319b5a43-after-plain-quote-1440-light.png

mailhtml-1372-f319b5a43-after-plain-quote-390-light.png

mailhtml-1372-f319b5a43-after-plain-quote-820-light.png

kayg commented

2026-10-07 14:33:03 +00:00

Author

Owner

Starting #1372 on job/mailhtml-1372, based on 0419170fe456b58614befcb12d9c9d255118f1c0 (origin/dev at job start). I am tracing the mail cache, provider body fetch path and reader fallback before adding the on-demand fetch.

Starting #1372 on `job/mailhtml-1372`, based on `0419170fe456b58614befcb12d9c9d255118f1c0` (origin/dev at job start). I am tracing the mail cache, provider body fetch path and reader fallback before adding the on-demand fetch.

kayg commented

2026-10-07 14:45:03 +00:00

Author

Owner

Finding for #1372: sync::fetch_window_attempt requests BODY.PEEK[]<0.65536>, and normalize_fetch stores only that prefix, truncates decoded parts, then sets body_truncated when RFC822.SIZE exceeds the returned prefix. MailReaderContent.svelte shows a cached-body-limit note and chooses plain text when sanitized HTML is absent. The existing sanitizer also has a 1 MiB input ceiling, so the required 3 MiB HTML fixture needs a larger bounded input ceiling while retaining the same active-content policy and parser limits. The full-body path will use bounded BODY.PEEK[] chunks and the existing per-User exact-body cache budget.

Finding for #1372: `sync::fetch_window_attempt` requests `BODY.PEEK[]<0.65536>`, and `normalize_fetch` stores only that prefix, truncates decoded parts, then sets `body_truncated` when RFC822.SIZE exceeds the returned prefix. `MailReaderContent.svelte` shows a cached-body-limit note and chooses plain text when sanitized HTML is absent. The existing sanitizer also has a 1 MiB input ceiling, so the required 3 MiB HTML fixture needs a larger bounded input ceiling while retaining the same active-content policy and parser limits. The full-body path will use bounded `BODY.PEEK[]` chunks and the existing per-User exact-body cache budget.

kayg commented

2026-10-07 15:05:57 +00:00

Author

Owner

The sanitizer regression was red before the change: large_mail_html_keeps_the_existing_sanitizer_policy failed at assert!(clean.contains("large message")) because the 1 MiB guard returned an empty body. I raised only the input byte ceiling to 25 MiB; CSS limits, node/depth limits and the existing allowlist remain unchanged. Afterward, cargo test -p calternal-plugin-mail html::tests -- --test-threads=4 passed: 15 passed, 0 failed.

The sanitizer regression was red before the change: `large_mail_html_keeps_the_existing_sanitizer_policy` failed at `assert!(clean.contains("large message"))` because the 1 MiB guard returned an empty body. I raised only the input byte ceiling to 25 MiB; CSS limits, node/depth limits and the existing allowlist remain unchanged. Afterward, `cargo test -p calternal-plugin-mail html::tests -- --test-threads=4` passed: 15 passed, 0 failed.

kayg referenced this issue from a commit

2026-10-07 15:06:31 +00:00

Allow bounded large Mail HTML through sanitizer (#1372)

kayg commented

2026-10-07 15:26:08 +00:00

Author

Owner

Finding: the first #1372 parser regression test showed that mail_parser::body_html() synthesizes HTML from a text/plain part. That would keep a no-HTML message in the iframe path. The shared body extractor now checks the actual MIME part type, so plain-only messages stay in the text fallback while sync and on-demand reads use the same interpretation.
Decision: treat only a MIME text/html part as body_html. This follows #1372 and keeps existing sanitization unchanged.

Finding: the first #1372 parser regression test showed that mail_parser::body_html() synthesizes HTML from a text/plain part. That would keep a no-HTML message in the iframe path. The shared body extractor now checks the actual MIME part type, so plain-only messages stay in the text fallback while sync and on-demand reads use the same interpretation. Decision: treat only a MIME text/html part as body_html. This follows #1372 and keeps existing sanitization unchanged.

kayg commented

2026-10-07 15:51:47 +00:00

Author

Owner

Finding: the first full Mail crate test run found that the schema inventory test did not list the new mail_reader_bodies table. I updated the inventory to include the intentional #1372 table.
The same parallel run reported a transport failure in delta_catch_up_shares_the_bounded_window_budget_and_resumes. Its focused rerun passed with --test-threads=1 in 47.87 s. The evidence points to shared-host load during the parallel suite, not a repeatable sync regression.

Finding: the first full Mail crate test run found that the schema inventory test did not list the new `mail_reader_bodies` table. I updated the inventory to include the intentional #1372 table. The same parallel run reported a transport failure in `delta_catch_up_shares_the_bounded_window_budget_and_resumes`. Its focused rerun passed with `--test-threads=1` in 47.87 s. The evidence points to shared-host load during the parallel suite, not a repeatable sync regression.

kayg commented

2026-10-07 15:58:51 +00:00

Author

Owner

Progress on #1372: the full body route uses owner-scoped message lookups, BODY.PEEK[] in bounded 256 KiB chunks, a 25 MiB cap, request deadlines, and the existing sanitizer and image proxy policy. Full MIME uses the shared per-User LRU budget. A focused cache test confirms that older proxy MIME is evicted before a newer reader body and that another User's cache is unchanged.

Evidence so far:

cargo test -p calternal-plugin-mail shared_body_budget_evicts_oldest_across_cache_tables_per_user -- --test-threads=1: 1 passed.
cargo test -p calternal-plugin-mail full_body_parser_keeps_large_html_and_plain_text_messages -- --test-threads=1: passed.
cargo test -p calternal-plugin-mail another_user_cannot_list_or_open_a_mail_account -- --test-threads=1: passed.
cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings: passed.

I am generating the API contract and completing the web reader and real-provider checks next.

Progress on #1372: the full body route uses owner-scoped message lookups, `BODY.PEEK[]` in bounded 256 KiB chunks, a 25 MiB cap, request deadlines, and the existing sanitizer and image proxy policy. Full MIME uses the shared per-User LRU budget. A focused cache test confirms that older proxy MIME is evicted before a newer reader body and that another User's cache is unchanged. Evidence so far: - `cargo test -p calternal-plugin-mail shared_body_budget_evicts_oldest_across_cache_tables_per_user -- --test-threads=1`: 1 passed. - `cargo test -p calternal-plugin-mail full_body_parser_keeps_large_html_and_plain_text_messages -- --test-threads=1`: passed. - `cargo test -p calternal-plugin-mail another_user_cannot_list_or_open_a_mail_account -- --test-threads=1`: passed. - `cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings`: passed. I am generating the API contract and completing the web reader and real-provider checks next.

kayg commented

2026-10-07 16:08:47 +00:00

Author

Owner

Finding on #1372: when Mail access changed during a full-body request, the reader cache invalidation left that old promise in the deduplication map. A new read reused it instead of starting under the current access revision. The regression test failed before the fix because apiFetch was called once instead of twice. invalidate() now clears pending full-body entries; the focused suite passes afterward:

Test Files  1 passed (1)
Tests       10 passed (10)

Finding on #1372: when Mail access changed during a full-body request, the reader cache invalidation left that old promise in the deduplication map. A new read reused it instead of starting under the current access revision. The regression test failed before the fix because `apiFetch` was called once instead of twice. `invalidate()` now clears pending full-body entries; the focused suite passes afterward: ``` Test Files 1 passed (1) Tests 10 passed (10) ```

kayg commented

2026-10-07 16:10:11 +00:00

Author

Owner

Follow-up for the access-change finding: invalidation now aborts the old request through its AbortController before clearing the per-User dedupe key. The regression also asserts that the old signal is aborted and that the fresh request starts. Focused result: 10 tests passed, 0 failed.

kayg commented

2026-10-07 16:16:21 +00:00

Author

Owner

Finding on #1372: the reader rendered shared RequestState while loading but did not pass its loading prop, which defaults to false. The progress line therefore stayed hidden. MailReaderContent.svelte now passes the state explicitly; the provider-backed e2e asserts that the status line appears while the body request is held.

Finding on #1372: the reader rendered shared `RequestState` while loading but did not pass its `loading` prop, which defaults to false. The progress line therefore stayed hidden. `MailReaderContent.svelte` now passes the state explicitly; the provider-backed e2e asserts that the status line appears while the body request is held.

kayg commented

2026-10-07 17:37:35 +00:00

Author

Owner

Progress on #1372 (branch mailhtml-1372): the production web build passed. The first perf-lint pass identified the new GET /messages/{id}/body route as missing from contracts/perf/registry.json and an unresolved Self::new constructor call in the existing provider error mapper. I added the exact route record and a narrowly scoped expiry pin for that constructor call; I will rerun the guard after regenerating the OpenAPI client.

Progress on #1372 (branch mailhtml-1372): the production web build passed. The first perf-lint pass identified the new `GET /messages/{id}/body` route as missing from `contracts/perf/registry.json` and an unresolved `Self::new` constructor call in the existing provider error mapper. I added the exact route record and a narrowly scoped expiry pin for that constructor call; I will rerun the guard after regenerating the OpenAPI client.

kayg commented

2026-10-07 18:26:22 +00:00

Author

Owner

#1372 progress: the real-provider E2E passed against the local TLS Dovecot fixture. It loaded the 3 MiB HTML part into the sandboxed iframe, confirmed the message stayed unseen, observed no CSP reports or remote requests, and verified plain-text fallback plus Retry. The fixture now waits for background sync to publish INBOX before selecting a message.

Before/after production screenshots are in artifacts/mailhtml/{before,after}/ at 390, 820 and 1440 px in light and dark mode, with macOS platform emulation. Captures wait for the reader content to finish loading.

The local debug-server profile is recorded in docs/perf/baseline.json and docs/perf/runs/mailhtml-1372-local.json. It used a 3 MiB MIME HTML part: 12 warm reads measured p50/p95 10,244.2/11,409.4 ms; the 4-request burst returned one 200 and three 429 responses. Host load was 31.73/36.45/46.76 before and 27.54/33.15/44.14 after, so latency is load-sensitive and is not a comparison against the perf VM.

#1372 progress: the real-provider E2E passed against the local TLS Dovecot fixture. It loaded the 3 MiB HTML part into the sandboxed iframe, confirmed the message stayed unseen, observed no CSP reports or remote requests, and verified plain-text fallback plus Retry. The fixture now waits for background sync to publish INBOX before selecting a message. Before/after production screenshots are in `artifacts/mailhtml/{before,after}/` at 390, 820 and 1440 px in light and dark mode, with macOS platform emulation. Captures wait for the reader content to finish loading. The local debug-server profile is recorded in `docs/perf/baseline.json` and `docs/perf/runs/mailhtml-1372-local.json`. It used a 3 MiB MIME HTML part: 12 warm reads measured p50/p95 10,244.2/11,409.4 ms; the 4-request burst returned one 200 and three 429 responses. Host load was 31.73/36.45/46.76 before and 27.54/33.15/44.14 after, so latency is load-sensitive and is not a comparison against the perf VM.

kayg commented

2026-10-07 18:50:12 +00:00

Author

Owner

#1372 complete

Branch: job/mailhtml-1372
Head SHA: f21e87977
git fetch origin && git merge origin/dev reported Already up to date. The branch contains no push or deployment. Commits: 8ce9a554e, 94ca358ac, 67f0d6fa1, 48d85b2aa, f21e87977.

Built

On open, a truncated or HTML-less message keeps its header and cached content visible while the reader fetches the complete body. The route streams MIME with BODY.PEEK[] in bounded chunks, enforces a 25 MiB cap and deadlines, and passes content through the existing sanitizer and image proxy. It keeps raw MIME in the shared per-User 128 MiB LRU and returns sanitized output with an exact ETag. Only one full-body parse runs per server process; saturation returns 429. The reader uses the existing bounded per-User cache, shows inline progress, supports Retry with text fallback, removes the obsolete cache-limit note and wraps long text URLs.

The TLS Dovecot E2E passed: the 3 MiB HTML part rendered in the sandboxed iframe; no script or remote request ran; manual read state stayed unchanged; the text-only message retried without an iframe; CSP reported zero violations. The plugin suite also covered sanitizer behavior, plain text and same-404 cross-User access.

Requested gate output (verbatim)

cd apps/web && bun run build — exit 0: Compressed 917 static variants; saved 21489555 bytes.
cargo fmt --check — exit 0; no output.
cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings — exit 0: Finished dev profile [unoptimized + debuginfo] target(s) in 46.89s
cargo test -p calternal-plugin-mail — test result: ok. 133 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 30.14s
bun run check — perf-lint: PASS; 0 violations; 22544 scoped exceptions; svelte-check found 0 errors and 1 warning in 1 file.
bun run test — Test Files 287 passed (287); Tests 2038 passed (2038); Duration 248.21s (transform 28%, environment 27%, import 22%, tests 17%, setup 5%).

Cleanup: Removed 18326 files, 12.4GiB total. The web production build output was removed.

Screenshots and performance

Before and after production screenshots cover 390, 820 and 1440 px in light and dark modes, with macOS platform emulation. They are in artifacts/mailhtml/before/ and artifacts/mailhtml/after/; PNGs are ignored and were not committed. The available fj CLI supports text comments only, so these files remain in the worktree for review.

The local debug-server profile is in docs/perf/baseline.json and docs/perf/runs/mailhtml-1372-local.json. With a 3 MiB MIME HTML part, 12 warm reads measured p50/p95 10,244.2/11,409.4 ms; the four-request burst returned one 200 and three 429s. Cold open was 11,605 ms. Host load was 31.73/36.45/46.76 before and 27.54/33.15/44.14 after. The perf VM was free but had no Node, Podman, Chromium or prepared Mail fixture, so this is load-sensitive local data, not a perf-VM comparison. Expected cost is one streamed parse and sanitization per uncached message, O(MIME bytes), capped at 25 MiB; LRU reuse avoids another provider fetch in the reader.

UX gaps

Closed: cached header/body remains visible; inline progress does not cover the header; Retry falls back to text; no-HTML mail renders text; old cached-limit note removed; long URLs wrap. The retry is the shared RequestState button. E2E exercised pointer retry; keyboard and touch retry were not separately scripted.

Decisions and remaining merge-round checks

Where DESIGN was silent, I used GET /api/v1/mail/messages/{id}/body, strong ETag/304 validation, one server parse at a time with 429 on saturation, and the existing per-User MIME LRU. No dependencies were added. Regenerating OpenAPI adds one path and one schema; the generator also changes key ordering throughout the JSON file, with no other semantic contract changes.

The four-hour job limit was reached after the requested gates. The additional server-crate checks are unfinished: cargo clippy -p calternal-server --all-targets -- -D warnings was stopped at the limit, and cargo test -p calternal-server was not started. Run both on the merge branch. The full adversarial matrices remain for the merge round under the 2026-10-02 verification policy.

Files

Reader: apps/web/src/lib/mail/MailReaderContent.svelte, MailView.svelte, mailCache.ts, remoteContent.ts, remoteContent.test.ts.
Mail backend: crates/plugins/mail/migrations/0021_reader_bodies.sql; src/cache/store.rs, html.rs, imap.rs, proxy.rs, proxy_tests.rs, proxy_transfers.rs, remote.rs, routes.rs, sync.rs.
Contract/client: contracts/action-policy.json, contracts/openapi.json, contracts/perf/adoption-1058.json, exceptions.json, registry.json; packages/api-client/src/generated.ts, index.ts, index.test.ts.
Fixtures and profiles: tests/adversarial/mail_sync_provider.py, apps/web/e2e/mailhtml-1372.mjs, bench/mailhtml-1372.mjs, docs/perf/baseline.json, docs/perf/runs/mailhtml-1372-local.json.
Design: docs/DESIGN.md.

#1372 complete Branch: `job/mailhtml-1372` Head SHA: `f21e87977` `git fetch origin && git merge origin/dev` reported `Already up to date.` The branch contains no push or deployment. Commits: `8ce9a554e`, `94ca358ac`, `67f0d6fa1`, `48d85b2aa`, `f21e87977`. ## Built On open, a truncated or HTML-less message keeps its header and cached content visible while the reader fetches the complete body. The route streams MIME with `BODY.PEEK[]` in bounded chunks, enforces a 25 MiB cap and deadlines, and passes content through the existing sanitizer and image proxy. It keeps raw MIME in the shared per-User 128 MiB LRU and returns sanitized output with an exact ETag. Only one full-body parse runs per server process; saturation returns 429. The reader uses the existing bounded per-User cache, shows inline progress, supports Retry with text fallback, removes the obsolete cache-limit note and wraps long text URLs. The TLS Dovecot E2E passed: the 3 MiB HTML part rendered in the sandboxed iframe; no script or remote request ran; manual read state stayed unchanged; the text-only message retried without an iframe; CSP reported zero violations. The plugin suite also covered sanitizer behavior, plain text and same-404 cross-User access. ## Requested gate output (verbatim) 1. `cd apps/web && bun run build` — exit 0: `Compressed 917 static variants; saved 21489555 bytes.` 2. `cargo fmt --check` — exit 0; no output. 3. `cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings` — exit 0: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 46.89s` 4. `cargo test -p calternal-plugin-mail` — `test result: ok. 133 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 30.14s` 5. `bun run check` — `perf-lint: PASS; 0 violations; 22544 scoped exceptions`; `svelte-check found 0 errors and 1 warning in 1 file`. 6. `bun run test` — `Test Files 287 passed (287)`; `Tests 2038 passed (2038)`; `Duration 248.21s (transform 28%, environment 27%, import 22%, tests 17%, setup 5%)`. Cleanup: `Removed 18326 files, 12.4GiB total`. The web production build output was removed. ## Screenshots and performance Before and after production screenshots cover 390, 820 and 1440 px in light and dark modes, with macOS platform emulation. They are in `artifacts/mailhtml/before/` and `artifacts/mailhtml/after/`; PNGs are ignored and were not committed. The available `fj` CLI supports text comments only, so these files remain in the worktree for review. The local debug-server profile is in `docs/perf/baseline.json` and `docs/perf/runs/mailhtml-1372-local.json`. With a 3 MiB MIME HTML part, 12 warm reads measured p50/p95 10,244.2/11,409.4 ms; the four-request burst returned one 200 and three 429s. Cold open was 11,605 ms. Host load was 31.73/36.45/46.76 before and 27.54/33.15/44.14 after. The perf VM was free but had no Node, Podman, Chromium or prepared Mail fixture, so this is load-sensitive local data, not a perf-VM comparison. Expected cost is one streamed parse and sanitization per uncached message, O(MIME bytes), capped at 25 MiB; LRU reuse avoids another provider fetch in the reader. ## UX gaps Closed: cached header/body remains visible; inline progress does not cover the header; Retry falls back to text; no-HTML mail renders text; old cached-limit note removed; long URLs wrap. The retry is the shared `RequestState` button. E2E exercised pointer retry; keyboard and touch retry were not separately scripted. ## Decisions and remaining merge-round checks Where DESIGN was silent, I used `GET /api/v1/mail/messages/{id}/body`, strong ETag/304 validation, one server parse at a time with 429 on saturation, and the existing per-User MIME LRU. No dependencies were added. Regenerating OpenAPI adds one path and one schema; the generator also changes key ordering throughout the JSON file, with no other semantic contract changes. The four-hour job limit was reached after the requested gates. The additional server-crate checks are unfinished: `cargo clippy -p calternal-server --all-targets -- -D warnings` was stopped at the limit, and `cargo test -p calternal-server` was not started. Run both on the merge branch. The full adversarial matrices remain for the merge round under the 2026-10-02 verification policy. ## Files - Reader: `apps/web/src/lib/mail/MailReaderContent.svelte`, `MailView.svelte`, `mailCache.ts`, `remoteContent.ts`, `remoteContent.test.ts`. - Mail backend: `crates/plugins/mail/migrations/0021_reader_bodies.sql`; `src/cache/store.rs`, `html.rs`, `imap.rs`, `proxy.rs`, `proxy_tests.rs`, `proxy_transfers.rs`, `remote.rs`, `routes.rs`, `sync.rs`. - Contract/client: `contracts/action-policy.json`, `contracts/openapi.json`, `contracts/perf/adoption-1058.json`, `exceptions.json`, `registry.json`; `packages/api-client/src/generated.ts`, `index.ts`, `index.test.ts`. - Fixtures and profiles: `tests/adversarial/mail_sync_provider.py`, `apps/web/e2e/mailhtml-1372.mjs`, `bench/mailhtml-1372.mjs`, `docs/perf/baseline.json`, `docs/perf/runs/mailhtml-1372-local.json`. - Design: `docs/DESIGN.md`.

kayg commented

2026-10-07 18:50:24 +00:00

Author

Owner

Correction to the gate-output line above: the exact clippy output was Finished dev profile [unoptimized + debuginfo] target(s) in 46.89s.

Correction to the gate-output line above: the exact clippy output was `Finished dev profile [unoptimized + debuginfo] target(s) in 46.89s`.

kayg commented

2026-10-08 02:14:54 +00:00

Author

Owner

Starting #1372 in branch job/mailhtml-1372. Base SHA: 0419170fe4. Start HEAD: f21e879770.

Starting #1372 in branch job/mailhtml-1372. Base SHA: 0419170fe456b58614befcb12d9c9d255118f1c0. Start HEAD: f21e87977013a1a9da5966dd287a3d0a8b04d77a.

kayg commented

2026-10-08 02:19:13 +00:00

Author

Owner

Finding: crates/plugins/mail/src/routes.rs reads cached raw MIME but reparses and sanitizes the full body on every request, then serializes and hashes the response before evaluating If-None-Match. The recorded profile made 12 unconditional GETs and used 119,570 ms server CPU (p50/p95 10,244/11,409 ms). I am caching the exact rendered JSON by raw-body digest and current remote-content/account access state; each request will still check current ownership and access.

kayg commented

2026-10-08 02:24:51 +00:00

Author

Owner

Correction to my prior CPU figure: bench/mailhtml-1372.mjs converted /proc jiffies with a factor 100 too large. The recorded 119,570 ms is not valid CPU evidence; the old p50/p95 wall times are valid. I corrected the counter conversion and will report the new profile. The source path still shows repeated parse/sanitize/serialize work on every unconditional warm GET.

kayg commented

2026-10-08 04:19:22 +00:00

Author

Owner

#1372 complete on job/mailhtml-1372

Head: 3feeed2b7291ce15ab6ff9a3a7832d06928d4807

Built

Full Mail bodies now use the per-User raw MIME cache and a bounded rendered JSON LRU. Warm reads reuse the parsed representation and its strong ETag; a matching If-None-Match returns an empty 304. Cold reads use IMAP BODY.PEEK, retain access and UID checks, and do not mark a message read.
Unstyled HTML uses the app reading font, size, line height, padding and theme colors through low-specificity defaults. Sender styles retain priority. Plain text wraps, uses the app theme and visually distinguishes contiguous quoted lines.
Extended the real-server Mail adversarial probe with sanitizer, ETag/304, warm-burst, malformed ID, missing ID and cross-User checks. The probe and its seeded MIME fixture pass syntax checks; the full adversarial matrix is deferred to the merge round.
Captured and attached 24 screenshots: styled newsletter, unstyled HTML, quoted plain text and 3 MiB HTML, at 390/820/1440 px, light/dark, with macOS platform emulation.

Performance

The previous local warm-read p95 was 11,409 ms. The new local profile used 10 cold and 10 warm samples for each body size. It ran on a loaded local host, not the perf VM.

3 MiB: cold p50/p95 7,492.6/8,741.9 ms; warm p50/p95 34.5/44.2 ms; warm response body 0 bytes.
50 KiB: cold p50/p95 95.6/498.6 ms; warm p50/p95 15.1/23.7 ms; warm response body 0 bytes.
Four concurrent 3 MiB conditional warm reads: p50/p95 57.4/73.7 ms; all returned 304.
Cold 3 MiB samples used 6,177 ms total server CPU across 10 reads, against 7,492.6 ms p50 latency, consistent with most cold latency being provider wait. Cold path cost is one bounded fetch (up to 25 MiB), one MIME parse/sanitize and one cache write. Warm path cost is a bounded cache lookup plus access/revision checks; matching ETags transfer no body bytes.
The perf VM was held by another job and did not have the local Node/browser runtime, so I used the local profile.

Gate output

cargo fmt --all -- --check
(exit 0; no output)

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 46.89s

cargo test -p calternal-plugin-mail -- --test-threads=4
test result: ok. 136 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out.

cargo clippy -p calternal-plugin --all-targets -- -D warnings
PASS

cargo test -p calternal-plugin -- --test-threads=4
test result: ok. 63 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out.

cargo clippy -p calternal-server --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 36.11s

cargo test -p calternal-server -- --test-threads=4
search_reconcile_calendar_write_profile files=20000 writes=32 p50_ms=14010 p95_ms=14210 p95_budget_ms=10000
Calendar write p95 14.210605296s exceeded 10s
test result: FAILED. 271 passed; 1 failed; 12 ignored; 0 measured; 0 filtered out; finished in 148.25s

error: test failed, to rerun pass `-p calternal-server --bin calternal-server`

bun run check
perf-lint: PASS; 0 violations; 22551 scoped exceptions
svelte-check found 0 errors and 0 warnings

bun run test
297 files and 2,119 tests passed

bun run build
PASS

Production E2E
PASS 3 MiB HTML loaded into the sandbox without setting Seen
PASS failure shows Retry and plain text; retry succeeds without an iframe
PASS after screenshots captured at phone, tablet and desktop widths with macOS shortcuts.
CSP REPORTS mailhtml-1372: 0 across 1 pages

Decisions

DESIGN has no Mail rule for unstyled HTML in dark mode. Unstyled messages use app theme colors; messages that set their own colors or background keep them.
Contiguous quoted plain-text lines use muted italic styling. Quote grouping caps at 1,024 segments to bound DOM work.
The reader and remote-content proxy share the per-User 128 MiB body-cache budget.

Known gaps

cargo test -p calternal-server had one load-sensitive calendar-write profile failure; the other 271 tests passed. The host was busy. No rerun was made.
The perf-VM measurement was unavailable; the local host profile is not a release-server measurement.
The new adversarial probe passed JS/Python syntax checks but was not run here, per merge-round verification policy.

For the merge round

Run bash tests/adversarial/run-split.sh. It must execute the extended Mail API probe against a real local server and prove the body endpoint’s sanitizer, strong ETag/304 behavior, concurrent warm-cache behavior, malformed/missing ID handling and cross-User 404 behavior.

#1372 complete on `job/mailhtml-1372` Head: `3feeed2b7291ce15ab6ff9a3a7832d06928d4807` ## Built - Full Mail bodies now use the per-User raw MIME cache and a bounded rendered JSON LRU. Warm reads reuse the parsed representation and its strong ETag; a matching `If-None-Match` returns an empty 304. Cold reads use IMAP `BODY.PEEK`, retain access and UID checks, and do not mark a message read. - Unstyled HTML uses the app reading font, size, line height, padding and theme colors through low-specificity defaults. Sender styles retain priority. Plain text wraps, uses the app theme and visually distinguishes contiguous quoted lines. - Extended the real-server Mail adversarial probe with sanitizer, ETag/304, warm-burst, malformed ID, missing ID and cross-User checks. The probe and its seeded MIME fixture pass syntax checks; the full adversarial matrix is deferred to the merge round. - Captured and attached 24 screenshots: styled newsletter, unstyled HTML, quoted plain text and 3 MiB HTML, at 390/820/1440 px, light/dark, with macOS platform emulation. ## Performance The previous local warm-read p95 was 11,409 ms. The new local profile used 10 cold and 10 warm samples for each body size. It ran on a loaded local host, not the perf VM. - 3 MiB: cold p50/p95 7,492.6/8,741.9 ms; warm p50/p95 34.5/44.2 ms; warm response body 0 bytes. - 50 KiB: cold p50/p95 95.6/498.6 ms; warm p50/p95 15.1/23.7 ms; warm response body 0 bytes. - Four concurrent 3 MiB conditional warm reads: p50/p95 57.4/73.7 ms; all returned 304. - Cold 3 MiB samples used 6,177 ms total server CPU across 10 reads, against 7,492.6 ms p50 latency, consistent with most cold latency being provider wait. Cold path cost is one bounded fetch (up to 25 MiB), one MIME parse/sanitize and one cache write. Warm path cost is a bounded cache lookup plus access/revision checks; matching ETags transfer no body bytes. - The perf VM was held by another job and did not have the local Node/browser runtime, so I used the local profile. ## Gate output ```text cargo fmt --all -- --check (exit 0; no output) cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 46.89s cargo test -p calternal-plugin-mail -- --test-threads=4 test result: ok. 136 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out. cargo clippy -p calternal-plugin --all-targets -- -D warnings PASS cargo test -p calternal-plugin -- --test-threads=4 test result: ok. 63 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out. cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 36.11s cargo test -p calternal-server -- --test-threads=4 search_reconcile_calendar_write_profile files=20000 writes=32 p50_ms=14010 p95_ms=14210 p95_budget_ms=10000 Calendar write p95 14.210605296s exceeded 10s test result: FAILED. 271 passed; 1 failed; 12 ignored; 0 measured; 0 filtered out; finished in 148.25s error: test failed, to rerun pass `-p calternal-server --bin calternal-server` bun run check perf-lint: PASS; 0 violations; 22551 scoped exceptions svelte-check found 0 errors and 0 warnings bun run test 297 files and 2,119 tests passed bun run build PASS Production E2E PASS 3 MiB HTML loaded into the sandbox without setting Seen PASS failure shows Retry and plain text; retry succeeds without an iframe PASS after screenshots captured at phone, tablet and desktop widths with macOS shortcuts. CSP REPORTS mailhtml-1372: 0 across 1 pages ``` ## Decisions - DESIGN has no Mail rule for unstyled HTML in dark mode. Unstyled messages use app theme colors; messages that set their own colors or background keep them. - Contiguous quoted plain-text lines use muted italic styling. Quote grouping caps at 1,024 segments to bound DOM work. - The reader and remote-content proxy share the per-User 128 MiB body-cache budget. ## Known gaps - `cargo test -p calternal-server` had one load-sensitive calendar-write profile failure; the other 271 tests passed. The host was busy. No rerun was made. - The perf-VM measurement was unavailable; the local host profile is not a release-server measurement. - The new adversarial probe passed JS/Python syntax checks but was not run here, per merge-round verification policy. ## For the merge round Run `bash tests/adversarial/run-split.sh`. It must execute the extended Mail API probe against a real local server and prove the body endpoint’s sanitizer, strong ETag/304 behavior, concurrent warm-cache behavior, malformed/missing ID handling and cross-User 404 behavior.

kayg referenced this issue

2026-10-08 04:21:46 +00:00

Server test: Calendar write p95 over 10 s with 20k files #1441

kayg referenced this issue

2026-10-08 05:30:11 +00:00

Mail Morph: switching messages closes the reader and never reopens it; rapid switching must be instant #1368

kayg referenced this issue

2026-10-08 05:46:20 +00:00

Mail: selecting a message fires about 75 requests #1449

kayg referenced this issue

2026-10-08 18:42:36 +00:00

Merge round 35b #1479