Mail: verify first sync actually runs after connecting; true toast wording, live progress, plain failure reasons, INFO logs

kayg commented

2026-10-01 09:42:46 +00:00

Owner

Owner report (2026-10-01, after connecting Fastmail on calternal.cloud)

Toast: "kaygdotorg@fastmail.com is connected. Mail sync is queued." Owner: "cursed text and probably wrong info? how can sync finish the moment it connects?"
(The smeared toast text is fixed on dev in 5882b4340: no glass text halo.)
Verify the Mail sync really runs (production, read-only; counts and timestamps only, never mail content):

after connecting, does a sync job start, fetch folders and messages, and finish?
what do Settings → Mail ("First sync is queued" / "Synced …") and the Mail tab show over time?
Production logs show no mail-sync lines at INFO, so add INFO-level progress logs (account id hash, folder count, message count, duration; no addresses or subjects).
Fix the UX:
The toast says what is true and useful: "Connected kaygdotorg@fastmail.com. Mail is syncing — follow progress in Mail." with an action "Open Mail".
Settings → Mail and the Mail sidebar show live progress during the first sync (folders done/total, messages fetched) and "Synced just now" only after a real completed sync. Never show "Synced" before one completed.
If the sync job fails (auth, TLS, network), show the reason in plain words on the account row, with "Try again".
Test: e2e against a local TLS IMAP test server (see #601) with 500 messages: connect → progress visible → completes → Mail lists the messages → the account row shows "Synced just now". A failure case shows the plain reason.

## Owner report (2026-10-01, after connecting Fastmail on calternal.cloud) Toast: "kaygdotorg@fastmail.com is connected. Mail sync is queued." Owner: "cursed text and probably wrong info? how can sync finish the moment it connects?" (The smeared toast text is fixed on dev in 5882b4340: no glass text halo.) **Verify the Mail sync really runs** (production, read-only; counts and timestamps only, never mail content): - after connecting, does a sync job start, fetch folders and messages, and finish? - what do Settings → Mail ("First sync is queued" / "Synced …") and the Mail tab show over time? - Production logs show no mail-sync lines at INFO, so add INFO-level progress logs (account id hash, folder count, message count, duration; no addresses or subjects). **Fix the UX:** - The toast says what is true and useful: "Connected kaygdotorg@fastmail.com. Mail is syncing — follow progress in Mail." with an action "Open Mail". - Settings → Mail and the Mail sidebar show live progress during the first sync (folders done/total, messages fetched) and "Synced just now" only after a real completed sync. Never show "Synced" before one completed. - If the sync job fails (auth, TLS, network), show the reason in plain words on the account row, with "Try again". **Test:** e2e against a local TLS IMAP test server (see #601) with 500 messages: connect → progress visible → completes → Mail lists the messages → the account row shows "Synced just now". A failure case shows the plain reason.

mail-synced-1440-dark.png

834 KiB

mail-synced-1440-light.png

725 KiB

mail-synced-390-dark.png

449 KiB

mail-synced-390-light.png

140 KiB

mail-synced-820-dark.png

460 KiB

mail-synced-820-light.png

737 KiB

settings-synced-1440-dark.png

220 KiB

settings-synced-1440-light.png

214 KiB

settings-synced-390-dark.png

83 KiB

settings-synced-390-light.png

77 KiB

settings-synced-820-dark.png

133 KiB

settings-synced-820-light.png

125 KiB

kayg referenced this issue

2026-10-01 09:53:49 +00:00

MAIL: Fastmail uses JMAP automatically (token auth, delta sync, push), same cache as IMAP #614

kayg commented

2026-10-01 10:01:28 +00:00

Author

Owner

Production evidence (orchestrator, 2026-10-01 ~12:05, read-only, counts only), owner's Fastmail account on 1631a7ef5:

mail_folders = 17; mail_sync_generations = 17, all 17 backfill_complete=1, min(low_water_uid)=0; mail_messages = 0, mail_memberships = 0. The INBOX has uid_validity 1782014281, uid_next 12659; Archive uid_next 8839.
Jobs: mail.sync jobs run and renew leases; one died after 3 attempts with "mail provider synchronization failed" (earlier, on the pre-TLS-fix build). On the current build (with the new step logging, 1631a7ef5), no step failure is logged. So the sync "succeeds" and marks every folder complete without fetching or storing a single message.
"Sync now" from Settings shows "Couldn't sync that Mail account."
So the bug is in sync_folder / the backfill window logic (crates/plugins/mail/src/sync.rs), e.g. a UID range computed empty (backfill_top_uid/low_water starting at 0, or 1:* vs uid_next-1), a FETCH whose items are all filtered out (a required attribute Fastmail does not return, e.g. MODSEQ, X-GM-*, or BODYSTRUCTURE parse), a CONDSTORE/QRESYNC path that returns nothing, or a fetch stream that is dropped. Also find why sync_now returns an error to the UI.
Reproduce without the owner's credentials: use a real IMAP server in a container (Dovecot or Stalwart) with 2,000 messages, TLS on (#601 harness), and Fastmail-like capabilities (CONDSTORE, QRESYNC, MOVE, SPECIAL-USE, LIST-STATUS, large UIDs with gaps: e.g. UIDs 1–12659 sparse). The existing tests evidently miss the case. Write the failing test first.
Fix → deploy → the owner clicks Sync now: messages must appear newest-first within seconds ("crazy fast": first 100 Inbox headers ≤ 2 s; backfill streams with progress). Never mark backfill complete when the fetched count is lower than the folder's message count (add an invariant + log).

**Production evidence (orchestrator, 2026-10-01 ~12:05, read-only, counts only), owner's Fastmail account on 1631a7ef5:** - `mail_folders` = 17; `mail_sync_generations` = 17, **all 17 `backfill_complete=1`, `min(low_water_uid)=0`**; `mail_messages` = **0**, `mail_memberships` = 0. The INBOX has `uid_validity` 1782014281, `uid_next` 12659; Archive `uid_next` 8839. - Jobs: `mail.sync` jobs run and renew leases; one died after 3 attempts with "mail provider synchronization failed" (earlier, on the pre-TLS-fix build). On the current build (with the new step logging, 1631a7ef5), **no step failure is logged**. So the sync "succeeds" and marks every folder complete **without fetching or storing a single message**. - "Sync now" from Settings shows "Couldn't sync that Mail account." **So the bug is in `sync_folder` / the backfill window logic (crates/plugins/mail/src/sync.rs)**, e.g. a UID range computed empty (`backfill_top_uid`/`low_water` starting at 0, or `1:*` vs `uid_next-1`), a FETCH whose items are all filtered out (a required attribute Fastmail does not return, e.g. MODSEQ, X-GM-*, or BODYSTRUCTURE parse), a CONDSTORE/QRESYNC path that returns nothing, or a fetch stream that is dropped. Also find why `sync_now` returns an error to the UI. **Reproduce without the owner's credentials:** use a real IMAP server in a container (Dovecot or Stalwart) with 2,000 messages, TLS on (#601 harness), and Fastmail-like capabilities (CONDSTORE, QRESYNC, MOVE, SPECIAL-USE, LIST-STATUS, large UIDs with gaps: e.g. UIDs 1–12659 sparse). The existing tests evidently miss the case. Write the failing test first. **Fix → deploy → the owner clicks Sync now:** messages must appear newest-first within seconds ("crazy fast": first 100 Inbox headers ≤ 2 s; backfill streams with progress). Never mark backfill complete when the fetched count is lower than the folder's message count (add an invariant + log).

kayg commented

2026-10-01 10:02:45 +00:00

Author

Owner

Started on job/mailsync-613, base 5882b4340e. Priority is a failing provider regression, the sync fix, per-crate gates and an atomic commit. Production access will remain read-only and limited to counts and timestamps.

Started on job/mailsync-613, base 5882b4340e7b6c1fa755567772484b19944cb1f0. Priority is a failing provider regression, the sync fix, per-crate gates and an atomic commit. Production access will remain read-only and limited to counts and timestamps.

kayg commented

2026-10-01 10:08:30 +00:00

Author

Owner

Finding: fetch_window sends multiple FETCH attributes without parentheses; async-imap forwards the string unchanged and parse_fetches discards tagged BAD/NO replies. The scripted provider accepted invalid syntax, so tests missed empty successful backfills. Sync now has a separate UI defect: the endpoint returns empty 202, but MailSection does not declare emptyResponseStatuses: [202]. Read-only production logs also show a cache SQL failure at 2026-10-01T10:04:57Z: unrecognized token backslash. Both account status update queries in this base contain literal backslashes.

kayg commented

2026-10-01 10:25:31 +00:00

Author

Owner

Regression evidence: the strict provider test failed with left: 0, right: 500 before the FETCH fix. After the fix, all 39 Mail tests pass, including tagged failure handling, completion rollback and repair of an already-complete empty generation. Real TLS Dovecot 2.4.1 with CONDSTORE, QRESYNC, MOVE, SPECIAL-USE and LIST-STATUS stored all 2,000 messages with sparse UIDs through 12658: first 100 = 660 ms locally, total = 23,744 ms, 159 windows. Dovecot accepts the old bare attribute syntax, so it does not prove Fastmail rejects that syntax; the exact Fastmail failure remains to be checked after deployment. No production messages or credentials were read. Decisions: preserve existing identities during repair, reject short completion in the same transaction, and set account completion before the bounded IDLE wait.

kayg referenced this issue from a commit

2026-10-01 10:28:45 +00:00

Fix rejected Mail fetches and repair incomplete backfills (#613)

kayg commented

2026-10-01 10:29:06 +00:00

Author

Owner

Priority sync slice committed: 1b04be06f. Read-only production count check at 2026-10-01T10:28:00Z confirms 17 folders, 17 generations marked complete, zero messages and zero memberships; mail.sync has two leased jobs. The fetched origin/dev adds step diagnostics and shortcut changes; resolving the sync.rs conflict keeps those diagnostics and the count invariant. No migration was added. Final per-crate gates and the local TLS performance profile are in progress. The perf VM lock was unavailable on the one nonblocking check, so measurements are local.

Priority sync slice committed: 1b04be06f. Read-only production count check at 2026-10-01T10:28:00Z confirms 17 folders, 17 generations marked complete, zero messages and zero memberships; mail.sync has two leased jobs. The fetched origin/dev adds step diagnostics and shortcut changes; resolving the sync.rs conflict keeps those diagnostics and the count invariant. No migration was added. Final per-crate gates and the local TLS performance profile are in progress. The perf VM lock was unavailable on the one nonblocking check, so measurements are local.

kayg referenced this issue

2026-10-01 10:35:37 +00:00

Vendored IMAP test expects provider bytes removed by the privacy patch #625

kayg referenced this issue from a commit

2026-10-01 10:38:28 +00:00

Merge origin/dev diagnostics and record Mail sync validation (#613)

kayg commented

2026-10-01 10:43:22 +00:00

Author

Owner

Priority sync slice finished. Head: 9c50871eda199ed69ef2fe3af8b44bec1bc5ebd5. Branch: job/mailsync-613.

Commits:

1b04be06f: fix rejected FETCH handling, complete-count checks, account status SQL, and generation repair; add regressions and TLS/provider performance fixtures.
b3eb4850f: merge the fetched origin/dev, keep its step diagnostics, and record validation.
9c50871ed: keep first-sync IDLE active after recording and logging real completion.

Built:

Parenthesized FETCH attribute lists for history and attachments.
Tagged FETCH failures now return an error with fixed text. They cannot look like empty successful pages.
The last backfill page must satisfy the provider message count in the same transaction that saves completion.
A falsely completed generation with missing memberships reopens in place. Message identities stay stable. The false account completion timestamp is cleared.
Partial windows no longer set the account completion timestamp. A real account completion is recorded before IDLE.
Fixed literal backslashes in both account status SQL updates.
INFO logs report an account ID hash, folder counts, page message counts, completion, and duration. They contain no addresses, subjects, credentials, or provider bytes.

Evidence:

Before the fix, the strict 500-message test failed with left: 0, right: 500.
Read-only production check at 2026-10-01T10:28:00Z: 17 folders, 17 completed generations, 0 messages, 0 memberships, 2 leased mail.sync jobs. No production content or credentials were read.
Real local TLS Dovecot 2.4.1: 2,000 messages, sparse UIDs through 12658, UIDNEXT 12659, CONDSTORE/QRESYNC/MOVE/SPECIAL-USE/LIST-STATUS. All messages entered the Index. The list query returns newest-first received times.
One bounded provider regression round covered rejection, short completion rollback, false-completion repair, real TLS fetching, and incorrect fixture password rejection. No new route was added.

Performance (local shared build host, fixed debug binary):

Serial first 100 messages: 251–755 ms. Three-run burst: 307–508 ms.
Serial page p50: 42.86–60.61 ms; p95: 146.45–236.37 ms. Full history: 15,581–20,888 ms.
Serial worker CPU: 18.33–21.46%; mean RSS: 20.14–20.97 MiB; peak RSS: 23.71–24.22 MiB.
Burst page p50: 44.59–45.84 ms; p95: 118.16–120.65 ms. Full history: 12,846–14,335 ms. Worker CPU: 24.86–26.74%; peak RSS: 23.55–24.67 MiB.
Perf VM lock unavailable on one nonblocking check. Local load average started at 27.32, 25.49, 27.14.
docs/perf/baseline.json has no comparable sparse TLS backfill profile. Its Mail account-list API baseline is p50 1.3 ms / p95 3.8 ms. No backfill regression comparison is possible.

Files:

crates/plugins/mail/src/sync.rs
crates/plugins/mail/src/cache.rs
crates/plugins/mail/src/cache/store.rs
crates/plugins/mail/src/imap.rs
crates/plugins/mail/vendor/async-imap/src/parse.rs
crates/plugins/mail/vendor/async-imap/CODEX_PATCHES.md
tests/adversarial/mail_sync_provider.py
tests/adversarial/Containerfile.mail-sync
tests/adversarial/mail-sync.md
bench/mail-sync.py
docs/perf/2026-10-01-mailsync-613.md

Gate output excerpts, verbatim:

cargo fmt --check: exit 0; no output.

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.24s

cargo test -p calternal-plugin-mail:

test result: ok. 39 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.97s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p async-imap --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.57s

cargo test -p async-imap:

assertion failed: session.noop().await.unwrap_err().to_string().contains("220 mail.example.org ESMTP Postcow")
test result: FAILED. 69 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s

The failure is filed as #625. The unchanged upstream NOOP test expects raw provider bytes. The existing privacy patch redacts them. Neither the test nor the redaction code changed in this job. The owner rule forbids changing this expectation without an explicit behavior decision, so it remains unchanged.

cargo test -p calternal-plugin-mail real_tls_provider_backfill -- --ignored --nocapture:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 40 filtered out; finished in 22.66s

Cleanup output, verbatim:

     Removed 5962 files, 1.9GiB total

No web build output exists. The test provider, temporary executable links, fixture keys, and copied test binary were removed. The worktree is clean. No push or deploy was done. Only the required origin/dev merge into the job branch was done.

Known gaps:

The exact Fastmail server failure needs confirmation after the orchestrator deploys this build and the owner runs Sync now. Dovecot accepts bare FETCH lists; the strict fixture proves the rejected-command defect, while Dovecot proves real TLS fetching and sparse backfill.
Dovecot ran as an isolated local process. The container recipe is added but not run. Container runtime checks did not provide a usable local runtime.
The toast, live progress UI, retry UI, and browser e2e remain deferred under the priority instruction to stop after the sync fix. Web gates and screenshots are not applicable to this Rust-only slice.
The reason Sync now shows an error is identified: the endpoint returns empty 202, but MailSection omits emptyResponseStatuses: [202]. The enqueue can succeed while JSON decoding fails. This UI fix remains in the follow-up slice.
The full server-process TLS wiring test requested by #601 is not part of this slice.

Decisions:

Repair the existing generation by replaying bounded UID windows and keeping identities stable; do not require the owner to reconnect.
Compare generation memberships with the provider folder count. A message can belong to more than one folder.
Record completed sync before the long IDLE wait, while keeping IDLE active on the first run.
Preserve the existing private-endpoint policy. Only the ignored provider test accepts loopback and a test CA.

Priority sync slice finished. Head: `9c50871eda199ed69ef2fe3af8b44bec1bc5ebd5`. Branch: `job/mailsync-613`. Commits: - `1b04be06f`: fix rejected FETCH handling, complete-count checks, account status SQL, and generation repair; add regressions and TLS/provider performance fixtures. - `b3eb4850f`: merge the fetched `origin/dev`, keep its step diagnostics, and record validation. - `9c50871ed`: keep first-sync IDLE active after recording and logging real completion. Built: - Parenthesized FETCH attribute lists for history and attachments. - Tagged FETCH failures now return an error with fixed text. They cannot look like empty successful pages. - The last backfill page must satisfy the provider message count in the same transaction that saves completion. - A falsely completed generation with missing memberships reopens in place. Message identities stay stable. The false account completion timestamp is cleared. - Partial windows no longer set the account completion timestamp. A real account completion is recorded before IDLE. - Fixed literal backslashes in both account status SQL updates. - INFO logs report an account ID hash, folder counts, page message counts, completion, and duration. They contain no addresses, subjects, credentials, or provider bytes. Evidence: - Before the fix, the strict 500-message test failed with `left: 0`, `right: 500`. - Read-only production check at 2026-10-01T10:28:00Z: 17 folders, 17 completed generations, 0 messages, 0 memberships, 2 leased mail.sync jobs. No production content or credentials were read. - Real local TLS Dovecot 2.4.1: 2,000 messages, sparse UIDs through 12658, UIDNEXT 12659, CONDSTORE/QRESYNC/MOVE/SPECIAL-USE/LIST-STATUS. All messages entered the Index. The list query returns newest-first received times. - One bounded provider regression round covered rejection, short completion rollback, false-completion repair, real TLS fetching, and incorrect fixture password rejection. No new route was added. Performance (local shared build host, fixed debug binary): - Serial first 100 messages: 251–755 ms. Three-run burst: 307–508 ms. - Serial page p50: 42.86–60.61 ms; p95: 146.45–236.37 ms. Full history: 15,581–20,888 ms. - Serial worker CPU: 18.33–21.46%; mean RSS: 20.14–20.97 MiB; peak RSS: 23.71–24.22 MiB. - Burst page p50: 44.59–45.84 ms; p95: 118.16–120.65 ms. Full history: 12,846–14,335 ms. Worker CPU: 24.86–26.74%; peak RSS: 23.55–24.67 MiB. - Perf VM lock unavailable on one nonblocking check. Local load average started at 27.32, 25.49, 27.14. - `docs/perf/baseline.json` has no comparable sparse TLS backfill profile. Its Mail account-list API baseline is p50 1.3 ms / p95 3.8 ms. No backfill regression comparison is possible. Files: - `crates/plugins/mail/src/sync.rs` - `crates/plugins/mail/src/cache.rs` - `crates/plugins/mail/src/cache/store.rs` - `crates/plugins/mail/src/imap.rs` - `crates/plugins/mail/vendor/async-imap/src/parse.rs` - `crates/plugins/mail/vendor/async-imap/CODEX_PATCHES.md` - `tests/adversarial/mail_sync_provider.py` - `tests/adversarial/Containerfile.mail-sync` - `tests/adversarial/mail-sync.md` - `bench/mail-sync.py` - `docs/perf/2026-10-01-mailsync-613.md` Gate output excerpts, verbatim: `cargo fmt --check`: exit 0; no output. `cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.24s ``` `cargo test -p calternal-plugin-mail`: ``` test result: ok. 39 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.97s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p async-imap --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.57s ``` `cargo test -p async-imap`: ``` assertion failed: session.noop().await.unwrap_err().to_string().contains("220 mail.example.org ESMTP Postcow") test result: FAILED. 69 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s ``` The failure is filed as #625. The unchanged upstream NOOP test expects raw provider bytes. The existing privacy patch redacts them. Neither the test nor the redaction code changed in this job. The owner rule forbids changing this expectation without an explicit behavior decision, so it remains unchanged. `cargo test -p calternal-plugin-mail real_tls_provider_backfill -- --ignored --nocapture`: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 40 filtered out; finished in 22.66s ``` Cleanup output, verbatim: ``` Removed 5962 files, 1.9GiB total ``` No web build output exists. The test provider, temporary executable links, fixture keys, and copied test binary were removed. The worktree is clean. No push or deploy was done. Only the required origin/dev merge into the job branch was done. Known gaps: - The exact Fastmail server failure needs confirmation after the orchestrator deploys this build and the owner runs Sync now. Dovecot accepts bare FETCH lists; the strict fixture proves the rejected-command defect, while Dovecot proves real TLS fetching and sparse backfill. - Dovecot ran as an isolated local process. The container recipe is added but not run. Container runtime checks did not provide a usable local runtime. - The toast, live progress UI, retry UI, and browser e2e remain deferred under the priority instruction to stop after the sync fix. Web gates and screenshots are not applicable to this Rust-only slice. - The reason Sync now shows an error is identified: the endpoint returns empty 202, but MailSection omits `emptyResponseStatuses: [202]`. The enqueue can succeed while JSON decoding fails. This UI fix remains in the follow-up slice. - The full server-process TLS wiring test requested by #601 is not part of this slice. Decisions: - Repair the existing generation by replaying bounded UID windows and keeping identities stable; do not require the owner to reconnect. - Compare generation memberships with the provider folder count. A message can belong to more than one folder. - Record completed sync before the long IDLE wait, while keeping IDLE active on the first run. - Preserve the existing private-endpoint policy. Only the ignored provider test accepts loopback and a test CA.

kayg referenced this issue from a commit

2026-10-01 10:44:54 +00:00

Keep first-sync IDLE active after recording real completion (#613)

kayg commented

2026-10-01 10:50:47 +00:00

Author

Owner

Starting Round 2 (UX) on branch job/mailsync-613, based on origin/dev at 9c50871eda. I am tracing the shared API response decoder and the existing Mail status UI before making the focused changes.

Starting Round 2 (UX) on branch job/mailsync-613, based on origin/dev at 9c50871eda199ed69ef2fe3af8b44bec1bc5ebd5. I am tracing the shared API response decoder and the existing Mail status UI before making the focused changes.

kayg referenced this issue

2026-10-01 10:52:12 +00:00

Mail: store every UID when one folder holds duplicate copies of a message (proper fix after the sync hotfix) #626

kayg commented

2026-10-01 10:53:41 +00:00

Author

Owner

Finding: packages/api-client/src/index.ts returns early for 204, but an empty 202 falls through to response.json(). The Mail sync POST returns StatusCode::ACCEPTED (202) with no body, so the shared decoder rejects after the server has queued the job. Settings currently labels the connection as queued; status is polled only for a tracked account ID, and MailSidebar does not read the sync status route.

kayg commented

2026-10-01 11:39:26 +00:00

Author

Owner

Finding for the requested TLS Dovecot e2e: the provider fixture binds only to loopback, while the production IMAP resolver rejects loopback for user-configured endpoints. I added an opt-in mail-test-provider build feature for the e2e binary. It allows only localhost on the explicitly configured fixture port and trusts only the supplied test CA; the default server build keeps the public-only rule.

Finding for the requested TLS Dovecot e2e: the provider fixture binds only to loopback, while the production IMAP resolver rejects loopback for user-configured endpoints. I added an opt-in `mail-test-provider` build feature for the e2e binary. It allows only `localhost` on the explicitly configured fixture port and trusts only the supplied test CA; the default server build keeps the public-only rule.

kayg commented

2026-10-01 12:05:14 +00:00

Author

Owner

E2E finding: the Settings connect flow authenticates against the TLS Dovecot fixture, saves the account, and returns the “Connected … Mail is syncing” toast. The real background worker then fails after 853 ms with the safe route reason “The IMAP server sent an invalid response.” The fixture’s current direct sync_folder test does not exercise sync_account or the worker job path, so I am tracing that difference now.

E2E finding: the Settings connect flow authenticates against the TLS Dovecot fixture, saves the account, and returns the “Connected … Mail is syncing” toast. The real background worker then fails after 853 ms with the safe route reason “The IMAP server sent an invalid response.” The fixture’s current direct `sync_folder` test does not exercise `sync_account` or the worker job path, so I am tracing that difference now.

kayg commented

2026-10-01 12:10:58 +00:00

Author

Owner

Root cause of the worker failure: rootless Podman exposed the host-owned Maildir as container UID 0 while Dovecot dropped to UID 1001. Its TLS LIST probe could succeed, but EXAMINE logged permission errors while updating the Maildir index and the worker returned “The IMAP server sent an invalid response.” The fixture now keeps its login sockets in the container filesystem and uses --userns=keep-id --user 0:0; the ignored real TLS backfill test passes again (2,000 messages, UID 12658, 159 windows, 23,870 ms total, page p50 84.51 ms / p95 149.39 ms).

Root cause of the worker failure: rootless Podman exposed the host-owned Maildir as container UID 0 while Dovecot dropped to UID 1001. Its TLS LIST probe could succeed, but EXAMINE logged permission errors while updating the Maildir index and the worker returned “The IMAP server sent an invalid response.” The fixture now keeps its login sockets in the container filesystem and uses `--userns=keep-id --user 0:0`; the ignored real TLS backfill test passes again (2,000 messages, UID 12658, 159 windows, 23,870 ms total, page p50 84.51 ms / p95 149.39 ms).

kayg commented

2026-10-01 12:37:52 +00:00

Author

Owner

Result

Implemented the #613 Mail sync UX and committed the work on job/mailsync-613.

The shared API client accepts empty 202 and 204 success replies while keeping JSON decoding strict for non-empty bodies.
Settings → Mail and the Mail sidebar share live folder/message progress from the existing status route. Completion appears only after last_sync_at; account rows show safe failure reasons and Try again. Sync now stays busy during an active attempt.
Connecting shows “Connected . Mail is syncing” with Open Mail. The worker records an attempt before provider I/O so the existing route can report first-sync progress.
Added a real production-build browser flow against TLS Dovecot: connect → live progress → first messages → 2,000 listed messages → real completion.

Files

Shared transport: packages/api-client/src/index.ts, packages/api-client/src/index.test.ts.
Mail UI and shared status reader: apps/web/src/lib/mail/{MailSidebar.svelte,MailSidebar.svelte.test.ts,syncStatus.ts,syncStatus.test.ts}, apps/web/src/routes/settings/mail/{MailSection.svelte,MailSection.svelte.test.ts}, apps/web/package.json.
Worker and isolated provider test: crates/plugins/mail/{Cargo.toml,src/cache.rs,src/cache/store.rs,src/imap.rs,src/sync.rs}, crates/calternal-server/Cargo.toml, tests/adversarial/{mail-sync.md,mail_sync_provider.py}, apps/web/e2e/mail-sync-613.mjs.

Browser evidence

The real production-build E2E passed:

PASS connected a real Mail account through the TLS Dovecot provider
PASS live folder and message progress appeared in the Mail sidebar
PASS 2,000 TLS fixture messages are listed and completion is real
PASS Mail screenshots captured at 390, 820, and 1440px in Light and Dark
CSP REPORTS mail-sync-613: 0 across 1 pages

Settings and Mail screenshots are attached to this issue in light and dark at 390, 820, and 1440 px:

Settings: 390 light, 390 dark, 820 light, 820 dark, 1440 light, 1440 dark.
Mail: 390 light, 390 dark, 820 light, 820 dark, 1440 light, 1440 dark.

Gates

bun run check output:

$ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
User browser caches use userStorage; only documented device/public-link exceptions remain.
Text sizes and UI shape values use shared role tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/mailsync-613/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run test final summary:

Test Files  150 passed (150)
     Tests  1017 passed (1017)
  Start at  14:26:32
  Duration  83.43s (transform 50%, environment 18%, import 18%, tests 10%, setup 4%)

Rust formatting and Clippy passed for calternal-plugin-mail (default and test-provider) and calternal-server; no formatting diagnostics or Clippy warnings. Mail crate tests passed in both configurations: 40 passed, 2 ignored, 0 failed. Server tests:

test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 15.07s

Production web build passed (✓ built in 35.67s). Real TLS provider backfill test output:

MAIL_TLS_PROFILE {"first_100_ms":900,"max_uid":12658,"messages":2000,"page_p50_ms":84.513194,"page_p95_ms":149.388679,"process_vm_hwm_kib":24732,"provider":"local TLS Dovecot; one window per call","uid_windows":159,"wall_ms":23870}
test sync::tests::real_tls_provider_backfill ... ok

After validation, cargo clean reported:

Removed 15598 files, 8.1GiB total

The web build/ and .svelte-kit/ outputs were removed. The worktree is clean.

Performance

The local debug profile is recorded in docs/perf/2026-10-01-mailsync-613.md. It ran three serial backfills and a burst of three, each with 2,000 messages. Serial full-history latency was 15.581–20.888 s; page p50 was 42.86–60.61 ms and p95 146.45–236.37 ms; worker peak RSS was 23.71–24.22 MiB. Burst full-history latency was 12.846–14.335 s; page p50 was 44.59–45.84 ms and p95 118.16–120.65 ms; worker peak RSS was 23.55–24.67 MiB. Load average was 27.32/25.49/27.14 before and 23.58/24.85/26.79 after. This is local because the perf VM lock was unavailable. docs/perf/baseline.json has no comparable sparse-TLS provider profile; its mail.accounts p50/p95 (1.3/3.8 ms) measures a different endpoint.

Decisions and known gaps

The design does not set a polling cadence or batch size. I chose a five-second poll and batches of three enabled accounts to bound concurrent status reads.
The design does not define how the local real-provider browser test trusts TLS. The opt-in mail-test-provider feature accepts only localhost at its configured fixture port and loads the fixture CA from a bounded hex environment value. Production builds do not enable the feature.
No functional gate failures remain. A directly comparable provider benchmark baseline is absent, as noted above; screenshots are attached for the orchestrator’s visual review.

Head SHA: f03440442e58a000495d3946de91f27e8052dae3.

## Result Implemented the #613 Mail sync UX and committed the work on `job/mailsync-613`. - The shared API client accepts empty `202` and `204` success replies while keeping JSON decoding strict for non-empty bodies. - Settings → Mail and the Mail sidebar share live folder/message progress from the existing status route. Completion appears only after `last_sync_at`; account rows show safe failure reasons and **Try again**. **Sync now** stays busy during an active attempt. - Connecting shows “Connected <account>. Mail is syncing” with **Open Mail**. The worker records an attempt before provider I/O so the existing route can report first-sync progress. - Added a real production-build browser flow against TLS Dovecot: connect → live progress → first messages → 2,000 listed messages → real completion. ## Files - Shared transport: `packages/api-client/src/index.ts`, `packages/api-client/src/index.test.ts`. - Mail UI and shared status reader: `apps/web/src/lib/mail/{MailSidebar.svelte,MailSidebar.svelte.test.ts,syncStatus.ts,syncStatus.test.ts}`, `apps/web/src/routes/settings/mail/{MailSection.svelte,MailSection.svelte.test.ts}`, `apps/web/package.json`. - Worker and isolated provider test: `crates/plugins/mail/{Cargo.toml,src/cache.rs,src/cache/store.rs,src/imap.rs,src/sync.rs}`, `crates/calternal-server/Cargo.toml`, `tests/adversarial/{mail-sync.md,mail_sync_provider.py}`, `apps/web/e2e/mail-sync-613.mjs`. ## Browser evidence The real production-build E2E passed: ```text PASS connected a real Mail account through the TLS Dovecot provider PASS live folder and message progress appeared in the Mail sidebar PASS 2,000 TLS fixture messages are listed and completion is real PASS Mail screenshots captured at 390, 820, and 1440px in Light and Dark CSP REPORTS mail-sync-613: 0 across 1 pages ``` Settings and Mail screenshots are attached to this issue in light and dark at 390, 820, and 1440 px: - Settings: [390 light](https://git.kayg.org/attachments/67011347-f21f-4210-9330-942e78070cca), [390 dark](https://git.kayg.org/attachments/1d7b191a-5eca-49ee-941b-ed86c02345d8), [820 light](https://git.kayg.org/attachments/c73f0574-087a-4870-aff0-020a41cfbb58), [820 dark](https://git.kayg.org/attachments/eb64dcc1-f11b-43a3-abb1-0a88a99225c0), [1440 light](https://git.kayg.org/attachments/4f4c1649-b612-4412-8b56-ae725ba577d7), [1440 dark](https://git.kayg.org/attachments/dcc506ce-ce58-41ae-b450-86677984daf8). - Mail: [390 light](https://git.kayg.org/attachments/498f6731-b619-4415-a4ad-1134b786e3c6), [390 dark](https://git.kayg.org/attachments/6e7a335a-fb85-40a3-8ab6-79ebec29bdd3), [820 light](https://git.kayg.org/attachments/84c4d8b5-3637-4e49-b9a5-424adee3260a), [820 dark](https://git.kayg.org/attachments/e5ba8e45-4225-4349-951e-74e4b3ef12d5), [1440 light](https://git.kayg.org/attachments/452dd0c8-8070-4def-8fae-c1655131ae8e), [1440 dark](https://git.kayg.org/attachments/4623cb13-57c2-4682-ad57-e6f64ecfb19a). ## Gates `bun run check` output: ```text $ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json User browser caches use userStorage; only documented device/public-link exceptions remain. Text sizes and UI shape values use shared role tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/mailsync-613/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run test` final summary: ```text Test Files 150 passed (150) Tests 1017 passed (1017) Start at 14:26:32 Duration 83.43s (transform 50%, environment 18%, import 18%, tests 10%, setup 4%) ``` Rust formatting and Clippy passed for `calternal-plugin-mail` (default and `test-provider`) and `calternal-server`; no formatting diagnostics or Clippy warnings. Mail crate tests passed in both configurations: 40 passed, 2 ignored, 0 failed. Server tests: ```text test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 15.07s ``` Production web build passed (`✓ built in 35.67s`). Real TLS provider backfill test output: ```text MAIL_TLS_PROFILE {"first_100_ms":900,"max_uid":12658,"messages":2000,"page_p50_ms":84.513194,"page_p95_ms":149.388679,"process_vm_hwm_kib":24732,"provider":"local TLS Dovecot; one window per call","uid_windows":159,"wall_ms":23870} test sync::tests::real_tls_provider_backfill ... ok ``` After validation, `cargo clean` reported: ```text Removed 15598 files, 8.1GiB total ``` The web `build/` and `.svelte-kit/` outputs were removed. The worktree is clean. ## Performance The local debug profile is recorded in `docs/perf/2026-10-01-mailsync-613.md`. It ran three serial backfills and a burst of three, each with 2,000 messages. Serial full-history latency was 15.581–20.888 s; page p50 was 42.86–60.61 ms and p95 146.45–236.37 ms; worker peak RSS was 23.71–24.22 MiB. Burst full-history latency was 12.846–14.335 s; page p50 was 44.59–45.84 ms and p95 118.16–120.65 ms; worker peak RSS was 23.55–24.67 MiB. Load average was 27.32/25.49/27.14 before and 23.58/24.85/26.79 after. This is local because the perf VM lock was unavailable. `docs/perf/baseline.json` has no comparable sparse-TLS provider profile; its `mail.accounts` p50/p95 (1.3/3.8 ms) measures a different endpoint. ## Decisions and known gaps - The design does not set a polling cadence or batch size. I chose a five-second poll and batches of three enabled accounts to bound concurrent status reads. - The design does not define how the local real-provider browser test trusts TLS. The opt-in `mail-test-provider` feature accepts only `localhost` at its configured fixture port and loads the fixture CA from a bounded hex environment value. Production builds do not enable the feature. - No functional gate failures remain. A directly comparable provider benchmark baseline is absent, as noted above; screenshots are attached for the orchestrator’s visual review. Head SHA: `f03440442e58a000495d3946de91f27e8052dae3`.

kayg referenced this issue

2026-10-01 13:59:20 +00:00

Log attachments: deleted file still shown, attachments not live, batch send not live, Log order, raw camera names #427

kayg referenced this issue

2026-10-01 15:27:52 +00:00

Log attachments: deleted file still shown, attachments not live, batch send not live, Log order, raw camera names #427

kayg referenced this issue

2026-10-01 18:28:59 +00:00

BETTER SETTINGS: organise Settings so it feels non-overwhelming and close to the User (grill pending); includes Connected Accounts #407

kayg commented

2026-10-02 02:09:30 +00:00

Author

Owner

Finding from the required post-merge local Mail protocol round: on this rootless Podman host, the documented container starts but Dovecot's static test user cannot write its bind-mounted Maildir. server.log reports euid=1001, Maildir owned by 0:0, then file_dotlock_open(...dovecot.index.log) failed: Permission denied; the Rust probe returns imap::Error::Protocol. --userns=keep-id fails earlier with fchown() failed for /fixture/run/login. For this local run, granting write permission to the generated test-only target/mail-provider/home fixture allowed the same probe to pass: 2,000 messages, 159 UID windows, 22.536 s total, 24,336 KiB peak RSS, page p50/p95 123.77/191.48 ms. The fixture and test keys were removed afterward. Please make the fixture setup work with rootless Podman or document the permission workaround.

Finding from the required post-merge local Mail protocol round: on this rootless Podman host, the documented container starts but Dovecot's static test user cannot write its bind-mounted Maildir. `server.log` reports `euid=1001`, Maildir owned by `0:0`, then `file_dotlock_open(...dovecot.index.log) failed: Permission denied`; the Rust probe returns `imap::Error::Protocol`. `--userns=keep-id` fails earlier with `fchown() failed for /fixture/run/login`. For this local run, granting write permission to the generated test-only `target/mail-provider/home` fixture allowed the same probe to pass: 2,000 messages, 159 UID windows, 22.536 s total, 24,336 KiB peak RSS, page p50/p95 123.77/191.48 ms. The fixture and test keys were removed afterward. Please make the fixture setup work with rootless Podman or document the permission workaround.

kayg referenced this issue

2026-10-02 02:13:01 +00:00

Calendar drag ghost: remove the thick snap line; inset the start/end times with card padding #612

kayg referenced this issue

2026-10-02 05:36:15 +00:00

PERF: Mail adopts shared revision, snapshot, mutation and delta contracts (#663) #672

kayg referenced this issue

2026-10-02 13:09:08 +00:00

PERF: separate Mail IDLE listeners from sync worker slots (#663) #753

kayg referenced this issue

2026-10-02 13:09:27 +00:00

PERF: bound Mail delta and expunge work per sync job (#663) #757

kayg referenced this issue

2026-10-02 13:09:54 +00:00

PERF: separate Mail IDLE waits from the three sync worker slots (#663) #763

kayg referenced this issue

2026-10-02 13:10:13 +00:00

Mail: read-state changes leave sidebar unread counts stale #771

kayg commented

2026-10-03 12:47:52 +00:00

Author

Owner

Audit against origin/dev: the Mail sync UX commits are present, but the latest post-merge TLS IMAP round only passed after changing permissions on the generated rootless-Podman Maildir fixture. The issue's real-server test still needs a supported fixture setup or a documented workaround, so it remains open.

kayg referenced this issue

2026-10-03 12:51:31 +00:00

Issue hygiene sweep 2026-10-03 #1008

kayg referenced this issue

2026-10-04 07:40:04 +00:00

Mail provider sync reports invalid responses after native Mail transfers #1037

kayg referenced this issue

2026-10-04 07:40:05 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

kayg referenced this issue

2026-10-04 07:48:25 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038