kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 803 Pull requests Projects Releases Packages Wiki Activity Actions 472

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

New issue
Closed
opened 2026-10-04 07:10:58 +00:00 by kayg · 78 comments
kayg commented 2026-10-04 07:10:58 +00:00
Owner
Copy link

Owner request (2026-10-04)

"This needs to be extensively tested. Try to make it crash by overloading it, going through the most bizarre of scenarios. We want it foolproof!"

Scope: the CalternalDAV mail proxy (#486, branch job/mailproxy-486 at 86f2e061e) and the Mail provider sync behind it (#1037).

What to do

Build a long-running stress and fault harness in tests/adversarial/ (extend mail_proxy.py; document in mail-sync.md) against a real local server with the local Dovecot TLS fixture. No Mac needed for this round. Every finding: fix with a regression test, or file a separate issue when it does not block a merge (CLAUDE.md "Adversarial testing").

Load

  • 50+ concurrent IMAP clients on one account and across 3 accounts; each runs mixed SELECT/FETCH/SEARCH/IDLE/APPEND/COPY/MOVE/STORE/EXPUNGE loops for 30+ minutes.
  • Mailboxes with 100k messages, a 50 MB message, a 0-byte message, 10k folders, very deep folder trees.
  • Initial-download storms (many clients fetching everything at once) while provider sync runs.
  • Bounded memory: record RSS and open file descriptors over the run; no unbounded growth.

Bizarre scenarios

  • Two clients moving the same message to different folders at the same instant; move + delete races; expunge during FETCH; rename a folder while another client is SELECTed in it; delete a folder with IDLE open on it.
  • Upstream misbehaviour: Dovecot restarted mid-command, connection dropped mid-literal, slow-loris upstream (1 byte/s), upstream returns BYE, NO, BAD, or garbage; UIDVALIDITY changes; duplicate Message-IDs; missing Message-ID.
  • Client protocol abuse: pipelined commands, huge literals, unterminated literals, invalid UTF-7 folder names, IMAP UTF8=ACCEPT names, NUL and control bytes, overlong lines, thousands of tags, IDLE without DONE, auth floods with wrong passwords (lockout must not lock the real User out of the web UI).
  • Unicode: RTL, combining marks, emoji and NFC/NFD-equivalent folder names; names that differ only by case.
  • Server restart and SIGKILL mid-transfer: no message lost or duplicated after recovery (check by Message-ID in every folder, upstream and in calternal's store).
  • Cross-user isolation: no client ever sees another User's folders or messages (CLAUDE.md, cross-user isolation is a merge blocker).

Pass criteria

No crash, panic, 5xx, deadlock or hang; no message lost or duplicated; every operation either succeeds or fails cleanly with a correct IMAP response; memory and descriptors stay bounded; provider sync recovers by itself. Report a table of every scenario with result and evidence.

## Owner request (2026-10-04) "This needs to be extensively tested. Try to make it crash by overloading it, going through the most bizarre of scenarios. We want it foolproof!" Scope: the CalternalDAV mail proxy (#486, branch job/mailproxy-486 at 86f2e061e) and the Mail provider sync behind it (#1037). ## What to do Build a long-running stress and fault harness in `tests/adversarial/` (extend `mail_proxy.py`; document in `mail-sync.md`) against a real local server with the local Dovecot TLS fixture. No Mac needed for this round. Every finding: fix with a regression test, or file a separate issue when it does not block a merge (CLAUDE.md "Adversarial testing"). ### Load - 50+ concurrent IMAP clients on one account and across 3 accounts; each runs mixed SELECT/FETCH/SEARCH/IDLE/APPEND/COPY/MOVE/STORE/EXPUNGE loops for 30+ minutes. - Mailboxes with 100k messages, a 50 MB message, a 0-byte message, 10k folders, very deep folder trees. - Initial-download storms (many clients fetching everything at once) while provider sync runs. - Bounded memory: record RSS and open file descriptors over the run; no unbounded growth. ### Bizarre scenarios - Two clients moving the same message to different folders at the same instant; move + delete races; expunge during FETCH; rename a folder while another client is SELECTed in it; delete a folder with IDLE open on it. - Upstream misbehaviour: Dovecot restarted mid-command, connection dropped mid-literal, slow-loris upstream (1 byte/s), upstream returns BYE, NO, BAD, or garbage; UIDVALIDITY changes; duplicate Message-IDs; missing Message-ID. - Client protocol abuse: pipelined commands, huge literals, unterminated literals, invalid UTF-7 folder names, IMAP UTF8=ACCEPT names, NUL and control bytes, overlong lines, thousands of tags, IDLE without DONE, auth floods with wrong passwords (lockout must not lock the real User out of the web UI). - Unicode: RTL, combining marks, emoji and NFC/NFD-equivalent folder names; names that differ only by case. - Server restart and SIGKILL mid-transfer: no message lost or duplicated after recovery (check by Message-ID in every folder, upstream and in calternal's store). - Cross-user isolation: no client ever sees another User's folders or messages (CLAUDE.md, cross-user isolation is a merge blocker). ## Pass criteria No crash, panic, 5xx, deadlock or hang; no message lost or duplicated; every operation either succeeds or fails cleanly with a correct IMAP response; memory and descriptors stay bounded; provider sync recovers by itself. Report a table of every scenario with result and evidence.
device-setup-1440-dark.png
238 KiB
device-setup-1440-light.png
597 KiB
device-setup-390-dark.png
238 KiB
folder-1440-dark.png
754 KiB
folder-1440-light.png
843 KiB
folder-390-dark.png
229 KiB
folder-390-light.png
241 KiB
folder-820-dark.png
439 KiB
folder-820-light.png
478 KiB
inbox-1440-dark.png
757 KiB
inbox-1440-light.png
842 KiB
inbox-390-dark.png
229 KiB
inbox-390-light.png
241 KiB
inbox-820-dark.png
437 KiB
inbox-820-light.png
439 KiB
reader-1440-dark.png
699 KiB
reader-1440-light.png
794 KiB
reader-390-dark.png
180 KiB
reader-390-light.png
194 KiB
reader-820-dark.png
409 KiB
reader-820-light.png
433 KiB
thread-390-light.png
194 KiB
thread-820-light.png
408 KiB
device-setup-390-light.png
141 KiB
device-setup-820-dark.png
238 KiB
device-setup-820-light.png
279 KiB
thread-1440-dark.png
700 KiB
thread-1440-light.png
793 KiB
thread-390-dark.png
180 KiB
thread-820-dark.png
391 KiB
native-read-settled.png
304 KiB
native-flagged.png
308 KiB
native-delete.png
300 KiB
native-archive.png
319 KiB
drag-selected.png
344 KiB
native-drag.png
311 KiB
applemail-accept-30ad9ce8-native-read-settled.png
304 KiB
applemail-accept-30ad9ce8-native-flagged.png
308 KiB
applemail-accept-30ad9ce8-native-archive.png
319 KiB
applemail-accept-30ad9ce8-native-delete.png
300 KiB
applemail-accept-30ad9ce8-drag-selected.png
344 KiB
applemail-accept-30ad9ce8-native-drag.png
311 KiB
applemail-accept-30ad9ce8-native-draft.png
85 KiB
applemail-accept-30ad9ce8-native-empty.png
161 KiB
applemail-accept-30ad9ce8-native-after-final-relaunch.png
242 KiB
applemail-accept-30ad9ce8-mail-download-start.png
158 KiB
applemail-accept-30ad9ce8-web-final-inbox.zip
2.9 MiB
applemail-accept-30ad9ce8-web-archive.zip
2.6 MiB
applemail-accept-30ad9ce8-web-drafts.zip
2.6 MiB
applemail-accept-30ad9ce8-web-final-trash.zip
2.6 MiB
applemail-accept-30ad9ce8-web-final-empty.zip
2.5 MiB
applemail-accept-30ad9ce8-receipts-1038.zip
17 KiB
applemail-accept-30ad9ce8-web-final-autumn.zip
2.8 MiB
columns-empty-desktop-tokyo-night-dark.png
429 KiB
columns-empty-tablet-paper-light.png
423 KiB
columns-empty-tablet-tokyo-night-dark.png
382 KiB
390-light.png
194 KiB
390-dark.png
174 KiB
820-light.png
676 KiB
merge-round-9-mail-layouts.zip
8.8 MiB
columns-empty-desktop-paper-light.png
553 KiB
columns-empty-desktop-tokyo-night-dark.png
434 KiB
columns-empty-tablet-paper-light.png
425 KiB
columns-empty-tablet-tokyo-night-dark.png
385 KiB
merge-round-9-polished-screenshots.zip
25 MiB
columns-empty-desktop-paper-light.png
549 KiB
1440-dark.png
702 KiB
merge-round-9-final-screenshots.zip
24 MiB
columns-empty-desktop-paper-light.png
549 KiB
columns-empty-desktop-tokyo-night-dark.png
429 KiB
columns-empty-tablet-paper-light.png
423 KiB
columns-empty-tablet-tokyo-night-dark.png
382 KiB
390-light.png
176 KiB
390-dark.png
163 KiB
820-light.png
299 KiB
820-dark.png
414 KiB
1440-light.png
1.1 MiB
1440-dark.png
426 KiB
mailstress-a-body-free-initial-sync-1440-dark.png
486 KiB
mailstress-a-body-free-initial-sync-1440-light.png
594 KiB
mailstress-a-body-free-initial-sync-390-dark.png
88 KiB
mailstress-a-body-free-initial-sync-390-light.png
80 KiB
mailstress-a-body-free-initial-sync-820-dark.png
374 KiB
mailstress-a-body-free-initial-sync-820-light.png
402 KiB
820-dark.png
446 KiB
1440-light.png
1.1 MiB
device-setup-1440-dark.png device-setup-1440-light.png device-setup-390-dark.png folder-1440-dark.png folder-1440-light.png folder-390-dark.png folder-390-light.png folder-820-dark.png folder-820-light.png inbox-1440-dark.png inbox-1440-light.png inbox-390-dark.png inbox-390-light.png inbox-820-dark.png inbox-820-light.png reader-1440-dark.png reader-1440-light.png reader-390-dark.png reader-390-light.png reader-820-dark.png reader-820-light.png thread-390-light.png thread-820-light.png device-setup-390-light.png device-setup-820-dark.png device-setup-820-light.png thread-1440-dark.png thread-1440-light.png thread-390-dark.png thread-820-dark.png native-read-settled.png native-flagged.png native-delete.png native-archive.png drag-selected.png native-drag.png applemail-accept-30ad9ce8-native-read-settled.png applemail-accept-30ad9ce8-native-flagged.png applemail-accept-30ad9ce8-native-archive.png applemail-accept-30ad9ce8-native-delete.png applemail-accept-30ad9ce8-drag-selected.png applemail-accept-30ad9ce8-native-drag.png applemail-accept-30ad9ce8-native-draft.png applemail-accept-30ad9ce8-native-empty.png applemail-accept-30ad9ce8-native-after-final-relaunch.png applemail-accept-30ad9ce8-mail-download-start.png columns-empty-desktop-tokyo-night-dark.png columns-empty-tablet-paper-light.png columns-empty-tablet-tokyo-night-dark.png 390-light.png 390-dark.png 820-light.png columns-empty-desktop-paper-light.png columns-empty-desktop-tokyo-night-dark.png columns-empty-tablet-paper-light.png columns-empty-tablet-tokyo-night-dark.png columns-empty-desktop-paper-light.png 1440-dark.png columns-empty-desktop-paper-light.png columns-empty-desktop-tokyo-night-dark.png columns-empty-tablet-paper-light.png columns-empty-tablet-tokyo-night-dark.png 390-light.png 390-dark.png 820-light.png 820-dark.png 1440-light.png 1440-dark.png mailstress-a-body-free-initial-sync-1440-dark.png mailstress-a-body-free-initial-sync-1440-light.png mailstress-a-body-free-initial-sync-390-dark.png mailstress-a-body-free-initial-sync-390-light.png mailstress-a-body-free-initial-sync-820-dark.png mailstress-a-body-free-initial-sync-820-light.png 820-dark.png 1440-light.png
kayg commented 2026-10-04 07:12:46 +00:00
Author
Owner
Copy link

Started job/mailstress-1038 from requested base 86f2e061e6. Read CLAUDE.md, CONTEXT.md, DESIGN sections 45/53, #486, #1037 and mail-sync.md. Extend the existing local-only mail_proxy.py harness; keep credentials and bodies private. #1037 owns provider parsing/timeouts. The job verification policy defers full adversarial matrices to the merge round; focused #1038 regressions run here. No pushes or deploys.

Started job/mailstress-1038 from requested base 86f2e061e6363d837d122574530e44caf843e47d. Read CLAUDE.md, CONTEXT.md, DESIGN sections 45/53, #486, #1037 and mail-sync.md. Extend the existing local-only mail_proxy.py harness; keep credentials and bodies private. #1037 owns provider parsing/timeouts. The job verification policy defers full adversarial matrices to the merge round; focused #1038 regressions run here. No pushes or deploys.
kayg commented 2026-10-04 07:22:10 +00:00
Author
Owner
Copy link

Harness slice committed: 493e74c58. Three finite harness tests pass. Real Dovecot TLS fixture verified three isolated upstream Users by one distinct Message-ID sentinel per User. Added optional bounded fixture sizes (1–100000 messages; up to 10000 folders), an 80-level hierarchy, 50 MiB and zero-byte items, duplicate/missing Message-ID items. These are fixture preparation facts, not calternal scenario passes. Server feature build finished in 5m 57s. Running focused three-client fixture smoke. Full matrices and 30-minute phases remain for the merge round under the shared verification policy.

Harness slice committed: 493e74c58. Three finite harness tests pass. Real Dovecot TLS fixture verified three isolated upstream Users by one distinct Message-ID sentinel per User. Added optional bounded fixture sizes (1–100000 messages; up to 10000 folders), an 80-level hierarchy, 50 MiB and zero-byte items, duplicate/missing Message-ID items. These are fixture preparation facts, not calternal scenario passes. Server feature build finished in 5m 57s. Running focused three-client fixture smoke. Full matrices and 30-minute phases remain for the merge round under the shared verification policy.
kayg commented 2026-10-04 07:28:22 +00:00
Author
Owner
Copy link

Starting continuation on job/mailstress-1038, base 86f2e061e, current head 7bc286e28. Reviewing the unfinished coordinator and local load tests. Full matrices and long runs are reserved for the merge round by the verification policy. This job runs focused reliability regressions, records sanitized evidence, and makes no pushes or deploys.

Scenario Result Evidence
Existing changes Review in progress 493e74c58, 7bc286e28, unfinished fixture coordinator
Local host sizing Available 94 GiB total, 85 GiB available at start; shared host
Starting continuation on `job/mailstress-1038`, base `86f2e061e`, current head `7bc286e28`. Reviewing the unfinished coordinator and local load tests. Full matrices and long runs are reserved for the merge round by the verification policy. This job runs focused reliability regressions, records sanitized evidence, and makes no pushes or deploys. | Scenario | Result | Evidence | |---|---|---| | Existing changes | Review in progress | `493e74c58`, `7bc286e28`, unfinished fixture coordinator | | Local host sizing | Available | 94 GiB total, 85 GiB available at start; shared host |
kayg commented 2026-10-04 07:29:46 +00:00
Author
Owner
Copy link

Harness review finding: required SEARCH/FETCH/STORE responses could be NO/BAD while the soak still reported PASS. Required commands now require OK. COPY/MOVE still permit legal pending-transfer conflicts, with checked receipts for accepted writes. Added a regression where SEARCH returns NO; it now fails and keeps private response text out of output. Receipt verification uses one bounded deadline for the phase. Focused Python tests: 5 passed. No server defect is established by this harness finding.

Harness review finding: required SEARCH/FETCH/STORE responses could be NO/BAD while the soak still reported PASS. Required commands now require OK. COPY/MOVE still permit legal pending-transfer conflicts, with checked receipts for accepted writes. Added a regression where SEARCH returns NO; it now fails and keeps private response text out of output. Receipt verification uses one bounded deadline for the phase. Focused Python tests: 5 passed. No server defect is established by this harness finding.
kayg commented 2026-10-04 07:40:05 +00:00
Author
Owner
Copy link

Timing caveat for the preceding #1038 evidence: the small stress fixture reused the default initial UID ceiling 12658. A 32-message initial Inbox therefore required about 159 80-UID backfill windows, before new sentinels. An ordinary-message retry without the Boundary folder also completed only two accounts in 180 seconds with zero reported errors. Stress fixtures now use dense initial UIDs; #613 keeps its original gaps and UIDNEXT, with a regression for both layouts. These timings are SLOW-only fixture evidence, not evidence that the 50 MiB message caused a provider defect. A final focused smoke uses the corrected UID range.

Timing caveat for the preceding #1038 evidence: the small stress fixture reused the default initial UID ceiling 12658. A 32-message initial Inbox therefore required about 159 80-UID backfill windows, before new sentinels. An ordinary-message retry without the Boundary folder also completed only two accounts in 180 seconds with zero reported errors. Stress fixtures now use dense initial UIDs; #613 keeps its original gaps and UIDNEXT, with a regression for both layouts. These timings are SLOW-only fixture evidence, not evidence that the 50 MiB message caused a provider defect. A final focused smoke uses the corrected UID range.
kayg commented 2026-10-04 07:48:25 +00:00
Author
Owner
Copy link

Continuation report — branch job/mailstress-1038, base 86f2e061e, head 94904b828b29ae72e3286ed627d403164bb86681.

READY FOR MERGE: no.

Built: an isolated TLS fixture coordinator that reuses real passkey/API/server helpers; bounded ordinary session pools, deadlines and cleanup; stricter command completion checks; exact upstream MIME and Flagged receipts; LIST/sentinel separation checks; resource samples after socket cleanup; dense stress UIDs with the original #613 gap fixture preserved. No production Rust code changed.

Atomic commits: b5c6fb625 (required-command failures and receipt budget), c472265c3 (stress UID range), 94904b828 (coordinator and receipt audit). Fetched and merged origin/dev once at 2014caf76; no pushes or deploys.

Files: tests/adversarial/mail_proxy.py, mail_stress.mjs, mail_sync_provider.py, test_mail_proxy.py, test_mail_sync_provider.py and mail-sync.md.

Scenario Result Evidence
Sustained load Incomplete Earlier three-session smoke passed five-second phases on one and three Users; no 30-minute run
Large/edge mailboxes Fixture regression passed; live acceptance incomplete 50 MiB, empty message, duplicate/missing Message-ID and 80-level hierarchy files checked; first sync attempt did not complete in 180 s
Concurrency correctness Not run No move/move, move/delete, expunge/fetch, selected-folder rename or IDLE/delete result
Upstream faults Not run No fault/restart/UIDVALIDITY acceptance result
Client input robustness Not run Unfinished malformed-input probe not retained; ordinary standard-client commands only
Unicode names Not run No Unicode namespace fixture added
Restart safety Incomplete Reconciliation restart intermittently left one ordinary Inbox at zero after 180 s; no transfer crash check
Account separation Earlier short smoke passed Three sentinel searches and account-label LIST checks; full authorization matrix deferred

Final focused live run: FAIL at initial-sync prerequisite. Corrected dense UIDs, Inbox sizes 64/32/32, no Boundary folder, no deep hierarchy. Status polls returned 200, completed_accounts=2, inbox_counts=[64,0,32], accounts_with_errors=0 through 180 seconds. Evidence and the earlier sparse-UID timing caveat are posted to #1037. No invalid-response text or server crash was established. Provider-sync code was not changed here. The strengthened MIME/Flagged receipt phase was not reached in this final run.

Earlier dense smoke: 12 read/IDLE cycles on one User and 13 across three Users; one accepted draft count receipt per phase. COPY/MOVE returned NO, so successful transfers are unproved. All three web sessions and readiness returned 200 after both phases. RSS 145788→182992 KiB and FDs 72→65 in the one-User phase; RSS 183200→265520 KiB and FDs 66→82 across three Users. These short samples do not prove bounded growth or a performance pass. The runtime binary was the pre-existing local-provider build in this worktree, not a fresh build of merged origin/dev.

Gates: cargo fmt --check exited 0 with no output. node --check tests/adversarial/mail_stress.mjs exited 0 with no output. No own Rust crates or web product code changed, so crate clippy/tests and bun product gates were not run. Focused Python gate output, verbatim:

test_refused_read_does_not_pass (test_mail_proxy.StressHarnessTests.test_refused_read_does_not_pass)
A healthy transport is insufficient when SEARCH fails (#1038). ... ok
test_rejects_header_control_in_run_identity (test_mail_proxy.StressHarnessTests.test_rejects_header_control_in_run_identity)
Generated Message-IDs must not introduce header lines (#1038). ... ok
test_rejects_remote_or_ambiguous_hosts (test_mail_proxy.StressHarnessTests.test_rejects_remote_or_ambiguous_hosts)
Do not permit this local load tool to target a remote service. ... ok
test_rejects_unbounded_or_fractional_pool (test_mail_proxy.StressHarnessTests.test_rejects_unbounded_or_fractional_pool)
Reject unbounded budgets and invalid pool sizes before sampling. ... ok
test_transport_failure_is_sanitized_and_fails (test_mail_proxy.StressHarnessTests.test_transport_failure_is_sanitized_and_fails)
Never publish an imaplib error string or call a broken run PASS. ... ok
test_default_gaps_and_refuse_existing_directory (test_mail_sync_provider.ProviderFixtureTests.test_default_gaps_and_refuse_existing_directory)
Keep the deployed #613 fixture contract when stress is not requested. ... ok
test_stress_boundaries_and_distinct_users (test_mail_sync_provider.ProviderFixtureTests.test_stress_boundaries_and_distinct_users)
Generate a small namespace but keep real large/empty boundary items. ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.507s

OK
Prepared fixture: 2000 messages, TLS port 29963, stress=False
Prepared fixture: 10 messages, TLS port 29963, stress=True

Cleanup: cargo clean output, verbatim:

     Removed 7238 files, 4.6GiB total

Web build output removed. No owned Dovecot container remained. Worktree clean. Doc comments in all touched files were reread.

Decisions: stress UIDs are dense; #613 keeps gaps. The focused smoke excludes boundary/deep data; the merge-round fixture keeps it. Ordinary load is sized to available RAM, capped at 12 sessions and a private 24-connection listener. Required commands require OK; COPY/MOVE may return clean NO, while BAD always fails. Credentials stay on child stdin and evidence contains only counts, types and resources. These are harness choices, not new product decisions.

Known gaps: intermittent initial sync; final exact-MIME/Flagged check not exercised live; successful COPY/MOVE unproved; long resource bounds, full boundary acceptance, race/fault/Unicode/transfer-restart scenarios unverified. UI/UX gaps: not applicable; no UI feature changed.

For the merge round: build its combined server with mail-test-provider and its production web app, then run the long ordinary reliability coordinator:

CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \
CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \
TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs

It must prove both 30-minute phases, accepted-write receipts, web availability and bounded resources. This coordinator does not implement the omitted fault/race scenarios. Use a fresh fixture and the CA/port setup in tests/adversarial/mail-sync.md for the existing transfer regression:

cargo test -p calternal-plugin-mail --features test-provider \
  real_tls_copy_delete_preserves_receipt_and_replays_once -- --ignored --nocapture

It must prove successful transfers and deletion with exact receipts. Combined server gates and full authorization matrices remain merge-round checks. #1038 is left open.

Continuation report — branch `job/mailstress-1038`, base `86f2e061e`, head `94904b828b29ae72e3286ed627d403164bb86681`. READY FOR MERGE: no. Built: an isolated TLS fixture coordinator that reuses real passkey/API/server helpers; bounded ordinary session pools, deadlines and cleanup; stricter command completion checks; exact upstream MIME and Flagged receipts; LIST/sentinel separation checks; resource samples after socket cleanup; dense stress UIDs with the original #613 gap fixture preserved. No production Rust code changed. Atomic commits: `b5c6fb625` (required-command failures and receipt budget), `c472265c3` (stress UID range), `94904b828` (coordinator and receipt audit). Fetched and merged origin/dev once at `2014caf76`; no pushes or deploys. Files: tests/adversarial/mail_proxy.py, mail_stress.mjs, mail_sync_provider.py, test_mail_proxy.py, test_mail_sync_provider.py and mail-sync.md. | Scenario | Result | Evidence | |---|---|---| | Sustained load | Incomplete | Earlier three-session smoke passed five-second phases on one and three Users; no 30-minute run | | Large/edge mailboxes | Fixture regression passed; live acceptance incomplete | 50 MiB, empty message, duplicate/missing Message-ID and 80-level hierarchy files checked; first sync attempt did not complete in 180 s | | Concurrency correctness | Not run | No move/move, move/delete, expunge/fetch, selected-folder rename or IDLE/delete result | | Upstream faults | Not run | No fault/restart/UIDVALIDITY acceptance result | | Client input robustness | Not run | Unfinished malformed-input probe not retained; ordinary standard-client commands only | | Unicode names | Not run | No Unicode namespace fixture added | | Restart safety | Incomplete | Reconciliation restart intermittently left one ordinary Inbox at zero after 180 s; no transfer crash check | | Account separation | Earlier short smoke passed | Three sentinel searches and account-label LIST checks; full authorization matrix deferred | Final focused live run: FAIL at initial-sync prerequisite. Corrected dense UIDs, Inbox sizes 64/32/32, no Boundary folder, no deep hierarchy. Status polls returned 200, completed_accounts=2, inbox_counts=[64,0,32], accounts_with_errors=0 through 180 seconds. Evidence and the earlier sparse-UID timing caveat are posted to #1037. No invalid-response text or server crash was established. Provider-sync code was not changed here. The strengthened MIME/Flagged receipt phase was not reached in this final run. Earlier dense smoke: 12 read/IDLE cycles on one User and 13 across three Users; one accepted draft count receipt per phase. COPY/MOVE returned NO, so successful transfers are unproved. All three web sessions and readiness returned 200 after both phases. RSS 145788→182992 KiB and FDs 72→65 in the one-User phase; RSS 183200→265520 KiB and FDs 66→82 across three Users. These short samples do not prove bounded growth or a performance pass. The runtime binary was the pre-existing local-provider build in this worktree, not a fresh build of merged origin/dev. Gates: cargo fmt --check exited 0 with no output. node --check tests/adversarial/mail_stress.mjs exited 0 with no output. No own Rust crates or web product code changed, so crate clippy/tests and bun product gates were not run. Focused Python gate output, verbatim: ```text test_refused_read_does_not_pass (test_mail_proxy.StressHarnessTests.test_refused_read_does_not_pass) A healthy transport is insufficient when SEARCH fails (#1038). ... ok test_rejects_header_control_in_run_identity (test_mail_proxy.StressHarnessTests.test_rejects_header_control_in_run_identity) Generated Message-IDs must not introduce header lines (#1038). ... ok test_rejects_remote_or_ambiguous_hosts (test_mail_proxy.StressHarnessTests.test_rejects_remote_or_ambiguous_hosts) Do not permit this local load tool to target a remote service. ... ok test_rejects_unbounded_or_fractional_pool (test_mail_proxy.StressHarnessTests.test_rejects_unbounded_or_fractional_pool) Reject unbounded budgets and invalid pool sizes before sampling. ... ok test_transport_failure_is_sanitized_and_fails (test_mail_proxy.StressHarnessTests.test_transport_failure_is_sanitized_and_fails) Never publish an imaplib error string or call a broken run PASS. ... ok test_default_gaps_and_refuse_existing_directory (test_mail_sync_provider.ProviderFixtureTests.test_default_gaps_and_refuse_existing_directory) Keep the deployed #613 fixture contract when stress is not requested. ... ok test_stress_boundaries_and_distinct_users (test_mail_sync_provider.ProviderFixtureTests.test_stress_boundaries_and_distinct_users) Generate a small namespace but keep real large/empty boundary items. ... ok ---------------------------------------------------------------------- Ran 7 tests in 0.507s OK Prepared fixture: 2000 messages, TLS port 29963, stress=False Prepared fixture: 10 messages, TLS port 29963, stress=True ``` Cleanup: cargo clean output, verbatim: ```text Removed 7238 files, 4.6GiB total ``` Web build output removed. No owned Dovecot container remained. Worktree clean. Doc comments in all touched files were reread. Decisions: stress UIDs are dense; #613 keeps gaps. The focused smoke excludes boundary/deep data; the merge-round fixture keeps it. Ordinary load is sized to available RAM, capped at 12 sessions and a private 24-connection listener. Required commands require OK; COPY/MOVE may return clean NO, while BAD always fails. Credentials stay on child stdin and evidence contains only counts, types and resources. These are harness choices, not new product decisions. Known gaps: intermittent initial sync; final exact-MIME/Flagged check not exercised live; successful COPY/MOVE unproved; long resource bounds, full boundary acceptance, race/fault/Unicode/transfer-restart scenarios unverified. UI/UX gaps: not applicable; no UI feature changed. For the merge round: build its combined server with mail-test-provider and its production web app, then run the long ordinary reliability coordinator: ```sh CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \ CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \ TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs ``` It must prove both 30-minute phases, accepted-write receipts, web availability and bounded resources. This coordinator does not implement the omitted fault/race scenarios. Use a fresh fixture and the CA/port setup in tests/adversarial/mail-sync.md for the existing transfer regression: ```sh cargo test -p calternal-plugin-mail --features test-provider \ real_tls_copy_delete_preserves_receipt_and_replays_once -- --ignored --nocapture ``` It must prove successful transfers and deletion with exact receipts. Combined server gates and full authorization matrices remain merge-round checks. #1038 is left open.
kayg commented 2026-10-04 07:51:45 +00:00
Author
Owner
Copy link

Reliability continuation started on job/mailstress-1038 at 94904b828b. First trace the silent initial-sync stall and the refused COPY/MOVE smoke. The local job/syncerr-1037 ref has no commits beyond job/mailproxy-486. Current verification policy assigns long soaks and full adversarial matrices to the merge round; focused regressions will run here. No pushes or deploys.

Reliability continuation started on job/mailstress-1038 at 94904b828b29ae72e3286ed627d403164bb86681. First trace the silent initial-sync stall and the refused COPY/MOVE smoke. The local job/syncerr-1037 ref has no commits beyond job/mailproxy-486. Current verification policy assigns long soaks and full adversarial matrices to the merge round; focused regressions will run here. No pushes or deploys.
kayg commented 2026-10-04 07:59:35 +00:00
Author
Owner
Copy link

#1038 continuation: deterministic restart recovery defect found in calternal-db's Worker::run, not the provider parser. Worker::run calls recover_expired_leases only once at startup. The server lease is 120 s. A process that restarts before an interrupted Mail sync lease expires leaves that job leased forever: after startup there is no lease recovery upkeep. Two other accounts can sync and enter IDLE while the third never starts, so no provider error is recorded.

A fresh deterministic regression queues three mail.sync jobs, leases one to an interrupted worker for 120 s, starts a replacement worker, observes the two fresh jobs complete, then expires the interrupted lease. The third job never completes. Verbatim result:

Mail sync lease expired after startup but was never recovered
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.12s
error: test failed, to rerun pass `-p calternal-db --test mailstress_restart`

The fresh local-provider live run also starts with completed_accounts=2, inbox_counts=[64,32,0], accounts_with_errors=0. Queue evidence at the deadline will follow. This is separate from #1037's invalid-response/download-timeout defect. No provider sync functions were changed. The local job/syncerr-1037 ref still has no commits beyond job/mailproxy-486. Proposed fix: periodic shared worker lease recovery. The job's cross-crate behavioral-change rule requires scope confirmation; that question is pending. No production fix has been applied yet.

#1038 continuation: deterministic restart recovery defect found in calternal-db's Worker::run, not the provider parser. Worker::run calls recover_expired_leases only once at startup. The server lease is 120 s. A process that restarts before an interrupted Mail sync lease expires leaves that job leased forever: after startup there is no lease recovery upkeep. Two other accounts can sync and enter IDLE while the third never starts, so no provider error is recorded. A fresh deterministic regression queues three mail.sync jobs, leases one to an interrupted worker for 120 s, starts a replacement worker, observes the two fresh jobs complete, then expires the interrupted lease. The third job never completes. Verbatim result: ```text Mail sync lease expired after startup but was never recovered test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.12s error: test failed, to rerun pass `-p calternal-db --test mailstress_restart` ``` The fresh local-provider live run also starts with completed_accounts=2, inbox_counts=[64,32,0], accounts_with_errors=0. Queue evidence at the deadline will follow. This is separate from #1037's invalid-response/download-timeout defect. No provider sync functions were changed. The local job/syncerr-1037 ref still has no commits beyond job/mailproxy-486. Proposed fix: periodic shared worker lease recovery. The job's cross-crate behavioral-change rule requires scope confirmation; that question is pending. No production fix has been applied yet.
kayg commented 2026-10-04 08:02:01 +00:00
Author
Owner
Copy link

Fresh server built from this branch; isolated generated Dovecot fixture. Initial sync stalled through 180 s with completed_accounts=2, inbox_counts=[64,32,0], accounts_with_errors=0. At the deadline all three mail.sync jobs were still leased and ALL THREE leases were expired:

{"scenario":"initial-sync","result":"FAIL","accounts_with_errors":0,"sync_error_owner":null}
{"scenario": "sync-queue-at-deadline", "states": [{"state": "leased", "count": 3, "expired_leases": 3}]}

This matches the deterministic regression and confirms the production failure mechanism: restart occurs while leases remain live; startup recovery finds none expired; all interrupted jobs stay leased after their deadlines. Accounts that had already committed their backfill look complete; the uncommitted account remains empty with no error because no new provider attempt runs. No provider parser or scheduling functions were changed. A single diagnostic restart after expiry is being used to isolate the COPY/MOVE receipt check. It will not count as automatic recovery.

Fresh server built from this branch; isolated generated Dovecot fixture. Initial sync stalled through 180 s with completed_accounts=2, inbox_counts=[64,32,0], accounts_with_errors=0. At the deadline all three mail.sync jobs were still leased and ALL THREE leases were expired: ```json {"scenario":"initial-sync","result":"FAIL","accounts_with_errors":0,"sync_error_owner":null} {"scenario": "sync-queue-at-deadline", "states": [{"state": "leased", "count": 3, "expired_leases": 3}]} ``` This matches the deterministic regression and confirms the production failure mechanism: restart occurs while leases remain live; startup recovery finds none expired; all interrupted jobs stay leased after their deadlines. Accounts that had already committed their backfill look complete; the uncommitted account remains empty with no error because no new provider attempt runs. No provider parser or scheduling functions were changed. A single diagnostic restart after expiry is being used to isolate the COPY/MOVE receipt check. It will not count as automatic recovery.
kayg commented 2026-10-04 08:08:11 +00:00
Author
Owner
Copy link

Reliability continuation — branch job/mailstress-1038, base 94904b828b29ae72e3286ed627d403164bb86681, head cae707d27234b7111a35c8f70dfa928acc308433.

READY FOR MERGE: no.

Built and fixed: the transfer fixture now requires successful COPY and MOVE after pending delivery. It retries only tagged NO within one shared deadline. It never repeats accepted or ambiguous writes. Exact MIME, Flagged state and source/destination counts are checked upstream AND through the proxy. Added aggregate queue evidence and one diagnostic restart which preserves the initial failure result.

Commits: a24a2365f (transfer sequencing and regression tests), cae707d27 (local receipts, expired-lease evidence and results). Files: tests/adversarial/mail_proxy.py, test_mail_proxy.py, mail_stress.mjs and mail-sync.md. Fetched and merged origin/dev once: Already up to date. No pushes or deploys.

A: ROOT CAUSE CONFIRMED, NOT FIXED. Worker::run in calternal-db calls recover_expired_leases only at startup. The server lease lasts 120 s. Restart before lease expiry leaves interrupted jobs leased indefinitely. A deterministic three-job regression failed after two fresh jobs completed and the interrupted lease expired after startup. Fresh live runs stayed at completed_accounts=2, inbox_counts=[64,32,0], accounts_with_errors=0 for 180 s. The deadline query found ALL THREE mail.sync leases expired. A diagnostic restart after expiry completed sync. This establishes manual recovery, not self recovery. Evidence is on #1037 and #1038.

The job instruction says: “Stop only when the change would alter another crate's behavior or design.” A shared Jobs recovery fix changes calternal-db behavior. The scope question remains unanswered; no production fix was applied. The deterministic source is preserved as artifacts/mailstress-1038/mailstress_restart.rs, and the proposed fix as lease-recovery.patch. The temporary failing test was removed from the crate; no failing regression is committed or hidden with ignore. The proposed patch is UNAPPLIED and UNVERIFIED. To use the regression with that patch, place it at crates/calternal-db/tests/mailstress_restart.rs and run cargo test -p calternal-db --test mailstress_restart.

#1037 overlap: none. No provider sync functions changed. Both local and fetched origin/job/syncerr-1037 refs have no commits beyond job/mailproxy-486. A is a shared Jobs lease-recovery defect, separate from provider parser errors and download timeouts.

B: FIXTURE ERROR FIXED AND LIVE RECEIPTS PASSED. APPEND creates a pending draft without a provider UID. Immediate COPY is refused. MOVE to a different destination also conflicts with a pending COPY (DESIGN §53). Waiting for explicit acceptance fixed the sequence without changing the proxy. Each focused phase accepted exactly one COPY and one MOVE. Exact MIME, Flagged state and folder counts matched upstream and locally: Drafts=0, Archive=1, Trash=1. The legal NO waits were counted: COPY 10 per phase; MOVE 11 and 9. Seven Python regressions passed, including acceptance without replay and failure on BAD/ambiguity/expired budget.

Scenario Result Evidence
1. Sustained load Short phases PASS; 30-minute phases deferred Three sessions; five-second command windows; 9 cycles per phase; 9.75 s and 11.21 s including delivery. All three web sessions and readiness returned 200. One-account RSS 138580→209832→210492 KiB, FDs 77→97→93. Three-account RSS 210556→248808→248736 KiB, FDs 94→96→93. No long-run bounded-growth claim.
2. Concurrency correctness Sequential transfer receipts PASS; race matrix not run Exact upstream and proxy COPY/MOVE receipts; simultaneous move/move, move/delete, fetch/expunge and folder lifecycle remain unverified.
3. Upstream faults Not run Restart/drop/slow/UIDVALIDITY matrix belongs to the merge round.
4. Client input robustness Ordinary commands PASS; input matrix not run SELECT/LIST/SEARCH/FETCH/IDLE/APPEND/STORE/COPY/MOVE/UID EXPUNGE completed; malformed/input/auth-flood cases unverified.
5. Unicode folder names Not run No RTL, combining, emoji, normalization or case acceptance claim.
6. Restart safety FAIL: expired leases strand sync Deterministic regression plus three expired live leases at 180 s; one manual restart recovers. Hard-kill during transfers remains unverified.
7. Account separation Short smoke PASS; full matrix deferred Three User sentinels and folder labels checked in six client sessions. Writer receipts are for User 0 only; not a full mutation/account matrix.

Verification: fresh production web build and cargo build -p calternal-server --features mail-test-provider succeeded. No production Rust or web source changed; crate clippy/full tests and bun product checks were not applicable. cargo fmt --check and node --check tests/adversarial/mail_stress.mjs exited 0 with no output. Focused Python gate output, verbatim:

test_refused_read_does_not_pass (test_mail_proxy.StressHarnessTests.test_refused_read_does_not_pass)
A healthy transport is insufficient when SEARCH fails (#1038). ... ok
test_rejects_header_control_in_run_identity (test_mail_proxy.StressHarnessTests.test_rejects_header_control_in_run_identity)
Generated Message-IDs must not introduce header lines (#1038). ... ok
test_rejects_remote_or_ambiguous_hosts (test_mail_proxy.StressHarnessTests.test_rejects_remote_or_ambiguous_hosts)
Do not permit this local load tool to target a remote service. ... ok
test_rejects_unbounded_or_fractional_pool (test_mail_proxy.StressHarnessTests.test_rejects_unbounded_or_fractional_pool)
Reject unbounded budgets and invalid pool sizes before sampling. ... ok
test_transfer_refusal_or_ambiguity_cannot_pass (test_mail_proxy.StressHarnessTests.test_transfer_refusal_or_ambiguity_cannot_pass)
BAD, a missing completion and disconnects never repeat writes (#1038). ... ok
test_transfer_waits_for_delivery_without_replaying_acceptance (test_mail_proxy.StressHarnessTests.test_transfer_waits_for_delivery_without_replaying_acceptance)
Pending drafts and COPY receipts must resolve before success (#1038). ... ok
test_transport_failure_is_sanitized_and_fails (test_mail_proxy.StressHarnessTests.test_transport_failure_is_sanitized_and_fails)
Never publish an imaplib error string or call a broken run PASS. ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.013s

OK

Provider fixture checks, verbatim:

test_default_gaps_and_refuse_existing_directory (test_mail_sync_provider.ProviderFixtureTests.test_default_gaps_and_refuse_existing_directory)
Keep the deployed #613 fixture contract when stress is not requested. ... ok
test_stress_boundaries_and_distinct_users (test_mail_sync_provider.ProviderFixtureTests.test_stress_boundaries_and_distinct_users)
Generate a small namespace but keep real large/empty boundary items. ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.397s

OK
Prepared fixture: 2000 messages, TLS port 29963, stress=False
Prepared fixture: 10 messages, TLS port 29963, stress=True

The deterministic diagnostic is evidence of the unfixed defect, NOT a passing gate. Output, verbatim:

   Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1.93s
     Running tests/mailstress_restart.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/mailstress_restart-4fbc9237aad16ee2)

running 1 test
test restart_before_mail_lease_expiry_does_not_strand_an_account ... FAILED

failures:

---- restart_before_mail_lease_expiry_does_not_strand_an_account stdout ----

thread 'restart_before_mail_lease_expiry_does_not_strand_an_account' (792686) panicked at crates/calternal-db/tests/mailstress_restart.rs:96:5:
Mail sync lease expired after startup but was never recovered
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    restart_before_mail_lease_expiry_does_not_strand_an_account

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.10s

error: test failed, to rerun pass `-p calternal-db --test mailstress_restart`

The initial live smoke exited 1 at the sync prerequisite. The diagnostic smoke also exited 1, correctly retaining initial-sync FAIL despite manual-recovery PASS and both transfer phases PASS. Safe count-only evidence is in artifacts/mailstress-1038/smoke.log and diagnostic-smoke.log.

Known gaps: A still needs a shared Jobs fix and a passing regression; long resource bounds and the requested race/fault/input/Unicode/transfer-kill/full-separation matrices remain unverified. No new harmless oddities were found. UX gaps closed/left: not applicable; no UI code changed. Doc comments in all changed files were reread. No dependencies or migrations changed.

Decisions: use one diagnostic restart after expiry to isolate B, while retaining overall failure. Use one bounded deadline for tagged-NO delivery waits; never repeat a possibly accepted write. These are fixture choices; no new product behavior was selected.

For the merge round, after fixing A and building its combined local-provider server and production web app:

CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \
CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \
TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs

Must prove both 30-minute phases, successful exact transfers, web availability and bounded resources. The current large fixture exceeds the provider's 512-folder discovery bound; 10,000-folder acceptance is not established. The coordinator does not implement the omitted matrices; those still need focused scenarios before complete QA can be claimed.

With a fresh fixture and private CA/port environment from tests/adversarial/mail-sync.md:

cargo test -p calternal-plugin-mail --features test-provider \
  real_tls_copy_delete_preserves_receipt_and_replays_once -- --ignored --nocapture

Must prove successful COPY/MOVE/delete receipts and durable replay. The combined branch must run its full Rust/web suites and tests/adversarial/run.sh for its existing authorization/robustness matrices; those do not substitute for the missing mail-specific fault/race cases.

Cleanup output, verbatim:

     Removed 8024 files, 5.1GiB total

Web build output removed. No owned Dovecot container remains. Worktree clean. Ignored evidence and proposed fix retained locally. #1038 stays open.

Reliability continuation — branch `job/mailstress-1038`, base `94904b828b29ae72e3286ed627d403164bb86681`, head `cae707d27234b7111a35c8f70dfa928acc308433`. READY FOR MERGE: no. Built and fixed: the transfer fixture now requires successful COPY and MOVE after pending delivery. It retries only tagged NO within one shared deadline. It never repeats accepted or ambiguous writes. Exact MIME, Flagged state and source/destination counts are checked upstream AND through the proxy. Added aggregate queue evidence and one diagnostic restart which preserves the initial failure result. Commits: `a24a2365f` (transfer sequencing and regression tests), `cae707d27` (local receipts, expired-lease evidence and results). Files: tests/adversarial/mail_proxy.py, test_mail_proxy.py, mail_stress.mjs and mail-sync.md. Fetched and merged origin/dev once: Already up to date. No pushes or deploys. A: ROOT CAUSE CONFIRMED, NOT FIXED. Worker::run in calternal-db calls recover_expired_leases only at startup. The server lease lasts 120 s. Restart before lease expiry leaves interrupted jobs leased indefinitely. A deterministic three-job regression failed after two fresh jobs completed and the interrupted lease expired after startup. Fresh live runs stayed at completed_accounts=2, inbox_counts=[64,32,0], accounts_with_errors=0 for 180 s. The deadline query found ALL THREE mail.sync leases expired. A diagnostic restart after expiry completed sync. This establishes manual recovery, not self recovery. Evidence is on #1037 and #1038. The job instruction says: “Stop only when the change would alter another crate's behavior or design.” A shared Jobs recovery fix changes calternal-db behavior. The scope question remains unanswered; no production fix was applied. The deterministic source is preserved as artifacts/mailstress-1038/mailstress_restart.rs, and the proposed fix as lease-recovery.patch. The temporary failing test was removed from the crate; no failing regression is committed or hidden with ignore. The proposed patch is UNAPPLIED and UNVERIFIED. To use the regression with that patch, place it at crates/calternal-db/tests/mailstress_restart.rs and run cargo test -p calternal-db --test mailstress_restart. #1037 overlap: none. No provider sync functions changed. Both local and fetched origin/job/syncerr-1037 refs have no commits beyond job/mailproxy-486. A is a shared Jobs lease-recovery defect, separate from provider parser errors and download timeouts. B: FIXTURE ERROR FIXED AND LIVE RECEIPTS PASSED. APPEND creates a pending draft without a provider UID. Immediate COPY is refused. MOVE to a different destination also conflicts with a pending COPY (DESIGN §53). Waiting for explicit acceptance fixed the sequence without changing the proxy. Each focused phase accepted exactly one COPY and one MOVE. Exact MIME, Flagged state and folder counts matched upstream and locally: Drafts=0, Archive=1, Trash=1. The legal NO waits were counted: COPY 10 per phase; MOVE 11 and 9. Seven Python regressions passed, including acceptance without replay and failure on BAD/ambiguity/expired budget. | Scenario | Result | Evidence | |---|---|---| | 1. Sustained load | Short phases PASS; 30-minute phases deferred | Three sessions; five-second command windows; 9 cycles per phase; 9.75 s and 11.21 s including delivery. All three web sessions and readiness returned 200. One-account RSS 138580→209832→210492 KiB, FDs 77→97→93. Three-account RSS 210556→248808→248736 KiB, FDs 94→96→93. No long-run bounded-growth claim. | | 2. Concurrency correctness | Sequential transfer receipts PASS; race matrix not run | Exact upstream and proxy COPY/MOVE receipts; simultaneous move/move, move/delete, fetch/expunge and folder lifecycle remain unverified. | | 3. Upstream faults | Not run | Restart/drop/slow/UIDVALIDITY matrix belongs to the merge round. | | 4. Client input robustness | Ordinary commands PASS; input matrix not run | SELECT/LIST/SEARCH/FETCH/IDLE/APPEND/STORE/COPY/MOVE/UID EXPUNGE completed; malformed/input/auth-flood cases unverified. | | 5. Unicode folder names | Not run | No RTL, combining, emoji, normalization or case acceptance claim. | | 6. Restart safety | FAIL: expired leases strand sync | Deterministic regression plus three expired live leases at 180 s; one manual restart recovers. Hard-kill during transfers remains unverified. | | 7. Account separation | Short smoke PASS; full matrix deferred | Three User sentinels and folder labels checked in six client sessions. Writer receipts are for User 0 only; not a full mutation/account matrix. | Verification: fresh production web build and cargo build -p calternal-server --features mail-test-provider succeeded. No production Rust or web source changed; crate clippy/full tests and bun product checks were not applicable. cargo fmt --check and node --check tests/adversarial/mail_stress.mjs exited 0 with no output. Focused Python gate output, verbatim: ```text test_refused_read_does_not_pass (test_mail_proxy.StressHarnessTests.test_refused_read_does_not_pass) A healthy transport is insufficient when SEARCH fails (#1038). ... ok test_rejects_header_control_in_run_identity (test_mail_proxy.StressHarnessTests.test_rejects_header_control_in_run_identity) Generated Message-IDs must not introduce header lines (#1038). ... ok test_rejects_remote_or_ambiguous_hosts (test_mail_proxy.StressHarnessTests.test_rejects_remote_or_ambiguous_hosts) Do not permit this local load tool to target a remote service. ... ok test_rejects_unbounded_or_fractional_pool (test_mail_proxy.StressHarnessTests.test_rejects_unbounded_or_fractional_pool) Reject unbounded budgets and invalid pool sizes before sampling. ... ok test_transfer_refusal_or_ambiguity_cannot_pass (test_mail_proxy.StressHarnessTests.test_transfer_refusal_or_ambiguity_cannot_pass) BAD, a missing completion and disconnects never repeat writes (#1038). ... ok test_transfer_waits_for_delivery_without_replaying_acceptance (test_mail_proxy.StressHarnessTests.test_transfer_waits_for_delivery_without_replaying_acceptance) Pending drafts and COPY receipts must resolve before success (#1038). ... ok test_transport_failure_is_sanitized_and_fails (test_mail_proxy.StressHarnessTests.test_transport_failure_is_sanitized_and_fails) Never publish an imaplib error string or call a broken run PASS. ... ok ---------------------------------------------------------------------- Ran 7 tests in 0.013s OK ``` Provider fixture checks, verbatim: ```text test_default_gaps_and_refuse_existing_directory (test_mail_sync_provider.ProviderFixtureTests.test_default_gaps_and_refuse_existing_directory) Keep the deployed #613 fixture contract when stress is not requested. ... ok test_stress_boundaries_and_distinct_users (test_mail_sync_provider.ProviderFixtureTests.test_stress_boundaries_and_distinct_users) Generate a small namespace but keep real large/empty boundary items. ... ok ---------------------------------------------------------------------- Ran 2 tests in 0.397s OK Prepared fixture: 2000 messages, TLS port 29963, stress=False Prepared fixture: 10 messages, TLS port 29963, stress=True ``` The deterministic diagnostic is evidence of the unfixed defect, NOT a passing gate. Output, verbatim: ```text Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db) Finished `test` profile [unoptimized + debuginfo] target(s) in 1.93s Running tests/mailstress_restart.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/mailstress_restart-4fbc9237aad16ee2) running 1 test test restart_before_mail_lease_expiry_does_not_strand_an_account ... FAILED failures: ---- restart_before_mail_lease_expiry_does_not_strand_an_account stdout ---- thread 'restart_before_mail_lease_expiry_does_not_strand_an_account' (792686) panicked at crates/calternal-db/tests/mailstress_restart.rs:96:5: Mail sync lease expired after startup but was never recovered note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: restart_before_mail_lease_expiry_does_not_strand_an_account test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.10s error: test failed, to rerun pass `-p calternal-db --test mailstress_restart` ``` The initial live smoke exited 1 at the sync prerequisite. The diagnostic smoke also exited 1, correctly retaining initial-sync FAIL despite manual-recovery PASS and both transfer phases PASS. Safe count-only evidence is in artifacts/mailstress-1038/smoke.log and diagnostic-smoke.log. Known gaps: A still needs a shared Jobs fix and a passing regression; long resource bounds and the requested race/fault/input/Unicode/transfer-kill/full-separation matrices remain unverified. No new harmless oddities were found. UX gaps closed/left: not applicable; no UI code changed. Doc comments in all changed files were reread. No dependencies or migrations changed. Decisions: use one diagnostic restart after expiry to isolate B, while retaining overall failure. Use one bounded deadline for tagged-NO delivery waits; never repeat a possibly accepted write. These are fixture choices; no new product behavior was selected. For the merge round, after fixing A and building its combined local-provider server and production web app: ```sh CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \ CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \ TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs ``` Must prove both 30-minute phases, successful exact transfers, web availability and bounded resources. The current large fixture exceeds the provider's 512-folder discovery bound; 10,000-folder acceptance is not established. The coordinator does not implement the omitted matrices; those still need focused scenarios before complete QA can be claimed. With a fresh fixture and private CA/port environment from tests/adversarial/mail-sync.md: ```sh cargo test -p calternal-plugin-mail --features test-provider \ real_tls_copy_delete_preserves_receipt_and_replays_once -- --ignored --nocapture ``` Must prove successful COPY/MOVE/delete receipts and durable replay. The combined branch must run its full Rust/web suites and tests/adversarial/run.sh for its existing authorization/robustness matrices; those do not substitute for the missing mail-specific fault/race cases. Cleanup output, verbatim: ```text Removed 8024 files, 5.1GiB total ``` Web build output removed. No owned Dovecot container remains. Worktree clean. Ignored evidence and proposed fix retained locally. #1038 stays open.
kayg commented 2026-10-04 09:04:37 +00:00
Author
Owner
Copy link

Reliability continuation started on job/mailstress-1038 at 6c03031e3. Read the previous report and DESIGN §53. Verify the merged #1042 lease fix with the existing deterministic restart regression and fresh three-User live smoke. Keep the existing harness unchanged. Report only measured results; the race/fault/input/Unicode matrices are not implemented by the coordinator. No pushes or deploys.

Reliability continuation started on job/mailstress-1038 at 6c03031e3. Read the previous report and DESIGN §53. Verify the merged #1042 lease fix with the existing deterministic restart regression and fresh three-User live smoke. Keep the existing harness unchanged. Report only measured results; the race/fault/input/Unicode matrices are not implemented by the coordinator. No pushes or deploys.
kayg commented 2026-10-04 09:07:07 +00:00
Author
Owner
Copy link

Finding recheck: defect A passes the existing deterministic restart regression on merged #1042 code. Two fresh jobs complete before the interrupted lease is expired; the third then completes on attempt 2 without a further restart. Runtime: 0.65 s. Result committed as 7a58cdc8a in tests/adversarial/mail-sync.md. Live three-User smoke is waiting for the local-provider server build. No provider sync change.

Long-fixture limitation: the unchanged coordinator creates 10,000 Bulk folders, while crates/plugins/mail/src/sync.rs sets MAX_MAILBOXES=512. It has no race/fault/Unicode/transfer-kill scenario phases. These are not completed results.

Finding recheck: defect A passes the existing deterministic restart regression on merged #1042 code. Two fresh jobs complete before the interrupted lease is expired; the third then completes on attempt 2 without a further restart. Runtime: 0.65 s. Result committed as 7a58cdc8a in tests/adversarial/mail-sync.md. Live three-User smoke is waiting for the local-provider server build. No provider sync change. Long-fixture limitation: the unchanged coordinator creates 10,000 Bulk folders, while crates/plugins/mail/src/sync.rs sets MAX_MAILBOXES=512. It has no race/fault/Unicode/transfer-kill scenario phases. These are not completed results.
kayg commented 2026-10-04 09:25:03 +00:00
Author
Owner
Copy link

Reliability continuation results — branch job/mailstress-1038, base 6c03031e319c6ce2a00101acc3f9e8130bc5d07c, head 4f83aef2b3b973aaa4ca8c10ade0a841683a9ed0.

READY FOR MERGE: no.

Built: fresh production web app and local-provider server; reran the existing deterministic lease regression, crate gates, fixture tests and fresh three-User smoke. The harness and production source are unchanged. Commits: 7a58cdc8a (lease recheck evidence), 4f83aef2b (live recovery and scenario results). Owned tracked file: tests/adversarial/mail-sync.md. No dependencies or migrations changed. No pushes or deploys.

A fresh production web build and local-provider server build passed. The
unchanged smoke coordinator restarted the Instance before lease expiry.
Twelve progress samples showed one completed account and Inbox counts
[64,32,0]. The next sample showed three completed accounts and
[64,32,32]. Initial sync passed within the 180-second deadline. No manual
recovery restart ran. No account reported a sync error. This confirms defect
A is fixed for this fixture. It does not check a hard kill during a transfer.

Scenario Result Evidence
0. Expired lease recovery PASS Deterministic restart regression and fresh three-User initial sync both passed without a manual restart
6. Hard kill during transfers Not run The unchanged coordinator has no transfer-kill phase
2. Concurrency correctness Partial Sequential exact COPY/MOVE receipts passed; no simultaneous mutation or folder lifecycle matrix ran
3. Upstream faults Not run The coordinator has no restart/drop/slow/UIDVALIDITY phase
7. Account separation Partial Three folder-label and message-sentinel checks passed in each short phase; transfer writes cover User 0 only
1. Sustained load Partial Two five-second command windows passed with three sessions; no 30-minute run or bounded-growth claim
4. Client input robustness Partial Ordinary SELECT/LIST/SEARCH/FETCH/IDLE/APPEND/STORE/COPY/MOVE/UID EXPUNGE passed; no malformed-input or failed-login matrix ran
5. Unicode names Not run The coordinator has no Unicode namespace phase

Each short phase accepted one APPEND, one COPY and one MOVE. Exact MIME,
Flagged state and receipt counts matched upstream and locally: Drafts zero,
Archive one, Trash one. All three authenticated web sessions and readiness
returned 200 after each phase. These checks prove retained sessions, not a
fresh sign-in during load.

Phase Elapsed including receipts RSS samples (KiB) Descriptor samples
One User 12.71 s 223124, 228172, 228456 97, 106, 103
Three Users 14.86 s 228584, 267664, 267596 104, 107, 103

Both phases completed seven read/IDLE cycles with no recorded failure.
COPY had ten clean NO waits per phase. MOVE had fourteen and fifteen. Each
transfer then completed once and passed both receipt checks. This short
series does not prove long-term resource bounds or the five-second delivery
target. The smoke selected three sessions and reported 79 GiB free.

Verification: cargo fmt --check passed with no output;
cargo clippy -p calternal-db --all-targets -- -D warnings passed;
cargo test -p calternal-db passed 46 tests, with one manual benchmark
ignored. Seven harness regressions and two provider-fixture tests passed.
node --check tests/adversarial/mail_stress.mjs passed with no output.
git fetch origin and git merge origin/dev ran once before final gates;
the merge reported Already up to date. No source, dependency or migration
changed in this continuation.

For the merge round: full suites and the long-run profile remain required
under the 2026-10-02 verification policy. The documented long command above
cannot reach the soak phases with its current 10,000-folder fixture. The
missing scenario matrices also need focused runs. Do not treat this smoke
as those results.

READY FOR MERGE: no. Defect A is verified fixed. The requested reliability
matrix remains incomplete. No new product behavior was selected. The harness
was not changed. No UI changed; UX gaps are not applicable.

Known gaps: no hard-kill transfer acceptance, simultaneous mutation/folder lifecycle matrix, upstream fault matrix, 30-minute resource-bound evidence, malformed-input/failed-login matrix, Unicode namespace matrix, full account mutation matrix or fresh web sign-in during load. The bounded ordinary smoke did not uncover a new production defect. No invalid-response/300-second-timeout evidence was seen by its sync status checks; no new #1037 diagnosis is claimed.

Decisions: retain the existing harness as instructed; do not label smoke as sustained load. Use the job's 2026-10-02 verification policy to defer long runs and full matrices to the merge round. The unchanged long fixture exceeds the existing folder discovery bound. No new product behavior was selected. UX gaps closed/left: not applicable; no UI source changed. Re-read the edited documentation before this report.

For the merge round:

CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \
CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \
TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs

This is the existing long-run command, not a passing result. Resolve its 10,000-folder versus 512-folder prerequisite before using it to prove two 30-minute phases, bounded resources and availability. It does not implement the missing scenario matrices; those need separate focused checks. Full combined gates and tests/adversarial/run.sh remain merge-round work.

Gate output below is verbatim. cargo fmt --check and node --check tests/adversarial/mail_stress.mjs exited 0 with no output. The live smoke exited 0; its count-only output is preserved locally in artifacts/mailstress-1038/round2-smoke.log. No screenshots were required: no UI changed.

Restart regression

   Compiling proc-macro2 v1.0.107
   Compiling quote v1.0.47
   Compiling unicode-ident v1.0.26
   Compiling libc v0.2.189
   Compiling typenum v1.20.1
   Compiling cfg-if v1.0.5
   Compiling stable_deref_trait v1.2.1
   Compiling getrandom v0.4.3
   Compiling smallvec v1.16.1
   Compiling version_check v0.9.5
   Compiling rand_core v0.10.1
   Compiling litemap v0.8.3
   Compiling writeable v0.6.4
   Compiling generic-array v0.14.9
   Compiling hybrid-array v0.4.15
   Compiling autocfg v1.5.1
   Compiling utf8_iter v1.0.4
   Compiling siphasher v1.0.3
   Compiling shlex v2.0.1
   Compiling syn v3.0.6
   Compiling find-msvc-tools v0.1.13
   Compiling icu_properties_data v2.3.0
   Compiling pin-project-lite v0.2.17
   Compiling icu_normalizer_data v2.3.0
   Compiling futures-core v0.3.34
   Compiling syn v2.0.119
   Compiling cc v1.4.7
   Compiling synstructure v0.14.0
   Compiling zerovec-derive v0.11.6
   Compiling displaydoc v0.2.7
   Compiling zerofrom-derive v0.1.8
   Compiling yoke-derive v0.8.3
   Compiling num-traits v0.2.19
   Compiling scopeguard v1.2.0
   Compiling zerofrom v0.1.8
   Compiling lock_api v0.4.14
   Compiling yoke v0.8.3
   Compiling once_cell v1.21.4
   Compiling futures-sink v0.3.34
   Compiling parking_lot_core v0.9.12
   Compiling serde_core v1.0.229
   Compiling zerovec v0.11.8
   Compiling zerotrie v0.2.5
   Compiling crypto-common v0.2.2
   Compiling equivalent v1.0.2
   Compiling memchr v2.8.3
   Compiling tinystr v0.8.4
   Compiling potential_utf v0.1.6
   Compiling crossbeam-utils v0.8.23
   Compiling icu_collections v2.3.0
   Compiling icu_locale_core v2.3.0
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling tokio-macros v2.7.2
   Compiling icu_provider v2.3.1
   Compiling socket2 v0.6.5
   Compiling mio v1.2.3
   Compiling thiserror v2.0.21
   Compiling bytes v1.12.1
   Compiling icu_properties v2.3.0
   Compiling icu_normalizer v2.3.0
   Compiling futures-task v0.3.34
   Compiling slab v0.4.12
   Compiling allocator-api2 v0.2.21
   Compiling idna_adapter v1.2.2
   Compiling futures-io v0.3.34
   Compiling rand_core v0.6.4
   Compiling pkg-config v0.3.34
   Compiling percent-encoding v2.3.2
   Compiling vcpkg v0.2.15
   Compiling serde v1.0.229
   Compiling foldhash v0.2.0
   Compiling rand v0.8.8
   Compiling form_urlencoded v1.2.2
   Compiling libsqlite3-sys v0.37.0
   Compiling hashbrown v0.16.1
   Compiling idna v1.1.0
   Compiling futures-util v0.3.34
   Compiling tokio v1.53.1
   Compiling digest v0.10.7
   Compiling parking_lot v0.12.5
   Compiling tracing-core v0.1.36
   Compiling tracing-attributes v0.1.31
   Compiling serde_derive v1.0.229
   Compiling thiserror-impl v2.0.21
   Compiling phf_shared v0.11.3
   Compiling inout v0.2.2
   Compiling cmov v0.5.4
   Compiling crc-catalog v2.5.0
   Compiling log v0.4.34
   Compiling hashbrown v0.17.1
   Compiling cpufeatures v0.2.17
   Compiling cpufeatures v0.3.1
   Compiling parking v2.2.1
   Compiling sha2 v0.10.9
   Compiling indexmap v2.14.2
   Compiling event-listener v5.4.2
   Compiling tracing v0.1.44
   Compiling crc v3.4.0
   Compiling ctutils v0.4.2
   Compiling phf_generator v0.11.3
   Compiling tokio-stream v0.1.19
   Compiling futures-intrusive v0.5.0
   Compiling crossbeam-queue v0.3.14
   Compiling url v2.5.8
   Compiling hashlink v0.11.1
   Compiling spin v0.9.9
   Compiling block-buffer v0.12.1
   Compiling iana-time-zone v0.1.65
   Compiling base64 v0.22.1
   Compiling zmij v1.0.23
   Compiling either v1.18.0
   Compiling chrono v0.4.45
   Compiling cipher v0.5.2
   Compiling sqlx-core v0.9.0
   Compiling flume v0.12.0
   Compiling phf_macros v0.11.3
   Compiling universal-hash v0.6.1
   Compiling futures-executor v0.3.34
   Compiling atoi v2.0.0
   Compiling futures-channel v0.3.34
   Compiling blake3 v1.8.7
   Compiling phf_shared v0.12.1
   Compiling serde_json v1.0.151
   Compiling chrono-tz v0.10.4
   Compiling rustix v1.1.5
   Compiling phf v0.12.1
   Compiling phf v0.11.3
   Compiling sqlx-sqlite v0.9.0
   Compiling poly1305 v0.9.1
   Compiling chacha20 v0.10.2
   Compiling aead v0.6.1
   Compiling constant_time_eq v0.4.2
   Compiling arrayvec v0.7.8
   Compiling itoa v1.0.18
   Compiling linux-raw-sys v0.12.1
   Compiling bitflags v2.13.2
   Compiling winnow v0.7.15
   Compiling cron v0.17.0
   Compiling chacha20poly1305 v0.11.0
   Compiling sqlx v0.9.0
   Compiling uuid v1.26.1
   Compiling fastrand v2.5.0
   Compiling tempfile v3.27.0
   Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 03s
     Running tests/mailstress_restart.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/mailstress_restart-4fbc9237aad16ee2)

running 1 test
test restart_before_mail_lease_expiry_does_not_strand_an_account ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.65s

calternal-db Clippy

    Blocking waiting for file lock on build directory
    Checking libc v0.2.189
    Checking cfg-if v1.0.5
    Checking typenum v1.20.1
    Checking zerofrom v0.1.8
    Checking stable_deref_trait v1.2.1
    Checking writeable v0.6.4
    Checking smallvec v1.16.1
    Checking yoke v0.8.3
    Checking rand_core v0.10.1
    Checking litemap v0.8.3
    Checking zerovec v0.11.8
    Checking zerotrie v0.2.5
    Checking getrandom v0.4.3
    Checking hybrid-array v0.4.15
    Checking futures-core v0.3.34
    Checking pin-project-lite v0.2.17
    Checking tinystr v0.8.4
    Checking potential_utf v0.1.6
    Checking utf8_iter v1.0.4
    Checking scopeguard v1.2.0
    Checking generic-array v0.14.9
    Checking icu_locale_core v2.3.0
    Checking icu_collections v2.3.0
    Checking lock_api v0.4.14
    Checking icu_properties_data v2.3.0
    Checking icu_normalizer_data v2.3.0
    Checking once_cell v1.21.4
    Checking futures-sink v0.3.34
    Checking num-traits v0.2.19
    Checking crypto-common v0.2.2
    Checking icu_provider v2.3.1
    Checking memchr v2.8.3
    Checking icu_normalizer v2.3.0
    Checking icu_properties v2.3.0
    Checking siphasher v1.0.3
    Checking equivalent v1.0.2
    Checking serde_core v1.0.229
    Checking parking_lot_core v0.9.12
    Checking block-buffer v0.10.4
    Checking crypto-common v0.1.6
    Checking socket2 v0.6.5
    Checking mio v1.2.3
    Checking idna_adapter v1.2.2
    Checking futures-io v0.3.34
    Checking slab v0.4.12
    Checking allocator-api2 v0.2.21
    Checking foldhash v0.2.0
    Checking percent-encoding v2.3.2
    Checking bytes v1.12.1
    Checking futures-task v0.3.34
    Checking hashbrown v0.16.1
    Checking form_urlencoded v1.2.2
    Checking futures-util v0.3.34
    Checking tokio v1.53.1
   Compiling phf_shared v0.11.3
    Checking idna v1.1.0
    Checking digest v0.10.7
    Checking parking_lot v0.12.5
    Checking crossbeam-utils v0.8.23
    Checking tracing-core v0.1.36
    Checking inout v0.2.2
    Checking hashbrown v0.17.1
    Checking parking v2.2.1
    Checking crc-catalog v2.5.0
    Checking cpufeatures v0.2.17
    Checking log v0.4.34
    Checking cmov v0.5.4
    Checking cpufeatures v0.3.1
    Checking tracing v0.1.44
    Checking ctutils v0.4.2
    Checking sha2 v0.10.9
    Checking crc v3.4.0
    Checking tokio-stream v0.1.19
    Checking indexmap v2.14.2
    Checking event-listener v5.4.2
    Checking serde v1.0.229
    Checking thiserror v2.0.21
    Checking crossbeam-queue v0.3.14
    Checking futures-intrusive v0.5.0
   Compiling phf_generator v0.11.3
    Checking url v2.5.8
    Checking hashlink v0.11.1
    Checking spin v0.9.9
    Checking block-buffer v0.12.1
    Checking either v1.18.0
    Checking iana-time-zone v0.1.65
    Checking base64 v0.22.1
    Checking flume v0.12.0
    Checking chrono v0.4.45
    Checking cipher v0.5.2
    Checking sqlx-core v0.9.0
    Checking libsqlite3-sys v0.37.0
   Compiling phf_macros v0.11.3
    Checking universal-hash v0.6.1
    Checking futures-executor v0.3.34
    Checking phf_shared v0.12.1
    Checking atoi v2.0.0
    Checking futures-channel v0.3.34
    Checking phf v0.12.1
    Checking phf v0.11.3
    Checking poly1305 v0.9.1
    Checking chacha20 v0.10.2
    Checking zmij v1.0.23
    Checking aead v0.6.1
    Checking sqlx-sqlite v0.9.0
    Checking linux-raw-sys v0.12.1
    Checking arrayvec v0.7.8
    Checking bitflags v2.13.2
    Checking itoa v1.0.18
    Checking winnow v0.7.15
    Checking constant_time_eq v0.4.2
    Checking serde_json v1.0.151
    Checking blake3 v1.8.7
    Checking sqlx v0.9.0
    Checking rustix v1.1.5
    Checking chacha20poly1305 v0.11.0
    Checking chrono-tz v0.10.4
    Checking uuid v1.26.1
    Checking cron v0.17.0
    Checking fastrand v2.5.0
    Checking tempfile v3.27.0
    Checking calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11m 09s

calternal-db tests

    Blocking waiting for file lock on build directory
   Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 8m 13s
     Running unittests src/lib.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/calternal_db-914a1a4b44b48b26)

running 24 tests
test integrations::tests::encrypts_and_binds_credential_to_user_and_account ... ok
test integrations::tests::older_shared_credential_payloads_keep_their_default_service_identity ... ok
test cron::tests::zoned_cron_keeps_its_wall_clock_time_across_daylight_saving ... ok
test db::tests::default_read_pool_covers_fifty_dav_clients ... ok
test integrations::tests::endpoint_identity_upgrade_preserves_accounts_and_mail_links ... ok
test integrations::tests::account_services_and_credentials_are_scoped_to_the_owner ... ok
test jobs::contention_tests::caller_transaction_controls_job_visibility ... ok
test integrations::tests::yahoo_is_stored_as_its_own_integration_provider ... ok
test sqlite::tests::saturated_or_closed_sqlite_pools_are_transient_service_errors ... ok
test sqlite::tests::transient_kinds_are_fixed_labels ... ok
test integrations::tests::mail_projection_links_enforce_owner_and_service_identity ... ok
test secrets::tests::named_secret_is_stable_and_first_candidate_wins ... ok
test secrets::tests::named_secret_can_be_replaced_and_cleared_without_reading_it_for_status ... ok
test integrations::tests::legacy_account_can_keep_all_services_disabled ... ok
test integrations::tests::yahoo_provider_migration_preserves_existing_accounts ... ok
test sqlite::tests::wrapped_sqlite_contention_retries_the_whole_operation ... ok
test secrets::tests::rejects_invalid_names_and_empty_candidates ... ok
test sqlite::tests::persistent_wrapped_sqlite_contention_stops_at_the_attempt_limit ... ok
test secrets::tests::secret_survives_database_reopen ... ok
test worker::tests::settlement_ignores_a_stolen_attempts_late_outcome ... ok
test worker::tests::disabled_handler_keeps_jobs_pending_and_finishes_active_work ... ok
test db::tests::live_writer_survives_queue_checkout_timeouts ... ok
test db::tests::reader_connections_see_a_thousand_immediate_writer_commits ... ok
test jobs::contention_tests::background_queue_write_waits_past_five_seconds_for_the_writer ... ok

test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.10s

     Running tests/mailstress_restart.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/mailstress_restart-4fbc9237aad16ee2)

running 1 test
test restart_before_mail_lease_expiry_does_not_strand_an_account ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s

     Running tests/queue.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/queue-bc7da7cdac84b88b)

running 21 tests
test enqueue_lease_throughput_microbenchmark ... ignored, manual enqueue plus lease throughput measurement
test failure_backoff_uses_fake_clock_and_dead_letters_at_limit ... ok
test expired_lease_is_recovered_and_old_owner_loses_lease ... ok
test queue_change_subscribers_receive_a_hint_after_state_changes ... ok
test deduplicates_pending_and_leased_jobs ... ok
test plugin_migrations_share_namespaces_and_check_applied_sql ... ok
test cron_enqueues_each_occurrence_once_and_keeps_one_active_job ... ok
test job_list_summaries_do_not_read_handler_payloads ... ok
test opens_wal_database_with_required_pragmas ... ok
test controlled_worker_observes_stop_at_handler_checkpoint ... ok
test expired_unrecovered_attempt_keeps_token_ownership ... ok
test owned_job_progress_is_private_and_cancellation_finishes_at_checkpoint ... ok
test live_writer_transaction_outlasts_lease_and_completes_once ... ok
test heartbeat_does_not_deadlock_a_handler_inside_a_write_transaction ... ok
test queue_pause_is_idempotent_persistent_and_blocks_new_leases ... ok
test worker_publishes_registered_kind_metadata_to_the_shared_queue ... ok
test queue_retry_run_now_and_clear_only_change_failed_jobs_of_one_kind ... ok
test stolen_attempt_late_completion_is_rejected_with_reused_worker_name ... ok
test workers_do_not_execute_a_job_twice ... ok
test snapshot_restores_as_a_readable_database ... ok
test slow_handler_renews_its_lease_and_is_not_stolen ... ok

test result: ok. 20 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.55s

     Running tests/sqlite_limits.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/sqlite_limits-ae16972a4cf57ed9)

running 1 test
test sqlite_pools_bound_readers_and_page_cache ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

   Doc-tests calternal_db

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Harness regression tests

test_refused_read_does_not_pass (test_mail_proxy.StressHarnessTests.test_refused_read_does_not_pass)
A healthy transport is insufficient when SEARCH fails (#1038). ... ok
test_rejects_header_control_in_run_identity (test_mail_proxy.StressHarnessTests.test_rejects_header_control_in_run_identity)
Generated Message-IDs must not introduce header lines (#1038). ... ok
test_rejects_remote_or_ambiguous_hosts (test_mail_proxy.StressHarnessTests.test_rejects_remote_or_ambiguous_hosts)
Do not permit this local load tool to target a remote service. ... ok
test_rejects_unbounded_or_fractional_pool (test_mail_proxy.StressHarnessTests.test_rejects_unbounded_or_fractional_pool)
Reject unbounded budgets and invalid pool sizes before sampling. ... ok
test_transfer_refusal_or_ambiguity_cannot_pass (test_mail_proxy.StressHarnessTests.test_transfer_refusal_or_ambiguity_cannot_pass)
BAD, a missing completion and disconnects never repeat writes (#1038). ... ok
test_transfer_waits_for_delivery_without_replaying_acceptance (test_mail_proxy.StressHarnessTests.test_transfer_waits_for_delivery_without_replaying_acceptance)
Pending drafts and COPY receipts must resolve before success (#1038). ... ok
test_transport_failure_is_sanitized_and_fails (test_mail_proxy.StressHarnessTests.test_transport_failure_is_sanitized_and_fails)
Never publish an imaplib error string or call a broken run PASS. ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.026s

OK

Provider fixture tests

test_default_gaps_and_refuse_existing_directory (test_mail_sync_provider.ProviderFixtureTests.test_default_gaps_and_refuse_existing_directory)
Keep the deployed #613 fixture contract when stress is not requested. ... ok
test_stress_boundaries_and_distinct_users (test_mail_sync_provider.ProviderFixtureTests.test_stress_boundaries_and_distinct_users)
Generate a small namespace but keep real large/empty boundary items. ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.653s

OK
Prepared fixture: 2000 messages, TLS port 29963, stress=False
Prepared fixture: 10 messages, TLS port 29963, stress=True

Cleanup

     Removed 8813 files, 5.2GiB total

Web build output removed. Test fixture keys were deleted by the coordinator. No owned Dovecot container remains. Worktree clean. #1038 stays open.

Reliability continuation results — branch `job/mailstress-1038`, base `6c03031e319c6ce2a00101acc3f9e8130bc5d07c`, head `4f83aef2b3b973aaa4ca8c10ade0a841683a9ed0`. READY FOR MERGE: no. Built: fresh production web app and local-provider server; reran the existing deterministic lease regression, crate gates, fixture tests and fresh three-User smoke. The harness and production source are unchanged. Commits: `7a58cdc8a` (lease recheck evidence), `4f83aef2b` (live recovery and scenario results). Owned tracked file: tests/adversarial/mail-sync.md. No dependencies or migrations changed. No pushes or deploys. A fresh production web build and local-provider server build passed. The unchanged smoke coordinator restarted the Instance before lease expiry. Twelve progress samples showed one completed account and Inbox counts `[64,32,0]`. The next sample showed three completed accounts and `[64,32,32]`. Initial sync passed within the 180-second deadline. No manual recovery restart ran. No account reported a sync error. This confirms defect A is fixed for this fixture. It does not check a hard kill during a transfer. | Scenario | Result | Evidence | |---|---|---| | 0. Expired lease recovery | PASS | Deterministic restart regression and fresh three-User initial sync both passed without a manual restart | | 6. Hard kill during transfers | Not run | The unchanged coordinator has no transfer-kill phase | | 2. Concurrency correctness | Partial | Sequential exact COPY/MOVE receipts passed; no simultaneous mutation or folder lifecycle matrix ran | | 3. Upstream faults | Not run | The coordinator has no restart/drop/slow/UIDVALIDITY phase | | 7. Account separation | Partial | Three folder-label and message-sentinel checks passed in each short phase; transfer writes cover User 0 only | | 1. Sustained load | Partial | Two five-second command windows passed with three sessions; no 30-minute run or bounded-growth claim | | 4. Client input robustness | Partial | Ordinary SELECT/LIST/SEARCH/FETCH/IDLE/APPEND/STORE/COPY/MOVE/UID EXPUNGE passed; no malformed-input or failed-login matrix ran | | 5. Unicode names | Not run | The coordinator has no Unicode namespace phase | Each short phase accepted one APPEND, one COPY and one MOVE. Exact MIME, Flagged state and receipt counts matched upstream and locally: Drafts zero, Archive one, Trash one. All three authenticated web sessions and readiness returned 200 after each phase. These checks prove retained sessions, not a fresh sign-in during load. | Phase | Elapsed including receipts | RSS samples (KiB) | Descriptor samples | |---|---|---|---| | One User | 12.71 s | 223124, 228172, 228456 | 97, 106, 103 | | Three Users | 14.86 s | 228584, 267664, 267596 | 104, 107, 103 | Both phases completed seven read/IDLE cycles with no recorded failure. COPY had ten clean NO waits per phase. MOVE had fourteen and fifteen. Each transfer then completed once and passed both receipt checks. This short series does not prove long-term resource bounds or the five-second delivery target. The smoke selected three sessions and reported 79 GiB free. Verification: `cargo fmt --check` passed with no output; `cargo clippy -p calternal-db --all-targets -- -D warnings` passed; `cargo test -p calternal-db` passed 46 tests, with one manual benchmark ignored. Seven harness regressions and two provider-fixture tests passed. `node --check tests/adversarial/mail_stress.mjs` passed with no output. `git fetch origin` and `git merge origin/dev` ran once before final gates; the merge reported `Already up to date.` No source, dependency or migration changed in this continuation. For the merge round: full suites and the long-run profile remain required under the 2026-10-02 verification policy. The documented long command above cannot reach the soak phases with its current 10,000-folder fixture. The missing scenario matrices also need focused runs. Do not treat this smoke as those results. READY FOR MERGE: no. Defect A is verified fixed. The requested reliability matrix remains incomplete. No new product behavior was selected. The harness was not changed. No UI changed; UX gaps are not applicable. Known gaps: no hard-kill transfer acceptance, simultaneous mutation/folder lifecycle matrix, upstream fault matrix, 30-minute resource-bound evidence, malformed-input/failed-login matrix, Unicode namespace matrix, full account mutation matrix or fresh web sign-in during load. The bounded ordinary smoke did not uncover a new production defect. No invalid-response/300-second-timeout evidence was seen by its sync status checks; no new #1037 diagnosis is claimed. Decisions: retain the existing harness as instructed; do not label smoke as sustained load. Use the job's 2026-10-02 verification policy to defer long runs and full matrices to the merge round. The unchanged long fixture exceeds the existing folder discovery bound. No new product behavior was selected. UX gaps closed/left: not applicable; no UI source changed. Re-read the edited documentation before this report. For the merge round: ```sh CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \ CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \ TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs ``` This is the existing long-run command, not a passing result. Resolve its 10,000-folder versus 512-folder prerequisite before using it to prove two 30-minute phases, bounded resources and availability. It does not implement the missing scenario matrices; those need separate focused checks. Full combined gates and `tests/adversarial/run.sh` remain merge-round work. Gate output below is verbatim. `cargo fmt --check` and `node --check tests/adversarial/mail_stress.mjs` exited 0 with no output. The live smoke exited 0; its count-only output is preserved locally in artifacts/mailstress-1038/round2-smoke.log. No screenshots were required: no UI changed. Restart regression ```text Compiling proc-macro2 v1.0.107 Compiling quote v1.0.47 Compiling unicode-ident v1.0.26 Compiling libc v0.2.189 Compiling typenum v1.20.1 Compiling cfg-if v1.0.5 Compiling stable_deref_trait v1.2.1 Compiling getrandom v0.4.3 Compiling smallvec v1.16.1 Compiling version_check v0.9.5 Compiling rand_core v0.10.1 Compiling litemap v0.8.3 Compiling writeable v0.6.4 Compiling generic-array v0.14.9 Compiling hybrid-array v0.4.15 Compiling autocfg v1.5.1 Compiling utf8_iter v1.0.4 Compiling siphasher v1.0.3 Compiling shlex v2.0.1 Compiling syn v3.0.6 Compiling find-msvc-tools v0.1.13 Compiling icu_properties_data v2.3.0 Compiling pin-project-lite v0.2.17 Compiling icu_normalizer_data v2.3.0 Compiling futures-core v0.3.34 Compiling syn v2.0.119 Compiling cc v1.4.7 Compiling synstructure v0.14.0 Compiling zerovec-derive v0.11.6 Compiling displaydoc v0.2.7 Compiling zerofrom-derive v0.1.8 Compiling yoke-derive v0.8.3 Compiling num-traits v0.2.19 Compiling scopeguard v1.2.0 Compiling zerofrom v0.1.8 Compiling lock_api v0.4.14 Compiling yoke v0.8.3 Compiling once_cell v1.21.4 Compiling futures-sink v0.3.34 Compiling parking_lot_core v0.9.12 Compiling serde_core v1.0.229 Compiling zerovec v0.11.8 Compiling zerotrie v0.2.5 Compiling crypto-common v0.2.2 Compiling equivalent v1.0.2 Compiling memchr v2.8.3 Compiling tinystr v0.8.4 Compiling potential_utf v0.1.6 Compiling crossbeam-utils v0.8.23 Compiling icu_collections v2.3.0 Compiling icu_locale_core v2.3.0 Compiling crypto-common v0.1.6 Compiling block-buffer v0.10.4 Compiling tokio-macros v2.7.2 Compiling icu_provider v2.3.1 Compiling socket2 v0.6.5 Compiling mio v1.2.3 Compiling thiserror v2.0.21 Compiling bytes v1.12.1 Compiling icu_properties v2.3.0 Compiling icu_normalizer v2.3.0 Compiling futures-task v0.3.34 Compiling slab v0.4.12 Compiling allocator-api2 v0.2.21 Compiling idna_adapter v1.2.2 Compiling futures-io v0.3.34 Compiling rand_core v0.6.4 Compiling pkg-config v0.3.34 Compiling percent-encoding v2.3.2 Compiling vcpkg v0.2.15 Compiling serde v1.0.229 Compiling foldhash v0.2.0 Compiling rand v0.8.8 Compiling form_urlencoded v1.2.2 Compiling libsqlite3-sys v0.37.0 Compiling hashbrown v0.16.1 Compiling idna v1.1.0 Compiling futures-util v0.3.34 Compiling tokio v1.53.1 Compiling digest v0.10.7 Compiling parking_lot v0.12.5 Compiling tracing-core v0.1.36 Compiling tracing-attributes v0.1.31 Compiling serde_derive v1.0.229 Compiling thiserror-impl v2.0.21 Compiling phf_shared v0.11.3 Compiling inout v0.2.2 Compiling cmov v0.5.4 Compiling crc-catalog v2.5.0 Compiling log v0.4.34 Compiling hashbrown v0.17.1 Compiling cpufeatures v0.2.17 Compiling cpufeatures v0.3.1 Compiling parking v2.2.1 Compiling sha2 v0.10.9 Compiling indexmap v2.14.2 Compiling event-listener v5.4.2 Compiling tracing v0.1.44 Compiling crc v3.4.0 Compiling ctutils v0.4.2 Compiling phf_generator v0.11.3 Compiling tokio-stream v0.1.19 Compiling futures-intrusive v0.5.0 Compiling crossbeam-queue v0.3.14 Compiling url v2.5.8 Compiling hashlink v0.11.1 Compiling spin v0.9.9 Compiling block-buffer v0.12.1 Compiling iana-time-zone v0.1.65 Compiling base64 v0.22.1 Compiling zmij v1.0.23 Compiling either v1.18.0 Compiling chrono v0.4.45 Compiling cipher v0.5.2 Compiling sqlx-core v0.9.0 Compiling flume v0.12.0 Compiling phf_macros v0.11.3 Compiling universal-hash v0.6.1 Compiling futures-executor v0.3.34 Compiling atoi v2.0.0 Compiling futures-channel v0.3.34 Compiling blake3 v1.8.7 Compiling phf_shared v0.12.1 Compiling serde_json v1.0.151 Compiling chrono-tz v0.10.4 Compiling rustix v1.1.5 Compiling phf v0.12.1 Compiling phf v0.11.3 Compiling sqlx-sqlite v0.9.0 Compiling poly1305 v0.9.1 Compiling chacha20 v0.10.2 Compiling aead v0.6.1 Compiling constant_time_eq v0.4.2 Compiling arrayvec v0.7.8 Compiling itoa v1.0.18 Compiling linux-raw-sys v0.12.1 Compiling bitflags v2.13.2 Compiling winnow v0.7.15 Compiling cron v0.17.0 Compiling chacha20poly1305 v0.11.0 Compiling sqlx v0.9.0 Compiling uuid v1.26.1 Compiling fastrand v2.5.0 Compiling tempfile v3.27.0 Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db) Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 03s Running tests/mailstress_restart.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/mailstress_restart-4fbc9237aad16ee2) running 1 test test restart_before_mail_lease_expiry_does_not_strand_an_account ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.65s ``` calternal-db Clippy ```text Blocking waiting for file lock on build directory Checking libc v0.2.189 Checking cfg-if v1.0.5 Checking typenum v1.20.1 Checking zerofrom v0.1.8 Checking stable_deref_trait v1.2.1 Checking writeable v0.6.4 Checking smallvec v1.16.1 Checking yoke v0.8.3 Checking rand_core v0.10.1 Checking litemap v0.8.3 Checking zerovec v0.11.8 Checking zerotrie v0.2.5 Checking getrandom v0.4.3 Checking hybrid-array v0.4.15 Checking futures-core v0.3.34 Checking pin-project-lite v0.2.17 Checking tinystr v0.8.4 Checking potential_utf v0.1.6 Checking utf8_iter v1.0.4 Checking scopeguard v1.2.0 Checking generic-array v0.14.9 Checking icu_locale_core v2.3.0 Checking icu_collections v2.3.0 Checking lock_api v0.4.14 Checking icu_properties_data v2.3.0 Checking icu_normalizer_data v2.3.0 Checking once_cell v1.21.4 Checking futures-sink v0.3.34 Checking num-traits v0.2.19 Checking crypto-common v0.2.2 Checking icu_provider v2.3.1 Checking memchr v2.8.3 Checking icu_normalizer v2.3.0 Checking icu_properties v2.3.0 Checking siphasher v1.0.3 Checking equivalent v1.0.2 Checking serde_core v1.0.229 Checking parking_lot_core v0.9.12 Checking block-buffer v0.10.4 Checking crypto-common v0.1.6 Checking socket2 v0.6.5 Checking mio v1.2.3 Checking idna_adapter v1.2.2 Checking futures-io v0.3.34 Checking slab v0.4.12 Checking allocator-api2 v0.2.21 Checking foldhash v0.2.0 Checking percent-encoding v2.3.2 Checking bytes v1.12.1 Checking futures-task v0.3.34 Checking hashbrown v0.16.1 Checking form_urlencoded v1.2.2 Checking futures-util v0.3.34 Checking tokio v1.53.1 Compiling phf_shared v0.11.3 Checking idna v1.1.0 Checking digest v0.10.7 Checking parking_lot v0.12.5 Checking crossbeam-utils v0.8.23 Checking tracing-core v0.1.36 Checking inout v0.2.2 Checking hashbrown v0.17.1 Checking parking v2.2.1 Checking crc-catalog v2.5.0 Checking cpufeatures v0.2.17 Checking log v0.4.34 Checking cmov v0.5.4 Checking cpufeatures v0.3.1 Checking tracing v0.1.44 Checking ctutils v0.4.2 Checking sha2 v0.10.9 Checking crc v3.4.0 Checking tokio-stream v0.1.19 Checking indexmap v2.14.2 Checking event-listener v5.4.2 Checking serde v1.0.229 Checking thiserror v2.0.21 Checking crossbeam-queue v0.3.14 Checking futures-intrusive v0.5.0 Compiling phf_generator v0.11.3 Checking url v2.5.8 Checking hashlink v0.11.1 Checking spin v0.9.9 Checking block-buffer v0.12.1 Checking either v1.18.0 Checking iana-time-zone v0.1.65 Checking base64 v0.22.1 Checking flume v0.12.0 Checking chrono v0.4.45 Checking cipher v0.5.2 Checking sqlx-core v0.9.0 Checking libsqlite3-sys v0.37.0 Compiling phf_macros v0.11.3 Checking universal-hash v0.6.1 Checking futures-executor v0.3.34 Checking phf_shared v0.12.1 Checking atoi v2.0.0 Checking futures-channel v0.3.34 Checking phf v0.12.1 Checking phf v0.11.3 Checking poly1305 v0.9.1 Checking chacha20 v0.10.2 Checking zmij v1.0.23 Checking aead v0.6.1 Checking sqlx-sqlite v0.9.0 Checking linux-raw-sys v0.12.1 Checking arrayvec v0.7.8 Checking bitflags v2.13.2 Checking itoa v1.0.18 Checking winnow v0.7.15 Checking constant_time_eq v0.4.2 Checking serde_json v1.0.151 Checking blake3 v1.8.7 Checking sqlx v0.9.0 Checking rustix v1.1.5 Checking chacha20poly1305 v0.11.0 Checking chrono-tz v0.10.4 Checking uuid v1.26.1 Checking cron v0.17.0 Checking fastrand v2.5.0 Checking tempfile v3.27.0 Checking calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db) Finished `dev` profile [unoptimized + debuginfo] target(s) in 11m 09s ``` calternal-db tests ```text Blocking waiting for file lock on build directory Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/mailstress-1038/crates/calternal-db) Finished `test` profile [unoptimized + debuginfo] target(s) in 8m 13s Running unittests src/lib.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/calternal_db-914a1a4b44b48b26) running 24 tests test integrations::tests::encrypts_and_binds_credential_to_user_and_account ... ok test integrations::tests::older_shared_credential_payloads_keep_their_default_service_identity ... ok test cron::tests::zoned_cron_keeps_its_wall_clock_time_across_daylight_saving ... ok test db::tests::default_read_pool_covers_fifty_dav_clients ... ok test integrations::tests::endpoint_identity_upgrade_preserves_accounts_and_mail_links ... ok test integrations::tests::account_services_and_credentials_are_scoped_to_the_owner ... ok test jobs::contention_tests::caller_transaction_controls_job_visibility ... ok test integrations::tests::yahoo_is_stored_as_its_own_integration_provider ... ok test sqlite::tests::saturated_or_closed_sqlite_pools_are_transient_service_errors ... ok test sqlite::tests::transient_kinds_are_fixed_labels ... ok test integrations::tests::mail_projection_links_enforce_owner_and_service_identity ... ok test secrets::tests::named_secret_is_stable_and_first_candidate_wins ... ok test secrets::tests::named_secret_can_be_replaced_and_cleared_without_reading_it_for_status ... ok test integrations::tests::legacy_account_can_keep_all_services_disabled ... ok test integrations::tests::yahoo_provider_migration_preserves_existing_accounts ... ok test sqlite::tests::wrapped_sqlite_contention_retries_the_whole_operation ... ok test secrets::tests::rejects_invalid_names_and_empty_candidates ... ok test sqlite::tests::persistent_wrapped_sqlite_contention_stops_at_the_attempt_limit ... ok test secrets::tests::secret_survives_database_reopen ... ok test worker::tests::settlement_ignores_a_stolen_attempts_late_outcome ... ok test worker::tests::disabled_handler_keeps_jobs_pending_and_finishes_active_work ... ok test db::tests::live_writer_survives_queue_checkout_timeouts ... ok test db::tests::reader_connections_see_a_thousand_immediate_writer_commits ... ok test jobs::contention_tests::background_queue_write_waits_past_five_seconds_for_the_writer ... ok test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.10s Running tests/mailstress_restart.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/mailstress_restart-4fbc9237aad16ee2) running 1 test test restart_before_mail_lease_expiry_does_not_strand_an_account ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s Running tests/queue.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/queue-bc7da7cdac84b88b) running 21 tests test enqueue_lease_throughput_microbenchmark ... ignored, manual enqueue plus lease throughput measurement test failure_backoff_uses_fake_clock_and_dead_letters_at_limit ... ok test expired_lease_is_recovered_and_old_owner_loses_lease ... ok test queue_change_subscribers_receive_a_hint_after_state_changes ... ok test deduplicates_pending_and_leased_jobs ... ok test plugin_migrations_share_namespaces_and_check_applied_sql ... ok test cron_enqueues_each_occurrence_once_and_keeps_one_active_job ... ok test job_list_summaries_do_not_read_handler_payloads ... ok test opens_wal_database_with_required_pragmas ... ok test controlled_worker_observes_stop_at_handler_checkpoint ... ok test expired_unrecovered_attempt_keeps_token_ownership ... ok test owned_job_progress_is_private_and_cancellation_finishes_at_checkpoint ... ok test live_writer_transaction_outlasts_lease_and_completes_once ... ok test heartbeat_does_not_deadlock_a_handler_inside_a_write_transaction ... ok test queue_pause_is_idempotent_persistent_and_blocks_new_leases ... ok test worker_publishes_registered_kind_metadata_to_the_shared_queue ... ok test queue_retry_run_now_and_clear_only_change_failed_jobs_of_one_kind ... ok test stolen_attempt_late_completion_is_rejected_with_reused_worker_name ... ok test workers_do_not_execute_a_job_twice ... ok test snapshot_restores_as_a_readable_database ... ok test slow_handler_renews_its_lease_and_is_not_stolen ... ok test result: ok. 20 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.55s Running tests/sqlite_limits.rs (/home/kayg/build/targets/mailstress-1038/debug/deps/sqlite_limits-ae16972a4cf57ed9) running 1 test test sqlite_pools_bound_readers_and_page_cache ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s Doc-tests calternal_db running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Harness regression tests ```text test_refused_read_does_not_pass (test_mail_proxy.StressHarnessTests.test_refused_read_does_not_pass) A healthy transport is insufficient when SEARCH fails (#1038). ... ok test_rejects_header_control_in_run_identity (test_mail_proxy.StressHarnessTests.test_rejects_header_control_in_run_identity) Generated Message-IDs must not introduce header lines (#1038). ... ok test_rejects_remote_or_ambiguous_hosts (test_mail_proxy.StressHarnessTests.test_rejects_remote_or_ambiguous_hosts) Do not permit this local load tool to target a remote service. ... ok test_rejects_unbounded_or_fractional_pool (test_mail_proxy.StressHarnessTests.test_rejects_unbounded_or_fractional_pool) Reject unbounded budgets and invalid pool sizes before sampling. ... ok test_transfer_refusal_or_ambiguity_cannot_pass (test_mail_proxy.StressHarnessTests.test_transfer_refusal_or_ambiguity_cannot_pass) BAD, a missing completion and disconnects never repeat writes (#1038). ... ok test_transfer_waits_for_delivery_without_replaying_acceptance (test_mail_proxy.StressHarnessTests.test_transfer_waits_for_delivery_without_replaying_acceptance) Pending drafts and COPY receipts must resolve before success (#1038). ... ok test_transport_failure_is_sanitized_and_fails (test_mail_proxy.StressHarnessTests.test_transport_failure_is_sanitized_and_fails) Never publish an imaplib error string or call a broken run PASS. ... ok ---------------------------------------------------------------------- Ran 7 tests in 0.026s OK ``` Provider fixture tests ```text test_default_gaps_and_refuse_existing_directory (test_mail_sync_provider.ProviderFixtureTests.test_default_gaps_and_refuse_existing_directory) Keep the deployed #613 fixture contract when stress is not requested. ... ok test_stress_boundaries_and_distinct_users (test_mail_sync_provider.ProviderFixtureTests.test_stress_boundaries_and_distinct_users) Generate a small namespace but keep real large/empty boundary items. ... ok ---------------------------------------------------------------------- Ran 2 tests in 0.653s OK Prepared fixture: 2000 messages, TLS port 29963, stress=False Prepared fixture: 10 messages, TLS port 29963, stress=True ``` Cleanup ```text Removed 8813 files, 5.2GiB total ``` Web build output removed. Test fixture keys were deleted by the coordinator. No owned Dovecot container remains. Worktree clean. #1038 stays open.
kayg commented 2026-10-04 09:29:09 +00:00
Author
Owner
Copy link

Continuation started on job/mailstress-1038, base 4f83aef2b3b973aaa4ca8c10ade0a841683a9ed0.

Read CLAUDE.md, CONTEXT.md, DESIGN sections 49 and 53, issue #1038, and the existing coordinator. The coordinator implements ordinary mixed-command soak and sequential exact transfer receipts; it does not implement the requested hard-kill, concurrent mutations, upstream fault or Unicode phases. No such acceptance is assumed.

The long fixture requests 10,000 Bulk folders, above the 512-folder discovery cap. Change that prerequisite to 400 Bulk folders (plus 80 implicit deep parents and special folders). This is a test fixture choice, not a product limit change. Build the production web app and local-provider server, then attempt the long run. Credentials and message bodies remain private.

Continuation started on `job/mailstress-1038`, base `4f83aef2b3b973aaa4ca8c10ade0a841683a9ed0`. Read CLAUDE.md, CONTEXT.md, DESIGN sections 49 and 53, issue #1038, and the existing coordinator. The coordinator implements ordinary mixed-command soak and sequential exact transfer receipts; it does not implement the requested hard-kill, concurrent mutations, upstream fault or Unicode phases. No such acceptance is assumed. The long fixture requests 10,000 Bulk folders, above the 512-folder discovery cap. Change that prerequisite to 400 Bulk folders (plus 80 implicit deep parents and special folders). This is a test fixture choice, not a product limit change. Build the production web app and local-provider server, then attempt the long run. Credentials and message bodies remain private.
kayg commented 2026-10-04 09:33:06 +00:00
Author
Owner
Copy link

Two coordinator prerequisite defects were found before the long run:

  1. It requested 10,000 Bulk folders, but sync.rs limits discovery to 512. The fixture now uses 400 Bulk folders. A generated-fixture regression counts physical names and implicit deep parents.
  2. It requested 100,000 messages and then added 32 sentinel messages. proxy.rs rejects SELECT above MAX_MESSAGES = 100_000. The long fixture now requests 99,968 ordinary messages and expects 100,000 total. A regression protects that sum.

These are fixture corrections only. No production limit or test expectation for server behavior changed. Commits: d1e478d4b and the next fixture-budget commit. No harness refactor or new scenario framework.

Two coordinator prerequisite defects were found before the long run: 1. It requested 10,000 Bulk folders, but `sync.rs` limits discovery to 512. The fixture now uses 400 Bulk folders. A generated-fixture regression counts physical names and implicit deep parents. 2. It requested 100,000 messages and then added 32 sentinel messages. `proxy.rs` rejects SELECT above `MAX_MESSAGES = 100_000`. The long fixture now requests 99,968 ordinary messages and expects 100,000 total. A regression protects that sum. These are fixture corrections only. No production limit or test expectation for server behavior changed. Commits: `d1e478d4b` and the next fixture-budget commit. No harness refactor or new scenario framework.
kayg commented 2026-10-04 09:57:24 +00:00
Author
Owner
Copy link

The full #1038 long-run attempt exited 1 before either soak phase. Fixture: 100,000 total Inbox messages on User 0, 32 each on Users 1 and 2; 400 Bulk folders plus the 80-level hierarchy and boundary messages. Session budget: 12, with 85 GiB free.

At the 900-second initial-sync deadline: Inbox counts [63520,32,32]; two completed accounts; two accounts had reported errors at some point. The current account errors had previously cleared in a read-only diagnostic. Queue: completed 7, dead 1, leased 3, pending 1; expired leases 0. The diagnostic restart also failed its 180-second completion deadline. It remained a failure, and the coordinator cleaned up its private Instance and provider.

A subsequent read-only diagnostic before cleanup found two dead jobs, each with three attempts and a coarse error class other. The Jobs payload and error text were not printed. sync.rs maps failed synchronization to the fixed job message mail provider synchronization failed, so job text does not reveal the provider cause. No claim that this is merely host load, a recovered transfer, or a crash.

The job merged origin/dev once (comment-only conflict, identical authentication code). A bounded continuation retains the same data and command checks: 30 minutes for initial sync, two full 30-minute phases, two-hour total deadline. This corrects the coordinator's arithmetic: its prior 75-minute whole-run budget left no setup/cleanup time after a full 15-minute prerequisite plus both phases. The initial failed attempt remains part of the final results.

The full #1038 long-run attempt exited 1 before either soak phase. Fixture: 100,000 total Inbox messages on User 0, 32 each on Users 1 and 2; 400 Bulk folders plus the 80-level hierarchy and boundary messages. Session budget: 12, with 85 GiB free. At the 900-second initial-sync deadline: Inbox counts `[63520,32,32]`; two completed accounts; two accounts had reported errors at some point. The current account errors had previously cleared in a read-only diagnostic. Queue: completed 7, dead 1, leased 3, pending 1; expired leases 0. The diagnostic restart also failed its 180-second completion deadline. It remained a failure, and the coordinator cleaned up its private Instance and provider. A subsequent read-only diagnostic before cleanup found two dead jobs, each with three attempts and a coarse error class `other`. The Jobs payload and error text were not printed. `sync.rs` maps failed synchronization to the fixed job message `mail provider synchronization failed`, so job text does not reveal the provider cause. No claim that this is merely host load, a recovered transfer, or a crash. The job merged origin/dev once (comment-only conflict, identical authentication code). A bounded continuation retains the same data and command checks: 30 minutes for initial sync, two full 30-minute phases, two-hour total deadline. This corrects the coordinator's arithmetic: its prior 75-minute whole-run budget left no setup/cleanup time after a full 15-minute prerequisite plus both phases. The initial failed attempt remains part of the final results.
kayg commented 2026-10-04 10:52:09 +00:00
Author
Owner
Copy link

Actual full-duration one-account result from #1038, ordinary fixture, 12 clients:

  • Elapsed including audit: 1845.99 s. All 12 clients connected. 16,693 SEARCH/FETCH/IDLE/EXPUNGE cycles completed.
  • 20 APPEND, draft-FETCH, STORE, COPY and MOVE commands each returned OK. COPY had 180 clean NO waits; MOVE had 177. No command-loop exception was recorded.
  • Original audit: 14 exact upstream receipts and 14 exact local receipts completed. One receipt-AssertionError was recorded. The helper records exception types only, so the exact assertion is unavailable. Overall phase result: FAIL. All three retained web sessions and readiness returned 200.
  • RSS first/last: 153300/263064 KiB; maximum 263064 KiB. Descriptors first/last: 76/96; maximum 143. During the steady command window, descriptors were mostly 135–139 and RSS about 240000–244000 KiB.

A separate read-only audit, while the three-account phase runs, checked all 20 first-phase identities. Local live memberships and flags matched Drafts=0, Archive=1, Trash=1 for every identity. Direct upstream checks found all 40 intentional destination copies, each with exact original MIME and Flagged state. Counts: identities 20, local_counts_ok 20, local_unflagged 0, upstream_counts_ok 20, upstream_bodies_ok 40, upstream_unflagged 0. Elapsed 1.08 s. No message body, Message-ID or credential was printed.

This rules out observed loss/duplication in those checked writes at the later audit time. It does not recheck exact bodies through the proxy or turn the original FAIL into PASS. A 45-second shared audit-budget exhaustion and a cold proxy-body fetch failure are both possible; cause is not established. proxy.rs::body has a 30-second cold upstream read limit. Do not attribute this solely to host load or change expectations to hide it.

The separate existing real Dovecot receipt regression passed: one test, 88 filtered out, 21.41 s. It proves COPY/delete or MOVE ordering and replay without duplicate receipts, not simultaneous mutation acceptance. The three-account 30-minute phase is still running.

Actual full-duration one-account result from #1038, ordinary fixture, 12 clients: - Elapsed including audit: 1845.99 s. All 12 clients connected. 16,693 SEARCH/FETCH/IDLE/EXPUNGE cycles completed. - 20 APPEND, draft-FETCH, STORE, COPY and MOVE commands each returned OK. COPY had 180 clean NO waits; MOVE had 177. No command-loop exception was recorded. - Original audit: 14 exact upstream receipts and 14 exact local receipts completed. One `receipt-AssertionError` was recorded. The helper records exception types only, so the exact assertion is unavailable. Overall phase result: FAIL. All three retained web sessions and readiness returned 200. - RSS first/last: 153300/263064 KiB; maximum 263064 KiB. Descriptors first/last: 76/96; maximum 143. During the steady command window, descriptors were mostly 135–139 and RSS about 240000–244000 KiB. A separate read-only audit, while the three-account phase runs, checked all 20 first-phase identities. Local live memberships and flags matched Drafts=0, Archive=1, Trash=1 for every identity. Direct upstream checks found all 40 intentional destination copies, each with exact original MIME and Flagged state. Counts: identities 20, local_counts_ok 20, local_unflagged 0, upstream_counts_ok 20, upstream_bodies_ok 40, upstream_unflagged 0. Elapsed 1.08 s. No message body, Message-ID or credential was printed. This rules out observed loss/duplication in those checked writes at the later audit time. It does not recheck exact bodies through the proxy or turn the original FAIL into PASS. A 45-second shared audit-budget exhaustion and a cold proxy-body fetch failure are both possible; cause is not established. `proxy.rs::body` has a 30-second cold upstream read limit. Do not attribute this solely to host load or change expectations to hide it. The separate existing real Dovecot receipt regression passed: one test, 88 filtered out, 21.41 s. It proves COPY/delete or MOVE ordering and replay without duplicate receipts, not simultaneous mutation acceptance. The three-account 30-minute phase is still running.
kayg commented 2026-10-04 11:19:57 +00:00
Author
Owner
Copy link

Final report — mailstress-1038

HEAD: 0dcfc98aa4f8da18cda3dc5d4330d36fd3e54d64. READY FOR MERGE: no.

Built: corrected folder/message budgets, finite full-run setup margin, an explicit ordinary-fixture full-duration profile, and fixed private-safe receipt failure codes with a red/green regression. Existing command, MIME, flag and receipt assertions remain strict.

Files: tests/adversarial/mail_stress.mjs, mail_proxy.py, test_mail_proxy.py, test_mail_sync_provider.py, and mail-sync.md; the calternal-auth/src/store.rs merge conflict retained the incoming comment over identical code. No production Mail, Jobs or server behavior changed.

Live ordinary soak used profile source a0192a88e and the merged server built at 1b55bbfe0. The later reporter change was unit-tested; the completed runs used the earlier reporter. The initial large attempt used the supplied Rust baseline 4f83aef2b. No live result is claimed for an unrun scenario.

Scenario results and verification

This job runs the reliability checks. The earlier merge-round deferrals do
not apply to this job. Results below do not certify the missing scenarios.

The large fixture now has 400 Bulk folders. The namespace test counts the
80 deep-folder parents and special folders too. It stays below the provider's
512-folder limit. The fixture has 99,968 ordinary Inbox messages plus 32
sentinels, for a total of 100,000. The old total of 100,032 exceeded SELECT's
100,000-message limit. Production limits did not change.

The first full attempt failed its 900-second initial-sync deadline. Inbox
counts were [63520,32,32]. Two accounts had completed. Two accounts had
reported an error during the attempt. A read-only check found no current
account error before the deadline. The queue had seven completed jobs, one
dead job, three leased jobs and one pending job. No lease had expired.
The diagnostic restart also failed its 180-second completion deadline.
Neither 30-minute command phase ran. This remains a failed result.

The job merged origin/dev once, as required. The authentication conflict
was a comment-only conflict with identical code. The incoming #1041 comment
was retained. The production web app and local-provider server rebuilt.
The continuation failed account creation with HTTP 504 on User 0. The
provider cause is not established. Both attempts are recorded on #1037.

The coordinator now allows 30 minutes for initial sync and two hours for
the full run. The old 75-minute total budget had no setup or cleanup margin
after a 15-minute prerequisite and two 30-minute phases. These test deadlines
do not change production deadlines or turn the first failure into a pass.

An explicit ordinary-fixture profile keeps the full session pool and both
30-minute command phases. It uses Inbox counts [64,32,32], 16 Bulk folders,
and no Boundary or deep folder. It never substitutes for the large fixture.
The host selected 12 clients and reported 88 GiB free for this run.
All three account calls returned 201; all App Password calls returned 200.
Initial sync passed without a manual recovery restart or a reported error.
Both full 30-minute command windows ran to completion. The overall process
exited 1 because the first phase failed its receipt audit. The second phase
passed. These are local shared-host results, not perf-VM measurements.

CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 \
CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \
CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \
TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs

The coordinator has no hard-kill transfer, simultaneous mutation, upstream
fault, malformed-input or Unicode phase. Its three-account phase uses three
Users with one Connected Account each. It does not test three Connected
Accounts in one User's unified view. The web check retains existing sessions;
it does not perform a fresh sign-in during load. These gaps remain explicit.

Phase Result Elapsed with audit Complete command cycles Accepted APPEND / COPY / MOVE Exact upstream / proxy receipts
One Connected Account, one User FAIL 1845.99 s 16,693 20 / 20 / 20 14 / 14; one receipt assertion failed
Three Connected Accounts, three Users PASS 1841.08 s 17,302 20 / 20 / 20 20 / 20; no exception

Each phase used 12 clients. All required command loops completed without an
exception. COPY had 180 and 221 clean NO waits; MOVE had 177 and 225.
Every transfer then returned OK. All three retained web sessions and readiness
returned 200 after each phase. A fresh sign-in during load was not checked.

Phase First / last / maximum RSS (KiB) First / last / maximum descriptors First / last steady five-minute mean RSS (KiB)
One account 153300 / 263064 / 263064 76 / 96 / 143 240795 / 244036
Three accounts 263260 / 308720 / 308720 100 / 99 / 130 283910 / 289265

The steady windows are seconds 60–360 and 1500–1800. Resource growth was
small during those windows. The samples do not prove a permanent memory
bound. No server crash or command-loop failure was observed in these phases.

The first audit failed after 14 of 20 receipts, with a 45-second shared
budget. It recorded only the assertion class. The cause is not established;
do not assume deadline exhaustion or a cold-body failure. A separate read-only
audit found all 20 Message-IDs in the right local live memberships and upstream
folders. Drafts had zero, Archive one and Trash one per identity. Flagged state
was intact in both projections. All 40 intentional upstream copies had exact
MIME. The audit took 1.08 s. It did not recheck the six remaining exact bodies
through the proxy, and it does not turn the original FAIL into PASS.

The read-only three-account audit found 20 identities, distributed [10,10,0]
across the Users. All local and upstream counts matched. All 40 upstream bodies
were exact and flagged. No local membership crossed an owner or account, and
no foreign upstream receipt appeared in any of the three accounts. Elapsed
time was 2.98 s. User 2 had no transfer writes; full authorization coverage
and three Connected Accounts in one User remain unverified.

The existing real Dovecot regression passed in 21.41 s. It checks COPY followed
by delete or MOVE, source removal, destination flags and repeated worker
delivery without duplicate receipts. It does not test simultaneous mutations.
The existing interrupted-lease restart regression also passed in 0.72 s.
It does not prove a hard kill during a transfer.

Requested scenario, in risk order Result Evidence or gap
6. Hard kill during transfers Not run Only the lease-recovery regression and graceful setup restart were checked
2. Concurrency correctness Partial Two writers, sequential transfer receipts and live worker replay passed; same-item races and folder lifecycle cases were not run
3. Upstream faults Not run Live sync errors and HTTP 504 were recorded on #1037; restart, drop, slow reply and UIDVALIDITY changes were not injected
7. Account separation Partial Three-User sentinels, labels, local ownership and upstream receipt absence passed; full matrix remains unverified
1. Sustained load Mixed Both 30-minute windows ran with 12 clients; first audit FAIL, second PASS; large prerequisites failed
4. Input robustness Partial Ordinary IMAP commands passed; pipelining, unfinished literals, invalid names, overlong lines, IDLE without DONE and failed-login cases were not run
5. Unicode names Not run No Unicode namespace scenario exists in the coordinator

These missing scenarios remain #1038 reliability work. They are not deferred
to a merge round. No exploit-style protocol-abuse probe was added or run.

The reporter now adds fixed codes for known receipt assertions. It still
fails the run and retains exception-class counters. Unknown strings and
non-string error arguments stay private. The regression failed before the
change and then passed. The live soak used the earlier reporter, so its lost
assertion detail cannot be recovered from the new code. No existing server
status, assertion or body/flag check was relaxed.

Verification excerpts, verbatim:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 37s
test result: ok. 95 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 66.90s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 88 filtered out; finished in 21.41s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.72s
Ran 12 tests in 1.043s

OK

The first line is cargo clippy -p calternal-auth --all-targets -- -D warnings.
The following lines are the Auth tests, the focused Mail receipt test, the
Jobs restart test and the Python fixture tests. cargo fmt --check and
node --check tests/adversarial/mail_stress.mjs exited 0 with no output.
Both production web builds and both local-provider server builds passed.
No Mail, Jobs or server Rust behavior changed in this continuation.

Decisions: use 400 Bulk folders and exactly 100,000 total Inbox messages for
the large profile; allow 30-minute setup and a two-hour total deadline; keep
an explicit ordinary-fixture profile for full-duration load results; report
only fixed assertion codes. These are test choices, not product decisions.
No UI changed. UX gaps closed or left are not applicable to this job.

READY FOR MERGE: no. The first receipt audit and large prerequisites failed,
and the requested reliability matrix remains incomplete. Evidence is on #1038
and #1037. No issue was closed. Private fixtures and keys were removed.

For the merge round (standard combined validation only): cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test -- --test-threads=4, and (cd apps/web && bun run check && bun run test -- --maxWorkers=2). These must validate the combined branch. The unrun #1038 scenarios above remain reliability work; this is not a deferral of their deliverable.

Cleanup: no generated private fixtures remain. Owned web build and .svelte-kit output was deleted. cargo clean output, verbatim:

     Removed 12213 files, 7.0GiB total

Working tree is clean. No push or deploy was run. No issue was closed. UX gaps closed/left: not applicable; no UI change.

Final report — mailstress-1038 HEAD: `0dcfc98aa4f8da18cda3dc5d4330d36fd3e54d64`. READY FOR MERGE: **no**. Built: corrected folder/message budgets, finite full-run setup margin, an explicit ordinary-fixture full-duration profile, and fixed private-safe receipt failure codes with a red/green regression. Existing command, MIME, flag and receipt assertions remain strict. Files: `tests/adversarial/mail_stress.mjs`, `mail_proxy.py`, `test_mail_proxy.py`, `test_mail_sync_provider.py`, and `mail-sync.md`; the `calternal-auth/src/store.rs` merge conflict retained the incoming comment over identical code. No production Mail, Jobs or server behavior changed. Live ordinary soak used profile source `a0192a88e` and the merged server built at `1b55bbfe0`. The later reporter change was unit-tested; the completed runs used the earlier reporter. The initial large attempt used the supplied Rust baseline `4f83aef2b`. No live result is claimed for an unrun scenario. ## Scenario results and verification This job runs the reliability checks. The earlier merge-round deferrals do not apply to this job. Results below do not certify the missing scenarios. The large fixture now has 400 Bulk folders. The namespace test counts the 80 deep-folder parents and special folders too. It stays below the provider's 512-folder limit. The fixture has 99,968 ordinary Inbox messages plus 32 sentinels, for a total of 100,000. The old total of 100,032 exceeded SELECT's 100,000-message limit. Production limits did not change. The first full attempt failed its 900-second initial-sync deadline. Inbox counts were `[63520,32,32]`. Two accounts had completed. Two accounts had reported an error during the attempt. A read-only check found no current account error before the deadline. The queue had seven completed jobs, one dead job, three leased jobs and one pending job. No lease had expired. The diagnostic restart also failed its 180-second completion deadline. Neither 30-minute command phase ran. This remains a failed result. The job merged `origin/dev` once, as required. The authentication conflict was a comment-only conflict with identical code. The incoming #1041 comment was retained. The production web app and local-provider server rebuilt. The continuation failed account creation with HTTP 504 on User 0. The provider cause is not established. Both attempts are recorded on #1037. The coordinator now allows 30 minutes for initial sync and two hours for the full run. The old 75-minute total budget had no setup or cleanup margin after a 15-minute prerequisite and two 30-minute phases. These test deadlines do not change production deadlines or turn the first failure into a pass. An explicit ordinary-fixture profile keeps the full session pool and both 30-minute command phases. It uses Inbox counts `[64,32,32]`, 16 Bulk folders, and no Boundary or deep folder. It never substitutes for the large fixture. The host selected 12 clients and reported 88 GiB free for this run. All three account calls returned 201; all App Password calls returned 200. Initial sync passed without a manual recovery restart or a reported error. Both full 30-minute command windows ran to completion. The overall process exited 1 because the first phase failed its receipt audit. The second phase passed. These are local shared-host results, not perf-VM measurements. ```sh CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 \ CALTERNAL_MAIL_STRESS_IMAGE=localhost/mail-sync-test-userns:latest \ CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/debug/calternal-server" \ TMPDIR="$PWD/target/tmp" node tests/adversarial/mail_stress.mjs ``` The coordinator has no hard-kill transfer, simultaneous mutation, upstream fault, malformed-input or Unicode phase. Its three-account phase uses three Users with one Connected Account each. It does not test three Connected Accounts in one User's unified view. The web check retains existing sessions; it does not perform a fresh sign-in during load. These gaps remain explicit. | Phase | Result | Elapsed with audit | Complete command cycles | Accepted APPEND / COPY / MOVE | Exact upstream / proxy receipts | |---|---|---|---|---|---| | One Connected Account, one User | FAIL | 1845.99 s | 16,693 | 20 / 20 / 20 | 14 / 14; one receipt assertion failed | | Three Connected Accounts, three Users | PASS | 1841.08 s | 17,302 | 20 / 20 / 20 | 20 / 20; no exception | Each phase used 12 clients. All required command loops completed without an exception. COPY had 180 and 221 clean NO waits; MOVE had 177 and 225. Every transfer then returned OK. All three retained web sessions and readiness returned 200 after each phase. A fresh sign-in during load was not checked. | Phase | First / last / maximum RSS (KiB) | First / last / maximum descriptors | First / last steady five-minute mean RSS (KiB) | |---|---|---|---| | One account | 153300 / 263064 / 263064 | 76 / 96 / 143 | 240795 / 244036 | | Three accounts | 263260 / 308720 / 308720 | 100 / 99 / 130 | 283910 / 289265 | The steady windows are seconds 60–360 and 1500–1800. Resource growth was small during those windows. The samples do not prove a permanent memory bound. No server crash or command-loop failure was observed in these phases. The first audit failed after 14 of 20 receipts, with a 45-second shared budget. It recorded only the assertion class. The cause is not established; do not assume deadline exhaustion or a cold-body failure. A separate read-only audit found all 20 Message-IDs in the right local live memberships and upstream folders. Drafts had zero, Archive one and Trash one per identity. Flagged state was intact in both projections. All 40 intentional upstream copies had exact MIME. The audit took 1.08 s. It did not recheck the six remaining exact bodies through the proxy, and it does not turn the original FAIL into PASS. The read-only three-account audit found 20 identities, distributed `[10,10,0]` across the Users. All local and upstream counts matched. All 40 upstream bodies were exact and flagged. No local membership crossed an owner or account, and no foreign upstream receipt appeared in any of the three accounts. Elapsed time was 2.98 s. User 2 had no transfer writes; full authorization coverage and three Connected Accounts in one User remain unverified. The existing real Dovecot regression passed in 21.41 s. It checks COPY followed by delete or MOVE, source removal, destination flags and repeated worker delivery without duplicate receipts. It does not test simultaneous mutations. The existing interrupted-lease restart regression also passed in 0.72 s. It does not prove a hard kill during a transfer. | Requested scenario, in risk order | Result | Evidence or gap | |---|---|---| | 6. Hard kill during transfers | Not run | Only the lease-recovery regression and graceful setup restart were checked | | 2. Concurrency correctness | Partial | Two writers, sequential transfer receipts and live worker replay passed; same-item races and folder lifecycle cases were not run | | 3. Upstream faults | Not run | Live sync errors and HTTP 504 were recorded on #1037; restart, drop, slow reply and UIDVALIDITY changes were not injected | | 7. Account separation | Partial | Three-User sentinels, labels, local ownership and upstream receipt absence passed; full matrix remains unverified | | 1. Sustained load | Mixed | Both 30-minute windows ran with 12 clients; first audit FAIL, second PASS; large prerequisites failed | | 4. Input robustness | Partial | Ordinary IMAP commands passed; pipelining, unfinished literals, invalid names, overlong lines, IDLE without DONE and failed-login cases were not run | | 5. Unicode names | Not run | No Unicode namespace scenario exists in the coordinator | These missing scenarios remain #1038 reliability work. They are not deferred to a merge round. No exploit-style protocol-abuse probe was added or run. The reporter now adds fixed codes for known receipt assertions. It still fails the run and retains exception-class counters. Unknown strings and non-string error arguments stay private. The regression failed before the change and then passed. The live soak used the earlier reporter, so its lost assertion detail cannot be recovered from the new code. No existing server status, assertion or body/flag check was relaxed. Verification excerpts, verbatim: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 37s test result: ok. 95 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 66.90s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 88 filtered out; finished in 21.41s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.72s Ran 12 tests in 1.043s OK ``` The first line is `cargo clippy -p calternal-auth --all-targets -- -D warnings`. The following lines are the Auth tests, the focused Mail receipt test, the Jobs restart test and the Python fixture tests. `cargo fmt --check` and `node --check tests/adversarial/mail_stress.mjs` exited 0 with no output. Both production web builds and both local-provider server builds passed. No Mail, Jobs or server Rust behavior changed in this continuation. Decisions: use 400 Bulk folders and exactly 100,000 total Inbox messages for the large profile; allow 30-minute setup and a two-hour total deadline; keep an explicit ordinary-fixture profile for full-duration load results; report only fixed assertion codes. These are test choices, not product decisions. No UI changed. UX gaps closed or left are not applicable to this job. READY FOR MERGE: no. The first receipt audit and large prerequisites failed, and the requested reliability matrix remains incomplete. Evidence is on #1038 and #1037. No issue was closed. Private fixtures and keys were removed. For the merge round (standard combined validation only): `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test -- --test-threads=4`, and `(cd apps/web && bun run check && bun run test -- --maxWorkers=2)`. These must validate the combined branch. The unrun #1038 scenarios above remain reliability work; this is not a deferral of their deliverable. Cleanup: no generated private fixtures remain. Owned web `build` and `.svelte-kit` output was deleted. `cargo clean` output, verbatim: ```text Removed 12213 files, 7.0GiB total ``` Working tree is clean. No push or deploy was run. No issue was closed. UX gaps closed/left: not applicable; no UI change.
kayg commented 2026-10-04 11:26:14 +00:00
Author
Owner
Copy link

Starting mailstress-a on job/mailstress-a, base 96bbb1b62. Scope: reproduce and fix the one-account 30-minute receipt failure and the 400-folder/100,000-message initial-sync failure. Local generated fixture only; count-only evidence, no credentials or message bodies.

Starting mailstress-a on job/mailstress-a, base 96bbb1b62. Scope: reproduce and fix the one-account 30-minute receipt failure and the 400-folder/100,000-message initial-sync failure. Local generated fixture only; count-only evidence, no credentials or message bodies.
kayg commented 2026-10-04 11:26:52 +00:00
Author
Owner
Copy link

Started mailstress-b on job/mailstress-b at base 96bbb1b62b. Scope: restart safety, ordinary concurrency, upstream recovery, Unicode and account separation on an isolated local fixture. Existing soak evidence does not cover these phases. No credentials or message bodies will be published.

Started mailstress-b on job/mailstress-b at base 96bbb1b62b38445a6a0dbbb4a82f1a3f4b2de9ec. Scope: restart safety, ordinary concurrency, upstream recovery, Unicode and account separation on an isolated local fixture. Existing soak evidence does not cover these phases. No credentials or message bodies will be published.
kayg commented 2026-10-04 11:31:16 +00:00
Author
Owner
Copy link

Finding (mailstress-a): inherited localhost/mail-sync-test-userns:latest is ARM64 on an x86-64 host. Fresh generated 100,000-message/400-Bulk-folder fixture: emulated login+LIST returned 487 folders after 36.012 s, exceeding Mail command deadline 30 s; native image on the same fixture returned login+LIST in 0.073 s and SELECT of 100,000 in 0.145 s. This establishes an environment cause for a large-creation timeout, not yet the old receipt assertion. Commits cf242b480 and 764be5b1a isolate the reruns, collect creation/sync RSS evidence and reject emulated images. Fresh full-duration reruns pending server build.

Finding (mailstress-a): inherited localhost/mail-sync-test-userns:latest is ARM64 on an x86-64 host. Fresh generated 100,000-message/400-Bulk-folder fixture: emulated login+LIST returned 487 folders after 36.012 s, exceeding Mail command deadline 30 s; native image on the same fixture returned login+LIST in 0.073 s and SELECT of 100,000 in 0.145 s. This establishes an environment cause for a large-creation timeout, not yet the old receipt assertion. Commits cf242b480 and 764be5b1a isolate the reruns, collect creation/sync RSS evidence and reject emulated images. Fresh full-duration reruns pending server build.
kayg commented 2026-10-04 11:38:28 +00:00
Author
Owner
Copy link

mailstress-b finding at 15114e0f2: the new restart audit used bracketed Message-ID equality against the Index. Provider sync stores the canonical bare Message-ID. A read-only query found one Drafts membership and one completed transfer after the first SIGKILL, while the old query found zero. Fixed both live-membership and pending-transfer queries to compare canonical forms; added a focused SQLite regression. This is a harness false negative, not a demonstrated product loss. The first live attempt remains failed; a corrected replay will use a fresh fixture.

mailstress-b finding at 15114e0f2: the new restart audit used bracketed Message-ID equality against the Index. Provider sync stores the canonical bare Message-ID. A read-only query found one Drafts membership and one completed transfer after the first SIGKILL, while the old query found zero. Fixed both live-membership and pending-transfer queries to compare canonical forms; added a focused SQLite regression. This is a harness false negative, not a demonstrated product loss. The first live attempt remains failed; a corrected replay will use a fresh fixture.
kayg commented 2026-10-04 11:43:33 +00:00
Author
Owner
Copy link

Native large fixture: all three account creations returned 201; latest User 0 took 14.092 s. Existing live Mail progress verified on the production build; six macOS-platform screenshots captured at 390/820/1440 in light/dark. Capture initially matched both mounted phone/desktop nav trees; harness now scopes the visible tree (65f475f23), without changing UI behavior. Full initial sync is running after the deliberate pre-lease-expiry restart, no account errors so far.

Native large fixture: all three account creations returned 201; latest User 0 took 14.092 s. Existing live Mail progress verified on the production build; six macOS-platform screenshots captured at 390/820/1440 in light/dark. Capture initially matched both mounted phone/desktop nav trees; harness now scopes the visible tree (65f475f23), without changing UI behavior. Full initial sync is running after the deliberate pre-lease-expiry restart, no account errors so far.
kayg commented 2026-10-04 11:52:01 +00:00
Author
Owner
Copy link

mailstress-b progress at 530d09a76: corrected local replay has passed APPEND at 0/100/500 ms, COPY at 0/100/500 ms, and MOVE at 0 ms after accepted replies. All seven queue-before-kill samples showed one queued transfer. Audit counts matched upstream, all three proxy views and the live Index. The 500 ms APPEND, 500 ms COPY and 0 ms MOVE audits needed automatic interrupted-lease recovery; no client replay or manual recovery restart ran. These are queued-delivery kills, not proven mid-literal or dispatched-receipt kills. Later cases now also audit all 128 original Inbox identities. Results are still in progress; no merge readiness claim.

mailstress-b progress at 530d09a76: corrected local replay has passed APPEND at 0/100/500 ms, COPY at 0/100/500 ms, and MOVE at 0 ms after accepted replies. All seven queue-before-kill samples showed one queued transfer. Audit counts matched upstream, all three proxy views and the live Index. The 500 ms APPEND, 500 ms COPY and 0 ms MOVE audits needed automatic interrupted-lease recovery; no client replay or manual recovery restart ran. These are queued-delivery kills, not proven mid-literal or dispatched-receipt kills. Later cases now also audit all 128 original Inbox identities. Results are still in progress; no merge readiness claim.
kayg commented 2026-10-04 11:53:54 +00:00
Author
Owner
Copy link

Large fixture continuation: PASS. Fresh native Dovecot: User 0 account creation 201 in 14.092 s; full initial sync after the deliberate lease-expiry restart completed in 732.712 s. Final Inbox counts [100000,32,32]. User 0 LIST has 487 folders (400 Bulk, 80 deep levels, six special folders and Boundary); all selectable folders complete as required by account backfill_complete. Sampled server peak RSS 339300 KiB (331.35 MiB); 10-second samples, local shared host. One account error was observed and cleared automatically; its class was not retained by the active coordinator. New commit 9d940d91d retains fixed error classes for subsequent runs. No manual recovery restart.

Existing User-visible progress passed on the production web build, macOS platform, 390/820/1440, light and dark. Screenshots for Claude review:

Large fixture continuation: PASS. Fresh native Dovecot: User 0 account creation 201 in 14.092 s; full initial sync after the deliberate lease-expiry restart completed in 732.712 s. Final Inbox counts [100000,32,32]. User 0 LIST has 487 folders (400 Bulk, 80 deep levels, six special folders and Boundary); all selectable folders complete as required by account backfill_complete. Sampled server peak RSS 339300 KiB (331.35 MiB); 10-second samples, local shared host. One account error was observed and cleared automatically; its class was not retained by the active coordinator. New commit 9d940d91d retains fixed error classes for subsequent runs. No manual recovery restart. Existing User-visible progress passed on the production web build, macOS platform, 390/820/1440, light and dark. Screenshots for Claude review: - [initial-sync-1440-dark.png](https://git.kayg.org/attachments/453431ac-4e4d-416a-a88e-c051253e2e72) - [initial-sync-1440-light.png](https://git.kayg.org/attachments/f70f6bee-a64a-4acf-85bd-0a7037297031) - [initial-sync-390-dark.png](https://git.kayg.org/attachments/0695b79a-9969-471e-b2e9-d78ed52327e3) - [initial-sync-390-light.png](https://git.kayg.org/attachments/78f56f11-8552-4489-822b-048c46f227c2) - [initial-sync-820-dark.png](https://git.kayg.org/attachments/567a76a9-eb5e-4a59-b66b-db6b8ccc0829) - [initial-sync-820-light.png](https://git.kayg.org/attachments/56871a4b-9252-4479-8d5c-489dd0e8fde8)
kayg commented 2026-10-04 11:58:32 +00:00
Author
Owner
Copy link

mailstress-b at adabd68ac: all twelve SIGKILL cases passed. MOVE-to-two-folders race returned NO/OK with exactly one Trash receipt. MOVE-vs-delete returned OK/NO with one Archive receipt. FETCH-vs-selective-EXPUNGE returned OK/OK, with no remaining receipt. Each race audit compared upstream, all three proxy views, the live Index, and all 128 original Inbox identities.

Folder lifecycle commands returned clean tagged NO: CREATE, RENAME while selected, and DELETE while another session held IDLE. The selected session still accepted NOOP. This is UNSUPPORTED coverage, not a successful folder race; filed #1055. The coordinator previously labelled its completed refusal check PASS even though the child correctly said UNSUPPORTED; adabd68ac preserves the child verdict in future summaries. Unicode rename/delete remain the same gap. No product defect or behavior change has been established in these cases. Remaining namespace and provider-recovery scenarios are still running or pending in this job.

mailstress-b at adabd68ac: all twelve SIGKILL cases passed. MOVE-to-two-folders race returned NO/OK with exactly one Trash receipt. MOVE-vs-delete returned OK/NO with one Archive receipt. FETCH-vs-selective-EXPUNGE returned OK/OK, with no remaining receipt. Each race audit compared upstream, all three proxy views, the live Index, and all 128 original Inbox identities. Folder lifecycle commands returned clean tagged NO: CREATE, RENAME while selected, and DELETE while another session held IDLE. The selected session still accepted NOOP. This is UNSUPPORTED coverage, not a successful folder race; filed #1055. The coordinator previously labelled its completed refusal check PASS even though the child correctly said UNSUPPORTED; adabd68ac preserves the child verdict in future summaries. Unicode rename/delete remain the same gap. No product defect or behavior change has been established in these cases. Remaining namespace and provider-recovery scenarios are still running or pending in this job.
kayg commented 2026-10-04 12:08:10 +00:00
Author
Owner
Copy link

mailstress-b provider subset, first attempt: a 35-second Dovecot pause plus immediate Dovecot restart passed the exact queued APPEND audit and all 128 original Inbox identities. No calternal restart or manual sync request ran. The subsequent UIDVALIDITY check failed with AssertionError; the old reporter suppressed its fixed assertion code. All three retained web sessions and readiness still returned 200. The whole subset exited 1. This is not an established provider-sync defect. A focused fresh-fixture replay now uses 5b2189e06, which adds fixed epoch assertion codes and a privacy regression. Unknown reply text remains suppressed. The failed result is retained in artifacts/mailstress-b/upstream-1.jsonl.

mailstress-b provider subset, first attempt: a 35-second Dovecot pause plus immediate Dovecot restart passed the exact queued APPEND audit and all 128 original Inbox identities. No calternal restart or manual sync request ran. The subsequent UIDVALIDITY check failed with AssertionError; the old reporter suppressed its fixed assertion code. All three retained web sessions and readiness still returned 200. The whole subset exited 1. This is not an established provider-sync defect. A focused fresh-fixture replay now uses 5b2189e06, which adds fixed epoch assertion codes and a privacy regression. Unknown reply text remains suppressed. The failed result is retained in artifacts/mailstress-b/upstream-1.jsonl.
kayg commented 2026-10-04 12:19:44 +00:00
Author
Owner
Copy link

Final mailstress-b report. Branch: job/mailstress-b. Base: 96bbb1b62b38445a6a0dbbb4a82f1a3f4b2de9ec. Head: cdadd0e69105ad2c6c3712dbe95dd7dfcc69a0b5.

Built: fixed local restart, two-session race, Unicode, User-pair namespace and provider recovery replays; receipt audits; privacy and fixture regressions. Files:

  • tests/adversarial/mail_proxy.py
  • tests/adversarial/mail_stress.mjs
  • tests/adversarial/mail_sync_provider.py
  • tests/adversarial/test_mail_proxy.py
  • tests/adversarial/test_mail_sync_provider.py
  • tests/adversarial/mail-sync.md

mailstress-b reliability results (#1038, 2026-10-04)

The main replay exited 0. The corrected provider subset exited 0. They used
three Users with one Connected Account each, a real local Dovecot TLS fixture,
and the server built from base 96bbb1b62 with mail-test-provider. No Rust
or UI behavior changed. No dependency or migration changed.

Assigned scenario Result Evidence and limit
APPEND, SIGKILL at 0 / 100 / 500 ms after OK PASS, queued delivery Three kills; one exact Drafts receipt each, upstream and in the proxy and live Index
COPY, SIGKILL at 0 / 100 / 500 ms after OK PASS, queued delivery Three kills; one Drafts source and one exact Archive receipt each
MOVE, SIGKILL at 0 / 100 / 500 ms after OK PASS, queued delivery Three kills; no Drafts source and one exact Trash receipt each
Selective EXPUNGE, SIGKILL at 0 / 100 / 500 ms after OK PASS, queued mutation Three kills; no remaining receipt in any checked folder
Kill during dispatched receipt or an upstream literal Not run All nine transfer samples were queued; all three EXPUNGE samples had one mutation
Two sessions MOVE one message to different folders PASS NO / OK; one Trash receipt, no source
MOVE versus delete on one message PASS OK / NO; one Archive receipt, no source
FETCH versus selective EXPUNGE PASS OK / OK; the receipt audit found no remaining message
RENAME while another session has the folder selected UNSUPPORTED Tagged NO; the selected session remained usable; #1055
DELETE while another session holds IDLE UNSUPPORTED Tagged NO; IDLE completed and NOOP worked; #1055
Dovecot paused for 35 s, then restarted PASS, recovery subset APPEND returned OK while unavailable; one exact receipt and all 128 original Inbox identities matched after restart
Dovecot restart inside a known provider command Not established The replay did not prove a provider command was in progress
Upstream TCP cut inside a literal Not run No dropped-literal fixture
Byte-rate-throttled upstream Not run A 35-second stall tested outage recovery, not a byte-rate throttle
UIDVALIDITY change PASS New epoch and committed live generation appeared without a manual sync request; new virtual UID; old Inbox UID returned no body; exact MIME matched upstream
Right-to-left folder name PASS, LIST / SELECT / MOVE One exact receipt; creation was upstream; rename and delete returned NO
Combining-mark folder name PASS, LIST / SELECT / MOVE One exact receipt; creation was upstream; rename and delete returned NO
Emoji folder name PASS, LIST / SELECT / MOVE One exact receipt; creation was upstream; rename and delete returned NO
NFC and NFD equivalent names PASS, distinct spellings Both folders remained selectable and each received its own exact message
Names that differ only by case PASS, distinct spellings Both folders remained selectable and each received its own exact message
Proxy CREATE / RENAME / DELETE of Unicode folders UNSUPPORTED Upstream created seven names per User; proxy folder administration remains #1055
Three-User LIST / SELECT / SEARCH / FETCH / COPY / APPEND matrix PASS First 72 checks used three special aliases; after epoch recovery, 222 checks covered every visible real-folder alias across all six ordered User pairs
Three Connected Accounts within one User Not run The fixture has one Connected Account per User; unified-view cross-account writes remain unverified
Two complete pipelined commands PASS Ordered NOOP and CAPABILITY tagged OK completions
Announced literal never completed Not run No unfinished-literal probe
Overlong command lines Not run No oversized-line probe
Invalid modified UTF-7 names Not run Only valid fixed Unicode spellings ran
IDLE without DONE Not run Folder lifecycle used an ordinary finite IDLE context
Failed-login flood and fresh web sign-in Not run Retained sessions were checked after provider faults; no fresh sign-in during failed logins

Each target audit checks all six special folders across all three proxy views
and upstream accounts. Unicode destination audits also check that real folder.
It compares exact MIME and live Index counts, and requires no pending transfer
for the target identity. COPY intentionally has two receipts in different
folders. MOVE has one. Accepted commands are never replayed.

The original-identity audit was added during this run. The first eight kill
results check their target identity only. The ninth kill and all EXPUNGE,
race and Unicode cases also check all 128 original Inbox identities through
both endpoints and the Index. A later successful audit does not change the
scope of an earlier one. Each audit checks its current target, not a full
history of all previous generated transfers.

Some kills interrupted leased Jobs. The audits waited for automatic lease
expiry and recovery; no second server restart or client replay was used.
These results establish queued-delivery restart safety. They do not establish
safety at every non-idempotent upstream receipt boundary.

The first main attempt failed the new Index audit because sync stores a bare
Message-ID and pending APPEND can store brackets. A real-audit regression now
covers both forms. The corrected main run passed. The first two provider
subsets failed the epoch fixture prerequisite. The fixed helper retained UID
mappings, keywords and MIME, rebuilt only the stopped Inbox indexes, and the
third subset passed. These failed attempts remain recorded as failures.
The epoch reporter now publishes only fixed assertion codes. Unknown strings
and non-string error arguments remain private.

All three retained web sessions and readiness returned 200 after the provider
subsets. This is not evidence of a fresh sign-in during an authentication flood.
No unexpected server exit was observed in the successful scoped cases.
The twelve SIGKILLs were intentional. No new product defect was established.
The harmless unsupported folder operations were filed as #1055.

Evidence stays in ignored artifacts/mailstress-b/: reliability-1.jsonl,
reliability-2.jsonl, upstream-1.jsonl, upstream-2.jsonl, upstream-3.jsonl,
and gate files. No credential, body or protocol transcript is in these reports.
All private fixture directories and keys were removed. The web build output
was removed. cargo clean reported:

     Removed 7238 files, 4.6GiB total

The required git fetch origin and git merge origin/dev ran once. Output:

Already up to date.

Final verification: cargo fmt --check,
node --check tests/adversarial/mail_stress.mjs and git diff --check
exited 0 with no output.
No Rust crate or web code changed, so per-crate clippy, Rust tests and web checks
are not applicable to this change. The production web build and the
local-provider server build passed before the live tests. The server output:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 19s

Python gate output, verbatim (TMPDIR was the worktree's target/tmp):

..................
----------------------------------------------------------------------
Ran 18 tests in 2.377s

OK
Prepared fixture: 2000 messages, TLS port 29963, stress=False
Prepared fixture: 1 messages, TLS port 29963, stress=True
Prepared fixture: 1 messages, TLS port 29963, stress=True
Prepared fixture: 10 messages, TLS port 29963, stress=True

Decisions: use the existing ordinary fixture and separate provider subset;
accept writes before the three fixed kill delays; allow 180 seconds for a
120-second interrupted lease and 420 seconds for normal five-minute provider
IDLE recovery; compare canonical Message-ID forms; retain control files while
changing only fixture UIDVALIDITY and rebuildable Inbox indexes. These are
test choices. No product behavior was selected. No performance profile was
required: this job changed only verification code and is not a performance issue.

UX gaps closed and left: not applicable; no UI changed. For the merge round:
no assigned scenario is deferred there. The unrun scenarios remain #1038
reliability work. No issue was closed. No push or deployment ran.

READY FOR MERGE: no. The scoped tests passed, but the complete requested
reliability matrix remains unverified.

Final mailstress-b report. Branch: `job/mailstress-b`. Base: `96bbb1b62b38445a6a0dbbb4a82f1a3f4b2de9ec`. Head: `cdadd0e69105ad2c6c3712dbe95dd7dfcc69a0b5`. Built: fixed local restart, two-session race, Unicode, User-pair namespace and provider recovery replays; receipt audits; privacy and fixture regressions. Files: - `tests/adversarial/mail_proxy.py` - `tests/adversarial/mail_stress.mjs` - `tests/adversarial/mail_sync_provider.py` - `tests/adversarial/test_mail_proxy.py` - `tests/adversarial/test_mail_sync_provider.py` - `tests/adversarial/mail-sync.md` ### mailstress-b reliability results (#1038, 2026-10-04) The main replay exited 0. The corrected provider subset exited 0. They used three Users with one Connected Account each, a real local Dovecot TLS fixture, and the server built from base `96bbb1b62` with `mail-test-provider`. No Rust or UI behavior changed. No dependency or migration changed. | Assigned scenario | Result | Evidence and limit | |---|---|---| | APPEND, SIGKILL at 0 / 100 / 500 ms after OK | PASS, queued delivery | Three kills; one exact Drafts receipt each, upstream and in the proxy and live Index | | COPY, SIGKILL at 0 / 100 / 500 ms after OK | PASS, queued delivery | Three kills; one Drafts source and one exact Archive receipt each | | MOVE, SIGKILL at 0 / 100 / 500 ms after OK | PASS, queued delivery | Three kills; no Drafts source and one exact Trash receipt each | | Selective EXPUNGE, SIGKILL at 0 / 100 / 500 ms after OK | PASS, queued mutation | Three kills; no remaining receipt in any checked folder | | Kill during dispatched receipt or an upstream literal | Not run | All nine transfer samples were queued; all three EXPUNGE samples had one mutation | | Two sessions MOVE one message to different folders | PASS | NO / OK; one Trash receipt, no source | | MOVE versus delete on one message | PASS | OK / NO; one Archive receipt, no source | | FETCH versus selective EXPUNGE | PASS | OK / OK; the receipt audit found no remaining message | | RENAME while another session has the folder selected | UNSUPPORTED | Tagged NO; the selected session remained usable; #1055 | | DELETE while another session holds IDLE | UNSUPPORTED | Tagged NO; IDLE completed and NOOP worked; #1055 | | Dovecot paused for 35 s, then restarted | PASS, recovery subset | APPEND returned OK while unavailable; one exact receipt and all 128 original Inbox identities matched after restart | | Dovecot restart inside a known provider command | Not established | The replay did not prove a provider command was in progress | | Upstream TCP cut inside a literal | Not run | No dropped-literal fixture | | Byte-rate-throttled upstream | Not run | A 35-second stall tested outage recovery, not a byte-rate throttle | | UIDVALIDITY change | PASS | New epoch and committed live generation appeared without a manual sync request; new virtual UID; old Inbox UID returned no body; exact MIME matched upstream | | Right-to-left folder name | PASS, LIST / SELECT / MOVE | One exact receipt; creation was upstream; rename and delete returned NO | | Combining-mark folder name | PASS, LIST / SELECT / MOVE | One exact receipt; creation was upstream; rename and delete returned NO | | Emoji folder name | PASS, LIST / SELECT / MOVE | One exact receipt; creation was upstream; rename and delete returned NO | | NFC and NFD equivalent names | PASS, distinct spellings | Both folders remained selectable and each received its own exact message | | Names that differ only by case | PASS, distinct spellings | Both folders remained selectable and each received its own exact message | | Proxy CREATE / RENAME / DELETE of Unicode folders | UNSUPPORTED | Upstream created seven names per User; proxy folder administration remains #1055 | | Three-User LIST / SELECT / SEARCH / FETCH / COPY / APPEND matrix | PASS | First 72 checks used three special aliases; after epoch recovery, 222 checks covered every visible real-folder alias across all six ordered User pairs | | Three Connected Accounts within one User | Not run | The fixture has one Connected Account per User; unified-view cross-account writes remain unverified | | Two complete pipelined commands | PASS | Ordered NOOP and CAPABILITY tagged OK completions | | Announced literal never completed | Not run | No unfinished-literal probe | | Overlong command lines | Not run | No oversized-line probe | | Invalid modified UTF-7 names | Not run | Only valid fixed Unicode spellings ran | | IDLE without DONE | Not run | Folder lifecycle used an ordinary finite IDLE context | | Failed-login flood and fresh web sign-in | Not run | Retained sessions were checked after provider faults; no fresh sign-in during failed logins | Each target audit checks all six special folders across all three proxy views and upstream accounts. Unicode destination audits also check that real folder. It compares exact MIME and live Index counts, and requires no pending transfer for the target identity. COPY intentionally has two receipts in different folders. MOVE has one. Accepted commands are never replayed. The original-identity audit was added during this run. The first eight kill results check their target identity only. The ninth kill and all EXPUNGE, race and Unicode cases also check all 128 original Inbox identities through both endpoints and the Index. A later successful audit does not change the scope of an earlier one. Each audit checks its current target, not a full history of all previous generated transfers. Some kills interrupted leased Jobs. The audits waited for automatic lease expiry and recovery; no second server restart or client replay was used. These results establish queued-delivery restart safety. They do not establish safety at every non-idempotent upstream receipt boundary. The first main attempt failed the new Index audit because sync stores a bare Message-ID and pending APPEND can store brackets. A real-audit regression now covers both forms. The corrected main run passed. The first two provider subsets failed the epoch fixture prerequisite. The fixed helper retained UID mappings, keywords and MIME, rebuilt only the stopped Inbox indexes, and the third subset passed. These failed attempts remain recorded as failures. The epoch reporter now publishes only fixed assertion codes. Unknown strings and non-string error arguments remain private. All three retained web sessions and readiness returned 200 after the provider subsets. This is not evidence of a fresh sign-in during an authentication flood. No unexpected server exit was observed in the successful scoped cases. The twelve SIGKILLs were intentional. No new product defect was established. The harmless unsupported folder operations were filed as #1055. Evidence stays in ignored `artifacts/mailstress-b/`: `reliability-1.jsonl`, `reliability-2.jsonl`, `upstream-1.jsonl`, `upstream-2.jsonl`, `upstream-3.jsonl`, and gate files. No credential, body or protocol transcript is in these reports. All private fixture directories and keys were removed. The web build output was removed. `cargo clean` reported: ```text Removed 7238 files, 4.6GiB total ``` The required `git fetch origin` and `git merge origin/dev` ran once. Output: ```text Already up to date. ``` Final verification: `cargo fmt --check`, `node --check tests/adversarial/mail_stress.mjs` and `git diff --check` exited 0 with no output. No Rust crate or web code changed, so per-crate clippy, Rust tests and web checks are not applicable to this change. The production web build and the local-provider server build passed before the live tests. The server output: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 19s ``` Python gate output, verbatim (`TMPDIR` was the worktree's `target/tmp`): ```text .................. ---------------------------------------------------------------------- Ran 18 tests in 2.377s OK Prepared fixture: 2000 messages, TLS port 29963, stress=False Prepared fixture: 1 messages, TLS port 29963, stress=True Prepared fixture: 1 messages, TLS port 29963, stress=True Prepared fixture: 10 messages, TLS port 29963, stress=True ``` Decisions: use the existing ordinary fixture and separate provider subset; accept writes before the three fixed kill delays; allow 180 seconds for a 120-second interrupted lease and 420 seconds for normal five-minute provider IDLE recovery; compare canonical Message-ID forms; retain control files while changing only fixture UIDVALIDITY and rebuildable Inbox indexes. These are test choices. No product behavior was selected. No performance profile was required: this job changed only verification code and is not a performance issue. UX gaps closed and left: not applicable; no UI changed. For the merge round: no assigned scenario is deferred there. The unrun scenarios remain #1038 reliability work. No issue was closed. No push or deployment ran. READY FOR MERGE: no. The scoped tests passed, but the complete requested reliability matrix remains unverified.
kayg commented 2026-10-04 12:22:29 +00:00
Author
Owner
Copy link

Continue #1038 reliability QA on branch job/mailstress-b, base/head cdadd0e69. Read CLAUDE.md, CONTEXT.md and DESIGN §45/49/53. Remaining scope: interrupted provider transfers, bounded slow-provider behavior, parser conformance/deadlines, login policy and three Connected Accounts under one User. Tests use private loopback fixtures and count/status-only evidence. No push, deploy or issue closure.

Continue #1038 reliability QA on branch `job/mailstress-b`, base/head `cdadd0e69`. Read CLAUDE.md, CONTEXT.md and DESIGN §45/49/53. Remaining scope: interrupted provider transfers, bounded slow-provider behavior, parser conformance/deadlines, login policy and three Connected Accounts under one User. Tests use private loopback fixtures and count/status-only evidence. No push, deploy or issue closure.
kayg commented 2026-10-04 12:34:39 +00:00
Author
Owner
Copy link

#1038 continuation finding: real TLS replay from base cdadd0e69 reported unfinished 1 MiB APPEND literal as EOF after 30.001 s (RSS delta 0 KiB), an overlong line as EOF after 0.005 s, and IDLE without DONE as EOF at the explicit local deadline. These are socket drops without IMAP BYE. The 50-command pipeline passed; an invalid modified-UTF-7 mailbox name returned tagged NO. A new focused Rust regression failed on the missing BYE before the fix. Root cause: CommandReader poisons failed reads and IDLE, but returns the error without sending a terminal protocol response. The fix emits only a fixed BYE, preserves the original error and never decodes the buffered bytes again. Gates and the corrected replay are pending.

#1038 continuation finding: real TLS replay from base `cdadd0e69` reported unfinished 1 MiB APPEND literal as EOF after 30.001 s (RSS delta 0 KiB), an overlong line as EOF after 0.005 s, and IDLE without DONE as EOF at the explicit local deadline. These are socket drops without IMAP BYE. The 50-command pipeline passed; an invalid modified-UTF-7 mailbox name returned tagged NO. A new focused Rust regression failed on the missing BYE before the fix. Root cause: CommandReader poisons failed reads and IDLE, but returns the error without sending a terminal protocol response. The fix emits only a fixed BYE, preserves the original error and never decodes the buffered bytes again. Gates and the corrected replay are pending.
kayg commented 2026-10-04 12:39:28 +00:00
Author
Owner
Copy link

One-account full rerun: PASS. Native fixture, 12 clients, full 1,800-second command window; elapsed with audit 1,807.33 s; 17,451 complete cycles; 20 APPEND / 20 COPY / 20 MOVE accepted; 20 upstream and 20 proxy receipts passed exact MIME, Flagged and folder counts. No probe failure; readiness and all three retained web sessions 200. RSS first/last/max 212336/123484/238980 KiB; descriptors 96/89/146. A comparison build reclaimed resident pages during this shared-host run; no permanent memory-bound claim. Audit now uses one read-only session per User without changing the 45-second deadline or receipt assertions (0ab4bb8c6, slow-sign-in regression 4c50ee44b). A full pre-1037 sync comparison with the original audit is running to recover the old assertion code; the historical log alone cannot identify it.

One-account full rerun: PASS. Native fixture, 12 clients, full 1,800-second command window; elapsed with audit 1,807.33 s; 17,451 complete cycles; 20 APPEND / 20 COPY / 20 MOVE accepted; 20 upstream and 20 proxy receipts passed exact MIME, Flagged and folder counts. No probe failure; readiness and all three retained web sessions 200. RSS first/last/max 212336/123484/238980 KiB; descriptors 96/89/146. A comparison build reclaimed resident pages during this shared-host run; no permanent memory-bound claim. Audit now uses one read-only session per User without changing the 45-second deadline or receipt assertions (0ab4bb8c6, slow-sign-in regression 4c50ee44b). A full pre-1037 sync comparison with the original audit is running to recover the old assertion code; the historical log alone cannot identify it.
kayg commented 2026-10-04 12:49:58 +00:00
Author
Owner
Copy link

Committed the parser/IDLE fix at 7d099f777 and the dispatched-command/login terminal response fix at e213d4bf0. Current verification harness head: 44d8bc6b3. New fault-boundary regressions prove cuts after 4,096 literal bytes, dispatch before close, and unaffected fixture-1 traffic. Required fetch/merge ran once: Already up to date.

Gates passed (OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4): fmt, per-crate IMAP and server clippy/tests. Server test output:

test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 23.66s

Python harness output:

.........................
----------------------------------------------------------------------
Ran 25 tests in 3.863s

OK

The first real cut-FETCH passed (4,096 bytes, tagged NO, zero partial cache rows, exact 1 MiB body recovered). Corrected full remaining-scenario replay is pending the local-provider binary rebuild. No readiness claim yet.

Committed the parser/IDLE fix at `7d099f777` and the dispatched-command/login terminal response fix at `e213d4bf0`. Current verification harness head: `44d8bc6b3`. New fault-boundary regressions prove cuts after 4,096 literal bytes, dispatch before close, and unaffected fixture-1 traffic. Required fetch/merge ran once: `Already up to date.` Gates passed (OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4): fmt, per-crate IMAP and server clippy/tests. Server test output: ``` test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 23.66s ``` Python harness output: ``` ......................... ---------------------------------------------------------------------- Ran 25 tests in 3.863s OK ``` The first real cut-FETCH passed (4,096 bytes, tagged NO, zero partial cache rows, exact 1 MiB body recovered). Corrected full remaining-scenario replay is pending the local-provider binary rebuild. No readiness claim yet.
kayg commented 2026-10-04 12:54:09 +00:00
Author
Owner
Copy link

Evidence correction: the first progress screenshot set used Inbox and could include generated body previews. I removed only those six mailstress-a attachments. Their links in my earlier comment are withdrawn. The coordinator now captures a real empty Bulk folder while showing account-wide live progress (no placeholder data and no message bodies). A replacement macOS 390/820/1440 light/dark set will follow after the active legacy comparison; one browser at a time. The completed large-sync and native 30-minute acceptance results remain valid.

Evidence correction: the first progress screenshot set used Inbox and could include generated body previews. I removed only those six mailstress-a attachments. Their links in my earlier comment are withdrawn. The coordinator now captures a real empty Bulk folder while showing account-wide live progress (no placeholder data and no message bodies). A replacement macOS 390/820/1440 light/dark set will follow after the active legacy comparison; one browser at a time. The completed large-sync and native 30-minute acceptance results remain valid.
kayg commented 2026-10-04 13:03:14 +00:00
Author
Owner
Copy link

The existing #1037 regression also passed on shipping source: cargo test -p calternal-plugin-mail --features test-provider interleaved_flags_fetch_is_ignored_after_uid_set_validation. Verbatim: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 91 filtered out; finished in 0.31s. Shipping Rust source is unchanged. Legacy comparison is still running the original full-duration command/audit behavior; no shortened substitution.

The existing #1037 regression also passed on shipping source: cargo test -p calternal-plugin-mail --features test-provider interleaved_flags_fetch_is_ignored_after_uid_set_validation. Verbatim: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 91 filtered out; finished in 0.31s. Shipping Rust source is unchanged. Legacy comparison is still running the original full-duration command/audit behavior; no shortened substitution.
kayg commented 2026-10-04 13:07:18 +00:00
Author
Owner
Copy link

Remaining replay at server e213d4bf0 / harness 1df0b8c27 passed all conformance, cut-FETCH, dispatch-FETCH and slow-provider checks. Exact results: unfinished literal BYE at 30.002 s, RSS delta 0 KiB; overlong line BYE at 0.002 s; invalid UTF-7 SELECT NO; IDLE BYE at 3.015 s; all 50 pipeline completions. A literal cut after 4,096 bytes returned NO in 0.335 s; dispatch cut returned NO in 0.308 s; both had zero partial cache rows and exact 1 MiB recovery. At 4 KiB/s, FETCH returned NO at 30.04 s. Another User's distinct upstream body miss took 0.408 s during throttle versus 1.374 s before.

APPEND fault is a delivery-recovery gap, filed #1056. The accepted complete draft remained exact at 1,048,576 bytes. Its journal stayed dispatched, with 1,048,576 raw bytes and no upstream receipt after 65 s; zero incomplete provider temporary files. DESIGN §53's no-replay receipt rule prevents a blind second APPEND. This invariant remains unchanged. READY FOR MERGE remains no.

The replay then failed the 180-second setup wait for the fifth active Connected Account: five rows, four synced, with three sync Jobs leased and pending work. Mail sync max_concurrency=3 covers five-minute IDLE as well as backfill. 07d4f23e6 adds an accounts/login-only replay and the established 420-second provider allowance. It keeps the other Users' accounts active; it does not request manual sync or repeat passed faults. The failed prerequisite remains recorded in completion-3.jsonl. The remaining account/login subset is running.

Remaining replay at server `e213d4bf0` / harness `1df0b8c27` passed all conformance, cut-FETCH, dispatch-FETCH and slow-provider checks. Exact results: unfinished literal BYE at 30.002 s, RSS delta 0 KiB; overlong line BYE at 0.002 s; invalid UTF-7 SELECT NO; IDLE BYE at 3.015 s; all 50 pipeline completions. A literal cut after 4,096 bytes returned NO in 0.335 s; dispatch cut returned NO in 0.308 s; both had zero partial cache rows and exact 1 MiB recovery. At 4 KiB/s, FETCH returned NO at 30.04 s. Another User's distinct upstream body miss took 0.408 s during throttle versus 1.374 s before. APPEND fault is a delivery-recovery gap, filed #1056. The accepted complete draft remained exact at 1,048,576 bytes. Its journal stayed `dispatched`, with 1,048,576 raw bytes and no upstream receipt after 65 s; zero incomplete provider temporary files. DESIGN §53's no-replay receipt rule prevents a blind second APPEND. This invariant remains unchanged. READY FOR MERGE remains no. The replay then failed the 180-second setup wait for the fifth active Connected Account: five rows, four synced, with three sync Jobs leased and pending work. Mail sync max_concurrency=3 covers five-minute IDLE as well as backfill. `07d4f23e6` adds an accounts/login-only replay and the established 420-second provider allowance. It keeps the other Users' accounts active; it does not request manual sync or repeat passed faults. The failed prerequisite remains recorded in completion-3.jsonl. The remaining account/login subset is running.
kayg commented 2026-10-04 13:22:22 +00:00
Author
Owner
Copy link

Recovered receipt failure by full legacy replay: FAIL after 15/20 upstream and 15/20 local receipts, fixed code receipt-budget-exhausted. Exact assertion: Receipt phase budget exhausted. Replay used pre-1037 sync + fetch type, the inherited ARM64 fixture on x86-64 and the original per-receipt audit; 12 clients completed the full 1,800-second window and 17,536 cycles, all 20 APPEND/COPY/MOVE sequences accepted. Elapsed with audit 1,846.58 s; no command-loop failure; readiness and all retained web sessions 200. The original 14/20 log lost its assertion detail, so this establishes the reproduced failure shape rather than recovering a missing stack trace. Root: audit work consumes the one shared 45-second deadline, amplified by emulation and repeated local TLS/sign-in. Current native + session reuse full rerun already PASS at 1,807.33 s, 20/20 exact upstream and proxy receipts. Deadlines and data assertions were not relaxed. Body-free screenshot replacement/full large confirmation now running.

Recovered receipt failure by full legacy replay: FAIL after 15/20 upstream and 15/20 local receipts, fixed code receipt-budget-exhausted. Exact assertion: Receipt phase budget exhausted. Replay used pre-1037 sync + fetch type, the inherited ARM64 fixture on x86-64 and the original per-receipt audit; 12 clients completed the full 1,800-second window and 17,536 cycles, all 20 APPEND/COPY/MOVE sequences accepted. Elapsed with audit 1,846.58 s; no command-loop failure; readiness and all retained web sessions 200. The original 14/20 log lost its assertion detail, so this establishes the reproduced failure shape rather than recovering a missing stack trace. Root: audit work consumes the one shared 45-second deadline, amplified by emulation and repeated local TLS/sign-in. Current native + session reuse full rerun already PASS at 1,807.33 s, 20/20 exact upstream and proxy receipts. Deadlines and data assertions were not relaxed. Body-free screenshot replacement/full large confirmation now running.
kayg commented 2026-10-04 13:40:41 +00:00
Author
Owner
Copy link

Body-free large confirmation: PASS. Native fixture on shipping Rust source; User 0 creation 201 in 11.763 s; full initial sync after setup restart 912.750 s; all three accounts complete; peak RSS including kernel high-water mark 336488 KiB (328.60 MiB); no account error observed. Existing live progress verified on the production build in a real empty Bulk folder. The DOM explicitly had zero message-body previews. These replace the withdrawn Inbox captures.

macOS platform, all required widths and themes, for Claude review:

Body-free large confirmation: PASS. Native fixture on shipping Rust source; User 0 creation 201 in 11.763 s; full initial sync after setup restart 912.750 s; all three accounts complete; peak RSS including kernel high-water mark 336488 KiB (328.60 MiB); no account error observed. Existing live progress verified on the production build in a real empty Bulk folder. The DOM explicitly had zero message-body previews. These replace the withdrawn Inbox captures. macOS platform, all required widths and themes, for Claude review: - [initial-sync-1440-dark.png](https://git.kayg.org/attachments/6d0ecddd-b456-44e8-8872-c7c25439a8a1) - [initial-sync-1440-light.png](https://git.kayg.org/attachments/50cf7039-3c99-40b4-9762-7afb0aa35710) - [initial-sync-390-dark.png](https://git.kayg.org/attachments/285ea32f-533f-4be0-9aa8-c6dbbafb4a30) - [initial-sync-390-light.png](https://git.kayg.org/attachments/1a475bba-f9e2-4cb3-93cb-fc5e45da01f1) - [initial-sync-820-dark.png](https://git.kayg.org/attachments/3b85b6a3-708c-4ed5-8b94-7c8c4f7adbf7) - [initial-sync-820-light.png](https://git.kayg.org/attachments/0c84cd7c-85e3-4c83-9791-f3a965e87ed9)
kayg commented 2026-10-04 13:52:14 +00:00
Author
Owner
Copy link

mailstress-a final report. READY FOR MERGE: yes (the two scoped prerequisites).

Branch: job/mailstress-a. Base: 96bbb1b62b. Head: a16257ec5c.
Fetched and merged origin/dev once before final checks: Already up to date.
No push, deploy or issue closure.

Built and fixed:

  • Require a fixture image that matches the host architecture. The inherited ARM64 image on x86-64 took 36.012 seconds for login and LIST, beyond the existing 30-second mail command deadline. Native creation succeeds. Keep the 512-folder guard: the large fixture has 487 LIST entries and 408 selectable folders.
  • Reuse one read-only receipt connection pair per User within the unchanged shared 45-second budget. Every MIME, Flagged, source/destination count and local projection check remains required. Add regression tests for slow sign-in, body/flag mismatch and architecture refusal.
  • Select full phases separately, record creation time and RSS/VmHWM, and capture existing live progress on the real production build at phone/tablet/desktop widths, light/dark, macOS platform.
  • No shipping Rust or UI module changed. No dependency or migration added. No test expectation or product limit weakened.
Scenario Result Evidence
Native one-account, full 1,800-second window, 12 clients PASS 17,451 complete cycles; all 20 APPEND/COPY/MOVE sequences; 20/20 upstream and local exact receipts; 1,807.33 seconds with audit
Native large fixture, first full run PASS Creation HTTP 201 in 14.092 seconds; backfill 732.712 seconds; sampled peak RSS 339,300 KiB (331.35 MiB)
Native large fixture, final confirmation PASS Creation HTTP 201 in 11.763 seconds; backfill 912.750 seconds; peak RSS/VmHWM 336,488 KiB (328.60 MiB); no account errors
Legacy provider code, emulated image, original audit, full 1,800-second window Diagnostic FAIL reproduced 17,536 cycles; all writes accepted; 15/20 upstream and local receipts; exact assertion Receipt phase budget exhausted, fixed code receipt-budget-exhausted; 1,846.58 seconds total
Same legacy server/image, corrected audit, 300-second comparison PASS 20/20 upstream and local receipts; one local audit connection; 329.77 seconds including audit

The native full run retained all three authenticated browser sessions and readiness at HTTP 200. RSS first/last/max: 212,336 / 123,484 / 238,980 KiB. FDs: 96 / 89 / 146. Resident pages were reclaimed during a concurrent diagnostic build; the lower final RSS is not a claimed performance improvement or permanent bound.

Both large runs completed all three accounts with final Inbox counts [100000,32,32], last_sync_at and backfill_complete. The large account also has five Boundary items beyond the 100,000 Inbox messages. No manual recovery restart ran after the deliberate setup restart. The first large run had a transient unclassified account error that cleared; the final run had none. Fixed error classes now retain safe evidence for later runs.

Historical failure qualification: the original 14/20 log lost its assertion detail. The full replay reproduces the same failure shape and identifies the budget assertion; it cannot restore the historical stack. This failure is audit overhead, not a missing-write assertion. Both the unchanged fixed server and the corrected harness completed the required native 30-minute phase.

Files:

  • tests/adversarial/mail_stress.mjs
  • tests/adversarial/mail_proxy.py
  • tests/adversarial/test_mail_proxy.py
  • tests/adversarial/test_mail_sync_provider.py
  • tests/adversarial/mail-sync.md

Gate output, verbatim:
cargo fmt --check: exit 0, no output.
node --check tests/adversarial/mail_stress.mjs: exit 0, no output.
git diff --check: exit 0, no output.
Python focused harness tests:

Ran 15 tests in 1.992s

OK

Inherited provider regression (cargo test -p calternal-plugin-mail --features test-provider interleaved_flags_fetch_is_ignored_after_uid_set_validation):

test sync::tests::interleaved_flags_fetch_is_ignored_after_uid_set_validation ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 91 filtered out; finished in 0.31s

Production web build:

> Using @sveltejs/adapter-static
  Wrote site to "build"
  ✔ done

Fixture-enabled server build:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 19s

No changed Rust crate or production web code required clippy/full crate or web suites under the current verification policy. The requested reliability scenarios ran to completion; none are deferred. Module/function comments were re-read. Atomic commits retain each working step.

UX gaps closed: the coordinator proves live account-wide backfill progress remains visible at 390, 820 and 1440 pixels, in both themes. It uses a real empty Bulk folder and asserts zero message-body previews. Earlier generated-preview captures were removed from this issue and replaced. Final captures for Claude review:

UX gaps left / known gaps: Claude visual review is pending. This scoped round does not certify all keyboard, touch or screen-reader flows, the sibling job's scenarios, or an indefinite memory bound. Measurements are finite local shared-host observations. The lost historical assertion remains unavailable.

Decisions: optional phase selection preserves default phases; use native fixtures; reuse read-only audit sessions under the original deadline; include VmHWM in peak evidence; capture progress from a real empty folder. These are harness choices; no new product design decision was needed.

Evidence remains in ignored artifacts/mailstress-a/: one-account-native.log, native-summary.json, large-native-round3.log, large-body-free.log, legacy-comparison.log, legacy-summary.json, legacy-reused-audit.log, legacy-reused-summary.json, focused check logs and final PNGs. No credentials or message bodies are reported or attached. No screenshots are committed.

Cleanup: cargo clean removed 8,685 files / 5.6 GiB; web build output and diagnostic server copies were removed. Worktree is clean.

READY FOR MERGE: yes

mailstress-a final report. READY FOR MERGE: yes (the two scoped prerequisites). Branch: job/mailstress-a. Base: 96bbb1b62b38445a6a0dbbb4a82f1a3f4b2de9ec. Head: a16257ec5c8a1fb1705c61dd0a8ed6a46dbd2dbf. Fetched and merged origin/dev once before final checks: Already up to date. No push, deploy or issue closure. Built and fixed: - Require a fixture image that matches the host architecture. The inherited ARM64 image on x86-64 took 36.012 seconds for login and LIST, beyond the existing 30-second mail command deadline. Native creation succeeds. Keep the 512-folder guard: the large fixture has 487 LIST entries and 408 selectable folders. - Reuse one read-only receipt connection pair per User within the unchanged shared 45-second budget. Every MIME, Flagged, source/destination count and local projection check remains required. Add regression tests for slow sign-in, body/flag mismatch and architecture refusal. - Select full phases separately, record creation time and RSS/VmHWM, and capture existing live progress on the real production build at phone/tablet/desktop widths, light/dark, macOS platform. - No shipping Rust or UI module changed. No dependency or migration added. No test expectation or product limit weakened. | Scenario | Result | Evidence | | --- | --- | --- | | Native one-account, full 1,800-second window, 12 clients | PASS | 17,451 complete cycles; all 20 APPEND/COPY/MOVE sequences; 20/20 upstream and local exact receipts; 1,807.33 seconds with audit | | Native large fixture, first full run | PASS | Creation HTTP 201 in 14.092 seconds; backfill 732.712 seconds; sampled peak RSS 339,300 KiB (331.35 MiB) | | Native large fixture, final confirmation | PASS | Creation HTTP 201 in 11.763 seconds; backfill 912.750 seconds; peak RSS/VmHWM 336,488 KiB (328.60 MiB); no account errors | | Legacy provider code, emulated image, original audit, full 1,800-second window | Diagnostic FAIL reproduced | 17,536 cycles; all writes accepted; 15/20 upstream and local receipts; exact assertion `Receipt phase budget exhausted`, fixed code `receipt-budget-exhausted`; 1,846.58 seconds total | | Same legacy server/image, corrected audit, 300-second comparison | PASS | 20/20 upstream and local receipts; one local audit connection; 329.77 seconds including audit | The native full run retained all three authenticated browser sessions and readiness at HTTP 200. RSS first/last/max: 212,336 / 123,484 / 238,980 KiB. FDs: 96 / 89 / 146. Resident pages were reclaimed during a concurrent diagnostic build; the lower final RSS is not a claimed performance improvement or permanent bound. Both large runs completed all three accounts with final Inbox counts [100000,32,32], last_sync_at and backfill_complete. The large account also has five Boundary items beyond the 100,000 Inbox messages. No manual recovery restart ran after the deliberate setup restart. The first large run had a transient unclassified account error that cleared; the final run had none. Fixed error classes now retain safe evidence for later runs. Historical failure qualification: the original 14/20 log lost its assertion detail. The full replay reproduces the same failure shape and identifies the budget assertion; it cannot restore the historical stack. This failure is audit overhead, not a missing-write assertion. Both the unchanged fixed server and the corrected harness completed the required native 30-minute phase. Files: - tests/adversarial/mail_stress.mjs - tests/adversarial/mail_proxy.py - tests/adversarial/test_mail_proxy.py - tests/adversarial/test_mail_sync_provider.py - tests/adversarial/mail-sync.md Gate output, verbatim: `cargo fmt --check`: exit 0, no output. `node --check tests/adversarial/mail_stress.mjs`: exit 0, no output. `git diff --check`: exit 0, no output. Python focused harness tests: ```text Ran 15 tests in 1.992s OK ``` Inherited provider regression (`cargo test -p calternal-plugin-mail --features test-provider interleaved_flags_fetch_is_ignored_after_uid_set_validation`): ```text test sync::tests::interleaved_flags_fetch_is_ignored_after_uid_set_validation ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 91 filtered out; finished in 0.31s ``` Production web build: ```text > Using @sveltejs/adapter-static Wrote site to "build" ✔ done ``` Fixture-enabled server build: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 19s ``` No changed Rust crate or production web code required clippy/full crate or web suites under the current verification policy. The requested reliability scenarios ran to completion; none are deferred. Module/function comments were re-read. Atomic commits retain each working step. UX gaps closed: the coordinator proves live account-wide backfill progress remains visible at 390, 820 and 1440 pixels, in both themes. It uses a real empty Bulk folder and asserts zero message-body previews. Earlier generated-preview captures were removed from this issue and replaced. Final captures for Claude review: - [initial-sync-1440-dark.png](https://git.kayg.org/attachments/6d0ecddd-b456-44e8-8872-c7c25439a8a1) - [initial-sync-1440-light.png](https://git.kayg.org/attachments/50cf7039-3c99-40b4-9762-7afb0aa35710) - [initial-sync-390-dark.png](https://git.kayg.org/attachments/285ea32f-533f-4be0-9aa8-c6dbbafb4a30) - [initial-sync-390-light.png](https://git.kayg.org/attachments/1a475bba-f9e2-4cb3-93cb-fc5e45da01f1) - [initial-sync-820-dark.png](https://git.kayg.org/attachments/3b85b6a3-708c-4ed5-8b94-7c8c4f7adbf7) - [initial-sync-820-light.png](https://git.kayg.org/attachments/0c84cd7c-85e3-4c83-9791-f3a965e87ed9) UX gaps left / known gaps: Claude visual review is pending. This scoped round does not certify all keyboard, touch or screen-reader flows, the sibling job's scenarios, or an indefinite memory bound. Measurements are finite local shared-host observations. The lost historical assertion remains unavailable. Decisions: optional phase selection preserves default phases; use native fixtures; reuse read-only audit sessions under the original deadline; include VmHWM in peak evidence; capture progress from a real empty folder. These are harness choices; no new product design decision was needed. Evidence remains in ignored artifacts/mailstress-a/: one-account-native.log, native-summary.json, large-native-round3.log, large-body-free.log, legacy-comparison.log, legacy-summary.json, legacy-reused-audit.log, legacy-reused-summary.json, focused check logs and final PNGs. No credentials or message bodies are reported or attached. No screenshots are committed. Cleanup: cargo clean removed 8,685 files / 5.6 GiB; web build output and diagnostic server copies were removed. Worktree is clean. READY FOR MERGE: yes
kayg commented 2026-10-04 13:59:06 +00:00
Author
Owner
Copy link

The completed three-Connected-Accounts replay found a transfer delivery defect. All 21 local checks passed across three retained sessions, and six cross-account MOVE commands returned NO. Upstream audits found two exact successful moves, but account 0 stayed queued through nine retry attempts. A count-only Index query showed that this queued receipt included the observed server-owned \Recent flag. The worker sent captured flags through the strict client-intent validator, which rejected the receipt before dispatch.

Regression imported_recent_is_not_captured_in_transfer_intent failed before the fix: captured flags included \Recent. The fix normalizes observed flags during capture and legacy journal replay, dropping only \Recent. Client flag validation remains strict. Mail and server gates and a corrected local replay follow. The interrupted APPEND delivery gap is separately filed in #1056; the full local draft is retained, and ambiguous dispatch must not be blindly replayed.

The completed three-Connected-Accounts replay found a transfer delivery defect. All 21 local checks passed across three retained sessions, and six cross-account MOVE commands returned NO. Upstream audits found two exact successful moves, but account 0 stayed queued through nine retry attempts. A count-only Index query showed that this queued receipt included the observed server-owned \\Recent flag. The worker sent captured flags through the strict client-intent validator, which rejected the receipt before dispatch. Regression `imported_recent_is_not_captured_in_transfer_intent` failed before the fix: captured flags included \\Recent. The fix normalizes observed flags during capture and legacy journal replay, dropping only \\Recent. Client flag validation remains strict. Mail and server gates and a corrected local replay follow. The interrupted APPEND delivery gap is separately filed in #1056; the full local draft is retained, and ambiguous dispatch must not be blindly replayed.
kayg commented 2026-10-04 14:06:43 +00:00
Author
Owner
Copy link

The transient-flag fix is committed at 77c9195c4. It normalizes observed flags at transfer capture and legacy receipt replay, while preserving strict client validation. The COPY/MOVE regression reproduced the old capture defect and now passes. The strict-client/legacy-observation regression also passes.

Gates passed with OPENSSL_NO_VENDOR=1 and low-memory Cargo settings: calternal-plugin-mail clippy -D warnings; 86 tests passed, 6 ignored; calternal-server clippy -D warnings; 170 tests passed, 6 ignored. The corrected feature server build passed. The fresh three-session account replay uses this build and is waiting for ordinary sync slots after adding Connected Accounts through the real route. The earlier parser/fault results stand; the APPEND recovery requirement remains failing and filed in #1056.

The transient-flag fix is committed at 77c9195c4. It normalizes observed flags at transfer capture and legacy receipt replay, while preserving strict client validation. The COPY/MOVE regression reproduced the old capture defect and now passes. The strict-client/legacy-observation regression also passes. Gates passed with OPENSSL_NO_VENDOR=1 and low-memory Cargo settings: calternal-plugin-mail clippy -D warnings; 86 tests passed, 6 ignored; calternal-server clippy -D warnings; 170 tests passed, 6 ignored. The corrected feature server build passed. The fresh three-session account replay uses this build and is waiting for ordinary sync slots after adding Connected Accounts through the real route. The earlier parser/fault results stand; the APPEND recovery requirement remains failing and filed in #1056.
kayg commented 2026-10-04 14:13:13 +00:00
Author
Owner
Copy link

Final continuation report for mailstress-b. Branch job/mailstress-b; starting head cdadd0e69; final head aa5d4f8bcf80aacd5670427c98d2a988ab48b732. READY FOR MERGE: no. Interrupted upstream APPEND delivery recovery remains failing and is filed in #1056.

Built: bounded loopback TLS fault fixture; remaining conformance, slow-provider, login-policy and three-Connected-Account replays; fixed missing terminal responses in parser/IDLE, command deadlines and rate refusals; fixed transient Recent stalling accepted COPY/MOVE delivery. Each product fix has focused regression coverage. No existing expectation was relaxed.

Files:

  • crates/calternal-imap/src/wire.rs
  • crates/calternal-imap/tests/session.rs
  • crates/calternal-imap/tests/wire.rs
  • crates/calternal-server/src/notes_imap.rs
  • crates/plugins/mail/src/proxy.rs
  • crates/plugins/mail/src/proxy_mutations.rs
  • crates/plugins/mail/src/proxy_transfers.rs
  • tests/adversarial/mail-sync.md
  • tests/adversarial/mail_fault_provider.py
  • tests/adversarial/mail_proxy.py
  • tests/adversarial/mail_stress.mjs
  • tests/adversarial/test_mail_fault_provider.py
  • tests/adversarial/test_mail_proxy.py

Remaining reliability results (#1038, 2026-10-04)

The local Dovecot fixture is native amd64, version 2.4.1-4 (7d8c0e5759).
The bodies are generated fixtures. No body, credential or protocol transcript
is included in this report. These results extend the earlier table; they do
not replace its recorded failures or expand the scope of the SIGKILL tests.

Assigned check Result Evidence
Upstream closes inside a 1 MiB FETCH literal PASS Fixture cut after 4,096 forwarded bytes. Tagged NO at 0.335 s; zero partial MIME cache rows. Exact full MIME recovered at 0.642 s.
Upstream closes after FETCH dispatch PASS Fixture confirmed dispatch before closing. Tagged NO at 0.308 s; zero partial MIME cache rows. Exact full MIME recovered at 0.640 s.
Upstream closes inside a 1 MiB APPEND literal FAIL, delivery recovery (#1056) Full local draft accepted; fixture forwarded 4,096 bytes. After service restoration and 65 s, no upstream receipt. Exact local draft and dispatched journal both retain 1,048,576 bytes; zero provider temporary files.
Upstream sends 4 KiB/s PASS Tagged NO at 30.040 s; zero partial MIME cache rows. Exact recovery at 0.538 s. Another User's uncached upstream FETCH took 0.408 s during the fault versus 1.374 s before it. Local timings, not a performance baseline.
Announced 1 MiB literal, no payload PASS after fix BYE at 30.002 s; measured RSS change 0 KiB. Framing allocates for received chunks, not the announced size. Focused timeout regression poisons the parser.
Command above 8 KiB limit PASS after fix BYE at 0.002 s; no input echoed. Focused regression checks rejection and reader poisoning.
Invalid modified-UTF-7 mailbox name PASS Tagged NO at 0.003 s.
IDLE without DONE PASS after fix BYE at 3.015 s using the explicit local three-second limit. Production remains 30 minutes.
50 pipelined commands PASS Fifty NOOP tagged OK completions.
Thirty wrong-password LOGIN attempts PASS after fix Seven tagged NO authentication failures, then 23 tagged NO UNAVAILABLE plus BYE rate refusals. All three retained IMAP sessions answered OK. Fresh passkey sign-in, readiness and all three retained web sessions returned 200.
Three Connected Accounts in one User PASS after fix Three retained sessions; 21 local checks, including six foreign MOVE refusals. Three own-account moves had exact upstream MIME, no foreign receipt and no source; recovery audit completed in 2.474 s.

completion-3.jsonl records the parser, transfer and slow-provider results.
accounts-3.jsonl records the failed upstream MOVE audit before the Recent
fix. The corrected account and login replay is recorded in accounts-4.jsonl. All fixture paths
and evidence files are below the worktree; evidence files are gitignored.

The first parser replay closed timed-out reads, oversized frames and expired
IDLE with bare EOF. The framing layer now sends a fixed bounded BYE and keeps
the failed reader poisoned. The server sends tagged NO and BYE on a dispatched
command deadline and on rate-limited LOGIN. Each fix has a focused regression.

The account replay found an accepted MOVE that stayed queued through nine
retry attempts. The saved flags included \Recent. Worker validation rejected
that flag before dispatch. Capture and legacy receipt replay now discard only
that transient observed flag. Client intent still rejects it. The new COPY
and MOVE capture regression failed before this fix and passes after it. A
second regression checks legacy normalization and strict client validation.
See RFC 3501 §2.3.2
for the session-only flag and its exclusion from STORE and APPEND.

The interrupted APPEND leaves a complete draft, not a partial message.
Automatic upstream recovery still fails. The receipt journal cannot safely
replay an unknown dispatched APPEND. Its no-replay regression remains intact.
Issue #1056 records the evidence and the need for a safe recovery path.
Do not treat retained local MIME as a successful upstream receipt.

Decisions for this replay: use a loopback TLS relay for counted faults; keep
production deadlines except for the explicit local IDLE limit; give normal
sync and transfer recovery 420 seconds; keep three sessions for one User and
all other Users' accounts enabled. No dependency, migration or UI changed.
DESIGN §53 gives a User one IMAP login for all Connected Accounts. Account
isolation here means a selected real folder and a transfer destination remain
within their Connected Account; it does not hide the User's other accounts
from LIST. These checks do not certify all RFC grammar or native-client behavior.

Verification output below is verbatim. Cargo commands use the required low-memory
settings, OPENSSL_NO_VENDOR=1, and the assigned target directory. The
production web build ran before server gates. No web source changed.

cargo fmt --check, node --check tests/adversarial/mail_stress.mjs, and
git diff --check exited 0 with no output.

cargo clippy -p calternal-imap --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 10s

cargo test -p calternal-imap:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 8.31s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.24s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.31s
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 34.55s

cargo test -p calternal-plugin-mail -- --test-threads=4:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 10.16s
test result: ok. 86 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 3.88s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.37s

cargo test -p calternal-server -- --test-threads=4:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 39.41s
test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.96s

cargo build -p calternal-server --features mail-test-provider:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 47.96s

python3 -m unittest discover -s tests/adversarial -p 'test_mail*.py':

..........................
----------------------------------------------------------------------
Ran 26 tests in 2.204s

OK
Prepared fixture: 2000 messages, TLS port 29963, stress=False
Prepared fixture: 1 messages, TLS port 29963, stress=True
Prepared fixture: 1 messages, TLS port 29963, stress=True
Prepared fixture: 10 messages, TLS port 29963, stress=True

Known gaps: #1056 retains complete draft MIME but cannot safely replay an ambiguous dispatched APPEND. This is not established local data loss or a partial-message leak. The profile keeps the upstream receipt assertion failing. The prior folder-administration gap remains #1055 and was outside this continuation. Full RFC grammar and native-client certification are not claimed.

UX gaps closed: none added; no UI changed. UX gap left: the retained interrupted draft has no delivery recovery action (#1056).

Decisions: test-only loopback relay with exact 1 MiB messages and counted cuts; explicit three-second local IDLE timeout, production unchanged; 420-second ordinary sync/recovery budget; per-DESIGN account-folder isolation under one User login, with three retained sessions. Preserve the no-replay invariant for uncertain APPEND rather than risk duplicate receipts. No dependency or migration changes; no performance profile because this is reliability verification, not a performance issue.

For the merge round: no assigned scenario was deferred. Requested checks ran to a recorded result, including the remaining failure. Screenshots are not applicable: no UI changed.

git fetch origin and git merge origin/dev ran once before final Rust gates:

Already up to date.

Cleanup: the coordinator removed its private Users, TLS keys, fixture directories, browser, container and server. The web production build output was removed. Cargo cleanup output follows. Working tree is clean; no push, deploy, merge of the job branch, or issue closure was done.

     Removed 16948 files, 9.0GiB total
Final continuation report for mailstress-b. Branch `job/mailstress-b`; starting head `cdadd0e69`; final head `aa5d4f8bcf80aacd5670427c98d2a988ab48b732`. READY FOR MERGE: **no**. Interrupted upstream APPEND delivery recovery remains failing and is filed in #1056. Built: bounded loopback TLS fault fixture; remaining conformance, slow-provider, login-policy and three-Connected-Account replays; fixed missing terminal responses in parser/IDLE, command deadlines and rate refusals; fixed transient Recent stalling accepted COPY/MOVE delivery. Each product fix has focused regression coverage. No existing expectation was relaxed. Files: - `crates/calternal-imap/src/wire.rs` - `crates/calternal-imap/tests/session.rs` - `crates/calternal-imap/tests/wire.rs` - `crates/calternal-server/src/notes_imap.rs` - `crates/plugins/mail/src/proxy.rs` - `crates/plugins/mail/src/proxy_mutations.rs` - `crates/plugins/mail/src/proxy_transfers.rs` - `tests/adversarial/mail-sync.md` - `tests/adversarial/mail_fault_provider.py` - `tests/adversarial/mail_proxy.py` - `tests/adversarial/mail_stress.mjs` - `tests/adversarial/test_mail_fault_provider.py` - `tests/adversarial/test_mail_proxy.py` ### Remaining reliability results (#1038, 2026-10-04) The local Dovecot fixture is native amd64, version `2.4.1-4 (7d8c0e5759)`. The bodies are generated fixtures. No body, credential or protocol transcript is included in this report. These results extend the earlier table; they do not replace its recorded failures or expand the scope of the SIGKILL tests. | Assigned check | Result | Evidence | |---|---|---| | Upstream closes inside a 1 MiB FETCH literal | PASS | Fixture cut after 4,096 forwarded bytes. Tagged NO at 0.335 s; zero partial MIME cache rows. Exact full MIME recovered at 0.642 s. | | Upstream closes after FETCH dispatch | PASS | Fixture confirmed dispatch before closing. Tagged NO at 0.308 s; zero partial MIME cache rows. Exact full MIME recovered at 0.640 s. | | Upstream closes inside a 1 MiB APPEND literal | FAIL, delivery recovery (#1056) | Full local draft accepted; fixture forwarded 4,096 bytes. After service restoration and 65 s, no upstream receipt. Exact local draft and dispatched journal both retain 1,048,576 bytes; zero provider temporary files. | | Upstream sends 4 KiB/s | PASS | Tagged NO at 30.040 s; zero partial MIME cache rows. Exact recovery at 0.538 s. Another User's uncached upstream FETCH took 0.408 s during the fault versus 1.374 s before it. Local timings, not a performance baseline. | | Announced 1 MiB literal, no payload | PASS after fix | BYE at 30.002 s; measured RSS change 0 KiB. Framing allocates for received chunks, not the announced size. Focused timeout regression poisons the parser. | | Command above 8 KiB limit | PASS after fix | BYE at 0.002 s; no input echoed. Focused regression checks rejection and reader poisoning. | | Invalid modified-UTF-7 mailbox name | PASS | Tagged NO at 0.003 s. | | IDLE without DONE | PASS after fix | BYE at 3.015 s using the explicit local three-second limit. Production remains 30 minutes. | | 50 pipelined commands | PASS | Fifty NOOP tagged OK completions. | | Thirty wrong-password LOGIN attempts | PASS after fix | Seven tagged NO authentication failures, then 23 tagged NO UNAVAILABLE plus BYE rate refusals. All three retained IMAP sessions answered OK. Fresh passkey sign-in, readiness and all three retained web sessions returned 200. | | Three Connected Accounts in one User | PASS after fix | Three retained sessions; 21 local checks, including six foreign MOVE refusals. Three own-account moves had exact upstream MIME, no foreign receipt and no source; recovery audit completed in 2.474 s. | `completion-3.jsonl` records the parser, transfer and slow-provider results. `accounts-3.jsonl` records the failed upstream MOVE audit before the Recent fix. The corrected account and login replay is recorded in `accounts-4.jsonl`. All fixture paths and evidence files are below the worktree; evidence files are gitignored. The first parser replay closed timed-out reads, oversized frames and expired IDLE with bare EOF. The framing layer now sends a fixed bounded BYE and keeps the failed reader poisoned. The server sends tagged NO and BYE on a dispatched command deadline and on rate-limited LOGIN. Each fix has a focused regression. The account replay found an accepted MOVE that stayed queued through nine retry attempts. The saved flags included `\Recent`. Worker validation rejected that flag before dispatch. Capture and legacy receipt replay now discard only that transient observed flag. Client intent still rejects it. The new COPY and MOVE capture regression failed before this fix and passes after it. A second regression checks legacy normalization and strict client validation. See [RFC 3501 §2.3.2](https://www.rfc-editor.org/rfc/rfc3501.html#section-2.3.2) for the session-only flag and its exclusion from STORE and APPEND. The interrupted APPEND leaves a complete draft, not a partial message. Automatic upstream recovery still fails. The receipt journal cannot safely replay an unknown dispatched APPEND. Its no-replay regression remains intact. Issue #1056 records the evidence and the need for a safe recovery path. Do not treat retained local MIME as a successful upstream receipt. Decisions for this replay: use a loopback TLS relay for counted faults; keep production deadlines except for the explicit local IDLE limit; give normal sync and transfer recovery 420 seconds; keep three sessions for one User and all other Users' accounts enabled. No dependency, migration or UI changed. DESIGN §53 gives a User one IMAP login for all Connected Accounts. Account isolation here means a selected real folder and a transfer destination remain within their Connected Account; it does not hide the User's other accounts from LIST. These checks do not certify all RFC grammar or native-client behavior. Verification output below is verbatim. Cargo commands use the required low-memory settings, `OPENSSL_NO_VENDOR=1`, and the assigned target directory. The production web build ran before server gates. No web source changed. `cargo fmt --check`, `node --check tests/adversarial/mail_stress.mjs`, and `git diff --check` exited 0 with no output. `cargo clippy -p calternal-imap --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 10s ``` `cargo test -p calternal-imap`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 8.31s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.24s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.31s test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s ``` `cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 34.55s ``` `cargo test -p calternal-plugin-mail -- --test-threads=4`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 10.16s test result: ok. 86 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 3.88s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.37s ``` `cargo test -p calternal-server -- --test-threads=4`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 39.41s test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.96s ``` `cargo build -p calternal-server --features mail-test-provider`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 47.96s ``` `python3 -m unittest discover -s tests/adversarial -p 'test_mail*.py'`: ```text .......................... ---------------------------------------------------------------------- Ran 26 tests in 2.204s OK Prepared fixture: 2000 messages, TLS port 29963, stress=False Prepared fixture: 1 messages, TLS port 29963, stress=True Prepared fixture: 1 messages, TLS port 29963, stress=True Prepared fixture: 10 messages, TLS port 29963, stress=True ``` Known gaps: #1056 retains complete draft MIME but cannot safely replay an ambiguous dispatched APPEND. This is not established local data loss or a partial-message leak. The profile keeps the upstream receipt assertion failing. The prior folder-administration gap remains #1055 and was outside this continuation. Full RFC grammar and native-client certification are not claimed. UX gaps closed: none added; no UI changed. UX gap left: the retained interrupted draft has no delivery recovery action (#1056). Decisions: test-only loopback relay with exact 1 MiB messages and counted cuts; explicit three-second local IDLE timeout, production unchanged; 420-second ordinary sync/recovery budget; per-DESIGN account-folder isolation under one User login, with three retained sessions. Preserve the no-replay invariant for uncertain APPEND rather than risk duplicate receipts. No dependency or migration changes; no performance profile because this is reliability verification, not a performance issue. For the merge round: no assigned scenario was deferred. Requested checks ran to a recorded result, including the remaining failure. Screenshots are not applicable: no UI changed. `git fetch origin` and `git merge origin/dev` ran once before final Rust gates: ```text Already up to date. ``` Cleanup: the coordinator removed its private Users, TLS keys, fixture directories, browser, container and server. The web production build output was removed. Cargo cleanup output follows. Working tree is clean; no push, deploy, merge of the job branch, or issue closure was done. ```text Removed 16948 files, 9.0GiB total ```
kayg commented 2026-10-04 14:52:47 +00:00
Author
Owner
Copy link

READY FOR MERGE: yes, for #1056 and its requested quick regressions.

Built interrupted draft APPEND recovery. The worker searches the pinned folder for its operation keyword, then resolves an uncertain draft by exact Message-ID, RFC822.SIZE and INTERNALDATE. A unique metadata candidate must also have exact MIME bytes. Checked absence permits draft re-delivery; a receipt confirms delivery without another APPEND. Missing identity, incomplete metadata, duplicate candidates or different content retain the local draft. Existing Jobs supply bounded retries, exponential backoff and visible failure. COPY/MOVE keep their marker-only no-replay invariant.

Branch: job/mailstress-b. Base: aa5d4f8bc. Merged job/mailstress-a a16257ec5 in 4577dc530, preserving both fixture fixes and phase selection. Fetched origin and merged origin/dev c39ffe5d9 before final gates (already up to date). Head: 6ea7361a30. No push, deploy or issue closure.

Files: crates/plugins/mail/src/proxy_transfers.rs; crates/plugins/mail/src/proxy.rs (minimal pub(crate) exposure of the existing bounded MIME fetch helper); docs/DESIGN.md §53; tests/adversarial/mail_fault_provider.py; tests/adversarial/test_mail_fault_provider.py; tests/adversarial/mail_stress.mjs; tests/adversarial/mail-sync.md. No dependency or migration change.

Requested scenario Result Evidence
APPEND: drop before upstream upload PASS One 1,048,576-byte client upload, no client replay; exactly one upstream copy, exact proxy MIME, empty journal
APPEND: drop mid-literal PASS Cut at 4,096 bytes; exactly one complete upstream copy, exact proxy MIME, empty journal
APPEND: upstream OK lost PASS Tagged OK withheld; exactly one complete upstream copy, exact proxy MIME, empty journal
SIGKILL APPEND / COPY / MOVE / EXPUNGE PASS 100 ms per command; exact receipts and 128 original Inbox identities across three Users
MOVE versus MOVE PASS NO / OK; one destination receipt
MOVE versus delete PASS OK / NO; one destination receipt
FETCH versus EXPUNGE PASS OK / OK; no remaining receipt
Interrupted FETCH PASS Dispatch cut and literal cut; NO with zero partial-body rows, then complete OK
Account separation PASS Three sessions, 21 checks, three exact upstream moves for three Connected Accounts under one User
Final health PASS Readiness 200 and all three retained web sessions 200

Evidence: artifacts/mailstress-b/repair-final.jsonl exited 0 on the exact-MIME-check build. Its three APPEND audits and all four kills plus races pass. Initial sync completed in 126.586 s, peak sampled RSS 208128 KiB, no account errors. artifacts/mailstress-b/repair-live.jsonl contains the earlier passing FETCH/account checks and all three APPEND boundaries. That combined run ended in a coordinator failure after the login-policy fixture filled its ten-attempt IP window and the next transfer phase attempted a fresh login. The coordinator now resets this fixture policy between phases. The failed run is retained; the final focused run skips the already-passed policy/account/FETCH phases. No existing product expectation changed. Six TLS fixture tests prove the three upload fault boundaries. The wire regression covers absence, exact match, substring mismatch, size/date mismatch, duplicate matches, missing identity, EOF and an older equal-length draft with identical metadata.

Decisions: reuse the journal and existing 100-attempt backoff queue; inspect at most 32 Message-ID candidates; require an exact MIME match for a unique metadata candidate; preserve missing-ID MIME instead of synthesizing identity. The owner-requested draft-only negative-search exception is recorded in DESIGN §53. No new UI is required for retained failures because Jobs already exposes owner-bound failed delivery.

Known gaps: ambiguous or missing identities remain retained rather than automatically delivered. The SIGKILL samples interrupted queued delivery, not an observed upstream commit. The broader #1038 load/platform matrix was not rerun; this report certifies the repair and requested quick regressions. The complete combined coordinator path after the policy reset was not repeated. UX gaps closed/left: not applicable; no UI changed. For the merge round: no requested repair scenario is deferred. This is not a performance issue; no perf-VM or Mac run was required.

Production web build and mail-test-provider server build passed. node --check tests/adversarial/mail_stress.mjs and git diff --check passed. Module/function comments were re-read. Cleanup completed: cargo clean removed 16948 files, 9.0GiB; web build output removed; worktree clean. The final commit only updates comments and verification documentation.

Verbatim gate summaries (all commands exited 0). cargo fmt --check produced no output. OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and worktree TMPDIR were set.

cargo clippy -p calternal-imap --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.20s

cargo test -p calternal-imap

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.88s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.26s

cargo test -p calternal-plugin-mail

test result: ok. 87 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 3.81s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.69s

cargo test -p calternal-server

test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.60s

python3 -m unittest discover -s tests/adversarial -p 'test_mail*.py'

...............................
----------------------------------------------------------------------
Ran 31 tests in 2.806s

OK
READY FOR MERGE: yes, for #1056 and its requested quick regressions. Built interrupted draft APPEND recovery. The worker searches the pinned folder for its operation keyword, then resolves an uncertain draft by exact Message-ID, RFC822.SIZE and INTERNALDATE. A unique metadata candidate must also have exact MIME bytes. Checked absence permits draft re-delivery; a receipt confirms delivery without another APPEND. Missing identity, incomplete metadata, duplicate candidates or different content retain the local draft. Existing Jobs supply bounded retries, exponential backoff and visible failure. COPY/MOVE keep their marker-only no-replay invariant. Branch: job/mailstress-b. Base: aa5d4f8bc. Merged job/mailstress-a a16257ec5 in 4577dc530, preserving both fixture fixes and phase selection. Fetched origin and merged origin/dev c39ffe5d9 before final gates (already up to date). Head: 6ea7361a30690ef56983f4b6991a13a3841a276a. No push, deploy or issue closure. Files: crates/plugins/mail/src/proxy_transfers.rs; crates/plugins/mail/src/proxy.rs (minimal pub(crate) exposure of the existing bounded MIME fetch helper); docs/DESIGN.md §53; tests/adversarial/mail_fault_provider.py; tests/adversarial/test_mail_fault_provider.py; tests/adversarial/mail_stress.mjs; tests/adversarial/mail-sync.md. No dependency or migration change. | Requested scenario | Result | Evidence | |---|---|---| | APPEND: drop before upstream upload | PASS | One 1,048,576-byte client upload, no client replay; exactly one upstream copy, exact proxy MIME, empty journal | | APPEND: drop mid-literal | PASS | Cut at 4,096 bytes; exactly one complete upstream copy, exact proxy MIME, empty journal | | APPEND: upstream OK lost | PASS | Tagged OK withheld; exactly one complete upstream copy, exact proxy MIME, empty journal | | SIGKILL APPEND / COPY / MOVE / EXPUNGE | PASS | 100 ms per command; exact receipts and 128 original Inbox identities across three Users | | MOVE versus MOVE | PASS | NO / OK; one destination receipt | | MOVE versus delete | PASS | OK / NO; one destination receipt | | FETCH versus EXPUNGE | PASS | OK / OK; no remaining receipt | | Interrupted FETCH | PASS | Dispatch cut and literal cut; NO with zero partial-body rows, then complete OK | | Account separation | PASS | Three sessions, 21 checks, three exact upstream moves for three Connected Accounts under one User | | Final health | PASS | Readiness 200 and all three retained web sessions 200 | Evidence: artifacts/mailstress-b/repair-final.jsonl exited 0 on the exact-MIME-check build. Its three APPEND audits and all four kills plus races pass. Initial sync completed in 126.586 s, peak sampled RSS 208128 KiB, no account errors. artifacts/mailstress-b/repair-live.jsonl contains the earlier passing FETCH/account checks and all three APPEND boundaries. That combined run ended in a coordinator failure after the login-policy fixture filled its ten-attempt IP window and the next transfer phase attempted a fresh login. The coordinator now resets this fixture policy between phases. The failed run is retained; the final focused run skips the already-passed policy/account/FETCH phases. No existing product expectation changed. Six TLS fixture tests prove the three upload fault boundaries. The wire regression covers absence, exact match, substring mismatch, size/date mismatch, duplicate matches, missing identity, EOF and an older equal-length draft with identical metadata. Decisions: reuse the journal and existing 100-attempt backoff queue; inspect at most 32 Message-ID candidates; require an exact MIME match for a unique metadata candidate; preserve missing-ID MIME instead of synthesizing identity. The owner-requested draft-only negative-search exception is recorded in DESIGN §53. No new UI is required for retained failures because Jobs already exposes owner-bound failed delivery. Known gaps: ambiguous or missing identities remain retained rather than automatically delivered. The SIGKILL samples interrupted queued delivery, not an observed upstream commit. The broader #1038 load/platform matrix was not rerun; this report certifies the repair and requested quick regressions. The complete combined coordinator path after the policy reset was not repeated. UX gaps closed/left: not applicable; no UI changed. For the merge round: no requested repair scenario is deferred. This is not a performance issue; no perf-VM or Mac run was required. Production web build and mail-test-provider server build passed. node --check tests/adversarial/mail_stress.mjs and git diff --check passed. Module/function comments were re-read. Cleanup completed: cargo clean removed 16948 files, 9.0GiB; web build output removed; worktree clean. The final commit only updates comments and verification documentation. Verbatim gate summaries (all commands exited 0). cargo fmt --check produced no output. OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and worktree TMPDIR were set. `cargo clippy -p calternal-imap --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.20s ``` `cargo test -p calternal-imap` ```text test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.88s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.26s ``` `cargo test -p calternal-plugin-mail` ```text test result: ok. 87 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 3.81s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.69s ``` `cargo test -p calternal-server` ```text test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.60s ``` `python3 -m unittest discover -s tests/adversarial -p 'test_mail*.py'` ```text ............................... ---------------------------------------------------------------------- Ran 31 tests in 2.806s OK ```
kayg commented 2026-10-04 15:49:55 +00:00
Author
Owner
Copy link

Started mail round #1038. Branch job/mailround-1038; base 4082669f7. Integrating job/mailstress-b at 6ea7361a3. The merge has conflicts in IMAP, provider sync, auth, Mail UI and generated contracts. I will preserve both feature sets, renumber Mail migrations, extend the production-schema upgrade test, then run the specified crate gates and full two-phase stress/fault pass. No push or deploy.

Started mail round #1038. Branch `job/mailround-1038`; base `4082669f7`. Integrating `job/mailstress-b` at `6ea7361a3`. The merge has conflicts in IMAP, provider sync, auth, Mail UI and generated contracts. I will preserve both feature sets, renumber Mail migrations, extend the production-schema upgrade test, then run the specified crate gates and full two-phase stress/fault pass. No push or deploy.
kayg commented 2026-10-04 16:04:26 +00:00
Author
Owner
Copy link

Integration findings on #1038:

  • Incoming Mail migrations are already 0012–0017, after 7b 0010–0011. Auth has a version-0013 collision; Mail usage moves to 0014, preserving Share invites at 0013.
  • The two Mail events implementations and registrations would conflict. The combined branch uses one owner-filtered, coalesced mail-change stream for sync and proxy commits.
  • The 7b bounded expunge prune must retain a pending transfer message and refresh preferred UID rows inside the transaction. The existing pending-receipt regression now executes that bounded production path with its original receipt assertions.
  • Device routing must forward the Notes fast STATUS/revision path. STATUS UNSEEN becomes a counter in the shared status type; Notes adds a zero counter (its existing Seen behavior). This is a minimal public contract addition in the Notes crate.
  • The Mail reader must preserve the incoming native-edit updates through the 7b retained cache. A shared stream refresh now reconciles bounded visible rows without blanking content or resetting read timers.
  • 31 focused Python harness tests passed. Rust is compiling the cold dependency graph; the web check found a stale performance ledger anchor in the integrated input. No stress acceptance is claimed yet.
Integration findings on #1038: - Incoming Mail migrations are already 0012–0017, after 7b 0010–0011. Auth has a version-0013 collision; Mail usage moves to 0014, preserving Share invites at 0013. - The two Mail events implementations and registrations would conflict. The combined branch uses one owner-filtered, coalesced `mail-change` stream for sync and proxy commits. - The 7b bounded expunge prune must retain a pending transfer message and refresh preferred UID rows inside the transaction. The existing pending-receipt regression now executes that bounded production path with its original receipt assertions. - Device routing must forward the Notes fast STATUS/revision path. STATUS UNSEEN becomes a counter in the shared status type; Notes adds a zero counter (its existing Seen behavior). This is a minimal public contract addition in the Notes crate. - The Mail reader must preserve the incoming native-edit updates through the 7b retained cache. A shared stream refresh now reconciles bounded visible rows without blanking content or resetting read timers. - 31 focused Python harness tests passed. Rust is compiling the cold dependency graph; the web check found a stale performance ledger anchor in the integrated input. No stress acceptance is claimed yet.
kayg commented 2026-10-04 16:11:51 +00:00
Author
Owner
Copy link

Correction to integration commit ed71a9322: the IMAP test had not finished when its message said tests passed. Its completed result was FAILED. 9 passed; 1 failed in the session suite. epoch_reset_requires_reselection_before_fetch found that the 7b revision fast path left the old selection active after UIDVALIDITY changed. The current patch clears that selection, using the same rule as the snapshot path; the existing expectation is unchanged. Server cargo check --features mail-test-provider passed. Focused web tests passed (3 files, 14 tests).

Fetched origin and merged origin/dev once before combined gates: Already up to date. Origin/dev is c39ffe5d9.

Correction to integration commit `ed71a9322`: the IMAP test had not finished when its message said tests passed. Its completed result was `FAILED. 9 passed; 1 failed` in the session suite. `epoch_reset_requires_reselection_before_fetch` found that the 7b revision fast path left the old selection active after UIDVALIDITY changed. The current patch clears that selection, using the same rule as the snapshot path; the existing expectation is unchanged. Server `cargo check --features mail-test-provider` passed. Focused web tests passed (3 files, 14 tests). Fetched origin and merged origin/dev once before combined gates: `Already up to date.` Origin/dev is `c39ffe5d9`.
kayg commented 2026-10-04 16:48:33 +00:00
Author
Owner
Copy link

Combined-branch verification finding (head 64a65c105; server binary built from the combined ed71a9322 tree plus the epoch invalidation fix):

  • Focused coordinator: all three real Users and accounts registered; initial sync PASS in 29.394 s. Both short command phases FAIL. APPEND, exact draft FETCH and STORE succeeded, but COPY returned NO until the 45-second receipt deadline (215 refusals in each phase). Web sessions remained valid. This is functional failure, not SLOW-only.
  • Restart replay: kill-APPEND-0, kill-APPEND-100 and kill-APPEND-500 PASS, with exact receipt audits. kill-COPY-0 FAIL after 98 audit attempts, fixed code audit-count. The coordinator stopped at its existing first-failed-audit assertion; remaining cases still require execution.
  • Auth gates: 120 passed, 0 failed, 3 ignored. Database gates passed. Notes gates passed. Server tests found a compile-only Arc comparison mismatch; correcting the representation preserves the expected string.

The full large-fixture coordinator is running. Refusal diagnostics now expose only fixed categories; unknown wire text stays private. READY FOR STAGING remains no until the receipt failures are resolved or recorded and the full requested verification completes.

Combined-branch verification finding (head 64a65c105; server binary built from the combined ed71a9322 tree plus the epoch invalidation fix): - Focused coordinator: all three real Users and accounts registered; initial sync PASS in 29.394 s. Both short command phases FAIL. APPEND, exact draft FETCH and STORE succeeded, but COPY returned NO until the 45-second receipt deadline (215 refusals in each phase). Web sessions remained valid. This is functional failure, not SLOW-only. - Restart replay: kill-APPEND-0, kill-APPEND-100 and kill-APPEND-500 PASS, with exact receipt audits. kill-COPY-0 FAIL after 98 audit attempts, fixed code audit-count. The coordinator stopped at its existing first-failed-audit assertion; remaining cases still require execution. - Auth gates: 120 passed, 0 failed, 3 ignored. Database gates passed. Notes gates passed. Server tests found a compile-only Arc<str> comparison mismatch; correcting the representation preserves the expected string. The full large-fixture coordinator is running. Refusal diagnostics now expose only fixed categories; unknown wire text stays private. READY FOR STAGING remains no until the receipt failures are resolved or recorded and the full requested verification completes.
kayg commented 2026-10-04 17:09:16 +00:00
Author
Owner
Copy link

Further combined-branch findings:

  • The preserved device chooser reset its selection correctly, but its Sync with Your Devices request still sent only CalDAV and Notes despite naming Mail. The request now includes an independent full Mail grant. The actual form regression passed once; the expanded file passed both authority-label cases and hit its default 5-second timeout in the unchanged grant case on the later busy-host run. No existing test timeout was changed.
  • Credential labels now keep a narrower plugin grant authoritative. A CalDAV-only read credential is described as Calendar, Reminders & Contacts with Can view, instead of claiming Mail, Notes and write access.
  • Combined Server clippy passed: Finished dev profile [unoptimized + debuginfo] target(s) in 16m 59s.
  • The Server suite reported 214 passed, 2 failed, 9 ignored. One failure is the existing 15-second startup-backfill deadline in its subprocess wrapper; its expectation is unchanged. The other is the new Mail preservation assertion from this job: migration 0014 intentionally sets the old metadata cursor to NULL, but the assertion decoded it as i64 and expected the old value 3. I changed this job's new assertion to Option/None and documented the migration's invariant. Epoch 77, delta cursor 9, flags and body remain pinned. The focused upgrade replay is running.

The large local fixture has not completed its history prerequisite yet. The two full 1,800-second ordinary-data phases and each remaining restart/fault profile are queued sequentially, with fresh fixtures after the first failed restart assertion. Existing test expectations remain visible; the final report will distinguish failures, SLOW-only results and acceptance gaps.

Further combined-branch findings: - The preserved device chooser reset its selection correctly, but its Sync with Your Devices request still sent only CalDAV and Notes despite naming Mail. The request now includes an independent full Mail grant. The actual form regression passed once; the expanded file passed both authority-label cases and hit its default 5-second timeout in the unchanged grant case on the later busy-host run. No existing test timeout was changed. - Credential labels now keep a narrower plugin grant authoritative. A CalDAV-only read credential is described as Calendar, Reminders & Contacts with Can view, instead of claiming Mail, Notes and write access. - Combined Server clippy passed: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 16m 59s`. - The Server suite reported 214 passed, 2 failed, 9 ignored. One failure is the existing 15-second startup-backfill deadline in its subprocess wrapper; its expectation is unchanged. The other is the new Mail preservation assertion from this job: migration 0014 intentionally sets the old metadata cursor to NULL, but the assertion decoded it as i64 and expected the old value 3. I changed this job's new assertion to Option<i64>/None and documented the migration's invariant. Epoch 77, delta cursor 9, flags and body remain pinned. The focused upgrade replay is running. The large local fixture has not completed its history prerequisite yet. The two full 1,800-second ordinary-data phases and each remaining restart/fault profile are queued sequentially, with fresh fixtures after the first failed restart assertion. Existing test expectations remain visible; the final report will distinguish failures, SLOW-only results and acceptance gaps.
kayg commented 2026-10-04 17:57:13 +00:00
Author
Owner
Copy link

The frozen feature binary reproduced a real scheduling defect: the server's shared Worker has a total limit of one, and mail.idle held that slot while mail.sync stayed pending. Accepted APPEND and STORE completed, but COPY returned the fixed pending-changes refusal until the command deadline. I am reserving a separate bounded IDLE Worker while retaining the one-slot ordinary background budget. A regression holds IDLE open and requires transfer and sync work to start before it closes.

The preceding ordinary-data soak completed its first 30-minute phase, which failed COPY, and began the second phase; I stopped that run after identifying the scheduling cause. Its partial second phase is not a full pass. The original large-mailbox prerequisite also did not complete within 30 minutes on this local busy host. Some earlier restart continuations were invalid: ordinary cargo server tests replaced the feature executable between restarts. Those setup failures are excluded from conclusions. Subsequent verification uses a separately copied feature executable with a recorded checksum.

The frozen feature binary reproduced a real scheduling defect: the server's shared Worker has a total limit of one, and `mail.idle` held that slot while `mail.sync` stayed pending. Accepted APPEND and STORE completed, but COPY returned the fixed `pending-changes` refusal until the command deadline. I am reserving a separate bounded IDLE Worker while retaining the one-slot ordinary background budget. A regression holds IDLE open and requires transfer and sync work to start before it closes. The preceding ordinary-data soak completed its first 30-minute phase, which failed COPY, and began the second phase; I stopped that run after identifying the scheduling cause. Its partial second phase is not a full pass. The original large-mailbox prerequisite also did not complete within 30 minutes on this local busy host. Some earlier restart continuations were invalid: ordinary cargo server tests replaced the feature executable between restarts. Those setup failures are excluded from conclusions. Subsequent verification uses a separately copied feature executable with a recorded checksum.
kayg commented 2026-10-04 18:06:51 +00:00
Author
Owner
Copy link

Committed the bounded Mail IDLE worker split as 1cc8383dc. It retains the existing single ordinary background slot. The runtime regression holds IDLE open and proves transfer and sync start before that wait ends. This regression and the production upgrade test pass in the final server suite.

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 32.00s
test wire::tests::mail_idle_does_not_occupy_the_ordinary_background_slot ... ok
test wire::upgrade_tests::production_schema_copy_upgrades_once_without_reinterpreting_notes_0028 ... ok
test result: FAILED. 216 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 69.50s

The remaining server failure is the unchanged 15-second startup deadline in startup_serves_http_while_upgrade_backfills_wait, invoked by live_apps_run_in_separate_processes. It reached 16.06 seconds on the shared host; its expectation is unchanged.

Correction to the preceding soak comment: the first phase was configured for 1,800 seconds, but every client failed before its deadline; its final summary is 1,501.72 seconds, not a completed 30 minutes. COPY's fixed category is message-unavailable, not pending-changes. The pool starvation is a separately confirmed queue defect. The next pinned-binary replay must establish which failures it fixes.

Committed the bounded Mail IDLE worker split as `1cc8383dc`. It retains the existing single ordinary background slot. The runtime regression holds IDLE open and proves transfer and sync start before that wait ends. This regression and the production upgrade test pass in the final server suite. ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 32.00s test wire::tests::mail_idle_does_not_occupy_the_ordinary_background_slot ... ok test wire::upgrade_tests::production_schema_copy_upgrades_once_without_reinterpreting_notes_0028 ... ok test result: FAILED. 216 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 69.50s ``` The remaining server failure is the unchanged 15-second startup deadline in `startup_serves_http_while_upgrade_backfills_wait`, invoked by `live_apps_run_in_separate_processes`. It reached 16.06 seconds on the shared host; its expectation is unchanged. Correction to the preceding soak comment: the first phase was configured for 1,800 seconds, but every client failed before its deadline; its final summary is 1,501.72 seconds, not a completed 30 minutes. COPY's fixed category is `message-unavailable`, not `pending-changes`. The pool starvation is a separately confirmed queue defect. The next pinned-binary replay must establish which failures it fixes.
kayg commented 2026-10-04 18:24:04 +00:00
Author
Owner
Copy link

The combined profile generator already includes Mail when the device credential grants it (build_notes_app_password_profile, #486), but the retained setup text told every User to switch Mail off. Fixed the text to follow the credential grants; Notes-only profiles keep their existing instructions. The existing populated Mail replay expectation for the Mail profile text is preserved.

Committed as 3dbc3d106. The device preset and authority-summary tests now all pass:

 Test Files  1 passed (1)
      Tests  3 passed (3)
perf-lint: PASS; 0 violations; 19340 scoped exceptions
svelte-check found 0 errors and 4 warnings in 3 files
✓ built in 34.74s
Compressed 534 static variants; saved 9823125 bytes.

Only the hashes of 24 existing scoped perf exceptions were rebound after the component syntax change. Their scopes, limits, owners, reasons and expiry dates are unchanged. The final screenshot replay will use this production build and show the Mail connection controls within the scrollable device panel.

The combined profile generator already includes Mail when the device credential grants it (`build_notes_app_password_profile`, #486), but the retained setup text told every User to switch Mail off. Fixed the text to follow the credential grants; Notes-only profiles keep their existing instructions. The existing populated Mail replay expectation for the Mail profile text is preserved. Committed as `3dbc3d106`. The device preset and authority-summary tests now all pass: ```text Test Files 1 passed (1) Tests 3 passed (3) perf-lint: PASS; 0 violations; 19340 scoped exceptions svelte-check found 0 errors and 4 warnings in 3 files ✓ built in 34.74s Compressed 534 static variants; saved 9823125 bytes. ``` Only the hashes of 24 existing scoped perf exceptions were rebound after the component syntax change. Their scopes, limits, owners, reasons and expiry dates are unchanged. The final screenshot replay will use this production build and show the Mail connection controls within the scrollable device panel.
kayg commented 2026-10-04 18:30:11 +00:00
Author
Owner
Copy link

Filed the unresolved sustained SELECT refusals as #1067: #1067 . The first pinned-binary phase requested 1,800 seconds but ended at 872.41 seconds when all 12 clients received SELECT NO. All 20 local and upstream write-receipt audits passed. Readiness and retained web sessions passed. The second phase is still running.

Added safe SELECT refusal categories in 7cc2a9c39, reusing the transfer allowlist. Unknown wire text remains private; no existing response or receipt expectation changed.

Ran 33 tests in 2.282s
OK

READY FOR STAGING: no while #1067 and the remaining verification gaps are unresolved.

Filed the unresolved sustained SELECT refusals as #1067: https://git.kayg.org/kayg/calternal/issues/1067 . The first pinned-binary phase requested 1,800 seconds but ended at 872.41 seconds when all 12 clients received SELECT NO. All 20 local and upstream write-receipt audits passed. Readiness and retained web sessions passed. The second phase is still running. Added safe SELECT refusal categories in `7cc2a9c39`, reusing the transfer allowlist. Unknown wire text remains private; no existing response or receipt expectation changed. ```text Ran 33 tests in 2.282s OK ``` READY FOR STAGING: no while #1067 and the remaining verification gaps are unresolved.
kayg commented 2026-10-04 18:58:25 +00:00
Author
Owner
Copy link

The pinned-binary three-account phase passed its complete requested duration:

{"scenario":"three-accounts","result":"PASS","web_sessions":"PASS"}

Its summary is 1,808.70 seconds, 12 clients, three Users with one Connected Account each, 17,293 SELECT OK, 17,273 SEARCH/FETCH/IDLE/EXPUNGE OK, and 20 accepted APPEND/STORE/COPY/MOVE sequences. All 20 exact local and upstream receipt audits passed; no client failure was recorded. RSS peaked at 309.00 MiB and open descriptors at 155 in this phase. The same binary failed the one-User/one-account phase at 872.41 seconds with 12 SELECT NO. Both results remain separate. This points to higher per-User concurrency in #1067; it does not prove the cause.

The job is continuing the real production screenshot replay, a diagnostic 50-client pass and the requested fresh-fixture restart/fault cases. The 50-client diagnostic is short and cannot replace the requested long run.

The pinned-binary three-account phase passed its complete requested duration: ```json {"scenario":"three-accounts","result":"PASS","web_sessions":"PASS"} ``` Its summary is 1,808.70 seconds, 12 clients, three Users with one Connected Account each, 17,293 SELECT OK, 17,273 SEARCH/FETCH/IDLE/EXPUNGE OK, and 20 accepted APPEND/STORE/COPY/MOVE sequences. All 20 exact local and upstream receipt audits passed; no client failure was recorded. RSS peaked at 309.00 MiB and open descriptors at 155 in this phase. The same binary failed the one-User/one-account phase at 872.41 seconds with 12 SELECT NO. Both results remain separate. This points to higher per-User concurrency in #1067; it does not prove the cause. The job is continuing the real production screenshot replay, a diagnostic 50-client pass and the requested fresh-fixture restart/fault cases. The 50-client diagnostic is short and cannot replace the requested long run.
kayg commented 2026-10-04 19:21:32 +00:00
Author
Owner
Copy link

All 30 production screenshots are attached to #1038: Inbox, folder, reader, thread and device setup; 390/820/1440 px; light/dark; macOS platform emulation. The completed-view replay exited 0 and the Mail setup captures show the connection controls. The profile limit remains five issuances per User; the replay reuses the existing two-User approach. No screenshot or review artifact is committed.

The screenshots expose a Settings sheet title/body mismatch after scrolling (#1068): #1068 . At phone width, Calendar Feeds/New feed appears above the App Password ready body. The screenshot evidence is for the orchestrator's visual review; the Mail job did not change the shared sheet behavior. Recheck on final 7b.

All twelve fresh-fixture SIGKILL cases pass: APPEND, COPY, MOVE and EXPUNGE at 0, 100 and 500 ms. Each audit checks three Users and the 128 unrelated baseline Inbox messages, as well as its exact accepted receipt. These samples interrupt queued delivery; the separate upstream literal fault scenarios cover in-flight upload boundaries. Same-item races and provider faults are running next.

All 30 production screenshots are attached to #1038: Inbox, folder, reader, thread and device setup; 390/820/1440 px; light/dark; macOS platform emulation. The completed-view replay exited 0 and the Mail setup captures show the connection controls. The profile limit remains five issuances per User; the replay reuses the existing two-User approach. No screenshot or review artifact is committed. The screenshots expose a Settings sheet title/body mismatch after scrolling (#1068): https://git.kayg.org/kayg/calternal/issues/1068 . At phone width, Calendar Feeds/New feed appears above the App Password ready body. The screenshot evidence is for the orchestrator's visual review; the Mail job did not change the shared sheet behavior. Recheck on final 7b. All twelve fresh-fixture SIGKILL cases pass: APPEND, COPY, MOVE and EXPUNGE at 0, 100 and 500 ms. Each audit checks three Users and the 128 unrelated baseline Inbox messages, as well as its exact accepted receipt. These samples interrupt queued delivery; the separate upstream literal fault scenarios cover in-flight upload boundaries. Same-item races and provider faults are running next.
kayg commented 2026-10-04 19:53:51 +00:00
Author
Owner
Copy link

READY FOR STAGING: no.

Committed locally on job/mailround-1038, head 0a3bf7a8eb5056730f9d7547ef65df562ed1b076. Base 4082669f7; integrated job/mailstress-b at 6ea7361a3; fetched and merged origin/dev once (already up to date at c39ffe5d9). No push or deploy.

Built

Combined the Mail proxy, provider sync and interrupted-APPEND repair with 7b durability and sharing. Kept Mail migrations 0010/0011 and added the incoming 0012–0017; moved Auth Mail usage to 0014 after Share invites 0013. Kept one #1041 authority check. Regenerated OpenAPI, action contracts and the API client. Extended the production-schema upgrade test. Preserved bounded backfill/expunge, immutable source sharing and generation-aware IMAP adapters.

Fixed selected IMAP epoch invalidation and separated bounded Mail IDLE from the one-slot ordinary background Worker. The runtime regression and real queued-write/restart audits pass. Mail live updates use the shared owner-bound event bus and existing detail/list caches. Device setup grants Mail independently of Notes, uses shared copy controls, describes effective authority and keeps Mail enabled when the generated profile grants it. Added focused device tests, safe probe diagnostics, fresh-case selectors, the finite 50-client profile and completed-view screenshot replay.

Files

  • crates/calternal-imap/src/{session,store,wire,mime}.rs and their tests; vendored async-imap FETCH attribute presence.
  • crates/plugins/mail/src/ cache, proxy, mutation/transfer, routes and sync modules; Mail migrations 0012–0017.
  • crates/calternal-server/src/ bootstrap, device adapters, integrations and upgrade tests; crates/calternal-auth/ API/store and migration 0014; crates/calternal-db/src/jobs.rs; Notes IMAP trait adaptation.
  • apps/web/src/lib/mail/{MailView.svelte,live.ts,live.test.ts}; apps/web/src/routes/settings/account/AppPasswordsGroup.svelte and its focused test; apps/web/e2e/mail-proxy-486.mjs.
  • Generated contracts/API client, Cargo.lock, scoped perf hash rebinding, bench/mail-*.py, tests/adversarial/mail*, related fixture tests and documentation. Complete 62-file list: artifacts/mailround-files.txt.

Verification matrix

Pinned fixture server SHA-256: 6bddc427158176be4f507fc4ff27be9ccd0256c2e22c38b5b956277d3b813f05. Rust product code includes 1cc8383dc; production web includes 3dbc3d106. Restarts used a separate copied executable so normal cargo tests could not replace the feature binary.

Scenario Result Evidence and limit
One account / one User / 12 clients, requested 30 min FAIL 872.41 s; 12 SELECT NO; all 20 local/upstream write receipts pass; #1067
Three accounts / three Users / 12 clients, requested 30 min PASS 1,808.70 s; 17,293 SELECT OK; 20 exact local/upstream receipts; retained web sessions pass
50 clients, both account groups PASS diagnostic Five-second budgets; 7.82/7.15 s including audits; five exact receipts each; not full-load acceptance
APPEND/COPY/MOVE/EXPUNGE SIGKILL at 0/100/500 ms PASS All 12 fresh cases; three Users and 128 original messages checked per case; queued cuts do not prove in-flight upload kills
MOVE/MOVE, MOVE/delete, FETCH/EXPUNGE races PASS OK/NO, OK/NO and OK/OK; exact receipt checks
Unicode and namespaces PASS Seven Unicode cases, 348 namespace checks
Folder create/rename/delete UNSUPPORTED Clean NO responses; no accepted operation
Pipeline PASS Two tagged completions in reliability, 50 in resilience
Provider restart / UIDVALIDITY PASS with limit Automatic recovery; restart did not establish an active interrupted command
Unfinished literal / overlong line / invalid UTF-7 / IDLE without DONE PASS Correct terminal replies; bounded local samples
Interrupted FETCH, including slow delivery PASS Exact recovery, zero partial-body rows; another User stays responsive
Interrupted APPEND before upload / inside literal / after upstream OK PASS Exact 1 MiB receipt at all three boundaries; empty journal and zero unfinished provider files
Three Connected Accounts under one User PASS Three exact upstream moves and 21 separation checks
Login policy and fresh web sign-in PASS Seven NO, 23 rate rejections; retained IMAP sessions and fresh web sign-in 200
Production screenshot replay PASS All 30 images attached: five views, 390/820/1440 px, both themes, macOS emulation, secrets masked
Populated Mail e2e PARTIAL Protocol checks, live BODY/read/unread/MOVE/EXPUNGE and 24 Mail views pass; existing phone sheet-title wait times out (#1068)
Focused real TLS regressions PASS 2,000-message backfill, exact body caching, durable copy/delete replay

Local stress samples remained bounded in these runs. The complete three-account phase peaks at 309.00 MiB RSS and 155 FDs. The isolated 2,000-message backfill regression reports folder page p50 118.965 ms / p95 579.987 ms, first 100 messages 1,071 ms, HWM 30,456 KiB. These are local debug/fixture observations, not a comparison with the release perf-VM baseline. The incoming hot-path bench profiles are retained. The latest verification policy limits separate perf measurements to performance issues.

UX gaps closed

Mail changes refresh retained rows and the shared detail cache without a browser reload. The ordinary device chooser now grants Mail and Notes separately; narrower credentials get accurate descriptions. Mail connection controls use the shared CopyableValue component. Profile instructions follow the actual Mail grant. The screenshot replay follows the visible chooser label, keeps issuance within the existing per-User limit and brings Mail connection controls into view.

UX gaps left / known gaps

  • #1067: #1067 . Sustained single-User SELECT refusals are unclassified. Do not call them SLOW-only without a classified reproduction. The safe diagnostic allowlist is now available.
  • #1068: #1068 . Phone Settings sheet shows Calendar Feeds/New feed above the App Password setup body after scrolling. Copy-link identity needs recheck on final 7b. An extra clipboard integration probe was inconclusive while waiting for a profile response; its uncommitted code was removed. No clipboard pass is claimed.
  • Three unchanged imported Mail tests fail: mail_events_are_authenticated_and_owner_filtered, idle_waits_have_a_separate_pool_from_mail_sync, interleaved_flags_fetch_is_ignored_after_uid_set_validation. They expect no initial event, exactly two handlers and unbounded SEARCH ALL. The orchestrator must resolve these expectations; they were not changed to force a pass.
  • The Server suite fails its existing 15-second startup deadline at 16.06 s. Production upgrade and IDLE scheduling regressions pass in that same suite. This is the observed SLOW-only gate failure.
  • The 100,000-message prerequisite reached 29,840 messages in 30 minutes before the IDLE fix. It was not repeated on the fixed server. The final revision has no complete 50-client long run, 50 MB/zero-byte/10,000-folder boundary matrix or download storm. Those acceptance gaps remain on #1038.
  • Earlier restart continuations built from a replaced default executable are invalid setup evidence and are excluded. Valid final restarts use the checksum above. Native Apple Mail was deliberately left for the later orchestrator pass, as the brief specifies.

Decisions

Keep the existing canonical mail-change bus and initial folder hint, bounded UID searches and 7b generation semantics. Reserve a bounded IDLE Worker while retaining one ordinary background slot. Keep incoming Mail migration numbers, which already follow 7b; renumber only the Auth collision. Preserve existing test assertions for owner review. Use two existing Users for profile screenshots, with no production quota increase. Expose a finite explicit 50-client private-instance profile; short diagnostics do not stand in for full acceptance. Only existing perf-exception hashes were rebound; scopes, limits, owners, reasons, expiry dates and count remain unchanged.

For the merge round

Merge final 7b as planned and recheck #1068. Run the normal combined gates, full web tests (cd apps/web && bun run test) and native Apple Mail acceptance under the required Mac VM lock. This job did run the requested mail fault/restart scenarios; its failed single-account load and incomplete large/50-client acceptance are recorded above rather than presented as a pass. Reproduce #1067 with the completed local fixture using CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_CLIENTS=50 node tests/adversarial/mail_stress.mjs; use the normal 1,800-second phases. The large prerequisite selector is CALTERNAL_MAIL_STRESS_PHASE=initial-sync node tests/adversarial/mail_stress.mjs with the pinned feature server.

Gate output, verbatim excerpts

cargo fmt --check exited 0 with no output. Cargo used the required debug, incremental, jobs and temporary-directory settings.

calternal-imap

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10m 32s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 34s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.13s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-mail

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 59.18s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 41s
test result: FAILED. 103 passed; 3 failed; 6 ignored; 0 measured; 0 filtered out; finished in 27.67s

calternal-server

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 32.00s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 17s
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 225 filtered out; finished in 16.06s
test result: FAILED. 216 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 69.50s

calternal-auth

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 01s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 04s
test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 202.36s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-db

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 07s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 56.97s
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.66s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.64s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.28s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.49s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.96s
test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.41s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-notes

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 28s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 10s
test result: ok. 262 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 248.89s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.19s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

async-imap

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 13m 41s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 12m 23s
test result: ok. 70 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s
test result: ok. 1 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 0.14s

Web check

perf-lint: PASS; 0 violations; 19340 scoped exceptions
svelte-check found 0 errors and 4 warnings in 3 files

Focused Mail Vitest

 Test Files  3 passed (3)
      Tests  14 passed (14)

Device preset/authority/profile Vitest

 Test Files  1 passed (1)
      Tests  3 passed (3)

Fixture unit tests

Ran 33 tests in 2.282s
OK
READY FOR STAGING: no. Committed locally on `job/mailround-1038`, head `0a3bf7a8eb5056730f9d7547ef65df562ed1b076`. Base `4082669f7`; integrated `job/mailstress-b` at `6ea7361a3`; fetched and merged `origin/dev` once (already up to date at `c39ffe5d9`). No push or deploy. **Built** Combined the Mail proxy, provider sync and interrupted-APPEND repair with 7b durability and sharing. Kept Mail migrations 0010/0011 and added the incoming 0012–0017; moved Auth Mail usage to 0014 after Share invites 0013. Kept one #1041 authority check. Regenerated OpenAPI, action contracts and the API client. Extended the production-schema upgrade test. Preserved bounded backfill/expunge, immutable source sharing and generation-aware IMAP adapters. Fixed selected IMAP epoch invalidation and separated bounded Mail IDLE from the one-slot ordinary background Worker. The runtime regression and real queued-write/restart audits pass. Mail live updates use the shared owner-bound event bus and existing detail/list caches. Device setup grants Mail independently of Notes, uses shared copy controls, describes effective authority and keeps Mail enabled when the generated profile grants it. Added focused device tests, safe probe diagnostics, fresh-case selectors, the finite 50-client profile and completed-view screenshot replay. **Files** - `crates/calternal-imap/src/{session,store,wire,mime}.rs` and their tests; vendored async-imap FETCH attribute presence. - `crates/plugins/mail/src/` cache, proxy, mutation/transfer, routes and sync modules; Mail migrations 0012–0017. - `crates/calternal-server/src/` bootstrap, device adapters, integrations and upgrade tests; `crates/calternal-auth/` API/store and migration 0014; `crates/calternal-db/src/jobs.rs`; Notes IMAP trait adaptation. - `apps/web/src/lib/mail/{MailView.svelte,live.ts,live.test.ts}`; `apps/web/src/routes/settings/account/AppPasswordsGroup.svelte` and its focused test; `apps/web/e2e/mail-proxy-486.mjs`. - Generated contracts/API client, Cargo.lock, scoped perf hash rebinding, `bench/mail-*.py`, `tests/adversarial/mail*`, related fixture tests and documentation. Complete 62-file list: `artifacts/mailround-files.txt`. **Verification matrix** Pinned fixture server SHA-256: `6bddc427158176be4f507fc4ff27be9ccd0256c2e22c38b5b956277d3b813f05`. Rust product code includes `1cc8383dc`; production web includes `3dbc3d106`. Restarts used a separate copied executable so normal cargo tests could not replace the feature binary. | Scenario | Result | Evidence and limit | |---|---|---| | One account / one User / 12 clients, requested 30 min | FAIL | 872.41 s; 12 SELECT NO; all 20 local/upstream write receipts pass; #1067 | | Three accounts / three Users / 12 clients, requested 30 min | PASS | 1,808.70 s; 17,293 SELECT OK; 20 exact local/upstream receipts; retained web sessions pass | | 50 clients, both account groups | PASS diagnostic | Five-second budgets; 7.82/7.15 s including audits; five exact receipts each; not full-load acceptance | | APPEND/COPY/MOVE/EXPUNGE SIGKILL at 0/100/500 ms | PASS | All 12 fresh cases; three Users and 128 original messages checked per case; queued cuts do not prove in-flight upload kills | | MOVE/MOVE, MOVE/delete, FETCH/EXPUNGE races | PASS | OK/NO, OK/NO and OK/OK; exact receipt checks | | Unicode and namespaces | PASS | Seven Unicode cases, 348 namespace checks | | Folder create/rename/delete | UNSUPPORTED | Clean NO responses; no accepted operation | | Pipeline | PASS | Two tagged completions in reliability, 50 in resilience | | Provider restart / UIDVALIDITY | PASS with limit | Automatic recovery; restart did not establish an active interrupted command | | Unfinished literal / overlong line / invalid UTF-7 / IDLE without DONE | PASS | Correct terminal replies; bounded local samples | | Interrupted FETCH, including slow delivery | PASS | Exact recovery, zero partial-body rows; another User stays responsive | | Interrupted APPEND before upload / inside literal / after upstream OK | PASS | Exact 1 MiB receipt at all three boundaries; empty journal and zero unfinished provider files | | Three Connected Accounts under one User | PASS | Three exact upstream moves and 21 separation checks | | Login policy and fresh web sign-in | PASS | Seven NO, 23 rate rejections; retained IMAP sessions and fresh web sign-in 200 | | Production screenshot replay | PASS | All 30 images attached: five views, 390/820/1440 px, both themes, macOS emulation, secrets masked | | Populated Mail e2e | PARTIAL | Protocol checks, live BODY/read/unread/MOVE/EXPUNGE and 24 Mail views pass; existing phone sheet-title wait times out (#1068) | | Focused real TLS regressions | PASS | 2,000-message backfill, exact body caching, durable copy/delete replay | Local stress samples remained bounded in these runs. The complete three-account phase peaks at 309.00 MiB RSS and 155 FDs. The isolated 2,000-message backfill regression reports folder page p50 118.965 ms / p95 579.987 ms, first 100 messages 1,071 ms, HWM 30,456 KiB. These are local debug/fixture observations, not a comparison with the release perf-VM baseline. The incoming hot-path bench profiles are retained. The latest verification policy limits separate perf measurements to performance issues. **UX gaps closed** Mail changes refresh retained rows and the shared detail cache without a browser reload. The ordinary device chooser now grants Mail and Notes separately; narrower credentials get accurate descriptions. Mail connection controls use the shared CopyableValue component. Profile instructions follow the actual Mail grant. The screenshot replay follows the visible chooser label, keeps issuance within the existing per-User limit and brings Mail connection controls into view. **UX gaps left / known gaps** - #1067: https://git.kayg.org/kayg/calternal/issues/1067 . Sustained single-User SELECT refusals are unclassified. Do not call them SLOW-only without a classified reproduction. The safe diagnostic allowlist is now available. - #1068: https://git.kayg.org/kayg/calternal/issues/1068 . Phone Settings sheet shows Calendar Feeds/New feed above the App Password setup body after scrolling. Copy-link identity needs recheck on final 7b. An extra clipboard integration probe was inconclusive while waiting for a profile response; its uncommitted code was removed. No clipboard pass is claimed. - Three unchanged imported Mail tests fail: `mail_events_are_authenticated_and_owner_filtered`, `idle_waits_have_a_separate_pool_from_mail_sync`, `interleaved_flags_fetch_is_ignored_after_uid_set_validation`. They expect no initial event, exactly two handlers and unbounded SEARCH ALL. The orchestrator must resolve these expectations; they were not changed to force a pass. - The Server suite fails its existing 15-second startup deadline at 16.06 s. Production upgrade and IDLE scheduling regressions pass in that same suite. This is the observed SLOW-only gate failure. - The 100,000-message prerequisite reached 29,840 messages in 30 minutes before the IDLE fix. It was not repeated on the fixed server. The final revision has no complete 50-client long run, 50 MB/zero-byte/10,000-folder boundary matrix or download storm. Those acceptance gaps remain on #1038. - Earlier restart continuations built from a replaced default executable are invalid setup evidence and are excluded. Valid final restarts use the checksum above. Native Apple Mail was deliberately left for the later orchestrator pass, as the brief specifies. **Decisions** Keep the existing canonical `mail-change` bus and initial folder hint, bounded UID searches and 7b generation semantics. Reserve a bounded IDLE Worker while retaining one ordinary background slot. Keep incoming Mail migration numbers, which already follow 7b; renumber only the Auth collision. Preserve existing test assertions for owner review. Use two existing Users for profile screenshots, with no production quota increase. Expose a finite explicit 50-client private-instance profile; short diagnostics do not stand in for full acceptance. Only existing perf-exception hashes were rebound; scopes, limits, owners, reasons, expiry dates and count remain unchanged. **For the merge round** Merge final 7b as planned and recheck #1068. Run the normal combined gates, full web tests (`cd apps/web && bun run test`) and native Apple Mail acceptance under the required Mac VM lock. This job did run the requested mail fault/restart scenarios; its failed single-account load and incomplete large/50-client acceptance are recorded above rather than presented as a pass. Reproduce #1067 with the completed local fixture using `CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_CLIENTS=50 node tests/adversarial/mail_stress.mjs`; use the normal 1,800-second phases. The large prerequisite selector is `CALTERNAL_MAIL_STRESS_PHASE=initial-sync node tests/adversarial/mail_stress.mjs` with the pinned feature server. **Gate output, verbatim excerpts** `cargo fmt --check` exited 0 with no output. Cargo used the required debug, incremental, jobs and temporary-directory settings. **calternal-imap** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 10m 32s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 34s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.13s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` **calternal-plugin-mail** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 59.18s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 41s test result: FAILED. 103 passed; 3 failed; 6 ignored; 0 measured; 0 filtered out; finished in 27.67s ``` **calternal-server** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 32.00s Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 17s test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 225 filtered out; finished in 16.06s test result: FAILED. 216 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 69.50s ``` **calternal-auth** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 01s Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 04s test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 202.36s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` **calternal-db** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 07s Finished `test` profile [unoptimized + debuginfo] target(s) in 56.97s test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.66s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.64s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.28s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.49s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.96s test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.41s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` **calternal-plugin-notes** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 28s Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 10s test result: ok. 262 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 248.89s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.19s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` **async-imap** ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 13m 41s Finished `test` profile [unoptimized + debuginfo] target(s) in 12m 23s test result: ok. 70 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s test result: ok. 1 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 0.14s ``` **Web check** ```text perf-lint: PASS; 0 violations; 19340 scoped exceptions svelte-check found 0 errors and 4 warnings in 3 files ``` **Focused Mail Vitest** ```text Test Files 3 passed (3) Tests 14 passed (14) ``` **Device preset/authority/profile Vitest** ```text Test Files 1 passed (1) Tests 3 passed (3) ``` **Fixture unit tests** ```text Ran 33 tests in 2.282s OK ```
kayg commented 2026-10-04 20:00:04 +00:00
Author
Owner
Copy link

Round 2 started on job/mailround-1038, prior head 0a3bf7a8eb5056730f9d7547ef65df562ed1b076. Merged origin/dev at 9fb9a4bfb2488152c83d50c55441ff5f43b572b2 cleanly; merge head b6269034026d7a0f3471b9740ab63bed69d4fc4b.

Investigating sustained single-account SELECT refusals (#1067), resolving three failing Mail regressions against DESIGN §§45/53/59, checking production migration upgrade, and rerunning the requested gates and 30-minute single-account phase. No push or deploy.

Round 2 started on `job/mailround-1038`, prior head `0a3bf7a8eb5056730f9d7547ef65df562ed1b076`. Merged `origin/dev` at `9fb9a4bfb2488152c83d50c55441ff5f43b572b2` cleanly; merge head `b6269034026d7a0f3471b9740ab63bed69d4fc4b`. Investigating sustained single-account SELECT refusals (#1067), resolving three failing Mail regressions against DESIGN §§45/53/59, checking production migration upgrade, and rerunning the requested gates and 30-minute single-account phase. No push or deploy.
kayg commented 2026-10-04 20:09:54 +00:00
Author
Owner
Copy link

Round 2 findings, DESIGN §§53/59:

  • mail_events_are_authenticated_and_owner_filtered must consume and assert the canonical initial mail-change / {"scope":"folders"} hint before proving a foreign User emits no hint. Owner filtering and exact content-free payload assertions remain.
  • idle_waits_have_a_separate_pool_from_mail_sync must pin all four handler pools: sync, IDLE, proxy mutation and proxy transfer (three each). Its scheduling check remains.
  • interleaved_flags_fetch_is_ignored_after_uid_set_validation must require exactly one bounded UID SEARCH UID 1:80, while retaining the exact 80-UID projection assertion. SEARCH ALL would violate bounded sync. Its opt-in profile follows the same command.
  • Migration numbering is already compatible: Mail 12–17 follow deployed 11; Auth 14 follows deployed 13. The upgrade regression now exercises both the older schema copy and deployed 7b prefix, keeps populated Mail/security rows and checks exact old receipts plus no second-start backup.
  • Startup limit remains 15 seconds. The locked measurement host is idle (load 0.00, 0.00, 0.00) and x86_64; the focused test will run there once its executable is built.
  • SELECT failure diagnostics now cover every fixed resource/deadline reason. Snapshot SQL failures use the shared content-free SQLite classifier; arbitrary provider, SQL and driver values are never logged.
Round 2 findings, DESIGN §§53/59: - `mail_events_are_authenticated_and_owner_filtered` must consume and assert the canonical initial `mail-change` / `{"scope":"folders"}` hint before proving a foreign User emits no hint. Owner filtering and exact content-free payload assertions remain. - `idle_waits_have_a_separate_pool_from_mail_sync` must pin all four handler pools: sync, IDLE, proxy mutation and proxy transfer (three each). Its scheduling check remains. - `interleaved_flags_fetch_is_ignored_after_uid_set_validation` must require exactly one bounded `UID SEARCH UID 1:80`, while retaining the exact 80-UID projection assertion. SEARCH ALL would violate bounded sync. Its opt-in profile follows the same command. - Migration numbering is already compatible: Mail 12–17 follow deployed 11; Auth 14 follows deployed 13. The upgrade regression now exercises both the older schema copy and deployed 7b prefix, keeps populated Mail/security rows and checks exact old receipts plus no second-start backup. - Startup limit remains 15 seconds. The locked measurement host is idle (load `0.00, 0.00, 0.00`) and x86_64; the focused test will run there once its executable is built. - SELECT failure diagnostics now cover every fixed resource/deadline reason. Snapshot SQL failures use the shared content-free SQLite classifier; arbitrary provider, SQL and driver values are never logged.
kayg commented 2026-10-04 20:22:59 +00:00
Author
Owner
Copy link

The three imported Mail failures are resolved against DESIGN §§53/59 in commit be539ef6d. The test contract changes are described in the preceding finding; no production limit or status assertion was weakened. The IDLE fixture now uses a stateful release barrier and proves sync, mutation and transfer all complete while three IDLE waits stay open.

Verbatim gate output:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10m 13s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 58s
test result: ok. 106 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 30.25s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The pinned current-dev TLS single-account replay is still running. No sustained SELECT repair is claimed until its outcome is classified.

The three imported Mail failures are resolved against DESIGN §§53/59 in commit `be539ef6d`. The test contract changes are described in the preceding finding; no production limit or status assertion was weakened. The IDLE fixture now uses a stateful release barrier and proves sync, mutation and transfer all complete while three IDLE waits stay open. Verbatim gate output: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 10m 13s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 58s test result: ok. 106 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 30.25s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` The pinned current-dev TLS single-account replay is still running. No sustained SELECT repair is claimed until its outcome is classified.
kayg commented 2026-10-04 20:30:18 +00:00
Author
Owner
Copy link

The production-prefix upgrade regression passes for both baselines. Mail 12–17 and Auth 14 remain the only pending migrations after production 7b (Auth 13, DB 15, Files 24, Mail 11, Notes 32). Old receipts, populated Mail data, Notes cursors and App Password authority survive; the second startup takes no backup.

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 226 filtered out; finished in 5.84s

Startup was measured on the idle x86_64 host at 10.69.69.63 with /root/perf.lock. No build ran there; it used the locally compiled test executable and production frontend assets. Two setup attempts failed immediately because the debug executable requires its frontend build directory and canonical parent path; these are setup failures, not startup timing samples. After supplying both, the existing focused test passes without changing its 15-second limit:

 22:29:29 up 2 days, 17:26,  0 users,  load average: 0.20, 0.17, 0.08
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 227 filtered out; finished in 3.34s
wall_seconds=3.40

This supports retaining the product and test deadline. The earlier 16.06-second busy-host observation does not justify a larger production limit. The local full server gate still runs in this round.

The production-prefix upgrade regression passes for both baselines. Mail 12–17 and Auth 14 remain the only pending migrations after production 7b (Auth 13, DB 15, Files 24, Mail 11, Notes 32). Old receipts, populated Mail data, Notes cursors and App Password authority survive; the second startup takes no backup. ```text test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 226 filtered out; finished in 5.84s ``` Startup was measured on the idle x86_64 host at 10.69.69.63 with `/root/perf.lock`. No build ran there; it used the locally compiled test executable and production frontend assets. Two setup attempts failed immediately because the debug executable requires its frontend build directory and canonical parent path; these are setup failures, not startup timing samples. After supplying both, the existing focused test passes without changing its 15-second limit: ```text 22:29:29 up 2 days, 17:26, 0 users, load average: 0.20, 0.17, 0.08 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 227 filtered out; finished in 3.34s wall_seconds=3.40 ``` This supports retaining the product and test deadline. The earlier 16.06-second busy-host observation does not justify a larger production limit. The local full server gate still runs in this round.
kayg commented 2026-10-04 20:47:20 +00:00
Author
Owner
Copy link

The current-dev single-account 30-minute phase passes against pinned feature executable SHA-256 9e046864e71e355412016b3bc73baa54f0faae9b2ad6206ef56211d17dc98bee.

Result: 1,808.72 seconds, 12 clients / one User / one Connected Account, 15,953 SELECT OK and 15,933 SEARCH/FETCH/IDLE/EXPUNGE OK. It accepts 20 APPEND/STORE/COPY/MOVE sequences; all 20 exact upstream and local receipt audits pass. No SELECT NO, client failure, receipt failure or snapshot SQL failure is recorded. Peak RSS is 298,568 KiB (291.57 MiB); peak descriptors 153. Retained web sessions and readiness pass.

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

This is a complete phase, not a short smoke run. It does not yet establish why the earlier revision failed. A historical-head comparison (0a3bf7a8e with only the content-free snapshot diagnostic patch) is now running with the same fixture and assertions. The earlier historical setup was deliberately stopped before its soak to keep one browser active; that interrupted setup is excluded from acceptance evidence. No production limits or required status/receipt assertions were changed, and #1067 is not labelled SLOW-only or resolved without a classified cause.

The current-dev single-account 30-minute phase passes against pinned feature executable SHA-256 `9e046864e71e355412016b3bc73baa54f0faae9b2ad6206ef56211d17dc98bee`. Result: 1,808.72 seconds, 12 clients / one User / one Connected Account, 15,953 SELECT OK and 15,933 SEARCH/FETCH/IDLE/EXPUNGE OK. It accepts 20 APPEND/STORE/COPY/MOVE sequences; all 20 exact upstream and local receipt audits pass. No SELECT NO, client failure, receipt failure or snapshot SQL failure is recorded. Peak RSS is 298,568 KiB (291.57 MiB); peak descriptors 153. Retained web sessions and readiness pass. ```json {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` This is a complete phase, not a short smoke run. It does not yet establish why the earlier revision failed. A historical-head comparison (`0a3bf7a8e` with only the content-free snapshot diagnostic patch) is now running with the same fixture and assertions. The earlier historical setup was deliberately stopped before its soak to keep one browser active; that interrupted setup is excluded from acceptance evidence. No production limits or required status/receipt assertions were changed, and #1067 is not labelled SLOW-only or resolved without a classified cause.
kayg commented 2026-10-04 21:07:36 +00:00
Author
Owner
Copy link

#1067: cached SELECT writer contention reproduced and repaired

The snapshot path checked out the single writer for every SELECT, including a fully cached, unchanged mailbox. A background transaction could therefore hold ordinary SELECT behind the command deadline. The bounded expunge cursor and split IDLE workers do not remove that dependency.

Added a deterministic regression that primes INBOX, holds an uncommitted background edit on the writer, and requires SELECT to finish while that transaction remains open. Before the repair:

test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... FAILED
unchanged SELECT must not check out the occupied writer: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.93s

After the repair:

test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.43s

Unchanged snapshots now use one consistent WAL reader transaction. The same SQL, ownership/service visibility joins, metadata bounds, pending-intent projection and fingerprint checks serve both passes. New UIDs, flags, visibility or view changes release the reader and use the single writer. Readers never advance durable clocks. No timeout or resource limit was increased.

The existing bench profile now holds the writer for serial cached SELECT samples and a 12-client burst, including its 100,000-message case. Full Mail/server gates and the fixed-build 30-minute phase are next.

The original 862-second run did not record the SQL error class, so this deterministic defect does not identify every refusal in that older run. The unmodified merged build already completed one 30-minute diagnostic phase; fixed-build acceptance will be reported separately. Historical comparison was stopped after Cargo reused incompatible artifacts from an archived source tree. Those stopped runs are excluded; no historical result is claimed.

#1067: cached SELECT writer contention reproduced and repaired The snapshot path checked out the single writer for every SELECT, including a fully cached, unchanged mailbox. A background transaction could therefore hold ordinary SELECT behind the command deadline. The bounded expunge cursor and split IDLE workers do not remove that dependency. Added a deterministic regression that primes INBOX, holds an uncommitted background edit on the writer, and requires SELECT to finish while that transaction remains open. Before the repair: ``` test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... FAILED unchanged SELECT must not check out the occupied writer: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.93s ``` After the repair: ``` test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.43s ``` Unchanged snapshots now use one consistent WAL reader transaction. The same SQL, ownership/service visibility joins, metadata bounds, pending-intent projection and fingerprint checks serve both passes. New UIDs, flags, visibility or view changes release the reader and use the single writer. Readers never advance durable clocks. No timeout or resource limit was increased. The existing bench profile now holds the writer for serial cached SELECT samples and a 12-client burst, including its 100,000-message case. Full Mail/server gates and the fixed-build 30-minute phase are next. The original 862-second run did not record the SQL error class, so this deterministic defect does not identify every refusal in that older run. The unmodified merged build already completed one 30-minute diagnostic phase; fixed-build acceptance will be reported separately. Historical comparison was stopped after Cargo reused incompatible artifacts from an archived source tree. Those stopped runs are excluded; no historical result is claimed.
kayg commented 2026-10-04 21:18:12 +00:00
Author
Owner
Copy link

The fixed-product server clippy passes. The second local server gate, concurrent with the required Mail soak on the shared host, hit only the existing startup deadline:

HTTP startup waited for an upgrade backfill: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out; finished in 15.15s
test result: FAILED. 218 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 106.38s

The first full round-2 server gate passed:

test result: ok. 219 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 120.55s

The requested idle-host measurement already passed the same focused test at load average 0.20 / 0.17 / 0.08 in 3.34 seconds (wall 3.40). No startup product change, test assertion change or limit increase is justified by these measurements. Retain the 15-second guarantee. This is a SLOW-only gate finding; it does not identify a Mail refusal, crash or data-loss fault. The final report will quote the latest local failure rather than call all gates clean.

The fixed-product server clippy passes. The second local server gate, concurrent with the required Mail soak on the shared host, hit only the existing startup deadline: ``` HTTP startup waited for an upgrade backfill: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out; finished in 15.15s test result: FAILED. 218 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 106.38s ``` The first full round-2 server gate passed: ``` test result: ok. 219 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 120.55s ``` The requested idle-host measurement already passed the same focused test at load average 0.20 / 0.17 / 0.08 in 3.34 seconds (wall 3.40). No startup product change, test assertion change or limit increase is justified by these measurements. Retain the 15-second guarantee. This is a SLOW-only gate finding; it does not identify a Mail refusal, crash or data-loss fault. The final report will quote the latest local failure rather than call all gates clean.
kayg commented 2026-10-04 21:19:21 +00:00
Author
Owner
Copy link

#1067: the 7b write topology is now covered by a before/after regression.

Compared mailstress-b (6ea7361a3) with the combined Mail head (0a3bf7a8e). SELECT's snapshot transaction was inherited unchanged. calternal-db changed from one application write connection to separate ordinary WAL/NORMAL and Security WAL/FULL connections plus checkpoint maintenance. The separate Security connection can hold SQLite's write lock without occupying the ordinary pool. Cached SELECT still tried to write, so it depended on both locks.

Extended the regression to hold the Security writer first and the Index writer second. Ran that same test against the pre-repair snapshot source in the current merged worktree, then restored the repair. The copied soak binary is independent and its checksum stayed unchanged.

Before repair, with the Security transaction held:

test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... FAILED
unchanged SELECT must not wait for Index or Security writes: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.24s

After repair, both held-writer cases pass inside the full Mail gate:

test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... ok
test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s

Clippy and fmt pass. The product repair is committed at 9c8d1e763; the topology regression and its instructions are committed separately. No durability policy, deadline or resource bound was weakened. This reproduces and removes an ordinary-read dependency exposed by the 7b topology. The older sustained refusal had no error class, so no specific SQLite code is assigned to that historical run. The fixed-build 1,800-second phase is still running.

#1067: the 7b write topology is now covered by a before/after regression. Compared `mailstress-b` (`6ea7361a3`) with the combined Mail head (`0a3bf7a8e`). SELECT's snapshot transaction was inherited unchanged. `calternal-db` changed from one application write connection to separate ordinary WAL/NORMAL and Security WAL/FULL connections plus checkpoint maintenance. The separate Security connection can hold SQLite's write lock without occupying the ordinary pool. Cached SELECT still tried to write, so it depended on both locks. Extended the regression to hold the Security writer first and the Index writer second. Ran that same test against the pre-repair snapshot source in the current merged worktree, then restored the repair. The copied soak binary is independent and its checksum stayed unchanged. Before repair, with the Security transaction held: ``` test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... FAILED unchanged SELECT must not wait for Index or Security writes: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.24s ``` After repair, both held-writer cases pass inside the full Mail gate: ``` test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... ok test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s ``` Clippy and fmt pass. The product repair is committed at `9c8d1e763`; the topology regression and its instructions are committed separately. No durability policy, deadline or resource bound was weakened. This reproduces and removes an ordinary-read dependency exposed by the 7b topology. The older sustained refusal had no error class, so no specific SQLite code is assigned to that historical run. The fixed-build 1,800-second phase is still running.
kayg commented 2026-10-04 21:43:53 +00:00
Author
Owner
Copy link

#1067: fixed-build single-account 30-minute phase PASS

Product source 9c8d1e7630aafae04c43036cba7b80fd64e2b932, copied feature executable SHA-256 d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2. The later commits add regression coverage and instructions only. The checksum stayed unchanged during gates and the controlled pre-repair regression.

Command: CALTERNAL_SERVER_BIN=<copied-feature-executable> CALTERNAL_MAIL_STRESS_IMAGE=localhost/mailstress-1038:latest CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_PHASE=one-account CALTERNAL_MAIL_STRESS_CLIENTS=12 node tests/adversarial/mail_stress.mjs.

One User, one Connected Account, 12 clients, full 1,800-second command phase plus receipt audits: 1,808.52 seconds total. 18,402 SELECT OK; 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each APPEND, STORE, COPY and MOVE succeed. All 20 upstream and 20 local exact-MIME/flags/folder receipts pass. Expected pending-source/destination NO responses were retried only after explicit refusal; accepted writes were never replayed.

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

The mixed-soak result has failures: {} and failure_reasons: {}. Peak RSS is 280,340 KiB (273.77 MiB), peak FDs 171. After client/receipt sockets close: 145,172 KiB RSS and 127 FDs. The descriptor series plateaus and then falls. No crash, hang or receipt discrepancy was found in this phase. These are local fixture/debug resource samples, not release performance baseline numbers.

Evidence: artifacts/round2-one-account-fixed.jsonl. The required #1067 continuation is now PASS. The larger #1038 acceptance gaps and startup SLOW-only gate remain recorded separately. A perf-VM profile and the final report are in progress; no staging deployment is authorized or performed.

#1067: fixed-build single-account 30-minute phase PASS Product source `9c8d1e7630aafae04c43036cba7b80fd64e2b932`, copied feature executable SHA-256 `d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2`. The later commits add regression coverage and instructions only. The checksum stayed unchanged during gates and the controlled pre-repair regression. Command: `CALTERNAL_SERVER_BIN=<copied-feature-executable> CALTERNAL_MAIL_STRESS_IMAGE=localhost/mailstress-1038:latest CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_PHASE=one-account CALTERNAL_MAIL_STRESS_CLIENTS=12 node tests/adversarial/mail_stress.mjs`. One User, one Connected Account, 12 clients, full 1,800-second command phase plus receipt audits: 1,808.52 seconds total. 18,402 SELECT OK; 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each APPEND, STORE, COPY and MOVE succeed. All 20 upstream and 20 local exact-MIME/flags/folder receipts pass. Expected pending-source/destination NO responses were retried only after explicit refusal; accepted writes were never replayed. ``` {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` The mixed-soak result has `failures: {}` and `failure_reasons: {}`. Peak RSS is 280,340 KiB (273.77 MiB), peak FDs 171. After client/receipt sockets close: 145,172 KiB RSS and 127 FDs. The descriptor series plateaus and then falls. No crash, hang or receipt discrepancy was found in this phase. These are local fixture/debug resource samples, not release performance baseline numbers. Evidence: `artifacts/round2-one-account-fixed.jsonl`. The required #1067 continuation is now PASS. The larger #1038 acceptance gaps and startup SLOW-only gate remain recorded separately. A perf-VM profile and the final report are in progress; no staging deployment is authorized or performed.
kayg commented 2026-10-04 22:08:32 +00:00
Author
Owner
Copy link

Final mailround2-1038 report.

Head: 8091d5788ad86aaee1da831801031110462c6f6d on job/mailround-1038. Worktree clean. No push, deploy or merge beyond the authorized origin/dev integration.

Mail round 2: #1038 and #1067

READY FOR STAGING: no.

Built

Merged remote origin/dev at 9fb9a4bfb2488152c83d50c55441ff5f43b572b2
into job/mailround-1038. The merge includes #1062, #1065 and Files/open
parity. There was no push, deploy or issue closure.

Kept the deployed migration prefix: Auth 13, DB 15, Files 24, Mail 11 and
Notes 32. Mail 12–17 and Auth 14 follow that prefix. Extended the existing
production-copy upgrade test. It now checks both the older production schema
and the deployed 7b prefix. Old receipts, Mail rows, Security state and Notes
cursors survive. Only seven proxy/Auth additions are pending after 7b. A
second startup makes no backup.

Repaired cached SELECT writer contention. The pre-7b Mail snapshot used a
writer transaction for every SELECT. 7b added a separate Security write
connection. An unchanged SELECT still needed the ordinary pool and SQLite's
write lock. It could therefore wait behind Index or Security work. The
snapshot now checks the complete committed view in one WAL reader
transaction. A change releases the reader and uses the single writer. Both
passes use the same SQL, ownership checks and limits. Readers never advance
durable clocks. The regression holds each writer separately. The old snapshot
fails; the repaired snapshot completes before either writer is released.

Added fixed-code diagnostics for SQL errors and every SELECT resource refusal.
The harness reports no arbitrary wire, SQL, credential or provider text.
Extended the existing bench profile with held-writer SELECT samples and a
12-client burst. The sampler also has a 100,000-message case.

Repaired the three stale Mail tests against the current contracts:

  • SSE: require the initial canonical folder refresh, the exact owner event
    and no foreign event. DESIGN §§53, 59.
  • Worker pools: pin sync, IDLE, mutation and transfer limits. A stateful
    barrier keeps all three IDLE waits open while ordinary jobs finish. §53.
  • FETCH recovery: require all 80 UIDs and exactly one bounded
    UID SEARCH UID 1:80. Keep membership validation. §§45, 59.

Formatted the imported Notes reconciliation test so the workspace fmt gate
passes. Its behavior and assertions stay unchanged.

Evidence

The product repair is 9c8d1e7630aafae04c43036cba7b80fd64e2b932.
The expanded topology regression is a6e0ff6a5. The acceptance executable
was copied before the server test build. Its SHA-256 is
d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2.
The later regression commit changes tests and instructions only.

The fixed-product single-account phase passes: one User, one Connected
Account, 12 clients, 1,808.52 seconds including receipt audits. It reports
18,402 SELECT OK and 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each
APPEND, STORE, COPY and MOVE succeed. All 20 exact upstream and 20 local
receipts pass. Required SELECT has no refusal. Expected pending-transfer NO
responses are retried only after explicit refusal.

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

There are no mixed-soak failures or failure reasons. Peak RSS is 280,340 KiB
(273.77 MiB), peak FDs 171. After client and receipt sockets close: 145,172
KiB RSS and 127 FDs. Readiness and retained web sessions pass. These are local
fixture/debug observations. Evidence is
artifacts/round2-one-account-fixed.jsonl.

The unmodified merged product also completed a diagnostic 1,808.72-second
phase before the repair: 15,953 SELECT OK, no failures, and 20 exact upstream
and local receipts. This is separate from fixed-product acceptance.

The old snapshot failed the held-Security-writer regression in 2.24 seconds.
The fixed full Mail suite passes that same test with both writers held in turn.
The older 872.41-second refusal did not record an SQL class. Do not assign a
specific SQLite error code to that historical run.

Startup was measured on the idle host at 10.69.69.63, with
/root/perf.lock. Load average was 0.20 / 0.17 / 0.08. The existing test
passed in 3.34 seconds; total wall time was 3.40 seconds. No build ran on
that host. Two initial attempts lacked debug frontend paths and failed
before measuring startup. They are excluded.

The first full local server suite passed: 219 passed, no failures. The later
fixed-product run, concurrent with the soak on the busy host, hit only the
existing startup deadline at 15.15 seconds. This is a SLOW-only result. The
15-second limit stays unchanged.

Release profile

The updated profile completes on the perf VM under /root/perf.lock. The
Mail test executable was built locally in release mode. The VM compiled no
code. Cached SELECT samples and each 12-client burst hold the writer occupied.

Profile Median per-run SELECT p50 / p95 Mean worker CPU Mean / peak worker RSS Largest 12-client SELECT duration
10,001 messages, 3 serial runs 134.51 / 176.76 ms 147.54% 111.03 / 153.00 MiB 1,696.68 ms
100,000 messages 1,953.88 / 5,540.95 ms 163.04% 848.05 / 1,205.99 MiB 18,724.86 ms
3 workers, each with 100,000 messages and its own Index 1,790.76 / 4,051.31 ms 113.24% 891.19 / 1,205.86 MiB 24,210.30 ms

CPU and RSS include seeding, metadata and queue phases. They are not
SELECT-only resource figures. CPU can exceed 100% because SQLite uses
multiple threads. The load averages inside the lock are
6.125 / 4.5923 / 2.1152 before and 7.5220 / 5.5894 / 3.3701 after. The
starting load includes the preceding debug diagnostic run. That incomplete
debug run was stopped and is excluded. Do not treat this as a cold quiet-host
baseline.

docs/perf/baseline.json has no comparable cached SELECT profile.
mail.accounts measures HTTP account listing (p50 1.3 ms / p95 3.8 ms).
It cannot establish a SELECT regression. The existing large-snapshot cost
remains visible and is filed as
#1070. This profile is not
real-provider 100,000-message sync or 50-client acceptance. Evidence is
artifacts/round2-select-perf-vm-release.log.

Files

Round-2 edits, excluding the incoming merge:

  • crates/plugins/mail/src/proxy.rs and proxy_tests.rs.
  • crates/plugins/mail/src/routes.rs and sync.rs.
  • crates/calternal-server/src/upgrade_tests.rs.
  • crates/plugins/notes/src/lib.rs (format only).
  • tests/adversarial/mail_proxy.py, test_mail_proxy.py, mail_stress.mjs
    and mail-sync.md.
  • bench/mail-sync.py and this report.

UX gaps closed

Unchanged cached Mail views can be read while Index or Security writes are
active. UID, revision and message identity checks remain intact. This round
adds no UI. The earlier job's production screenshots remain its visual
evidence; this round makes no new visual claim.

UX gaps left and known gaps

  • #1068 remains: phone Settings sheet title and Copy link checks.
  • The latest busy-host server gate has one startup SLOW-only failure.
  • #1038 still has no complete 50-client long run on both account groups,
    100,000-message initial-sync acceptance, 50 MB/zero-byte/10,000-folder
    matrix or download storm on the final product. Earlier short diagnostics
    do not replace these cases. These gaps prevent a full #1038 acceptance
    claim and READY FOR STAGING remains no.
  • The older SELECT refusal was unclassified. The held-writer defect is
    reproduced and repaired; no exact error code is inferred for that run.
  • Historical comparison attempts were stopped when Cargo reused artifacts
    from an archived source tree. They are excluded. The controlled regression
    uses the old snapshot source in the current worktree and is valid.
  • #1070 tracks large cached SELECT latency and memory. There is no
    comparable baseline to establish a regression.
  • Four existing Svelte warnings remain. There are no Svelte errors.

Decisions

Use a checked WAL reader pass for unchanged snapshots. Keep all changes on
the single writer. This is an implementation choice consistent with
DESIGN §§2, 53, 59. Do not change durability, deadlines or cache bounds.

Use the canonical SSE hint, four bounded worker pools and bounded UID search
already implemented on dev. Update the stale test contracts while keeping or
strengthening what they prove.

Keep the startup product and 15-second test limit. The idle measurement does
not justify a limit increase. Report the later busy-host failure verbatim.

Measure the existing component profile in release mode on the perf VM. Keep
its cache timings separate from the local debug protocol phase. File the
large-snapshot performance gap as #1070. Do not call it a baseline regression
without comparable endpoint, data and build measurements.

For the merge round

Run the combined web suite with cd apps/web && bun run test and the combined
startup gate with cargo test -p calternal-server -- --test-threads=4. The
startup test must show that upgrade backfills do not delay HTTP readiness.
Recheck #1068. These do not replace the completed requested single-account
phase. The original #1038 large-data and 50-client gaps remain listed above.

Gate output, verbatim excerpts

cargo fmt --check exits 0 with no output. Cargo commands use
CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0,
CARGO_BUILD_JOBS=4 and the worktree target/tmp. The preset target directory
was not changed. Tests use --test-threads=4 unless a focused test says 1.

calternal-imap

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 21s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 35.52s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.35s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

calternal-plugin-mail

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.80s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 16.17s
test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 42.71s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 00s
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out; finished in 15.15s
test result: FAILED. 218 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 106.38s

calternal-auth

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 26s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 43s
test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 133.11s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-db

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.23s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 36.98s
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.44s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.26s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.44s
test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.25s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-notes

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 47s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 59s
test result: ok. 264 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 509.86s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.69s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Web check

perf-lint: parse Rust product sources
perf-lint: parse browser product sources
perf-lint: validate coverage and architecture
perf-lint: PASS; 0 violations; 19340 scoped exceptions
svelte-check found 0 errors and 4 warnings in 3 files

Focused Vitest

 Test Files  4 passed (4)
      Tests  18 passed (18)

Probe diagnostics

python3 -m unittest discover -s tests/adversarial -p test_mail_proxy.py

......................
----------------------------------------------------------------------
Ran 22 tests in 0.216s

OK

Cleanup

The owned local and perf-VM executables, temporary fixtures and web build
output are removed. Review artifacts remain in the ignored artifacts/
directory. No screenshot or binary is committed. cargo clean exits 0:

     Removed 26944 files, 20.4GiB total
Final mailround2-1038 report. Head: `8091d5788ad86aaee1da831801031110462c6f6d` on `job/mailround-1038`. Worktree clean. No push, deploy or merge beyond the authorized origin/dev integration. # Mail round 2: #1038 and #1067 READY FOR STAGING: no. ## Built Merged remote `origin/dev` at `9fb9a4bfb2488152c83d50c55441ff5f43b572b2` into `job/mailround-1038`. The merge includes #1062, #1065 and Files/open parity. There was no push, deploy or issue closure. Kept the deployed migration prefix: Auth 13, DB 15, Files 24, Mail 11 and Notes 32. Mail 12–17 and Auth 14 follow that prefix. Extended the existing production-copy upgrade test. It now checks both the older production schema and the deployed 7b prefix. Old receipts, Mail rows, Security state and Notes cursors survive. Only seven proxy/Auth additions are pending after 7b. A second startup makes no backup. Repaired cached SELECT writer contention. The pre-7b Mail snapshot used a writer transaction for every SELECT. 7b added a separate Security write connection. An unchanged SELECT still needed the ordinary pool and SQLite's write lock. It could therefore wait behind Index or Security work. The snapshot now checks the complete committed view in one WAL reader transaction. A change releases the reader and uses the single writer. Both passes use the same SQL, ownership checks and limits. Readers never advance durable clocks. The regression holds each writer separately. The old snapshot fails; the repaired snapshot completes before either writer is released. Added fixed-code diagnostics for SQL errors and every SELECT resource refusal. The harness reports no arbitrary wire, SQL, credential or provider text. Extended the existing bench profile with held-writer SELECT samples and a 12-client burst. The sampler also has a 100,000-message case. Repaired the three stale Mail tests against the current contracts: - SSE: require the initial canonical folder refresh, the exact owner event and no foreign event. DESIGN §§53, 59. - Worker pools: pin sync, IDLE, mutation and transfer limits. A stateful barrier keeps all three IDLE waits open while ordinary jobs finish. §53. - FETCH recovery: require all 80 UIDs and exactly one bounded `UID SEARCH UID 1:80`. Keep membership validation. §§45, 59. Formatted the imported Notes reconciliation test so the workspace fmt gate passes. Its behavior and assertions stay unchanged. ## Evidence The product repair is `9c8d1e7630aafae04c43036cba7b80fd64e2b932`. The expanded topology regression is `a6e0ff6a5`. The acceptance executable was copied before the server test build. Its SHA-256 is `d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2`. The later regression commit changes tests and instructions only. The fixed-product single-account phase passes: one User, one Connected Account, 12 clients, 1,808.52 seconds including receipt audits. It reports 18,402 SELECT OK and 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each APPEND, STORE, COPY and MOVE succeed. All 20 exact upstream and 20 local receipts pass. Required SELECT has no refusal. Expected pending-transfer NO responses are retried only after explicit refusal. ```text {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` There are no mixed-soak failures or failure reasons. Peak RSS is 280,340 KiB (273.77 MiB), peak FDs 171. After client and receipt sockets close: 145,172 KiB RSS and 127 FDs. Readiness and retained web sessions pass. These are local fixture/debug observations. Evidence is `artifacts/round2-one-account-fixed.jsonl`. The unmodified merged product also completed a diagnostic 1,808.72-second phase before the repair: 15,953 SELECT OK, no failures, and 20 exact upstream and local receipts. This is separate from fixed-product acceptance. The old snapshot failed the held-Security-writer regression in 2.24 seconds. The fixed full Mail suite passes that same test with both writers held in turn. The older 872.41-second refusal did not record an SQL class. Do not assign a specific SQLite error code to that historical run. Startup was measured on the idle host at `10.69.69.63`, with `/root/perf.lock`. Load average was 0.20 / 0.17 / 0.08. The existing test passed in 3.34 seconds; total wall time was 3.40 seconds. No build ran on that host. Two initial attempts lacked debug frontend paths and failed before measuring startup. They are excluded. The first full local server suite passed: 219 passed, no failures. The later fixed-product run, concurrent with the soak on the busy host, hit only the existing startup deadline at 15.15 seconds. This is a SLOW-only result. The 15-second limit stays unchanged. ## Release profile The updated profile completes on the perf VM under `/root/perf.lock`. The Mail test executable was built locally in release mode. The VM compiled no code. Cached SELECT samples and each 12-client burst hold the writer occupied. | Profile | Median per-run SELECT p50 / p95 | Mean worker CPU | Mean / peak worker RSS | Largest 12-client SELECT duration | |---|---|---|---|---| | 10,001 messages, 3 serial runs | 134.51 / 176.76 ms | 147.54% | 111.03 / 153.00 MiB | 1,696.68 ms | | 100,000 messages | 1,953.88 / 5,540.95 ms | 163.04% | 848.05 / 1,205.99 MiB | 18,724.86 ms | | 3 workers, each with 100,000 messages and its own Index | 1,790.76 / 4,051.31 ms | 113.24% | 891.19 / 1,205.86 MiB | 24,210.30 ms | CPU and RSS include seeding, metadata and queue phases. They are not SELECT-only resource figures. CPU can exceed 100% because SQLite uses multiple threads. The load averages inside the lock are 6.125 / 4.5923 / 2.1152 before and 7.5220 / 5.5894 / 3.3701 after. The starting load includes the preceding debug diagnostic run. That incomplete debug run was stopped and is excluded. Do not treat this as a cold quiet-host baseline. `docs/perf/baseline.json` has no comparable cached SELECT profile. `mail.accounts` measures HTTP account listing (p50 1.3 ms / p95 3.8 ms). It cannot establish a SELECT regression. The existing large-snapshot cost remains visible and is filed as [#1070](https://git.kayg.org/kayg/calternal/issues/1070). This profile is not real-provider 100,000-message sync or 50-client acceptance. Evidence is `artifacts/round2-select-perf-vm-release.log`. ## Files Round-2 edits, excluding the incoming merge: - `crates/plugins/mail/src/proxy.rs` and `proxy_tests.rs`. - `crates/plugins/mail/src/routes.rs` and `sync.rs`. - `crates/calternal-server/src/upgrade_tests.rs`. - `crates/plugins/notes/src/lib.rs` (format only). - `tests/adversarial/mail_proxy.py`, `test_mail_proxy.py`, `mail_stress.mjs` and `mail-sync.md`. - `bench/mail-sync.py` and this report. ## UX gaps closed Unchanged cached Mail views can be read while Index or Security writes are active. UID, revision and message identity checks remain intact. This round adds no UI. The earlier job's production screenshots remain its visual evidence; this round makes no new visual claim. ## UX gaps left and known gaps - #1068 remains: phone Settings sheet title and Copy link checks. - The latest busy-host server gate has one startup SLOW-only failure. - #1038 still has no complete 50-client long run on both account groups, 100,000-message initial-sync acceptance, 50 MB/zero-byte/10,000-folder matrix or download storm on the final product. Earlier short diagnostics do not replace these cases. These gaps prevent a full #1038 acceptance claim and READY FOR STAGING remains no. - The older SELECT refusal was unclassified. The held-writer defect is reproduced and repaired; no exact error code is inferred for that run. - Historical comparison attempts were stopped when Cargo reused artifacts from an archived source tree. They are excluded. The controlled regression uses the old snapshot source in the current worktree and is valid. - #1070 tracks large cached SELECT latency and memory. There is no comparable baseline to establish a regression. - Four existing Svelte warnings remain. There are no Svelte errors. ## Decisions Use a checked WAL reader pass for unchanged snapshots. Keep all changes on the single writer. This is an implementation choice consistent with DESIGN §§2, 53, 59. Do not change durability, deadlines or cache bounds. Use the canonical SSE hint, four bounded worker pools and bounded UID search already implemented on dev. Update the stale test contracts while keeping or strengthening what they prove. Keep the startup product and 15-second test limit. The idle measurement does not justify a limit increase. Report the later busy-host failure verbatim. Measure the existing component profile in release mode on the perf VM. Keep its cache timings separate from the local debug protocol phase. File the large-snapshot performance gap as #1070. Do not call it a baseline regression without comparable endpoint, data and build measurements. ## For the merge round Run the combined web suite with `cd apps/web && bun run test` and the combined startup gate with `cargo test -p calternal-server -- --test-threads=4`. The startup test must show that upgrade backfills do not delay HTTP readiness. Recheck #1068. These do not replace the completed requested single-account phase. The original #1038 large-data and 50-client gaps remain listed above. ## Gate output, verbatim excerpts `cargo fmt --check` exits 0 with no output. Cargo commands use `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4` and the worktree `target/tmp`. The preset target directory was not changed. Tests use `--test-threads=4` unless a focused test says 1. ### calternal-imap ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 21s Finished `test` profile [unoptimized + debuginfo] target(s) in 35.52s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.35s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s ``` ### calternal-plugin-mail ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.80s Finished `test` profile [unoptimized + debuginfo] target(s) in 16.17s test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-server ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 42.71s Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 00s test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out; finished in 15.15s test result: FAILED. 218 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 106.38s ``` ### calternal-auth ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 26s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 43s test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 133.11s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-db ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.23s Finished `test` profile [unoptimized + debuginfo] target(s) in 36.98s test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.44s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.26s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.44s test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.25s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-plugin-notes ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 47s Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 59s test result: ok. 264 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 509.86s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.69s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### Web check ```text perf-lint: parse Rust product sources perf-lint: parse browser product sources perf-lint: validate coverage and architecture perf-lint: PASS; 0 violations; 19340 scoped exceptions svelte-check found 0 errors and 4 warnings in 3 files ``` ### Focused Vitest ```text Test Files 4 passed (4) Tests 18 passed (18) ``` ### Probe diagnostics `python3 -m unittest discover -s tests/adversarial -p test_mail_proxy.py` ```text ...................... ---------------------------------------------------------------------- Ran 22 tests in 0.216s OK ``` ## Cleanup The owned local and perf-VM executables, temporary fixtures and web build output are removed. Review artifacts remain in the ignored `artifacts/` directory. No screenshot or binary is committed. `cargo clean` exits 0: ```text Removed 26944 files, 20.4GiB total ```
kayg commented 2026-10-04 22:11:49 +00:00
Author
Owner
Copy link

Started applemail-accept on job/applemail-accept, base 8091d5788a. I will use a fresh local Dovecot TLS fixture with 2,000 messages, install the generated device-setup profile on the locked Mac VM, and audit native edits with read-only upstream receipts and production web screenshots. No pushes or deployments.

Started applemail-accept on job/applemail-accept, base 8091d5788ad86aaee1da831801031110462c6f6d. I will use a fresh local Dovecot TLS fixture with 2,000 messages, install the generated device-setup profile on the locked Mac VM, and audit native edits with read-only upstream receipts and production web screenshots. No pushes or deployments.
kayg commented 2026-10-04 22:44:26 +00:00
Author
Owner
Copy link

Native acceptance update: the generated profile installed on macOS 27.0 (26A428). Apple Mail sees the lab account, the real Inbox has 2,009 rows and EmptyMe has two. The old lab profile was removed; the unrelated managed profile was retained. The Mac lock remains held.

Setup findings: non-interactive NetBird VNC and IMAP forwards ended during setup. Interactive forwards are now in use and both native ports were checked. The browser resolves calternal.lab, but the screenshot helper's separate HTTP client does not; the new coordinator saves appearance through the authenticated browser API. Six production Inbox screenshots now cover 390/820/1440, light/dark, macOS.

The native flag edit has one read-only upstream receipt with Seen, Flagged and the Apple Mail flag keyword. Full initial body download is still running; no full-download pass is claimed. The separate read check remains pending. The only Mail crash report remains the old Mail-2026-10-04-103511.ips. Acceptance helpers and their read-only audit regression are committed through d15bf6416. No product code has changed.

Native acceptance update: the generated profile installed on macOS 27.0 (26A428). Apple Mail sees the lab account, the real Inbox has 2,009 rows and EmptyMe has two. The old lab profile was removed; the unrelated managed profile was retained. The Mac lock remains held. Setup findings: non-interactive NetBird VNC and IMAP forwards ended during setup. Interactive forwards are now in use and both native ports were checked. The browser resolves calternal.lab, but the screenshot helper's separate HTTP client does not; the new coordinator saves appearance through the authenticated browser API. Six production Inbox screenshots now cover 390/820/1440, light/dark, macOS. The native flag edit has one read-only upstream receipt with Seen, Flagged and the Apple Mail flag keyword. Full initial body download is still running; no full-download pass is claimed. The separate read check remains pending. The only Mail crash report remains the old Mail-2026-10-04-103511.ips. Acceptance helpers and their read-only audit regression are committed through d15bf6416. No product code has changed.
kayg commented 2026-10-04 23:25:17 +00:00
Author
Owner
Copy link

Real Apple Mail acceptance completed on job/applemail-accept.

Base: 8091d5788ad86aaee1da831801031110462c6f6d. Head: 30ad9ce8cb3982feb88c6b4d246fd5bf92f2b499. origin/dev was fetched and merged once before gates; it was already included (9fb9a4bfb2488152c83d50c55441ff5f43b572b2). No push or deploy.

Built: a private native acceptance coordinator, read-only upstream receipt auditor and its regression test, native AppleScript selection/move driver, VNC multi-message drag support, and documented lab recovery rules/results. No production Rust or web source changed. No product defect was found in the requested action results.

Files: tests/adversarial/apple_mail_accept.mjs, apple_mail_native.applescript, apple_mail_receipts.py, apple_mail_vnc.py, test_apple_mail_receipts.py, and mail-sync.md.

Environment: real macOS 27.0 (26A428), Apple Mail 16.0 (3901.100.1.1.11), isolated Dovecot TLS fixture, branch-built mail-test-provider server and real production web build. Binary SHA-256 is in the receipts bundle. All generated fixture identities were audited with EXAMINE + BODY.PEEK. Each seeded identity has one receipt with its original size and SHA-256. Final counts: Inbox 2,002; Archive 1; Drafts 1; Trash 3; Autumn 5; EmptyMe 0. No transfer remains pending.

Step Result Evidence
Device profile + initial download PASS: macOS System Settings installed the generated device profile. Native download indicator cleared; 2,000 fixture messages + 11 cases accounted for. mail-download-start.png; native-download-complete-counts.txt in receipts
Read PASS: One Inbox receipt has Seen; exact baseline MIME. native-read-settled.png
Flag PASS: One Inbox receipt has Flagged and Seen; web flags agree. native-flagged.png
Archive PASS: One exact Archive receipt; Inbox source absent. native-archive.png
Folder move PASS: One exact Autumn receipt; Inbox source absent. native-after-final-relaunch.png
Delete to Trash PASS: One exact Trash receipt; Inbox source absent. native-delete.png
Drag two messages PASS: Real pointer drag of a rendered two-row selection. One exact Autumn receipt per message; both Inbox sources absent. drag-selected.png, native-drag.png
Draft PASS: One Drafts receipt with Draft flag. Generated recipient and body checked with read-only BODY.PEEK. native-draft.png
Empty mailbox PASS: EmptyMe count 0; both messages have one exact Trash receipt. native-empty.png
Quit/relaunch mid-sync PASS: Native quit/relaunch during initial download and again with queued work. Final real and unified native counts agree. final-quit-relaunch.txt and final-native-counts-crashes.txt in receipts
Ethernet off/on during move PASS: Ethernet Disabled for 18 s. Native move dispatched during outage. After tunnel restoration, one exact Autumn receipt; source absent. network-interrupt.txt, network-interrupted-receipts.json, network-final-receipts.json in receipts
Server restart during activity PASS: Interrupted transfer completed on attempt 2 in 142.381 s from creation. No error or pending transfers. One exact Autumn receipt; source absent. restart-job-receipt.json and final-confirmed-receipts.json in receipts
Mail crashes after every step PASS: No new report. Only Mail-2026-10-04-103511.ips existed before and after the session. Per-step checks retained locally; final-native-counts-crashes.txt in receipts

Read-only receipts, exact MIME assertions, native counts, interruption logs, Job receipt and focused gates.

Production web evidence: each bundle has 390, 820 and 1440 px in light and dark, with macOS platform emulation: Inbox/read/flag, Archive, Autumn/moves/drag/recovery, Drafts, Trash/delete/empty, EmptyMe. Screenshots are review artifacts; none are committed.

Gate output, verbatim:

cargo fmt --check: no output, exit 0.
node --check tests/adversarial/apple_mail_accept.mjs: no output, exit 0.
git diff --check: no output, exit 0.

python3 -m unittest discover -s tests/adversarial -p 'test_apple_mail_receipts.py':

.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK

Branch server build (OPENSSL_NO_VENDOR=1, required low-memory Cargo environment):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 58s

Production web build final line:

Compressed 534 static variants; saved 9822299 bytes.

No Rust crate or web application source was changed, so per-crate clippy/test and focused Vitest gates were not applicable. The native scenarios requested by this verification job all ran here.

Known gaps: initial full-download completion uses the cleared native indicator and counts; it does not claim a native MIME hash for every one of the 2,000 fixture messages. The restart timing does not prove a cut inside upstream COPY upload. The recovered transfer took 142.381 s from creation; this run does not set a performance baseline. SMTP sending is deferred by DESIGN §53 and was not requested.

Findings and fixes to the lab: a bulk raw-source AppleEvent during download blocked Mail's main thread while it waited for rawSource fetches. The query was stopped, Mail restarted and the checks continued after download settled. No new crash report resulted. Separate NetBird reverse tunnels were required: combining reverse forwards mapped the listeners to the final SMTP destination. Correct IMAP greeting was checked after restoring separate tunnels. These are documented automation findings, not claims of production crashes.

Decisions: use 11 isolated named messages in addition to the 2,000-message fixture so each action has one stable Message-ID. Interpret empty mailbox as deleting every message in the isolated EmptyMe folder to Trash. Use native AppleScript moves for scheduled interruptions and real VNC pointer/keyboard actions for the requested drag and ordinary toolbar actions. No undecided product design was implemented.

UX gaps closed: none in production code; this is acceptance work. UX gaps left: none found in the requested native actions.

Cleanup: generated lab profile removed; after Mail relaunch the lab account list was empty. The unrelated managed profile remains. Ethernet is enabled. Mac private profile/helper files removed; own tunnels, VNC, server and provider stopped; Mac lock released. All retained private lab attempts and User data removed. Cargo cleanup output:

     Removed 8434 files, 7.5GiB total

Web build output removed. Local evidence stays under artifacts/applemail-accept/.

APPLE MAIL ACCEPTANCE: pass

Real Apple Mail acceptance completed on `job/applemail-accept`. Base: `8091d5788ad86aaee1da831801031110462c6f6d`. Head: `30ad9ce8cb3982feb88c6b4d246fd5bf92f2b499`. `origin/dev` was fetched and merged once before gates; it was already included (`9fb9a4bfb2488152c83d50c55441ff5f43b572b2`). No push or deploy. Built: a private native acceptance coordinator, read-only upstream receipt auditor and its regression test, native AppleScript selection/move driver, VNC multi-message drag support, and documented lab recovery rules/results. No production Rust or web source changed. No product defect was found in the requested action results. Files: `tests/adversarial/apple_mail_accept.mjs`, `apple_mail_native.applescript`, `apple_mail_receipts.py`, `apple_mail_vnc.py`, `test_apple_mail_receipts.py`, and `mail-sync.md`. Environment: real macOS 27.0 (26A428), Apple Mail 16.0 (3901.100.1.1.11), isolated Dovecot TLS fixture, branch-built `mail-test-provider` server and real production web build. Binary SHA-256 is in the receipts bundle. All generated fixture identities were audited with EXAMINE + BODY.PEEK. Each seeded identity has one receipt with its original size and SHA-256. Final counts: Inbox 2,002; Archive 1; Drafts 1; Trash 3; Autumn 5; EmptyMe 0. No transfer remains pending. | Step | Result | Evidence | |---|---|---| | Device profile + initial download | PASS: macOS System Settings installed the generated device profile. Native download indicator cleared; 2,000 fixture messages + 11 cases accounted for. | [mail-download-start.png](https://git.kayg.org/attachments/b22b404f-5d6b-40c3-8df6-d2459de4b93a); native-download-complete-counts.txt in receipts | | Read | PASS: One Inbox receipt has Seen; exact baseline MIME. | [native-read-settled.png](https://git.kayg.org/attachments/2641efe4-edbd-451a-a99d-2881a20770a3) | | Flag | PASS: One Inbox receipt has Flagged and Seen; web flags agree. | [native-flagged.png](https://git.kayg.org/attachments/44056b3e-803d-49df-9e4e-f4748c13376c) | | Archive | PASS: One exact Archive receipt; Inbox source absent. | [native-archive.png](https://git.kayg.org/attachments/ebc55b0a-4608-4bb4-96ed-9bc2d7c47f1b) | | Folder move | PASS: One exact Autumn receipt; Inbox source absent. | [native-after-final-relaunch.png](https://git.kayg.org/attachments/12c1dc02-b6ba-436d-b295-fbe4fdb4a2dd) | | Delete to Trash | PASS: One exact Trash receipt; Inbox source absent. | [native-delete.png](https://git.kayg.org/attachments/4db9e8d2-93cf-4ba2-bc7c-14c2ce090bda) | | Drag two messages | PASS: Real pointer drag of a rendered two-row selection. One exact Autumn receipt per message; both Inbox sources absent. | [drag-selected.png](https://git.kayg.org/attachments/67ebcdbb-4bb9-4f2a-846c-21a569541759), [native-drag.png](https://git.kayg.org/attachments/174d2763-840b-4747-8356-c25a50d6e8a7) | | Draft | PASS: One Drafts receipt with Draft flag. Generated recipient and body checked with read-only BODY.PEEK. | [native-draft.png](https://git.kayg.org/attachments/e9f36b4f-0332-45c3-a2a9-a8ecdf9ffa8a) | | Empty mailbox | PASS: EmptyMe count 0; both messages have one exact Trash receipt. | [native-empty.png](https://git.kayg.org/attachments/c977289c-cf1b-4366-85c5-a18ced2bead8) | | Quit/relaunch mid-sync | PASS: Native quit/relaunch during initial download and again with queued work. Final real and unified native counts agree. | final-quit-relaunch.txt and final-native-counts-crashes.txt in receipts | | Ethernet off/on during move | PASS: Ethernet Disabled for 18 s. Native move dispatched during outage. After tunnel restoration, one exact Autumn receipt; source absent. | network-interrupt.txt, network-interrupted-receipts.json, network-final-receipts.json in receipts | | Server restart during activity | PASS: Interrupted transfer completed on attempt 2 in 142.381 s from creation. No error or pending transfers. One exact Autumn receipt; source absent. | restart-job-receipt.json and final-confirmed-receipts.json in receipts | | Mail crashes after every step | PASS: No new report. Only Mail-2026-10-04-103511.ips existed before and after the session. | Per-step checks retained locally; final-native-counts-crashes.txt in receipts | [Read-only receipts, exact MIME assertions, native counts, interruption logs, Job receipt and focused gates](https://git.kayg.org/attachments/1c31ee6f-55e3-4a35-8ab5-f2f63b58f98d). Production web evidence: each bundle has 390, 820 and 1440 px in light and dark, with macOS platform emulation: [Inbox/read/flag](https://git.kayg.org/attachments/ffc18348-2360-401f-ad39-dd1615ea3d8d), [Archive](https://git.kayg.org/attachments/fbe23491-d6cb-4b34-a1eb-9c2a72600914), [Autumn/moves/drag/recovery](https://git.kayg.org/attachments/dfca2a2d-2f93-4eef-817a-53bf187b7a5b), [Drafts](https://git.kayg.org/attachments/ee41ed28-1dfd-420e-88ee-20e5a5e85565), [Trash/delete/empty](https://git.kayg.org/attachments/c1eb92ea-e73e-4423-a934-4dade170dccf), [EmptyMe](https://git.kayg.org/attachments/3fc6eb79-ceb9-4fdd-8e04-83594edc9160). Screenshots are review artifacts; none are committed. Gate output, verbatim: `cargo fmt --check`: no output, exit 0. `node --check tests/adversarial/apple_mail_accept.mjs`: no output, exit 0. `git diff --check`: no output, exit 0. `python3 -m unittest discover -s tests/adversarial -p 'test_apple_mail_receipts.py'`: ``` . ---------------------------------------------------------------------- Ran 1 test in 0.002s OK ``` Branch server build (`OPENSSL_NO_VENDOR=1`, required low-memory Cargo environment): ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 58s ``` Production web build final line: ``` Compressed 534 static variants; saved 9822299 bytes. ``` No Rust crate or web application source was changed, so per-crate clippy/test and focused Vitest gates were not applicable. The native scenarios requested by this verification job all ran here. Known gaps: initial full-download completion uses the cleared native indicator and counts; it does not claim a native MIME hash for every one of the 2,000 fixture messages. The restart timing does not prove a cut inside upstream COPY upload. The recovered transfer took 142.381 s from creation; this run does not set a performance baseline. SMTP sending is deferred by DESIGN §53 and was not requested. Findings and fixes to the lab: a bulk raw-source AppleEvent during download blocked Mail's main thread while it waited for rawSource fetches. The query was stopped, Mail restarted and the checks continued after download settled. No new crash report resulted. Separate NetBird reverse tunnels were required: combining reverse forwards mapped the listeners to the final SMTP destination. Correct IMAP greeting was checked after restoring separate tunnels. These are documented automation findings, not claims of production crashes. Decisions: use 11 isolated named messages in addition to the 2,000-message fixture so each action has one stable Message-ID. Interpret empty mailbox as deleting every message in the isolated EmptyMe folder to Trash. Use native AppleScript moves for scheduled interruptions and real VNC pointer/keyboard actions for the requested drag and ordinary toolbar actions. No undecided product design was implemented. UX gaps closed: none in production code; this is acceptance work. UX gaps left: none found in the requested native actions. Cleanup: generated lab profile removed; after Mail relaunch the lab account list was empty. The unrelated managed profile remains. Ethernet is enabled. Mac private profile/helper files removed; own tunnels, VNC, server and provider stopped; Mac lock released. All retained private lab attempts and User data removed. Cargo cleanup output: ``` Removed 8434 files, 7.5GiB total ``` Web build output removed. Local evidence stays under `artifacts/applemail-accept/`. APPLE MAIL ACCEPTANCE: pass
kayg commented 2026-10-05 01:01:24 +00:00
Author
Owner
Copy link

Merge round 9 started on job/merge-round-9, base 2b6c77c14. Integrate mail round 5a5b067a0, Mail UI d6982f4e9, and Files identity fix da2eb5f91; preserve production migrations and run the requested merge-round gates. No push or deploy.

Merge round 9 started on `job/merge-round-9`, base `2b6c77c14`. Integrate mail round `5a5b067a0`, Mail UI `d6982f4e9`, and Files identity fix `da2eb5f91`; preserve production migrations and run the requested merge-round gates. No push or deploy.
kayg commented 2026-10-05 01:12:03 +00:00
Author
Owner
Copy link

Integration is committed: mail merge 5aa12d1a7, UI merge 93cb77054, Files cherry-pick a3767ba87, bounded soak duration 4f2811892. git fetch origin && git merge origin/dev returned Already up to date.

Production prefix is preserved: Auth 13, DB 15, Files 24, Mail 11, Notes 32. Auth adds 14; Mail adds 12–17. Both production upgrade tests remain in the Server suite.

Completed gates:

fmt exit=0
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 05s
 Test Files  226 passed (226)
      Tests  1523 passed (1523)
 Test Files  21 passed (21)
      Tests  433 passed (433)
Ran 34 tests in 2.870s

OK

The Files identity change requires source-ledger reconciliation beyond hash replacement because its query and call syntax changed. Exact tested in-memory/Index call bindings will remove conservative unresolved-call findings; real filesystem metadata stays scoped debt. The ratchet will not increase. Rust tests and the feature-enabled Server build continue.

Integration is committed: mail merge `5aa12d1a7`, UI merge `93cb77054`, Files cherry-pick `a3767ba87`, bounded soak duration `4f2811892`. `git fetch origin && git merge origin/dev` returned `Already up to date.` Production prefix is preserved: Auth 13, DB 15, Files 24, Mail 11, Notes 32. Auth adds 14; Mail adds 12–17. Both production upgrade tests remain in the Server suite. Completed gates: ``` fmt exit=0 Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 05s Test Files 226 passed (226) Tests 1523 passed (1523) Test Files 21 passed (21) Tests 433 passed (433) Ran 34 tests in 2.870s OK ``` The Files identity change requires source-ledger reconciliation beyond hash replacement because its query and call syntax changed. Exact tested in-memory/Index call bindings will remove conservative unresolved-call findings; real filesystem metadata stays scoped debt. The ratchet will not increase. Rust tests and the feature-enabled Server build continue.
kayg commented 2026-10-05 01:20:27 +00:00
Author
Owner
Copy link

Mail layouts found a real shared gesture conflict: Split divider drag received pointerdown and one pointermove, then the enclosing pull gesture took pointer capture. The divider painted 392 px but persisted the previous 376 px. Existing Split height survives a reload failed unchanged.

Fix: the separator stops pointerdown propagation after accepting its primary-button drag. Root capture listeners still receive input-mode events. Ancestor pull gestures cannot arm for that drag. A focused regression checks both capture-listener delivery and absence of the ancestor pull event, then checks the settled height. No existing expectation is changed.

Notes idle passed with 700 Notes, zero idle Files events and four Files events for one edit. Web check passed:

svelte-check found 0 errors and 4 warnings in 3 files
Mail layouts found a real shared gesture conflict: Split divider drag received pointerdown and one pointermove, then the enclosing pull gesture took pointer capture. The divider painted 392 px but persisted the previous 376 px. Existing `Split height survives a reload` failed unchanged. Fix: the separator stops pointerdown propagation after accepting its primary-button drag. Root capture listeners still receive input-mode events. Ancestor pull gestures cannot arm for that drag. A focused regression checks both capture-listener delivery and absence of the ancestor pull event, then checks the settled height. No existing expectation is changed. Notes idle passed with 700 Notes, zero idle Files events and four Files events for one edit. Web check passed: ``` svelte-check found 0 errors and 4 warnings in 3 files ```
kayg commented 2026-10-05 01:27:49 +00:00
Author
Owner
Copy link

Mail layouts passed after fixing the shared separator/pull gesture conflict. The unchanged persistence assertion now passes.

Review artifacts (macOS emulation; 390, 820, 1440 px; light and dark):

The first set includes an app-update notice caused by the earlier copied server/web identity mismatch. A matched build is ready; the final set will replace this evidence after the running Mail soak frees the browser. Visual approval stays with the orchestrator.

Mail layouts passed after fixing the shared separator/pull gesture conflict. The unchanged persistence assertion now passes. Review artifacts (macOS emulation; 390, 820, 1440 px; light and dark): - [merge-round-9-mail-layouts.zip](https://git.kayg.org/attachments/83f351a3-27d9-4ba1-81ea-ac4b29fa9b28) - [columns-empty-desktop-paper-light.png](https://git.kayg.org/attachments/5c9cb5e8-782e-49b4-b00d-eee2d0c1a5bd) - [columns-empty-desktop-tokyo-night-dark.png](https://git.kayg.org/attachments/09c1f87d-9166-4a79-9ac3-c51ba728717a) - [columns-empty-tablet-paper-light.png](https://git.kayg.org/attachments/ece592d3-52ba-49d3-95b5-562aa57367c9) - [columns-empty-tablet-tokyo-night-dark.png](https://git.kayg.org/attachments/5cd57623-8df0-4a1c-a2f2-54ee4db2b0a1) The first set includes an app-update notice caused by the earlier copied server/web identity mismatch. A matched build is ready; the final set will replace this evidence after the running Mail soak frees the browser. Visual approval stays with the orchestrator.
kayg commented 2026-10-05 01:40:32 +00:00
Author
Owner
Copy link

All requested crate gates completed. No Rust test expectation was changed by this integration. Both production-schema upgrade tests passed.

Verbatim gate completion lines:

calternal-imap

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.12s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-mail

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 05s
test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.52s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-files

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 58s
test result: ok. 239 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 168.21s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-auth

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 30.34s
test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 78.49s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-db

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.22s
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.07s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.54s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s
test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.21s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 03s
test result: ok. 221 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 57.68s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 25.06s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s

calternal-plugin-notes

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.00s
test result: ok. 265 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 168.67s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

One-account stress passed with 12 clients for 600 seconds, no command failures, and no cached snapshot failures. Three-account stress continues. The remaining live checks are serialized.

All requested crate gates completed. No Rust test expectation was changed by this integration. Both production-schema upgrade tests passed. Verbatim gate completion lines: `calternal-imap` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.12s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `calternal-plugin-mail` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 05s test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.52s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `calternal-plugin-files` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 58s test result: ok. 239 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 168.21s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `calternal-auth` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 30.34s test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 78.49s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `calternal-db` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.22s test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.07s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.54s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.21s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `calternal-server` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 03s test result: ok. 221 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 57.68s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 25.06s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s ``` `calternal-plugin-notes` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.00s test result: ok. 265 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 168.67s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` One-account stress passed with 12 clients for 600 seconds, no command failures, and no cached snapshot failures. Three-account stress continues. The remaining live checks are serialized.
kayg commented 2026-10-05 02:13:36 +00:00
Author
Owner
Copy link

The live authorization matrix sent 2,734 requests over 392 operations. It found two HTTP 500 responses on GET /api/v1/groups for anonymous and Public link requests: Axum rejected the missing PluginRequestContext before the handler could return its documented 403. The route now checks User identity and data scope before extension extraction. A focused Axum regression covers absent identity, Public link identity, limited scope and a valid User. Server gates and the live matrix will verify the fix.

The screenshot alignment check also found App Password key icons centered against wrapped descriptions. The caller now uses SettingsRow title alignment and the existing Connected Accounts cap-box CSS recipe. No runtime layout measurement was added.

The live authorization matrix sent 2,734 requests over 392 operations. It found two HTTP 500 responses on GET /api/v1/groups for anonymous and Public link requests: Axum rejected the missing PluginRequestContext before the handler could return its documented 403. The route now checks User identity and data scope before extension extraction. A focused Axum regression covers absent identity, Public link identity, limited scope and a valid User. Server gates and the live matrix will verify the fix. The screenshot alignment check also found App Password key icons centered against wrapped descriptions. The caller now uses SettingsRow title alignment and the existing Connected Accounts cap-box CSS recipe. No runtime layout measurement was added.
kayg commented 2026-10-05 02:24:14 +00:00
Author
Owner
Copy link

Fixed the two GET /api/v1/groups missing-context HTTP 500 responses in d20966258. The Axum regression rejects absent identity, Public link identity and missing data scope with 403; a valid User reaches the handler. Full calternal-server recheck: 222 main tests passed, 9 ignored; source guard integration 1 passed; private Index permissions 1 passed. Clippy -D warnings passed. Both production-copy upgrade tests passed again. The initial source-guard recheck caught fingerprints changed by a later module comment; refreshed the same scopes and limits, then the full server suite passed.

App Password cap-height alignment is committed separately as 42d045777. Its 3 focused tests passed; bun run check reports 0 errors and the same 4 warnings. Source guard: PASS; 0 violations; 18996 scoped exceptions. Matching production builds/screenshots and the full web recheck are in progress.

The 50-client ten-minute one-account phase passed, including retained web sessions and no snapshot failures. Three-account phase is running. The 12-client one/three-account diagnostic phases, all 12 SIGKILL receipt recovery cases, provider restart/UIDVALIDITY recovery, literal-cut repair regressions, Mail proxy focused check, Mail layouts and cross-User matrix also passed. This does not claim the full 30-minute/large-fixture acceptance requested in the original issue.

Fixed the two GET /api/v1/groups missing-context HTTP 500 responses in d20966258. The Axum regression rejects absent identity, Public link identity and missing data scope with 403; a valid User reaches the handler. Full calternal-server recheck: 222 main tests passed, 9 ignored; source guard integration 1 passed; private Index permissions 1 passed. Clippy -D warnings passed. Both production-copy upgrade tests passed again. The initial source-guard recheck caught fingerprints changed by a later module comment; refreshed the same scopes and limits, then the full server suite passed. App Password cap-height alignment is committed separately as 42d045777. Its 3 focused tests passed; bun run check reports 0 errors and the same 4 warnings. Source guard: PASS; 0 violations; 18996 scoped exceptions. Matching production builds/screenshots and the full web recheck are in progress. The 50-client ten-minute one-account phase passed, including retained web sessions and no snapshot failures. Three-account phase is running. The 12-client one/three-account diagnostic phases, all 12 SIGKILL receipt recovery cases, provider restart/UIDVALIDITY recovery, literal-cut repair regressions, Mail proxy focused check, Mail layouts and cross-User matrix also passed. This does not claim the full 30-minute/large-fixture acceptance requested in the original issue.
kayg commented 2026-10-05 02:34:59 +00:00
Author
Owner
Copy link

The requested 50-client ten-minute phases both passed on the copied merged server:

  • One User / Connected Account: 614.69 s with audits; 25,120 SELECT OK; 25,070 each SEARCH/FETCH/IDLE/EXPUNGE OK; 50 each successful APPEND/STORE/COPY/MOVE; 50 exact upstream and local receipts. No failures, failure reasons or snapshot refusals. Retained web sessions PASS.
  • Three Users / Connected Accounts: 612.84 s with audits; 25,708 SELECT OK; 25,658 each SEARCH/FETCH/IDLE/EXPUNGE OK; 50 each successful APPEND/STORE/COPY/MOVE; 50 exact upstream and local receipts. No failures, failure reasons or snapshot refusals. Retained web sessions PASS.
  • Local debug resources: one-account peak/end RSS 354,628/156,728 KiB, peak/end FDs 255/201; three-account peak/end RSS 202,908/202,908 KiB, peak/end FDs 255/195. These observations cover ten-minute phases, not a long-term memory bound or a release perf baseline.

Full web recheck passed: 226 files, 1,524 tests. Matching production web/server are tied to d209662581; server SHA-256 463ffe23a2ae1e3df5955fde0a4608d3b29f42e8ab2c2e88464bfd3ecae8854a. Contract regeneration changed no files. Corrected authorization, API robustness and final screenshot refresh are queued serially. No push or deploy.

The requested 50-client ten-minute phases both passed on the copied merged server: - One User / Connected Account: 614.69 s with audits; 25,120 SELECT OK; 25,070 each SEARCH/FETCH/IDLE/EXPUNGE OK; 50 each successful APPEND/STORE/COPY/MOVE; 50 exact upstream and local receipts. No failures, failure reasons or snapshot refusals. Retained web sessions PASS. - Three Users / Connected Accounts: 612.84 s with audits; 25,708 SELECT OK; 25,658 each SEARCH/FETCH/IDLE/EXPUNGE OK; 50 each successful APPEND/STORE/COPY/MOVE; 50 exact upstream and local receipts. No failures, failure reasons or snapshot refusals. Retained web sessions PASS. - Local debug resources: one-account peak/end RSS 354,628/156,728 KiB, peak/end FDs 255/201; three-account peak/end RSS 202,908/202,908 KiB, peak/end FDs 255/195. These observations cover ten-minute phases, not a long-term memory bound or a release perf baseline. Full web recheck passed: 226 files, 1,524 tests. Matching production web/server are tied to d209662581756d6050e62b5c566ac9a03379c4be; server SHA-256 463ffe23a2ae1e3df5955fde0a4608d3b29f42e8ab2c2e88464bfd3ecae8854a. Contract regeneration changed no files. Corrected authorization, API robustness and final screenshot refresh are queued serially. No push or deploy.
kayg commented 2026-10-05 02:44:49 +00:00
Author
Owner
Copy link

The one broad API round found a Finder probe transport failure: an oversized AppleDouble upload threw Broken pipe before the client read the expected 413. Its status and zero-byte cleanup expectations remain unchanged. The standalone client now reads a queued rejection after BrokenPipeError/ConnectionResetError; an absent response stays a finding. This matches the existing DAV early-response invariant (#471). Added two regressions. All 23 DAV contract/transport unit tests pass. A focused real-server DAV replay is queued to verify the original expectations.

Other current broad-round findings are labelled SLOW (Photos setup and Calendar duplicate concurrency). No SLOW threshold or existing expectation was changed.

The one broad API round found a Finder probe transport failure: an oversized AppleDouble upload threw Broken pipe before the client read the expected 413. Its status and zero-byte cleanup expectations remain unchanged. The standalone client now reads a queued rejection after BrokenPipeError/ConnectionResetError; an absent response stays a finding. This matches the existing DAV early-response invariant (#471). Added two regressions. All 23 DAV contract/transport unit tests pass. A focused real-server DAV replay is queued to verify the original expectations. Other current broad-round findings are labelled SLOW (Photos setup and Calendar duplicate concurrency). No SLOW threshold or existing expectation was changed.
kayg commented 2026-10-05 02:54:50 +00:00
Author
Owner
Copy link

Final merge-round-9 report

HEAD: 269b1b51b5774d79de08bfb31d058b3d248a7e7b
Branch: job/merge-round-9
Base: 2b6c77c14
Audit: docs/audits/merge-round-9.md

Merge round 9: Mail proxy release

READY FOR STAGING: yes, for this merge brief. Full #1038 acceptance is not claimed.

Built

Integrated job/mailround-1038 at 5a5b067a0 and job/mailui-1078
at d6982f4e9 into job/merge-round-9 from 2b6c77c14.
Cherry-picked the Files upload identity repair da2eb5f91 as a3767ba87.
Fetched origin and merged origin/dev once before the final gates. Git
reported Already up to date. There was no push, deploy or issue closure.

The release includes the CalternalDAV Mail proxy, provider sync repairs,
restart recovery, cached SELECT readers, the phone Settings title repair,
Mail columns empty state, enabled Tabs in the tray and folder loading Retry.
A sibling write no longer invalidates an upload when the folder identity,
inode and parent fingerprint still match.

Kept deployed migration prefixes: Auth 13, DB 15, Files 24, Mail 11 and
Notes 32. Auth 14 and Mail 12–17 follow those prefixes. Both production
schema-copy upgrade tests pass. OpenAPI, action and client contracts were
regenerated from code.

Fixed two findings during this round. Pull-down gestures could steal pointer
capture from a panel separator. The resize action now keeps the accepted
pointer gesture. A regression checks capture and bubble behavior. Anonymous
and Public link requests to GET /api/v1/groups got HTTP 500 before the
handler could reject them. The route now checks identity and data scope
before required extension extraction. Its Axum regression checks missing
identity, Public link identity, limited scope and a valid User.

The Finder probe now reads an early HTTP rejection after upload closure.
Its new regressions distinguish a queued 413 from a missing response. The
existing 413 and zero-byte cleanup expectations were kept. The focused live
DAV replay passed.

The Files performance ledger uses exact pure-operation and SQL bindings.
Filesystem observations remain unresolved debt. The combined ledger has
18,996 scoped exceptions, down from 19,340. Unresolved call sites fell from
13,616 to 13,272. No limit or ratchet ceiling was raised.

UX gaps closed

  • Empty Mail columns show a real empty state.
  • The tray shows every enabled Tab.
  • Failed folder loading has Retry.
  • Phone Settings retains its title and Copy link action.
  • Split width survives a completed pointer drag. The existing e2e expectation
    was kept.
  • App Password key icons use SettingsRow title alignment and the existing
    Connected Accounts cap-box CSS recipe when descriptions wrap.
  • Device read, unread, MOVE and EXPUNGE changes appear in the live web view
    without a reload. App Password revocation denies IMAP and SMTP access.

Decisions

  • Added a validated 600-second phase option for this brief's ten-minute soak.
    The full acceptance default stays at 1,800 seconds. The option cannot
    shorten a normal phase below 600 seconds.
  • Kept the existing 403 contract for Groups. Reject before Axum extraction.
  • Used the existing SettingsRow title-alignment option and cap-box recipe.
    No runtime layout measurement was added.
  • Used exact bindings for existing pure operations and Index SQL chains.
    Did not classify filesystem observations as pure.

Known gaps and UX gaps left

This report covers the merge brief. It does not close #1038 or claim its full
acceptance matrix. The 30-minute 50-client phases, 100,000-message initial
sync, 50 MB and zero-byte messages, 10,000 folders, deep trees and complete
download storms were not repeated on this merged build. The complete upstream
slow-delivery/BYE/NO/BAD/garbage matrix was not replayed here. The bounded fault
fixture used 1 MiB messages. #1070 tracks large cached SELECT latency and
memory. These local debug results are not a release performance baseline.

CREATE, RENAME and DELETE return clean unsupported IMAP responses. Successful
folder lifecycle races were not exercised. The provider restart check did
not establish an active interrupted command. The separate literal-cut checks
cover FETCH and APPEND interruption. Four existing Svelte warnings remain.
The cross-User matrix lists 36 identifier routes without a seeded local item.
The broad API round returned exit 1 with 20 SLOW findings and two other
findings. The Finder client defect is fixed in 27416d47d. The collection
discovery timeout is filed as #1081.
The focused DAV replay passed the unchanged 100-resource and Finder checks.
The timeout cause is not established. No crash, 5xx, data loss or denial
failure was observed in that broad round.
No owner Apple Mail or real macOS VM check was repeated in this worktree;
the integrated branch carries the earlier acceptance evidence. Screenshots
use macOS platform emulation. Claude must review visual quality.

Live scenarios

All probes use a throwaway local server and fixture. No provider account or
production data was used. The copied executables prevent Cargo test builds
from replacing a server used by a restart probe.

Scenario Result Evidence and limit
12 clients, one account, 600-second phase PASS 604.98 seconds with audits; 6,281 SELECT OK; 20 exact upstream/local write receipts; no failures
12 clients, three Users/accounts, 600-second phase PASS 606.57 seconds with audits; 6,302 SELECT OK; 20 exact upstream/local write receipts; no failures
50 clients, one account, 600-second phase PASS 614.69 seconds with audits; 25,120 SELECT OK; 50 exact upstream/local write receipts; no failures
50 clients, three Users/accounts, 600-second phase PASS 612.84 seconds with audits; 25,708 SELECT OK; 50 exact upstream/local write receipts; no failures
SIGKILL recovery PASS APPEND, COPY, MOVE and EXPUNGE at 0/100/500 ms: all 12 cases; exact MIME identity and receipt audits
Same-item races PASS MOVE/MOVE, MOVE/delete and EXPUNGE/FETCH; exact receipt audits
Unicode namespace PASS Seven spellings; 348 namespace checks; two ordered pipeline completions
Provider restart and UIDVALIDITY PASS with limit Automatic epoch recovery; 222 namespace checks; interrupted active command was not established
FETCH literal cuts PASS Cuts at 4,096 bytes and before data; clean NO, then recovery; zero partial body rows
APPEND cuts PASS Before upload, mid-literal and after upstream OK; exact 1 MiB messages; queue empty; zero incomplete provider files
Protocol abuse PASS 50-command pipeline, unfinished literal, overlong line, invalid UTF-7 and IDLE without DONE
Three Connected Accounts under one User PASS Three sessions, 21 local checks and three exact upstream MOVE receipts
Bad device password flood PASS 30 attempts: 7 NO, 23 LIMITED; retained IMAP sessions OK; fresh web sign-in HTTP 200
Mail proxy focused regression PASS TLS fixture with 2,000 messages; metadata/part/partial FETCH; STORE→COPY, MOVE, exact APPEND, selective EXPUNGE, CLOSE; revocation
Mail layouts PASS Existing split persistence assertion retained; 24 production screenshots
Authorization matrix PASS after fix 392 operations, 2,734 requests, 21 App Password scope classes; valid/malformed bodies on 17 privileged routes; the two Groups 500s are fixed
Cross-User matrix PASS 392 operations classified; 183 replayed; 848 ID comparisons; 107 Job/Mail/quota comparisons; zero denial failures
API robustness COMPLETE, findings kept One broad round: 22 findings (20 SLOW, Finder client transport fixed, collection discovery timeout filed as #1081); focused DAV replay passes
Notes idle PASS 700 Notes, 60 seconds, zero Files events, zero SSE change frames; one edit yields four events
Production schema upgrade PASS Both deployed-prefix and older production-copy tests; repeat startup remains safe
Python Mail harness PASS 34 tests; Apple receipt parser: one test
DAV transport regression PASS 23 tests, including queued 413 and missing-response controls; focused real-server replay passes unchanged expectations
Cached SELECT with writers held PASS unchanged_select_does_not_wait_for_index_or_security_writer

The Notes idle probe used the earlier integrated binary, SHA-256
67db6ef1b205100098300c6718ec170d3c8d2bbe8123044a754475f4b805429e.
Later product changes affect panel gestures, App Password alignment and the
Groups guard. Notes code did not change. Its local host load was
17.93 / 15.29 / 12.35. The result checks idle event behavior, not a performance
budget. No perf VM measurement was made in this merge round.

Reproduction

Rust commands used CARGO_PROFILE_DEV_DEBUG=line-tables-only,
CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and a worktree target/tmp.
The preset CARGO_TARGET_DIR was kept. Each crate was tested separately.
The web test command used --maxWorkers=2; server tests used
--test-threads=4.

The Mail soak used the native TLS fixture image, a copied server with
mail-test-provider, CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1,
CALTERNAL_MAIL_STRESS_SECONDS=600 and CALTERNAL_MAIL_STRESS_CLIENTS=50.
Restart cases used CALTERNAL_MAIL_RELIABILITY=1. Provider restarts added
CALTERNAL_MAIL_RELIABILITY_UPSTREAM=1. Repair cases used
CALTERNAL_MAIL_RESILIENCE=1 CALTERNAL_MAIL_REPAIR_REGRESSION=1.
All execute node tests/adversarial/mail_stress.mjs.

Mail proxy regressions use CALTERNAL_MAIL_PROXY_POPULATED=1 node apps/web/e2e/mail-proxy-486.mjs. Layouts use node apps/web/e2e/mail-layouts.mjs. Both set CALTERNAL_E2E_MAC=1.
Authorization uses AUTHZ_MATRIX_ONLY=1 ADVERSARIAL_SKIP_WEB_BUILD=1 ADVERSARIAL_SERVER_BIN=<copied-binary> bash tests/adversarial/run.sh.
Cross-User and API runs reuse the same fixture runner with only its selected
probe command changed to xuser_matrix.py or direct-backend attack.py.
No probe assertion was changed.

Soak resources

50-client phase Peak RSS (KiB) End RSS (KiB) Peak FDs End FDs
1 User(s)/account(s) 354628 156728 255 201
3 User(s)/account(s) 202908 202908 255 195

These are local debug observations over the requested ten-minute phases.
They do not prove a long-term memory bound. Both phases retained valid web
sessions and had zero snapshot failures. Expected pending-transfer NO
responses remained clean refusals; exact write receipts passed.

Verified product

Product source: d209662581756d6050e62b5c566ac9a03379c4be.
Server SHA-256: 463ffe23a2ae1e3df5955fde0a4608d3b29f42e8ab2c2e88464bfd3ecae8854a.
The production web build ID matches that product commit. The later commit
27416d47d changes only the DAV probe and its tests. The final report commit
does not change the product.

Gates: verbatim output

cargo fmt --check returned exit 0 with no output.

Each Rust crate ran cargo clippy -p <crate> --all-targets -- -D warnings
and cargo test -p <crate>. The final server run added -- --test-threads=4.

calternal-imap

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.12s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-mail

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 05s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 54s
test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.52s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-files

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 58s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 41s
test result: ok. 239 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 168.21s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-auth

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 30.34s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 47.35s
test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 78.49s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-db

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.22s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 17.46s
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.07s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.54s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s
test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.21s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 02s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 34s
test result: ok. 222 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 46.94s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 30.51s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.21s

calternal-plugin-notes

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.00s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 56s
test result: ok. 265 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 168.67s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

bun run check

svelte-check found 0 errors and 4 warnings in 3 files

Final source guard

perf-lint: PASS; 0 violations; 18996 scoped exceptions

bun run test --maxWorkers=2

Ran 131 tests in 0.046s
OK
Ran 7 tests across 1 file. [767.00ms]
 Test Files  226 passed (226)
      Tests  1524 passed (1524)

packages/editor: bun run test --maxWorkers=2

 Test Files  21 passed (21)
      Tests  433 passed (433)

Panel resize focused regression

 Test Files  1 passed (1)
      Tests  3 passed (3)

App Password focused regression

 Test Files  1 passed (1)
      Tests  3 passed (3)

Mail harness unit tests

Ran 34 tests in 2.870s
OK

Apple Mail receipt parser

Ran 1 test in 0.002s
OK

Files

The complete changed-file list against 2b6c77c14 follows. It includes
the two integrated branches and the Files cherry-pick. Review artifacts are
ignored and are attached to #1038. No screenshots were committed.

CONTEXT.md
Cargo.lock
apps/web/e2e/mail-layouts.mjs
apps/web/e2e/mail-proxy-486.mjs
apps/web/src/lib/actions/edgeResize.test.ts
apps/web/src/lib/actions/edgeResize.ts
apps/web/src/lib/components/OverlaySurface.svelte.test.ts
apps/web/src/lib/mail/MailSidebar.svelte
apps/web/src/lib/mail/MailSidebar.svelte.test.ts
apps/web/src/lib/mail/MailView.svelte
apps/web/src/lib/mail/live.test.ts
apps/web/src/lib/mail/live.ts
apps/web/src/lib/navigation.test.ts
apps/web/src/lib/navigation.ts
apps/web/src/lib/plugins/user-enable.test.ts
apps/web/src/lib/plugins/user-enable.ts
apps/web/src/routes/settings/[...path]/+page.svelte
apps/web/src/routes/settings/account/AppPasswordsGroup.svelte
apps/web/src/routes/settings/account/AppPasswordsGroup.svelte.test.ts
bench/blaze.md
bench/blaze.mjs
bench/blaze.test.mjs
bench/mail-folder-page.py
bench/mail-sync.py
contracts/actions.json
contracts/openapi.json
contracts/perf/exceptions.json
contracts/perf/ratchet.json
contracts/perf/registry.json
crates/calternal-auth/migrations/0014_mail_app_password_usage.sql
crates/calternal-auth/src/api.rs
crates/calternal-auth/src/store.rs
crates/calternal-db/src/jobs.rs
crates/calternal-imap/src/lib.rs
crates/calternal-imap/src/mime.rs
crates/calternal-imap/src/session.rs
crates/calternal-imap/src/store.rs
crates/calternal-imap/src/wire.rs
crates/calternal-imap/tests/mail.rs
crates/calternal-imap/tests/mime.rs
crates/calternal-imap/tests/session.rs
crates/calternal-imap/tests/wire.rs
crates/calternal-server/src/device_imap.rs
crates/calternal-server/src/integrations.rs
crates/calternal-server/src/integrations_review.rs
crates/calternal-server/src/notes_imap.rs
crates/calternal-server/src/notes_submission.rs
crates/calternal-server/src/upgrade_tests.rs
crates/calternal-server/src/wire.rs
crates/calternal-server/src/wire/groups.rs
crates/plugins/files/src/index.rs
crates/plugins/files/src/lib.rs
crates/plugins/mail/Cargo.toml
crates/plugins/mail/migrations/0012_mail_proxy.sql
crates/plugins/mail/migrations/0013_folder_page_index.sql
crates/plugins/mail/migrations/0014_proxy_metadata.sql
crates/plugins/mail/migrations/0015_proxy_mutations.sql
crates/plugins/mail/migrations/0016_proxy_transfers.sql
crates/plugins/mail/migrations/0017_proxy_projection.sql
crates/plugins/mail/src/cache.rs
crates/plugins/mail/src/cache/store.rs
crates/plugins/mail/src/lib.rs
crates/plugins/mail/src/proxy.rs
crates/plugins/mail/src/proxy_mutations.rs
crates/plugins/mail/src/proxy_tests.rs
crates/plugins/mail/src/proxy_transfers.rs
crates/plugins/mail/src/routes.rs
crates/plugins/mail/src/sync.rs
crates/plugins/mail/vendor/async-imap/src/types/fetch.rs
crates/plugins/notes/src/imap.rs
docs/DESIGN.md
docs/audits/mailround2-1038.md
docs/audits/merge-round-9.md
packages/api-client/src/generated.ts
packages/ui/src/components/OverlaySurface.svelte
tests/adversarial/apple_mail_accept.mjs
tests/adversarial/apple_mail_native.applescript
tests/adversarial/apple_mail_receipts.py
tests/adversarial/apple_mail_vnc.py
tests/adversarial/mail-sync.md
tests/adversarial/mail_fault_provider.py
tests/adversarial/mail_proxy.py
tests/adversarial/mail_stress.mjs
tests/adversarial/mail_sync_provider.py
tests/adversarial/test_apple_mail_receipts.py
tests/adversarial/test_dav_probe.py
tests/adversarial/test_mail_fault_provider.py
tests/adversarial/test_mail_proxy.py
tests/adversarial/test_mail_sync_provider.py
tests/adversarial/webdav.py

Live checks: verbatim output

50-client phases

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}
{"scenario":"three-accounts","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

SIGKILL recovery

{"scenario":"kill-APPEND-0","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-APPEND-100","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-APPEND-500","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-COPY-0","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-COPY-100","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-COPY-500","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-MOVE-0","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-MOVE-100","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-MOVE-500","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-EXPUNGE-0","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-EXPUNGE-100","result":"PASS","signal":"SIGKILL"}
{"scenario":"kill-EXPUNGE-500","result":"PASS","signal":"SIGKILL"}

Mail proxy focused regression

PASS populated provider started with 2,000 fixture messages
PASS populated Mail projection and Connected Account reconciliation
PASS mail: IMAP namespace and authenticated SMTP refusal
PASS notes: IMAP namespace and authenticated SMTP refusal
PASS combined: IMAP namespace and authenticated SMTP refusal
PASS other: IMAP namespace and authenticated SMTP refusal
PASS caldav: IMAP and SMTP authentication denied
PASS populated Mail: metadata/part/partial FETCH, queued STORE → COPY, MOVE, exact draft APPEND, selective EXPUNGE, CLOSE
PASS device live edit: read
PASS device live edit: unread
PASS device live edit: move
PASS device live edit: delete
PASS live web projection: device BODY[]/read/unread/MOVE/EXPUNGE without browser reload
PASS production Mail views: inbox/folder/message/thread at 390/820/1440, light/dark, macOS platform
PASS production App Password scope labels: 390/820/1440, light/dark, macOS platform
PASS revoked: IMAP and SMTP authentication denied

Mail layouts

Mail layouts e2e passed; screenshots are in /home/kayg/Developer/calternal-wt/merge-round-9/artifacts/mail-layouts

Production schema upgrades

test wire::upgrade_tests::production_7b_schema_copy_upgrades_only_pending_mail_and_auth_migrations ... ok
test wire::upgrade_tests::production_schema_copy_upgrades_once_without_reinterpreting_notes_0028 ... ok

Authorization matrix

Authorization matrix: 392 OpenAPI operations; 2734 requests across 4 base identities plus invalid/stale session probes and 21 App Password scope classes; valid/malformed session bodies on 17 privileged body routes; App Passwords use valid bodies; policy classes {'public': 20, 'public_link': 11, 'user': 314, 'admin': 47}

Cross-User matrix

Two-User OpenAPI matrix: 392 operations classified; 183 operations replayed; 848 A-ID vs missing-ID comparisons across B, C, D and anonymous; 36 identifier routes classified with no local fixture factory; median absolute timing delta 3.1 ms
Job/Mail/quota ownership checks: 107 comparisons; 0 denial failures

Focused DAV replay

DAV 100-href Apple multiget: 18045 request bytes, 100 responses, 0.158s
WebDAV scripted probes passed

DAV transport unit tests

Ran 23 tests in 0.049s
OK

Notes idle probe

{
  "success": true,
  "load": [
    17.9267578125,
    15.29296875,
    12.3466796875
  ],
  "build": {
    "source_commit": "a3767ba87696cd81f2e5c88775ffd98ae57a24fc",
    "binary_sha256": "67db6ef1b205100098300c6718ec170d3c8d2bbe8123044a754475f4b805429e"
  },
  "notes": 700,
  "idle_seconds": 60.0,
  "files_events": 0,
  "files_events_per_minute": 0.0,
  "sse_change_frames": 0,
  "response_profiles": {
    "status": 404,
    "body_bytes": 57,
    "samples_per_case": 20,
    "cases": [
      {
        "p50_ms": 6.885,
        "p95_ms": 28.97
      },
      {
        "p50_ms": 6.865,
        "p95_ms": 37.746
      },
      {
        "p50_ms": 6.691,
        "p95_ms": 16.322
      }
    ]
  },
  "single_edit_files_events": 4
}

Broad API round: result retained

==== FINDINGS 22
 - DAV 100-href area discovery :: the imported area did not become a Calendar collection
 - DAV Files scripted probe :: exited 1
 - tag setup upload bytes Photos/2026/2026-10-05/burst-015.jpg :: SLOW 5.2s status 204
 - tag setup upload bytes Photos/2026/2026-10-05/burst-023.jpg :: SLOW 5.1s status 204
 - Journal attachment append concurrency 7 :: SLOW 5.1s status 200
 - Journal attachment append concurrency 9 :: SLOW 5.6s status 200
 - Journal attachment append concurrency 10 :: SLOW 6.0s status 200
 - Journal attachment append concurrency 11 :: SLOW 5.4s status 200
 - Calendar duplicate concurrency 5 :: SLOW 9.1s status 201
 - Calendar duplicate concurrency 7 :: SLOW 6.8s status 201
 - Calendar duplicate concurrency 8 :: SLOW 5.3s status 201
 - Calendar duplicate concurrency 9 :: SLOW 5.9s status 201
 - Calendar duplicate concurrency 10 :: SLOW 12.4s status 201
 - Calendar duplicate concurrency 11 :: SLOW 8.5s status 201
 - Calendar duplicate concurrency 12 :: SLOW 11.1s status 201
 - Calendar duplicate concurrency 13 :: SLOW 10.8s status 201
 - Calendar duplicate concurrency 14 :: SLOW 13.4s status 201
 - Calendar duplicate concurrency 15 :: SLOW 14.3s status 201
 - Calendar duplicate concurrency 16 :: SLOW 11.5s status 201
 - Calendar duplicate concurrency 17 :: SLOW 9.4s status 201
 - Calendar duplicate concurrency 18 :: SLOW 10.1s status 201
 - Calendar duplicate concurrency 19 :: SLOW 11.9s status 201

Review artifacts

The latest production screenshot set supersedes the earlier sets. It uses
macOS emulation at 390, 820 and 1440 px in both themes. The key-icon cap
alignment was inspected at each width. Visual approval belongs to Claude.

Cleanup

cargo clean returned exit 0. Its output was:

Removed 25823 files, 23.1GiB total

Removed the ignored production web build, SvelteKit output and local fixture
temporary directory after all live checks finished. Each directory was
checked against the worktree boundary and tracked-file list. The screenshots
and gate logs remain in ignored artifacts/.

Final merge-round-9 report HEAD: `269b1b51b5774d79de08bfb31d058b3d248a7e7b` Branch: `job/merge-round-9` Base: `2b6c77c14` Audit: `docs/audits/merge-round-9.md` # Merge round 9: Mail proxy release READY FOR STAGING: yes, for this merge brief. Full #1038 acceptance is not claimed. ## Built Integrated `job/mailround-1038` at `5a5b067a0` and `job/mailui-1078` at `d6982f4e9` into `job/merge-round-9` from `2b6c77c14`. Cherry-picked the Files upload identity repair `da2eb5f91` as `a3767ba87`. Fetched `origin` and merged `origin/dev` once before the final gates. Git reported `Already up to date.` There was no push, deploy or issue closure. The release includes the CalternalDAV Mail proxy, provider sync repairs, restart recovery, cached SELECT readers, the phone Settings title repair, Mail columns empty state, enabled Tabs in the tray and folder loading Retry. A sibling write no longer invalidates an upload when the folder identity, inode and parent fingerprint still match. Kept deployed migration prefixes: Auth 13, DB 15, Files 24, Mail 11 and Notes 32. Auth 14 and Mail 12–17 follow those prefixes. Both production schema-copy upgrade tests pass. OpenAPI, action and client contracts were regenerated from code. Fixed two findings during this round. Pull-down gestures could steal pointer capture from a panel separator. The resize action now keeps the accepted pointer gesture. A regression checks capture and bubble behavior. Anonymous and Public link requests to `GET /api/v1/groups` got HTTP 500 before the handler could reject them. The route now checks identity and data scope before required extension extraction. Its Axum regression checks missing identity, Public link identity, limited scope and a valid User. The Finder probe now reads an early HTTP rejection after upload closure. Its new regressions distinguish a queued 413 from a missing response. The existing 413 and zero-byte cleanup expectations were kept. The focused live DAV replay passed. The Files performance ledger uses exact pure-operation and SQL bindings. Filesystem observations remain unresolved debt. The combined ledger has 18,996 scoped exceptions, down from 19,340. Unresolved call sites fell from 13,616 to 13,272. No limit or ratchet ceiling was raised. ## UX gaps closed - Empty Mail columns show a real empty state. - The tray shows every enabled Tab. - Failed folder loading has Retry. - Phone Settings retains its title and Copy link action. - Split width survives a completed pointer drag. The existing e2e expectation was kept. - App Password key icons use SettingsRow title alignment and the existing Connected Accounts cap-box CSS recipe when descriptions wrap. - Device read, unread, MOVE and EXPUNGE changes appear in the live web view without a reload. App Password revocation denies IMAP and SMTP access. ## Decisions - Added a validated 600-second phase option for this brief's ten-minute soak. The full acceptance default stays at 1,800 seconds. The option cannot shorten a normal phase below 600 seconds. - Kept the existing 403 contract for Groups. Reject before Axum extraction. - Used the existing SettingsRow title-alignment option and cap-box recipe. No runtime layout measurement was added. - Used exact bindings for existing pure operations and Index SQL chains. Did not classify filesystem observations as pure. ## Known gaps and UX gaps left This report covers the merge brief. It does not close #1038 or claim its full acceptance matrix. The 30-minute 50-client phases, 100,000-message initial sync, 50 MB and zero-byte messages, 10,000 folders, deep trees and complete download storms were not repeated on this merged build. The complete upstream slow-delivery/BYE/NO/BAD/garbage matrix was not replayed here. The bounded fault fixture used 1 MiB messages. #1070 tracks large cached SELECT latency and memory. These local debug results are not a release performance baseline. CREATE, RENAME and DELETE return clean unsupported IMAP responses. Successful folder lifecycle races were not exercised. The provider restart check did not establish an active interrupted command. The separate literal-cut checks cover FETCH and APPEND interruption. Four existing Svelte warnings remain. The cross-User matrix lists 36 identifier routes without a seeded local item. The broad API round returned exit 1 with 20 SLOW findings and two other findings. The Finder client defect is fixed in `27416d47d`. The collection discovery timeout is filed as [#1081](https://git.kayg.org/kayg/calternal/issues/1081). The focused DAV replay passed the unchanged 100-resource and Finder checks. The timeout cause is not established. No crash, 5xx, data loss or denial failure was observed in that broad round. No owner Apple Mail or real macOS VM check was repeated in this worktree; the integrated branch carries the earlier acceptance evidence. Screenshots use macOS platform emulation. Claude must review visual quality. ## Live scenarios All probes use a throwaway local server and fixture. No provider account or production data was used. The copied executables prevent Cargo test builds from replacing a server used by a restart probe. | Scenario | Result | Evidence and limit | |---|---|---| | 12 clients, one account, 600-second phase | PASS | 604.98 seconds with audits; 6,281 SELECT OK; 20 exact upstream/local write receipts; no failures | | 12 clients, three Users/accounts, 600-second phase | PASS | 606.57 seconds with audits; 6,302 SELECT OK; 20 exact upstream/local write receipts; no failures | | 50 clients, one account, 600-second phase | PASS | 614.69 seconds with audits; 25,120 SELECT OK; 50 exact upstream/local write receipts; no failures | | 50 clients, three Users/accounts, 600-second phase | PASS | 612.84 seconds with audits; 25,708 SELECT OK; 50 exact upstream/local write receipts; no failures | | SIGKILL recovery | PASS | APPEND, COPY, MOVE and EXPUNGE at 0/100/500 ms: all 12 cases; exact MIME identity and receipt audits | | Same-item races | PASS | MOVE/MOVE, MOVE/delete and EXPUNGE/FETCH; exact receipt audits | | Unicode namespace | PASS | Seven spellings; 348 namespace checks; two ordered pipeline completions | | Provider restart and UIDVALIDITY | PASS with limit | Automatic epoch recovery; 222 namespace checks; interrupted active command was not established | | FETCH literal cuts | PASS | Cuts at 4,096 bytes and before data; clean NO, then recovery; zero partial body rows | | APPEND cuts | PASS | Before upload, mid-literal and after upstream OK; exact 1 MiB messages; queue empty; zero incomplete provider files | | Protocol abuse | PASS | 50-command pipeline, unfinished literal, overlong line, invalid UTF-7 and IDLE without DONE | | Three Connected Accounts under one User | PASS | Three sessions, 21 local checks and three exact upstream MOVE receipts | | Bad device password flood | PASS | 30 attempts: 7 NO, 23 LIMITED; retained IMAP sessions OK; fresh web sign-in HTTP 200 | | Mail proxy focused regression | PASS | TLS fixture with 2,000 messages; metadata/part/partial FETCH; STORE→COPY, MOVE, exact APPEND, selective EXPUNGE, CLOSE; revocation | | Mail layouts | PASS | Existing split persistence assertion retained; 24 production screenshots | | Authorization matrix | PASS after fix | 392 operations, 2,734 requests, 21 App Password scope classes; valid/malformed bodies on 17 privileged routes; the two Groups 500s are fixed | | Cross-User matrix | PASS | 392 operations classified; 183 replayed; 848 ID comparisons; 107 Job/Mail/quota comparisons; zero denial failures | | API robustness | COMPLETE, findings kept | One broad round: 22 findings (20 SLOW, Finder client transport fixed, collection discovery timeout filed as #1081); focused DAV replay passes | | Notes idle | PASS | 700 Notes, 60 seconds, zero Files events, zero SSE change frames; one edit yields four events | | Production schema upgrade | PASS | Both deployed-prefix and older production-copy tests; repeat startup remains safe | | Python Mail harness | PASS | 34 tests; Apple receipt parser: one test | | DAV transport regression | PASS | 23 tests, including queued 413 and missing-response controls; focused real-server replay passes unchanged expectations | | Cached SELECT with writers held | PASS | `unchanged_select_does_not_wait_for_index_or_security_writer` | The Notes idle probe used the earlier integrated binary, SHA-256 `67db6ef1b205100098300c6718ec170d3c8d2bbe8123044a754475f4b805429e`. Later product changes affect panel gestures, App Password alignment and the Groups guard. Notes code did not change. Its local host load was 17.93 / 15.29 / 12.35. The result checks idle event behavior, not a performance budget. No perf VM measurement was made in this merge round. ## Reproduction Rust commands used `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4` and a worktree `target/tmp`. The preset `CARGO_TARGET_DIR` was kept. Each crate was tested separately. The web test command used `--maxWorkers=2`; server tests used `--test-threads=4`. The Mail soak used the native TLS fixture image, a copied server with `mail-test-provider`, `CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1`, `CALTERNAL_MAIL_STRESS_SECONDS=600` and `CALTERNAL_MAIL_STRESS_CLIENTS=50`. Restart cases used `CALTERNAL_MAIL_RELIABILITY=1`. Provider restarts added `CALTERNAL_MAIL_RELIABILITY_UPSTREAM=1`. Repair cases used `CALTERNAL_MAIL_RESILIENCE=1 CALTERNAL_MAIL_REPAIR_REGRESSION=1`. All execute `node tests/adversarial/mail_stress.mjs`. Mail proxy regressions use `CALTERNAL_MAIL_PROXY_POPULATED=1 node apps/web/e2e/mail-proxy-486.mjs`. Layouts use `node apps/web/e2e/mail-layouts.mjs`. Both set `CALTERNAL_E2E_MAC=1`. Authorization uses `AUTHZ_MATRIX_ONLY=1 ADVERSARIAL_SKIP_WEB_BUILD=1 ADVERSARIAL_SERVER_BIN=<copied-binary> bash tests/adversarial/run.sh`. Cross-User and API runs reuse the same fixture runner with only its selected probe command changed to `xuser_matrix.py` or direct-backend `attack.py`. No probe assertion was changed. ## Soak resources | 50-client phase | Peak RSS (KiB) | End RSS (KiB) | Peak FDs | End FDs | |---|---:|---:|---:|---:| | 1 User(s)/account(s) | 354628 | 156728 | 255 | 201 | | 3 User(s)/account(s) | 202908 | 202908 | 255 | 195 | These are local debug observations over the requested ten-minute phases. They do not prove a long-term memory bound. Both phases retained valid web sessions and had zero snapshot failures. Expected pending-transfer NO responses remained clean refusals; exact write receipts passed. ## Verified product Product source: `d209662581756d6050e62b5c566ac9a03379c4be`. Server SHA-256: `463ffe23a2ae1e3df5955fde0a4608d3b29f42e8ab2c2e88464bfd3ecae8854a`. The production web build ID matches that product commit. The later commit `27416d47d` changes only the DAV probe and its tests. The final report commit does not change the product. ## Gates: verbatim output `cargo fmt --check` returned exit 0 with no output. Each Rust crate ran `cargo clippy -p <crate> --all-targets -- -D warnings` and `cargo test -p <crate>`. The final server run added `-- --test-threads=4`. ### calternal-imap ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.12s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-plugin-mail ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 05s Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 54s test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.52s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-plugin-files ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 58s Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 41s test result: ok. 239 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 168.21s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-auth ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 30.34s Finished `test` profile [unoptimized + debuginfo] target(s) in 47.35s test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 78.49s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-db ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.22s Finished `test` profile [unoptimized + debuginfo] target(s) in 17.46s test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.07s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.54s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.21s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-server ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 02s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 34s test result: ok. 222 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 46.94s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 30.51s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.21s ``` ### calternal-plugin-notes ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.00s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 56s test result: ok. 265 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 168.67s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### bun run check ```text svelte-check found 0 errors and 4 warnings in 3 files ``` ### Final source guard ```text perf-lint: PASS; 0 violations; 18996 scoped exceptions ``` ### bun run test --maxWorkers=2 ```text Ran 131 tests in 0.046s OK Ran 7 tests across 1 file. [767.00ms] Test Files 226 passed (226) Tests 1524 passed (1524) ``` ### packages/editor: bun run test --maxWorkers=2 ```text Test Files 21 passed (21) Tests 433 passed (433) ``` ### Panel resize focused regression ```text Test Files 1 passed (1) Tests 3 passed (3) ``` ### App Password focused regression ```text Test Files 1 passed (1) Tests 3 passed (3) ``` ### Mail harness unit tests ```text Ran 34 tests in 2.870s OK ``` ### Apple Mail receipt parser ```text Ran 1 test in 0.002s OK ``` ## Files The complete changed-file list against `2b6c77c14` follows. It includes the two integrated branches and the Files cherry-pick. Review artifacts are ignored and are attached to #1038. No screenshots were committed. ```text CONTEXT.md Cargo.lock apps/web/e2e/mail-layouts.mjs apps/web/e2e/mail-proxy-486.mjs apps/web/src/lib/actions/edgeResize.test.ts apps/web/src/lib/actions/edgeResize.ts apps/web/src/lib/components/OverlaySurface.svelte.test.ts apps/web/src/lib/mail/MailSidebar.svelte apps/web/src/lib/mail/MailSidebar.svelte.test.ts apps/web/src/lib/mail/MailView.svelte apps/web/src/lib/mail/live.test.ts apps/web/src/lib/mail/live.ts apps/web/src/lib/navigation.test.ts apps/web/src/lib/navigation.ts apps/web/src/lib/plugins/user-enable.test.ts apps/web/src/lib/plugins/user-enable.ts apps/web/src/routes/settings/[...path]/+page.svelte apps/web/src/routes/settings/account/AppPasswordsGroup.svelte apps/web/src/routes/settings/account/AppPasswordsGroup.svelte.test.ts bench/blaze.md bench/blaze.mjs bench/blaze.test.mjs bench/mail-folder-page.py bench/mail-sync.py contracts/actions.json contracts/openapi.json contracts/perf/exceptions.json contracts/perf/ratchet.json contracts/perf/registry.json crates/calternal-auth/migrations/0014_mail_app_password_usage.sql crates/calternal-auth/src/api.rs crates/calternal-auth/src/store.rs crates/calternal-db/src/jobs.rs crates/calternal-imap/src/lib.rs crates/calternal-imap/src/mime.rs crates/calternal-imap/src/session.rs crates/calternal-imap/src/store.rs crates/calternal-imap/src/wire.rs crates/calternal-imap/tests/mail.rs crates/calternal-imap/tests/mime.rs crates/calternal-imap/tests/session.rs crates/calternal-imap/tests/wire.rs crates/calternal-server/src/device_imap.rs crates/calternal-server/src/integrations.rs crates/calternal-server/src/integrations_review.rs crates/calternal-server/src/notes_imap.rs crates/calternal-server/src/notes_submission.rs crates/calternal-server/src/upgrade_tests.rs crates/calternal-server/src/wire.rs crates/calternal-server/src/wire/groups.rs crates/plugins/files/src/index.rs crates/plugins/files/src/lib.rs crates/plugins/mail/Cargo.toml crates/plugins/mail/migrations/0012_mail_proxy.sql crates/plugins/mail/migrations/0013_folder_page_index.sql crates/plugins/mail/migrations/0014_proxy_metadata.sql crates/plugins/mail/migrations/0015_proxy_mutations.sql crates/plugins/mail/migrations/0016_proxy_transfers.sql crates/plugins/mail/migrations/0017_proxy_projection.sql crates/plugins/mail/src/cache.rs crates/plugins/mail/src/cache/store.rs crates/plugins/mail/src/lib.rs crates/plugins/mail/src/proxy.rs crates/plugins/mail/src/proxy_mutations.rs crates/plugins/mail/src/proxy_tests.rs crates/plugins/mail/src/proxy_transfers.rs crates/plugins/mail/src/routes.rs crates/plugins/mail/src/sync.rs crates/plugins/mail/vendor/async-imap/src/types/fetch.rs crates/plugins/notes/src/imap.rs docs/DESIGN.md docs/audits/mailround2-1038.md docs/audits/merge-round-9.md packages/api-client/src/generated.ts packages/ui/src/components/OverlaySurface.svelte tests/adversarial/apple_mail_accept.mjs tests/adversarial/apple_mail_native.applescript tests/adversarial/apple_mail_receipts.py tests/adversarial/apple_mail_vnc.py tests/adversarial/mail-sync.md tests/adversarial/mail_fault_provider.py tests/adversarial/mail_proxy.py tests/adversarial/mail_stress.mjs tests/adversarial/mail_sync_provider.py tests/adversarial/test_apple_mail_receipts.py tests/adversarial/test_dav_probe.py tests/adversarial/test_mail_fault_provider.py tests/adversarial/test_mail_proxy.py tests/adversarial/test_mail_sync_provider.py tests/adversarial/webdav.py ``` ## Live checks: verbatim output ### 50-client phases ```text {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} {"scenario":"three-accounts","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` ### SIGKILL recovery ```text {"scenario":"kill-APPEND-0","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-APPEND-100","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-APPEND-500","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-COPY-0","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-COPY-100","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-COPY-500","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-MOVE-0","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-MOVE-100","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-MOVE-500","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-EXPUNGE-0","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-EXPUNGE-100","result":"PASS","signal":"SIGKILL"} {"scenario":"kill-EXPUNGE-500","result":"PASS","signal":"SIGKILL"} ``` ### Mail proxy focused regression ```text PASS populated provider started with 2,000 fixture messages PASS populated Mail projection and Connected Account reconciliation PASS mail: IMAP namespace and authenticated SMTP refusal PASS notes: IMAP namespace and authenticated SMTP refusal PASS combined: IMAP namespace and authenticated SMTP refusal PASS other: IMAP namespace and authenticated SMTP refusal PASS caldav: IMAP and SMTP authentication denied PASS populated Mail: metadata/part/partial FETCH, queued STORE → COPY, MOVE, exact draft APPEND, selective EXPUNGE, CLOSE PASS device live edit: read PASS device live edit: unread PASS device live edit: move PASS device live edit: delete PASS live web projection: device BODY[]/read/unread/MOVE/EXPUNGE without browser reload PASS production Mail views: inbox/folder/message/thread at 390/820/1440, light/dark, macOS platform PASS production App Password scope labels: 390/820/1440, light/dark, macOS platform PASS revoked: IMAP and SMTP authentication denied ``` ### Mail layouts ```text Mail layouts e2e passed; screenshots are in /home/kayg/Developer/calternal-wt/merge-round-9/artifacts/mail-layouts ``` ### Production schema upgrades ```text test wire::upgrade_tests::production_7b_schema_copy_upgrades_only_pending_mail_and_auth_migrations ... ok test wire::upgrade_tests::production_schema_copy_upgrades_once_without_reinterpreting_notes_0028 ... ok ``` ### Authorization matrix ```text Authorization matrix: 392 OpenAPI operations; 2734 requests across 4 base identities plus invalid/stale session probes and 21 App Password scope classes; valid/malformed session bodies on 17 privileged body routes; App Passwords use valid bodies; policy classes {'public': 20, 'public_link': 11, 'user': 314, 'admin': 47} ``` ### Cross-User matrix ```text Two-User OpenAPI matrix: 392 operations classified; 183 operations replayed; 848 A-ID vs missing-ID comparisons across B, C, D and anonymous; 36 identifier routes classified with no local fixture factory; median absolute timing delta 3.1 ms Job/Mail/quota ownership checks: 107 comparisons; 0 denial failures ``` ### Focused DAV replay ```text DAV 100-href Apple multiget: 18045 request bytes, 100 responses, 0.158s WebDAV scripted probes passed ``` ### DAV transport unit tests ```text Ran 23 tests in 0.049s OK ``` ### Notes idle probe ```json { "success": true, "load": [ 17.9267578125, 15.29296875, 12.3466796875 ], "build": { "source_commit": "a3767ba87696cd81f2e5c88775ffd98ae57a24fc", "binary_sha256": "67db6ef1b205100098300c6718ec170d3c8d2bbe8123044a754475f4b805429e" }, "notes": 700, "idle_seconds": 60.0, "files_events": 0, "files_events_per_minute": 0.0, "sse_change_frames": 0, "response_profiles": { "status": 404, "body_bytes": 57, "samples_per_case": 20, "cases": [ { "p50_ms": 6.885, "p95_ms": 28.97 }, { "p50_ms": 6.865, "p95_ms": 37.746 }, { "p50_ms": 6.691, "p95_ms": 16.322 } ] }, "single_edit_files_events": 4 } ``` ### Broad API round: result retained ```text ==== FINDINGS 22 - DAV 100-href area discovery :: the imported area did not become a Calendar collection - DAV Files scripted probe :: exited 1 - tag setup upload bytes Photos/2026/2026-10-05/burst-015.jpg :: SLOW 5.2s status 204 - tag setup upload bytes Photos/2026/2026-10-05/burst-023.jpg :: SLOW 5.1s status 204 - Journal attachment append concurrency 7 :: SLOW 5.1s status 200 - Journal attachment append concurrency 9 :: SLOW 5.6s status 200 - Journal attachment append concurrency 10 :: SLOW 6.0s status 200 - Journal attachment append concurrency 11 :: SLOW 5.4s status 200 - Calendar duplicate concurrency 5 :: SLOW 9.1s status 201 - Calendar duplicate concurrency 7 :: SLOW 6.8s status 201 - Calendar duplicate concurrency 8 :: SLOW 5.3s status 201 - Calendar duplicate concurrency 9 :: SLOW 5.9s status 201 - Calendar duplicate concurrency 10 :: SLOW 12.4s status 201 - Calendar duplicate concurrency 11 :: SLOW 8.5s status 201 - Calendar duplicate concurrency 12 :: SLOW 11.1s status 201 - Calendar duplicate concurrency 13 :: SLOW 10.8s status 201 - Calendar duplicate concurrency 14 :: SLOW 13.4s status 201 - Calendar duplicate concurrency 15 :: SLOW 14.3s status 201 - Calendar duplicate concurrency 16 :: SLOW 11.5s status 201 - Calendar duplicate concurrency 17 :: SLOW 9.4s status 201 - Calendar duplicate concurrency 18 :: SLOW 10.1s status 201 - Calendar duplicate concurrency 19 :: SLOW 11.9s status 201 ``` ## Review artifacts The latest production screenshot set supersedes the earlier sets. It uses macOS emulation at 390, 820 and 1440 px in both themes. The key-icon cap alignment was inspected at each width. Visual approval belongs to Claude. - [merge-round-9-polished-screenshots.zip](https://git.kayg.org/attachments/f1f6df04-a983-4b72-889b-475d8f13dd10) - [columns-empty-desktop-paper-light.png](https://git.kayg.org/attachments/1e0b457d-5cb7-44a6-baa0-785f8be2845c) - [columns-empty-desktop-tokyo-night-dark.png](https://git.kayg.org/attachments/ff1c5dec-a71d-4af4-a5d5-e1a28ddb2038) - [columns-empty-tablet-paper-light.png](https://git.kayg.org/attachments/e9a50da5-cce4-4b84-9c9f-3dc8f14cd2f6) - [columns-empty-tablet-tokyo-night-dark.png](https://git.kayg.org/attachments/73a0d734-85d4-4be1-949b-beaed287bea2) - [390-light.png](https://git.kayg.org/attachments/b8dbfd6e-182a-4cd2-98dd-e1138faa52ae) - [390-dark.png](https://git.kayg.org/attachments/71b33e77-d769-43fc-8e92-247cf9d87197) - [820-light.png](https://git.kayg.org/attachments/37085bb1-0755-4fb8-b3e5-b4cf33a07f40) - [820-dark.png](https://git.kayg.org/attachments/6a565406-8f62-45ff-9a45-70755259861a) - [1440-light.png](https://git.kayg.org/attachments/8a5573c9-dc04-4c9f-8f97-d42bbb72d82b) - [1440-dark.png](https://git.kayg.org/attachments/3cad6048-f6c6-441b-841d-d43fc7abd95a) ## Cleanup `cargo clean` returned exit 0. Its output was: ```text Removed 25823 files, 23.1GiB total ``` Removed the ignored production web build, SvelteKit output and local fixture temporary directory after all live checks finished. Each directory was checked against the worktree boundary and tracked-file list. The screenshots and gate logs remain in ignored `artifacts/`.
kayg commented 2026-10-05 03:08:42 +00:00
Author
Owner
Copy link

Deployed to production 2026-10-05 ~04:40 CEST in round 9 (269b1b51b). Includes the mail proxy (CalternalDAV, real Apple Mail acceptance PASS on the Mac VM), provider sync fixes, the stress-round fixes, #1067, #1068, #1078 and the Files upload identity repair. Staging healthy first; production healthy in 18 s; Auth 14 and Mail 17 migrations applied; change events 0/30 s; no expired leases.

Deployed to production 2026-10-05 ~04:40 CEST in round 9 (269b1b51b). Includes the mail proxy (CalternalDAV, real Apple Mail acceptance PASS on the Mac VM), provider sync fixes, the stress-round fixes, #1067, #1068, #1078 and the Files upload identity repair. Staging healthy first; production healthy in 18 s; Auth 14 and Mail 17 migrations applied; change events 0/30 s; no expired leases.
kayg closed this issue 2026-10-05 03:08:47 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
wip/rev2-webperf
wip/rev2-money-ident
wip/restyle-notes
wip/previewcard-1098
wip/palette2-1123
wip/palette-1093
wip/onboard2-1141
wip/onboard-1141.aborted-early
wip/onboard-1141
wip/nlpchip-1127
wip/morph-1104
wip/merge-round-7c5
wip/merge-round-7c4
job/notifloop-1194
wip/merge-round-7c3
wip/merge-round-7c2
wip/merge-round-7c
wip/mchrome-1084
wip/mailghost2-1094
wip/mailghost-1094
wip/kbpreview2-1118
wip/kbpreview-1118
wip/kanban-1092
wip/importhang-1121
wip/hiderev-1153
wip/hide4-1153
wip/hide3-1153
wip/hide2-1153
wip/hide-1153
wip/editreg-1132
wip/editorrail3-1113
wip/editorrail2-1113
wip/editorrail-1113
wip/e2e-b2-1071
wip/e2e-b-1071
wip/draw4-1101
wip/draw3-1101
wip/draw2-1101
wip/draw-1101
wip/directory-1199-r
wip/directory-1199
wip/delete-1119
wip/collabrev-1197
wip/collabloss-1197
wip/cards2-1083
wip/cards-1083
wip/canvas-visual
wip/canvasvis2-976
wip/calhdr-1112
wip/calcards-1115
wip/browserfix
wip/blocks-1125
wip/allday-1107
wip/agenda-decks
wip/agenda-1086
wip/adv7c-1105
wip/txentry-1198
wip/trayicons2-1095
wip/trayicons-1095
job/onboard-1141
wip/sidebar3-1094
job/collabloss-1197
job/hide-1153
job/perf-1124
job/perf2-1124
job/tocrail-1191
job/restyle-settings
wip/restyle-settings
job/segmented-1200
wip/notifloop-1194
job/tagperf-1186
wip/tagperf-1186
wip/segmented-1200
job/restyle-files
job/tagdnd-1187
job/merge30
job/cards-1179
wip/cards2-1179
wip/cards-1179
wip/tocrail-1191
wip/tagdnd-1187
wip/restyle-files
wip/perf-1124
wip/merge30j
job/restyle-notes
job/wizchoices-1140
job/adv-1202
wip/wizchoices-1140
wip/restyle-1190
job/moneyfmt-1180
job/txentry-1198
wip/moneyfmt2-1180
wip/moneyfmt-1180-r
wip/moneyfmt-1180
job/pillglass-1189
job/flags-1181
wip/flags-1181
job/restyle-1190
job/restyle-mailmoney
job/restyle-search
job/settingsreg-1195
job/wizard-1140
site/website
wip/wizardrev2-1140
wip/wizardrev-1140
wip/wizard5-1140
wip/wizard4-1140
wip/wizard3-1140
wip/wizard2-1140
wip/wizard-1140
wip/pillglass-1189
wip/settingsreg-1195
job/merge29
job/fu-1171
wip/merge29j
wip/fu-1171
job/fu-1166
job/directory-1199
job/proflog-1204
job/txresearch-1188
wip/fu-1166
job/merge28
job/search-1066
wip/search-1066
wip/merge28j
job/gateslot-1182
job/bulkimport-1157
job/mailnet-1160
wip/mailnetrev-1160
wip/mailnet-1160
wip/bulkrev-1157
wip/bulkimport-1157
job/startup-1161
wip/startup-1161
job/merge27
job/linkcards-1151
wip/linkcards3-1151
wip/linkcards2-1151
wip/linkcards-1151
job/traydate-1144
wip/traydate3-1144
wip/traydate2-1144
wip/traydate-1144
job/draw-1101
wip/merge27j
job/blockpill-1152
wip/blockpill3-1152
wip/blockpill2-1152
wip/blockpill-1152
job/minihover-1149
wip/minihover2-1149
wip/minihover-1149
job/merge25
wip/merge25-r
wip/merge25b
wip/merge25
job/inspector-1129
job/tags-1110
wip/inspector3-1129
wip/inspector2-1129
wip/inspector-1129
wip/tagsrev-1110
wip/tags2-1110
wip/tags-1110
job/dates-1148
wip/datesrev-1148
wip/dates2-1148
wip/dates-1148
job/licence-1145
wip/licence2-1145
wip/licence-1145
job/selfhost-1156
job/merge23
wip/merge23
job/tagfilter-1109
wip/tagfilter2-1109
wip/tagfilter-1109
job/kbd-1134
wip/kbd2-1134
wip/kbd-1134
job/palfoot-1137
wip/selfhost-1156
wip/palfoot2-1137
wip/palfoot-1137
job/toggle-1158
wip/toggle-1158
job/kbpreview-1118
job/docratchet-1155
job/perflint-1133
job/devtests-1159
wip/docratchet-1155
wip/devtests-1159
job/segv-1136
wip/toast-1142
wip/segv-1136
job/toast-1142
job/blockreload-1147
wip/blockreload-1147
job/font-1150
wip/font-1150
job/importui-1120
job/minimonth-1149
wip/importui-1120
wip/minimonth-1149
job/depcheck-1146
wip/perflint-1133
wip/depcheck-1146
job/calcards-1115
job/blocks-1125
job/plus-1128
job/shift-1138
wip/plus2-1128
wip/plus-1128
wip/shift-1138
job/moneyfid-1130
job/editorrail-1113
wip/moneyrev-1130
wip/moneyfid-1130
job/noext-851
wip/noext-851
wip/noext3-851
wip/noext2-851
job/week-1135
wip/week-1135
job/editreg-1132
job/smoke-1122
wip/smoke-1122
job/docs-1143
job/palette2-1123
job/calhdr-1112
job/nlpchip-1127
job/mailghost-1094
job/reconnect-1131
wip/reconnect-1131
job/trayicons-1095
job/delete-1119
job/importhang-1121
job/cards-1083
job/palette-1093
job/mchrome-1084
job/e2e-a-1071
job/canvas-visual
job/previewcard-1098
job/allday-1107
wip/e2e-a2-1071
wip/e2e-a-1071
job/e2e-b-1071
job/adv7c-1105
job/kanban-1092
job/agenda-1086
job/merge-round-7c
job/morph-1104
wip/surfaces-p2
job/merge-round-9
wip/merge-round-9
job/7cfix-small
wip/7cfix-small
job/mailui-1078
job/merge-round-8
wip/merge-round-8
wip/mailui-1078
job/mailround-1038
job/applemail-accept
wip/settitle-1068
wip/mailround2-1038
wip/mailround-1038
wip/e2e-7b
job/crash-1069
wip/crash-1069
job/searchlost-1066
wip/searchlost-1066
job/7b-reconcile
job/flake-1065
wip/flake-1065
wip/merge-round-7b7
wip/merge-round-7b6
wip/merge-round-7b5
wip/merge-round-7b4
wip/7b-reconcile
job/appupdate-1059
job/nfd-1044
wip/appupdate-1059
job/e2e-7b
job/loop-1062
wip/loop-1062
job/pdfprev-1045
job/invtoggle-1053
wip/pdfprev-1045
wip/nfd-1044
wip/invtoggle-1053
job/7bfix-e2e
job/mailstress-b
wip/7bfix-e2e
wip/mailstress-b
job/7bfix-adv
wip/7bfix-adv
job/mailstress-a
job/stack-1054
wip/stack-1054
wip/mailstress-a
job/mailstress-1038
wip/mailstress-1038
job/upload500-1051
wip/upload500-1051
job/share-1034
wip/share-1034
job/syncerr-1037
job/7bfix-photos
wip/7bfix-photos
job/paste-1036
job/setside-1039
wip/setside-1039
wip/paste-1036
job/lease-1042
wip/syncerr-1037
wip/lease-1042
job/7bfix-data
job/passkeybind-1043
wip/apprevoke-1041
job/invite-1035
wip/invite-1035
job/merge-round-7b2
wip/merge-round-7b2
job/mailproxy-486
job/apprevoke-1041
job/rebuild-1033
job/pillborder-1029
wip/pillborder-1029
wip/mailproxy-486
wip/applemail-486
job/headless-998
wip/headless-998
job/groups-1028
wip/groups-1028
job/rebuildwarn-1016
wip/rebuildwarn-1016
job/startup-1011
wip/startup-1011
job/monthpill-1009
job/bgthumb-1025
job/sharetitle-1012
wip/monthpill-1009
wip/bgthumb-1025
wip/sharetitle-1012
job/canvas-cards-977
wip/canvas-cards-977
job/canvas-pencil-978
job/canvas-sketch-990
wip/canvas-sketch-990
wip/canvas-pencil-978
job/canvas-files-989
wip/canvas-files-989
job/canvas-collab-991
wip/canvas-collab-991
job/weekscroll-1018
wip/weekscroll-1018
wip/canvas-core-976
job/canvas-core-976
job/round-drag
wip/round-drag
job/round-settings
job/browserfix
wip/oapi-974
job/oapi-974
job/hist2-integrate
job/mailhtml-726
wip/mailhtml-726
wip/hist2-integrate
job/moneyfu-984
job/drag-1015
wip/drag-1015
job/rename-1017
wip/rename-1017
job/hist2-api
wip/hist2-api
job/oneacct-1014
wip/oneacct-1014
wip/moneyfu-984
job/hist2-bench
job/hist2-restore
wip/hist2-bench
job/hist2-write
job/hotfix-724
wip/hotfix-724
wip/hist2-write
wip/hist2-restore
job/hist2-store
job/hist2-ui
wip/hist2-ui
wip/hist2-store
job/searchstarve-965
job/shutdown-963
wip/shutdown-963
wip/pubedit-981
job/pubedit-981
job/analytics-973
wip/searchstarve-965
job/authflash-850
job/weeklane-969
job/pvtitle-1004
job/hist-975
wip/authflash-850
job/voicepill-617
wip/pvtitle-1004
job/headring-1003
wip/weeklane-969
wip/voicepill-617
wip/headring-1003
wip/analytics-973
job/agentscope-980
wip/thumbsandbox-988
job/thumbsandbox-988
wip/hist-975
job/links-856
wip/links-856
job/davetag-966
wip/davetag-966
job/filesstorm-1000
job/hoverpad-725
wip/filesstorm-1000
job/ffmpegblas-993
job/merge-round-7a
wip/hoverpad-725
wip/ffmpegblas-993
job/nowdot-1002
wip/verify-7a
job/noteid-857
wip/nowdot-1002
wip/noteid-857
wip/merge-round-7a
wip/agentscope-980
job/imapedge
job/a11yfix2
wip/imapedge-941
wip/imapedge
wip/a11yfix2
job/notetask-986
job/logheading
wip/logheading-998
job/textthumb-652
job/photolive-987
wip/photolive-987
job/davactive-983
job/savefix-985
job/tabicons-607
wip/davactive-983
wip/tabicons-607
wip/notetask-986
wip/savefix-985
job/dirid-627
job/buildspeed-1007
wip/dirid-627
job/agenda-decks
job/perfguards-impl
job/undo-a11y
wip/undo-a11y
job/mailperf
job/wal-824
wip/settings-50
job/settings-50
job/notesfilter-606
wip/notesfilter-606
job/surfaces-p2
wip/wal-824
job/maillayouts
wip/mailperf
wip/maillayouts
job/taskmeta-659
job/money-ident
wip/money-ident
wip/taskmeta-659
job/errstates
wip/perfguards-impl
job/headings-881
wip/headings-881
wip/errstates
job/voice-619
job/gaps-827
job/notesperf
wip/notesperf
wip/voice-619
job/hddsql-549
job/perf-stream-668
wip/perf-stream-668
wip/deeplinks-fix
job/deeplinks-fix
job/authfix
job/docsfix-rust
wip/docsfix-rust
job/webperf
job/docsfix-web
job/datafix2
job/webdav-lock-476
job/copyfix
wip/copyfix
wip/webperf
job/focus-658
wip/protofix
job/mediafix
job/protofix
wip/mediafix
job/agentfix
job/hhmm-724
wip/agentfix
job/undo-722
job/reuse
wip/webdav-lock-476
wip/reuse
job/scopefix
job/datafix
wip/hhmm-724
wip/undo-722
job/surfaces-p1
wip/hddsql-549
job/voicememos-618
wip/datafix2
wip/surfaces-p1
job/fix-940
wip/fix-940
job/blaze-surfaces
wip/datafix
wip/blaze-surfaces
job/taskday-655
job/linknav-639
wip/linknav-639
wip/gaps-827
job/isolation-707
job/audiophotos-720
wip/audiophotos-720
job/advfind-664
wip/voicememos-618
wip/taskday-655
wip/isolation-707
wip/advfind-664
wip/scopefix
wip/focus-658
job/testgaps
wip/testgaps
job/overscroll-718
wip/authfix
job/deps
wip/overscroll-718
job/rev2-agentfix
job/rev2-money-ident
job/rev2-mailperf
wip/deps
job/hardening-728
wip/hardening-728
job/searchgen-832
wip/searchgen-832
job/photopw-849
job/mailsql-825
wip/photopw-849
job/sharefix
wip/sharefix
job/rev2-mailhtml-726
job/rev2-perfguards
job/copyval-723
job/lightglass-r2
wip/lightglass-r2
wip/docsfix-web
job/copy-audit
job/macinterop-staging-r2
job/design-sync
job/rev2-taskmeta-659
job/rev2-webperf
job/docs-audit
job/rev2-advfind-664
job/rev2-mailproxy-486
job/states-audit
job/rev2-datafix
job/design-drift
job/test-gaps
job/rev2-voicememos-618
job/rev2-mediafix
job/rev2-deps
job/rev2-datafix2
job/licence-audit
job/issue-hygiene
job/rev2-protofix
job/rev2-voice-619
job/rev2-isolation-707
job/rev2-surfaces-p1
job/deeplink-audit2
job/rev2-audiophotos-720
wip/test-gaps
job/rev2-overscroll-718
job/rev2-undo-722
wip/states-audit
job/rev2-dropmd-719
job/rev2-linknav-639
job/merge-7b-plan
wip/merge-7b-plan
job/rev2-taskday-655
wip/mailsql-825
job/rev2-webdav-lock-476
job/rev2-browserfix
wip/design-drift
job/rev2-hddsql-549
wip/deeplink-audit2
job/rev2-scopefix
job/rev2-authfix
job/rev2-hardening-728
job/rev2-wal-824
job/rev2-sharefix
job/calsidebar-638
job/chrome-audit
job/ioperf
wip/ioperf
wip/chrome-audit
wip/calsidebar-638
job/dropmd-719
wip/dropmd-719
job/ocr-build
wip/ocr-build
job/blaze-settings
wip/copyval-723
job/toastring-721
wip/toastring-721
job/deployfix-732
wip/deployfix-732
wip/blaze-settings
job/money-import-recheck
job/rev-a11y
job/perf-arch-db
job/rev-7b-data
wip/textthumb-652
wip/perf-arch-db
job/sec-protocols
job/sidehdr-660
job/rev-7b-security
job/research-surfaces
job/rev-design-gaps
job/rev-mcp-api
wip/sidehdr-660
job/perf-arch-memory
wip/sec-protocols
job/perf-arch-bundle
job/snapedge-714
wip/rev-mcp-api
job/sec-supplychain
wip/research-surfaces
job/perf-arch-sync
job/rev-consistency
job/perf-arch-server
wip/perf-arch-server
wip/perf-arch-memory
job/perf-arch-io
job/perf-arch-client
job/sec-fs
job/sec-mcp-scopes
job/sec-sharing
job/perf-guards
job/sec-browser
job/sec-admin-deploy
job/sec-auth
wip/snapedge-714
job/bgpicker-717
wip/perf-arch-bundle
wip/money-import-recheck
job/advsetup-654
wip/bgpicker-717
wip/advsetup-654
job/burst-709
job/kbdcaps-710
job/app-pw-chooser
wip/burst-709
wip/app-pw-chooser
job/imaptest-625
wip/kbdcaps-710
job/fix-499
wip/fix-499
job/perf-mut-667
job/calimg-589
job/perf-snap-666
wip/calimg-589
wip/perf-snap-666
wip/perf-mut-667
job/perf-cache-665
wip/perf-cache-665
job/voicefiles-620
wip/voicefiles-620
job/admin-burst-705
wip/admin-burst-705
job/voicememos-review
wip/voicememos-review
wip/ryw-653
job/ryw-653
job/writeonopen-661
job/instant-663
wip/writeonopen-661
job/money-import-review
wip/money-import-review
wip/importjs-610
review/integrations-407-round6
wip/integrations-review
job/dragghost-612
wip/dragghost-612
job/integrations
wip/integrations
job/decider-656
job/merge-round-6
job/perf-rerun
wip/merge-round-6
job/integrations-review-round5
job/selalign-576
wip/selalign-576
job/mcp-events-491
job/files-631
job/cal-e2e-569
wip/cal-e2e-569
job/reload-423
wip/reload-423
wip/mcp-events-491
wip/files-631
job/notesbridge-644
wip/notesbridge-644
job/editor-series
job/calcard-series
wip/calcard-series
job/mcp-events-review-491
wip/mcp-events-review
wip/editor-series
job/quirks-546
job/integrations-recheck
job/tocrail-636
wip/tocrail-636
wip/quirks-546
wip/reminders-643
job/reminders-643
wip/davscale-573
job/davscale-573
job/integrations-review
wip/ocr-eval-584
job/ocr-eval-584
job/esc-537
wip/esc-537
job/toastname-586
wip/toastname-586
job/submenu-579
wip/submenu-579
job/tasks-mode
wip/tasks-mode
job/agentdocs-630
job/dupwrite-634
wip/agentdocs-630
wip/dupwrite-634
job/lightglass-588
wip/lightglass-588
job/tabswitch-549
job/ghosttask-623
wip/ghosttask-623
job/toaststack-616
job/weekstate-609
job/mailsync-613
wip/mailsync-613
wip/weekstate-609
job/maildup-626
wip/tabswitch-549
wip/maildup-626
wip/toaststack-616
job/motion-611
wip/motion-611
job/tlstest-601
wip/tlstest-601
job/perf-495
job/floating-sheet
wip/floating-sheet
job/remdup-585
wip/remdup-585
job/fix-502
wip/fix-502
job/attachplay-622
job/perf-batch
wip/perf-batch-563
wip/perf-495
hotfix/mail-sync-diag
job/mail-m3
wip/mail-m3
job/attach-poof-603
job/calhover-608
job/editorbar-604
job/mentions-605
job/merge-round-4
job/allday-514
wip/merge-round-4
wip/allday-514
job/merge-round-4a
wip/merge-round-4a
job/sharestack-580
job/fix-501
wip/sharestack-580
wip/fix-501
job/perf-batch-563
job/apw-cache-review
wip/apw-cache-review
job/probe-520
wip/probe-520
job/mac-393
wip/mac-393
job/header-571
job/flake-513
wip/flake-513
job/docs-thumb-547
wip/header-571
job/webcal-572
wip/webcal-572
wip/shortcuts-542
job/shortcuts-542
wip/docs-thumb-547
job/caldav-stress
wip/caldav-stress
wip/sweep-478
job/apw-cache-512
wip/apw-cache-512
job/money-empty-540
wip/restart-505
wip/money-empty-540
wip/fix-510
job/restart-505
job/fix-503
job/perf-496
wip/perf-496
job/fix-498
wip/fix-498
job/info-inspector-465
wip/info-inspector-465
job/fix-510
job/fix-507
wip/fix-507
wip/fix-503
job/fix-493
job/money-kinds
wip/money-kinds
job/hygiene-548
job/merge-round-3
wip/fix-493
job/drag-snap-536
wip/merge-round-3
wip/merge-round-0930
wip/drag-snap-536
job/align-538
wip/align-538
job/bg-flash
wip/bg-flash
job/money-import
job/search-count-544
wip/search-count-544
wip/money-import
job/settings-key-541
wip/settings-key-541
job/toast-539
job/preview-421
wip/preview-421
wip/toast-539
job/tasks-500-531
job/title-plain-526
wip/title-plain-526
wip/tasks-500-531
job/notes-bridge
wip/parity-484
job/parity-484
job/files-slow
job/crash-525
wip/notes-bridge
wip/files-slow
wip/crash-525
job/kbd-motion-527
wip/bg-422
job/analytics-504
wip/analytics-504
wip/kbd-motion-527
job/upload-pill-523
wip/upload-pill-523
wip/tray-order
job/tray-order
wip/overflow-mid
wip/merge-round-2
job/perf-494
wip/perf-494
wip/mcp-fast-492
wip/motion-477
wip/asr-ab-489
wip/theme-variants-506
wip/overflow-511
wip/week-header-508
wip/attach-427
job/dav-delete-471
job/iso-435
wip/iso-435
wip/files-sel-keys
wip/dav-delete-471
job/align-253
job/siwc-490
wip/siwc-490
job/money-kinds-review
wip/align-253
wip/money-kinds-review
job/small-bugs-3
wip/overlay-title-487
wip/multiget-500
wip/hidden-420
wip/webcal-ui
wip/webcal-431
job/perf-367
job/location
wip/small-bugs-3
wip/location
wip/perf-367
wip/admin-deny-483
job/tag-unicode-473
wip/tag-unicode-473
job/blur-436
wip/photos-470
wip/blur-436
wip/small-bugs-4
wip/hunt-20260930
wip/settings-hdr-482
wip/chips-416
job/dedup-375
wip/dedup-375
job/doc-stack
wip/doc-stack
job/tokens-literals
wip/tokens-literals
job/jobs-leftovers
wip/send-fast
wip/paste-467
wip/money-numbers
job/money-plugin
wip/money-plugin
job/break-dav
wip/merge-batch
wip/crossday-469
wip/mac-verify
wip/mail-m2
wip/break-dav
wip/money-review2
job/money-md
job/modes-424
wip/money-md
wip/jobs-leftovers
job/agenda-413
wip/agenda-413
wip/modes-424
job/recog-417
wip/recog-417
wip/bounce-425
wip/ab-384-luna
job/webdav-perf
wip/webdav-perf
job/toast-ring
wip/toast-ring
job/money-review
wip/money-review
wip/micro-motion
wip/settings-card
wip/minical
job/notes-imap-428
job/least-priv
wip/ui-small-2
wip/flaky-426
wip/drag-end-418
job/jank
wip/jank
wip/least-priv
wip/docs-site
job/agenda
job/sec-batch
wip/sec-batch
wip/per-user-index
job/area-calendars
wip/area-calendars
job/parity
wip/parity
job/documents-research
wip/documents-research
job/test-infra
job/reminders-sync
wip/small-bugs-2
wip/reminders-sync
wip/gestures
job/google-oauth
wip/tags-merge
wip/tags
job/e2e-theme
wip/e2e-theme
job/icon-align
wip/test-infra
wip/select-align
wip/editor-385
job/voice
wip/webdav
job/webdav
job/app-pw-ui
job/editor-integrity
wip/editor-integrity
wip/voice
wip/quota
wip/cal-followups
wip/icon-align
job/composer-scale
wip/composer-scale
job/jobs-page
wip/jobs-page
job/hig-type
wip/hig-type
wip/app-pw-ui
job/motion-spring
job/mcp
wip/motion-spring
wip/mcp
job/small-bugs
wip/push-hosts
job/profile-sign
wip/touch-369
wip/profile-sign
job/mobile-focus
wip/mobile-focus
wip/ui-polish-354
wip/small-bugs
wip/dup-task
job/toast-polish
job/app-pw-scopes
wip/toast-polish
wip/app-pw-scopes
wip/cli-agent
wip/selection-pills
job/preview-attach
wip/preview-attach
job/dav-proppatch
wip/dav-proppatch
wip/cal-switcher
job/atomic-race
wip/atomic-race
job/photos-shared
wip/photos-shared
wip/cal-grid
wip/note-rewrite
wip/search-rebuild
job/mail-m1
job/paperless-import
wip/paperless-import
wip/mail-m1
wip/hidden-activity
wip/search-d
wip/pricing-research
wip/cursors
wip/auto-scheme
job/single-pills
wip/single-pills
wip/xuser-matrix
wip/money-format
wip/app-pw-setup
wip/purge-dos
wip/vault-health
wip/caldav-apple
wip/xuser-audit
wip/e2e-green
wip/tabbar
wip/adv-harness
wip/maple-mono
job/search-fix
wip/search-fix
wip/search-perf-c
job/adv-harness
wip/sidebar-headers
job/glass
wip/temp-index
job/polish
wip/polish
wip/file-protocols
wip/money-research
wip/glass
wip/voice-models
wip/collab-redo
job/voice-research
wip/hunt-20260928
wip/notes-actions-research
wip/search-pad
wip/search-perf
wip/search-sticky
wip/editor-undo
wip/chrome-rules
wip/motion
wip/appearance-research
wip/appearance
wip/audit-bugs
wip/cal-glass
wip/block-actions
wip/authz-order
wip/event-stripes
wip/chrome-sidebar
wip/auth-flaky
wip/robust-2
wip/gate-fix
wip/menu-blur
wip/import-calternaljs
wip/tray-fix
job/import-calternaljs
wip/index-order
wip/audit-fixes
wip/search-chevrons
research/mail
wip/phone-chrome
wip/dedup-break
wip/csp
wip/ui-audit
wip/select-toast
wip/perf
wip/flat-layout
wip/fonts
wip/event-tint
wip/sync-converge
wip/data-split
wip/glass-audit
wip/robustness
wip/sync-chaos
wip/search-thumbs
wip/fuzz
wip/menu-icons
wip/search-pill
wip/sync-changing
wip/heading-links
wip/date-formats
wip/a11y
wip/break-editor
wip/e2e-fix
wip/settings-sections
wip/sync-root-guard
wip/search-palette
wip/share-edit
job/toasts
wip/toasts
wip/cont-analytics
wip/authz-review
wip/popovers
wip/overlay-glass
wip/change-feed
wip/editor-modes
wip/composer-align
wip/cont-agenda
wip/agenda-merge
job/agent-conventions
wip/agent-conventions
wip/backend-misc
job/route-audit
wip/route-audit
wip/ui-batch
wip/heif-hardening
wip/grid-resize
wip/ask-page
wip/webmcp
job/deeplink-audit
wip/deeplinks
wip/shortcuts
wip/cont-tz-days
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#1038
No description provided.