Mail draft delivery needs recovery after an interrupted upstream APPEND literal

kayg commented

2026-10-04 13:02:14 +00:00

Owner

Found by the remaining #1038 loopback TLS reliability replay on job/mailstress-b, server fix commit e213d4bf0, harness 1df0b8c27. No production data or credentials were used.

A complete 1 MiB draft APPEND received proxy OK. The TLS fixture forwarded the upstream APPEND literal announcement and 4,096 payload bytes, then closed both fixture connections before the full literal. The fixed fixture event confirms append-literal-cut, 4,096 bytes. The coordinator restored normal upstream service and did not replay the client APPEND.

After a 65-second recovery wait, the upstream exact receipt audit failed: no draft receipt. The proxy exact-MIME audit passed at 1,048,576 bytes. The journal contains one dispatched row with 1,048,576 raw bytes. The provider has zero incomplete Maildir temporary files. No partial message or lost local draft was observed. This is a delivery-recovery gap, not a crash or established data loss.

Root cause in crates/plugins/mail/src/proxy_transfers.rs: delivery commits state='dispatched' before upstream APPEND. An I/O failure leaves that state. reconcile("dispatched", &[]) returns Upstream receipt is still unknown; it cannot authorize another APPEND. There is no visible recovery action for this retained draft. The existing focused uncertain_dispatch_never_replays_non_idempotent_commands test enforces the no-replay invariant.

DESIGN §53 deliberately forbids a second COPY/APPEND after an unknown dispatched receipt. Blind automatic replay would conflict with that decision and could duplicate a draft when the provider committed before losing its response. Keep this invariant. Choose and implement a recovery path that resolves the incomplete-literal case while preserving exact MIME, stable virtual UID and no duplicate upstream receipt. The #1038 automatic-recovery expectation is not met by retaining the local draft alone.

Regression evidence is in the fixed bounded CALTERNAL_MAIL_RESILIENCE=1 node tests/adversarial/mail_stress.mjs profile. The fixture's own TLS tests prove that the cut happens inside a literal. append-state checks retained exact MIME and journal byte counts. upstream-audit remains a failing requirement. No existing test expectation was changed.

Found by the remaining #1038 loopback TLS reliability replay on `job/mailstress-b`, server fix commit `e213d4bf0`, harness `1df0b8c27`. No production data or credentials were used. A complete 1 MiB draft APPEND received proxy OK. The TLS fixture forwarded the upstream APPEND literal announcement and 4,096 payload bytes, then closed both fixture connections before the full literal. The fixed fixture event confirms `append-literal-cut`, 4,096 bytes. The coordinator restored normal upstream service and did not replay the client APPEND. After a 65-second recovery wait, the upstream exact receipt audit failed: no draft receipt. The proxy exact-MIME audit passed at 1,048,576 bytes. The journal contains one `dispatched` row with 1,048,576 raw bytes. The provider has zero incomplete Maildir temporary files. No partial message or lost local draft was observed. This is a delivery-recovery gap, not a crash or established data loss. Root cause in `crates/plugins/mail/src/proxy_transfers.rs`: delivery commits `state='dispatched'` before upstream APPEND. An I/O failure leaves that state. `reconcile("dispatched", &[])` returns `Upstream receipt is still unknown`; it cannot authorize another APPEND. There is no visible recovery action for this retained draft. The existing focused `uncertain_dispatch_never_replays_non_idempotent_commands` test enforces the no-replay invariant. DESIGN §53 deliberately forbids a second COPY/APPEND after an unknown dispatched receipt. Blind automatic replay would conflict with that decision and could duplicate a draft when the provider committed before losing its response. Keep this invariant. Choose and implement a recovery path that resolves the incomplete-literal case while preserving exact MIME, stable virtual UID and no duplicate upstream receipt. The #1038 automatic-recovery expectation is not met by retaining the local draft alone. Regression evidence is in the fixed bounded `CALTERNAL_MAIL_RESILIENCE=1 node tests/adversarial/mail_stress.mjs` profile. The fixture's own TLS tests prove that the cut happens inside a literal. `append-state` checks retained exact MIME and journal byte counts. `upstream-audit` remains a failing requirement. No existing test expectation was changed.

kayg referenced this issue

2026-10-04 13:07:18 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

kayg referenced this issue

2026-10-04 13:59:06 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

kayg referenced this issue

2026-10-04 14:06:43 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

kayg referenced this issue

2026-10-04 14:13:13 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

kayg commented

2026-10-04 14:16:46 +00:00

Author

Owner

Started repair and verification on job/mailstress-b, base aa5d4f8bc. Combine job/mailstress-a a16257ec5 with the resilience phase selectors. Investigate bounded draft-only recovery with Message-ID, exact size and INTERNALDATE evidence. COPY/MOVE retain their no-replay invariant. No production mailbox or credentials are used.

Started repair and verification on job/mailstress-b, base aa5d4f8bc. Combine job/mailstress-a a16257ec5 with the resilience phase selectors. Investigate bounded draft-only recovery with Message-ID, exact size and INTERNALDATE evidence. COPY/MOVE retain their no-replay invariant. No production mailbox or credentials are used.

kayg commented

2026-10-04 14:23:00 +00:00

Author

Owner

Repair commits fb4df811c and d4d97aff6 are ready for real-provider verification. Draft-only recovery checks a complete HEADER Message-ID SEARCH, then exact ENVELOPE Message-ID, RFC822.SIZE and INTERNALDATE, in the pinned folder epoch. At most 32 candidates are inspected; absent identity, incomplete metadata and duplicate matches retain the draft. Marker recovery and COPY/MOVE no-replay are unchanged. Existing Jobs provide 100 attempts, exponential delay (1 s up to 1 hour), retained local bytes and the visible failed Job state.

Decisions: reuse the existing journal and Jobs queue; no migration or dependency. Implement the owner-requested negative-search authorization for drafts only, as documented in DESIGN §53. Do not synthesize Message-IDs or change MIME. The focused wire regression covers absence, exact match, substring mismatch, size/date mismatch, duplicate matches and EOF. Fixture tests prove cuts before upload, at 4096 bytes inside a literal, and after upstream tagged OK without forwarding that response. All six fixture tests pass. Real-server audits remain in progress.

Repair commits fb4df811c and d4d97aff6 are ready for real-provider verification. Draft-only recovery checks a complete HEADER Message-ID SEARCH, then exact ENVELOPE Message-ID, RFC822.SIZE and INTERNALDATE, in the pinned folder epoch. At most 32 candidates are inspected; absent identity, incomplete metadata and duplicate matches retain the draft. Marker recovery and COPY/MOVE no-replay are unchanged. Existing Jobs provide 100 attempts, exponential delay (1 s up to 1 hour), retained local bytes and the visible failed Job state. Decisions: reuse the existing journal and Jobs queue; no migration or dependency. Implement the owner-requested negative-search authorization for drafts only, as documented in DESIGN §53. Do not synthesize Message-IDs or change MIME. The focused wire regression covers absence, exact match, substring mismatch, size/date mismatch, duplicate matches and EOF. Fixture tests prove cuts before upload, at 4096 bytes inside a literal, and after upstream tagged OK without forwarding that response. All six fixture tests pass. Real-server audits remain in progress.

kayg commented

2026-10-04 14:35:55 +00:00

Author

Owner

Interrupted APPEND repair now passes on the real local TLS provider. The focused replay accepted one 1,048,576-byte client upload per boundary, with no client replay. All three upstream audits found exactly one exact-MIME draft; all three proxy audits matched the same complete bytes and found an empty transfer journal. Boundaries: no upstream APPEND sent; cut after 4,096 literal bytes; upstream tagged OK withheld. Zero incomplete Dovecot temporary files. Both interrupted FETCH checks also passed (clean NO, zero partial-body rows, then complete OK). Rust fmt, clippy and tests for calternal-imap, calternal-plugin-mail and calternal-server passed; 31 Python harness tests passed. Account separation, SIGKILL and races remain in progress. Evidence: artifacts/mailstress-b/repair-live.jsonl. No production mailbox was used.

kayg commented

2026-10-04 14:38:30 +00:00

Author

Owner

Review found that Message-ID, size and INTERNALDATE alone can identify an older draft version with different equal-length content. Added a regression candidate with identical metadata and different MIME. Recovery now requires the unique metadata candidate's exact bytes, through the existing bounded MIME fetch helper. A mismatch retains the local draft and fails delivery instead of adopting the older version. No existing expectation changed. Mail clippy and all 87 active tests pass. Commit includes a minimal pub(crate) exposure of the existing helper; its existing reader behavior is unchanged. DESIGN §53 and harness documentation now state the additional exact-byte check.

kayg commented

2026-10-04 14:52:46 +00:00

Author

Owner

READY FOR MERGE: yes, for #1056 and its requested quick regressions.

Built interrupted draft APPEND recovery. The worker searches the pinned folder for its operation keyword, then resolves an uncertain draft by exact Message-ID, RFC822.SIZE and INTERNALDATE. A unique metadata candidate must also have exact MIME bytes. Checked absence permits draft re-delivery; a receipt confirms delivery without another APPEND. Missing identity, incomplete metadata, duplicate candidates or different content retain the local draft. Existing Jobs supply bounded retries, exponential backoff and visible failure. COPY/MOVE keep their marker-only no-replay invariant.

Branch: job/mailstress-b. Base: aa5d4f8bc. Merged job/mailstress-a a16257ec5 in 4577dc530, preserving both fixture fixes and phase selection. Fetched origin and merged origin/dev c39ffe5d9 before final gates (already up to date). Head: 6ea7361a30. No push, deploy or issue closure.

Files: crates/plugins/mail/src/proxy_transfers.rs; crates/plugins/mail/src/proxy.rs (minimal pub(crate) exposure of the existing bounded MIME fetch helper); docs/DESIGN.md §53; tests/adversarial/mail_fault_provider.py; tests/adversarial/test_mail_fault_provider.py; tests/adversarial/mail_stress.mjs; tests/adversarial/mail-sync.md. No dependency or migration change.

Requested scenario	Result	Evidence
APPEND: drop before upstream upload	PASS	One 1,048,576-byte client upload, no client replay; exactly one upstream copy, exact proxy MIME, empty journal
APPEND: drop mid-literal	PASS	Cut at 4,096 bytes; exactly one complete upstream copy, exact proxy MIME, empty journal
APPEND: upstream OK lost	PASS	Tagged OK withheld; exactly one complete upstream copy, exact proxy MIME, empty journal
SIGKILL APPEND / COPY / MOVE / EXPUNGE	PASS	100 ms per command; exact receipts and 128 original Inbox identities across three Users
MOVE versus MOVE	PASS	NO / OK; one destination receipt
MOVE versus delete	PASS	OK / NO; one destination receipt
FETCH versus EXPUNGE	PASS	OK / OK; no remaining receipt
Interrupted FETCH	PASS	Dispatch cut and literal cut; NO with zero partial-body rows, then complete OK
Account separation	PASS	Three sessions, 21 checks, three exact upstream moves for three Connected Accounts under one User
Final health	PASS	Readiness 200 and all three retained web sessions 200

Evidence: artifacts/mailstress-b/repair-final.jsonl exited 0 on the exact-MIME-check build. Its three APPEND audits and all four kills plus races pass. Initial sync completed in 126.586 s, peak sampled RSS 208128 KiB, no account errors. artifacts/mailstress-b/repair-live.jsonl contains the earlier passing FETCH/account checks and all three APPEND boundaries. That combined run ended in a coordinator failure after the login-policy fixture filled its ten-attempt IP window and the next transfer phase attempted a fresh login. The coordinator now resets this fixture policy between phases. The failed run is retained; the final focused run skips the already-passed policy/account/FETCH phases. No existing product expectation changed. Six TLS fixture tests prove the three upload fault boundaries. The wire regression covers absence, exact match, substring mismatch, size/date mismatch, duplicate matches, missing identity, EOF and an older equal-length draft with identical metadata.

Decisions: reuse the journal and existing 100-attempt backoff queue; inspect at most 32 Message-ID candidates; require an exact MIME match for a unique metadata candidate; preserve missing-ID MIME instead of synthesizing identity. The owner-requested draft-only negative-search exception is recorded in DESIGN §53. No new UI is required for retained failures because Jobs already exposes owner-bound failed delivery.

Known gaps: ambiguous or missing identities remain retained rather than automatically delivered. The SIGKILL samples interrupted queued delivery, not an observed upstream commit. The broader #1038 load/platform matrix was not rerun; this report certifies the repair and requested quick regressions. The complete combined coordinator path after the policy reset was not repeated. UX gaps closed/left: not applicable; no UI changed. For the merge round: no requested repair scenario is deferred. This is not a performance issue; no perf-VM or Mac run was required.

Production web build and mail-test-provider server build passed. node --check tests/adversarial/mail_stress.mjs and git diff --check passed. Module/function comments were re-read. Cleanup completed: cargo clean removed 16948 files, 9.0GiB; web build output removed; worktree clean. The final commit only updates comments and verification documentation.

Verbatim gate summaries (all commands exited 0). cargo fmt --check produced no output. OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and worktree TMPDIR were set.

cargo clippy -p calternal-imap --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.20s

cargo test -p calternal-imap

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.88s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.26s

cargo test -p calternal-plugin-mail

test result: ok. 87 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 3.81s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.69s

cargo test -p calternal-server

test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.60s

python3 -m unittest discover -s tests/adversarial -p 'test_mail*.py'

...............................
----------------------------------------------------------------------
Ran 31 tests in 2.806s

OK

READY FOR MERGE: yes, for #1056 and its requested quick regressions. Built interrupted draft APPEND recovery. The worker searches the pinned folder for its operation keyword, then resolves an uncertain draft by exact Message-ID, RFC822.SIZE and INTERNALDATE. A unique metadata candidate must also have exact MIME bytes. Checked absence permits draft re-delivery; a receipt confirms delivery without another APPEND. Missing identity, incomplete metadata, duplicate candidates or different content retain the local draft. Existing Jobs supply bounded retries, exponential backoff and visible failure. COPY/MOVE keep their marker-only no-replay invariant. Branch: job/mailstress-b. Base: aa5d4f8bc. Merged job/mailstress-a a16257ec5 in 4577dc530, preserving both fixture fixes and phase selection. Fetched origin and merged origin/dev c39ffe5d9 before final gates (already up to date). Head: 6ea7361a30690ef56983f4b6991a13a3841a276a. No push, deploy or issue closure. Files: crates/plugins/mail/src/proxy_transfers.rs; crates/plugins/mail/src/proxy.rs (minimal pub(crate) exposure of the existing bounded MIME fetch helper); docs/DESIGN.md §53; tests/adversarial/mail_fault_provider.py; tests/adversarial/test_mail_fault_provider.py; tests/adversarial/mail_stress.mjs; tests/adversarial/mail-sync.md. No dependency or migration change. | Requested scenario | Result | Evidence | |---|---|---| | APPEND: drop before upstream upload | PASS | One 1,048,576-byte client upload, no client replay; exactly one upstream copy, exact proxy MIME, empty journal | | APPEND: drop mid-literal | PASS | Cut at 4,096 bytes; exactly one complete upstream copy, exact proxy MIME, empty journal | | APPEND: upstream OK lost | PASS | Tagged OK withheld; exactly one complete upstream copy, exact proxy MIME, empty journal | | SIGKILL APPEND / COPY / MOVE / EXPUNGE | PASS | 100 ms per command; exact receipts and 128 original Inbox identities across three Users | | MOVE versus MOVE | PASS | NO / OK; one destination receipt | | MOVE versus delete | PASS | OK / NO; one destination receipt | | FETCH versus EXPUNGE | PASS | OK / OK; no remaining receipt | | Interrupted FETCH | PASS | Dispatch cut and literal cut; NO with zero partial-body rows, then complete OK | | Account separation | PASS | Three sessions, 21 checks, three exact upstream moves for three Connected Accounts under one User | | Final health | PASS | Readiness 200 and all three retained web sessions 200 | Evidence: artifacts/mailstress-b/repair-final.jsonl exited 0 on the exact-MIME-check build. Its three APPEND audits and all four kills plus races pass. Initial sync completed in 126.586 s, peak sampled RSS 208128 KiB, no account errors. artifacts/mailstress-b/repair-live.jsonl contains the earlier passing FETCH/account checks and all three APPEND boundaries. That combined run ended in a coordinator failure after the login-policy fixture filled its ten-attempt IP window and the next transfer phase attempted a fresh login. The coordinator now resets this fixture policy between phases. The failed run is retained; the final focused run skips the already-passed policy/account/FETCH phases. No existing product expectation changed. Six TLS fixture tests prove the three upload fault boundaries. The wire regression covers absence, exact match, substring mismatch, size/date mismatch, duplicate matches, missing identity, EOF and an older equal-length draft with identical metadata. Decisions: reuse the journal and existing 100-attempt backoff queue; inspect at most 32 Message-ID candidates; require an exact MIME match for a unique metadata candidate; preserve missing-ID MIME instead of synthesizing identity. The owner-requested draft-only negative-search exception is recorded in DESIGN §53. No new UI is required for retained failures because Jobs already exposes owner-bound failed delivery. Known gaps: ambiguous or missing identities remain retained rather than automatically delivered. The SIGKILL samples interrupted queued delivery, not an observed upstream commit. The broader #1038 load/platform matrix was not rerun; this report certifies the repair and requested quick regressions. The complete combined coordinator path after the policy reset was not repeated. UX gaps closed/left: not applicable; no UI changed. For the merge round: no requested repair scenario is deferred. This is not a performance issue; no perf-VM or Mac run was required. Production web build and mail-test-provider server build passed. node --check tests/adversarial/mail_stress.mjs and git diff --check passed. Module/function comments were re-read. Cleanup completed: cargo clean removed 16948 files, 9.0GiB; web build output removed; worktree clean. The final commit only updates comments and verification documentation. Verbatim gate summaries (all commands exited 0). cargo fmt --check produced no output. OPENSSL_NO_VENDOR=1, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and worktree TMPDIR were set. `cargo clippy -p calternal-imap --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.20s ``` `cargo test -p calternal-imap` ```text test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.88s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.26s ``` `cargo test -p calternal-plugin-mail` ```text test result: ok. 87 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 3.81s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.69s ``` `cargo test -p calternal-server` ```text test result: ok. 170 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 14.60s ``` `python3 -m unittest discover -s tests/adversarial -p 'test_mail*.py'` ```text ............................... ---------------------------------------------------------------------- Ran 31 tests in 2.806s OK ```

kayg referenced this issue

2026-10-04 14:52:47 +00:00

Mail proxy: crash-proof it under overload and bizarre scenarios #1038

kayg commented

2026-10-05 03:08:44 +00:00

Author

Owner

Deployed to production 2026-10-05 ~04:40 CEST in round 9 (269b1b51b). Includes the mail proxy (CalternalDAV, real Apple Mail acceptance PASS on the Mac VM), provider sync fixes, the stress-round fixes, #1067, #1068, #1078 and the Files upload identity repair. Staging healthy first; production healthy in 18 s; Auth 14 and Mail 17 migrations applied; change events 0/30 s; no expired leases.

Deployed to production 2026-10-05 ~04:40 CEST in round 9 (269b1b51b). Includes the mail proxy (CalternalDAV, real Apple Mail acceptance PASS on the Mac VM), provider sync fixes, the stress-round fixes, #1067, #1068, #1078 and the Files upload identity repair. Staging healthy first; production healthy in 18 s; Auth 14 and Mail 17 migrations applied; change events 0/30 s; no expired leases.

kayg closed this issue

2026-10-05 03:08:49 +00:00

Rows
Columns

Mail draft delivery needs recovery after an interrupted upstream APPEND literal #1056