kayg/calternal

Watch

Fork 0

TESTS: real TLS in the server process (IMAP/SMTP/webcal/accounts) + startup and post-deploy TLS self-check, after the rustls provider outage #601

New issue

Open

opened 2026-10-01 08:33:07 +00:00 by kayg · 3 comments

kayg commented

2026-10-01 08:33:07 +00:00

Owner

Cause: every outbound TLS client panicked ("Could not automatically determine the process-level CryptoProvider"), because both rustls providers (aws-lc-rs, ring) are compiled in. The fix is hotfix 4dc9ca394. No test caught it: the Mail/Calendar tests connect to local servers without TLS, and nothing exercised ClientConfig::builder() in the real server process.
Job: make this class of failure impossible to ship:

A TLS integration test in the server process: start a local IMAP test server (and an HTTPS webcal feed) with TLS using a test CA. Inject the test CA into the client's root store through a test-only hook. Run the real connect paths from a test that boots the server's wiring (not only the plugin unit). Cover IMAP TLS, IMAP STARTTLS, SMTP validation, webcal/CalDAV-account HTTPS, and the Connected Accounts "Test connection".
A startup self-check: on boot, the server builds one TLS client config for each outbound path and logs a clear error (and fails health) if any panics or errors, so a broken build never reaches users silently.
A post-deploy smoke in deploy/deploy-cloud.sh: after health, run a TLS handshake from inside the container to a public endpoint (e.g. imap.fastmail.com:993) through the same client code (a CLI subcommand calternal-server selfcheck tls), with no credentials.
A dependency guard: a test that fails if rustls::ClientConfig::builder() (implicit provider) is used anywhere in the workspace. Require builder_with_provider or a shared helper. One shared helper for TLS client configs (reuse).
Per-crate gates. Time limit 3 h.

## Found by the owner (2026-10-01): Fastmail sign-in failed with "The provider could not be reached securely" **Cause:** every outbound TLS client panicked ("Could not automatically determine the process-level CryptoProvider"), because both rustls providers (aws-lc-rs, ring) are compiled in. The fix is hotfix 4dc9ca394. **No test caught it:** the Mail/Calendar tests connect to local servers **without TLS**, and nothing exercised `ClientConfig::builder()` in the real server process. **Job:** make this class of failure impossible to ship: 1. **A TLS integration test in the server process:** start a local IMAP test server (and an HTTPS webcal feed) **with TLS** using a test CA. Inject the test CA into the client's root store through a test-only hook. Run the real connect paths from a test that boots the server's wiring (not only the plugin unit). Cover IMAP TLS, IMAP STARTTLS, SMTP validation, webcal/CalDAV-account HTTPS, and the Connected Accounts "Test connection". 2. A **startup self-check**: on boot, the server builds one TLS client config for each outbound path and logs a clear error (and fails health) if any panics or errors, so a broken build never reaches users silently. 3. A **post-deploy smoke** in `deploy/deploy-cloud.sh`: after health, run a TLS handshake from inside the container to a public endpoint (e.g. imap.fastmail.com:993) through the same client code (a CLI subcommand `calternal-server selfcheck tls`), with no credentials. 4. A dependency guard: a test that fails if `rustls::ClientConfig::builder()` (implicit provider) is used anywhere in the workspace. Require `builder_with_provider` or a shared helper. One shared helper for TLS client configs (reuse). Per-crate gates. Time limit 3 h.

kayg commented

2026-10-01 08:37:27 +00:00

Author

Owner

Started work on branch job/tlstest-601 from base SHA 4dc9ca394dfef2587b0833a1a59e256349591bed. I am tracing every outbound TLS client and the server startup/deploy paths before adding shared config construction, TLS integration coverage, and self-check commands for #601.

Started work on branch `job/tlstest-601` from base SHA `4dc9ca394dfef2587b0833a1a59e256349591bed`. I am tracing every outbound TLS client and the server startup/deploy paths before adding shared config construction, TLS integration coverage, and self-check commands for #601.

kayg commented

2026-10-01 08:43:30 +00:00

Author

Owner

Finding: only crates/plugins/mail/src/imap.rs builds an explicit rustls provider today. CalDAV and webcal each create independent reqwest clients, while SMTP validation only resolves the endpoint and does not negotiate TLS. I will keep SMTP behavior at the current milestone and make the startup check build its TLS config explicitly; the integration test will cover SMTP endpoint validation through the real Connected Accounts test route.

Finding: only `crates/plugins/mail/src/imap.rs` builds an explicit rustls provider today. CalDAV and webcal each create independent reqwest clients, while SMTP validation only resolves the endpoint and does not negotiate TLS. I will keep SMTP behavior at the current milestone and make the startup check build its TLS config explicitly; the integration test will cover SMTP endpoint validation through the real Connected Accounts test route.

kayg referenced this issue

2026-10-01 09:42:46 +00:00

Mail: verify first sync actually runs after connecting; true toast wording, live progress, plain failure reasons, INFO logs #613

kayg referenced this issue

2026-10-01 10:01:28 +00:00

Mail: verify first sync actually runs after connecting; true toast wording, live progress, plain failure reasons, INFO logs #613

kayg referenced this issue

2026-10-01 10:43:22 +00:00

Mail: verify first sync actually runs after connecting; true toast wording, live progress, plain failure reasons, INFO logs #613

kayg commented

2026-10-01 10:54:09 +00:00

Author

Owner

Finished: tlstest-601

Head SHA: b9df74fb08bd7a643371bbb40ab1a3282787e88e

Commits:

efcba53fb fix(plugins): use shared outbound TLS config
894b2158a feat(server): check outbound TLS before readiness
b9df74fb0 docs: clarify TLS test invariants

Built

Added calternal_plugin::tls::client_config(), which chooses aws-lc-rs explicitly and uses the shared Mozilla WebPKI roots. Mail IMAP, CalDAV account discovery, and webcal feed clients now use it.
Added server startup checks for IMAP implicit TLS and STARTTLS, SMTP validation, Connected Accounts “Test connection”, webcal HTTPS, and CalDAV HTTPS. Each path logs a labeled error; startup returns an error before binding if a check fails or panics.
Added a server-wiring integration test with a local test CA, TLS IMAP and STARTTLS servers, and an HTTPS feed/CalDAV fixture. The isolated integration test passed.
Added calternal-server selfcheck tls and wired deploy/deploy-cloud.sh to run a credential-free TLS handshake to imap.fastmail.com:993 inside the container after health.
Added a workspace Rust-source guard against implicit ClientConfig::builder() calls.

Files

Cargo.lock; crates/calternal-plugin/Cargo.toml; crates/calternal-plugin/src/lib.rs; crates/calternal-plugin/src/tls.rs; crates/plugins/mail/Cargo.toml; crates/plugins/mail/src/imap.rs; crates/plugins/mail/src/lib.rs; crates/plugins/calendar/Cargo.toml; crates/plugins/calendar/src/client/mod.rs; crates/plugins/calendar/src/feeds/mod.rs; crates/plugins/calendar/src/feeds/subscriptions.rs; crates/plugins/calendar/src/lib.rs; crates/calternal-server/Cargo.toml; crates/calternal-server/src/main.rs; crates/calternal-server/src/tls.rs; crates/calternal-server/src/wire.rs; crates/calternal-server/src/wire/tls_integration.rs; deploy/deploy-cloud.sh.

Decisions and known gaps

SMTP endpoint validation currently performs DNS and address-policy checks only; it has no SMTP TLS transport. I preserved that behavior. Startup builds the shared TLS config under the SMTP validation label, and the Connected Accounts integration request exercises SMTP endpoint validation.
The shared helper uses bundled Mozilla WebPKI roots, matching the existing IMAP trust roots; Calendar clients now use the same roots.
The public Fastmail handshake was not run against a deployment. The deploy smoke is wired and bash -n deploy/deploy-cloud.sh passed. No deploy was performed.
The required origin/dev fetch and merge was done once before final gates at merge commit e963219d1. The tracking ref advanced during the job; the branch is now 7 commits behind it. I did not fetch or merge again, per the one-merge rule.

Gates (verbatim result lines)

cargo fmt --check
(no output; exit 0)

cargo clippy -p calternal-plugin --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s
cargo test -p calternal-plugin
Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 43s
test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 23.07s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-calendar --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 01s
cargo test -p calternal-plugin-calendar
Finished `test` profile [unoptimized + debuginfo] target(s) in 22m 17s
test result: ok. 83 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.75s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 45s
cargo test -p calternal-plugin-mail
Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 50s
test result: ok. 36 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.62s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 42s
cargo test -p calternal-server
Finished `test` profile [unoptimized + debuginfo] target(s) in 15m 27s
test result: ok. 108 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 14.04s

wire::tests::tls_integration::outbound_tls_routes_work_with_test_ca --exact --ignored --test-threads=1
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 111 filtered out; finished in 1.33s

bash -n deploy/deploy-cloud.sh
(no output; exit 0)
git diff --check
(no output; exit 0)
cargo clean
Removed 18322 files, 9.2GiB total

The worktree is clean. No push, deploy, or merge was performed after the required origin/dev merge.

## Finished: tlstest-601 Head SHA: `b9df74fb08bd7a643371bbb40ab1a3282787e88e` Commits: - `efcba53fb` fix(plugins): use shared outbound TLS config - `894b2158a` feat(server): check outbound TLS before readiness - `b9df74fb0` docs: clarify TLS test invariants ### Built - Added `calternal_plugin::tls::client_config()`, which chooses aws-lc-rs explicitly and uses the shared Mozilla WebPKI roots. Mail IMAP, CalDAV account discovery, and webcal feed clients now use it. - Added server startup checks for IMAP implicit TLS and STARTTLS, SMTP validation, Connected Accounts “Test connection”, webcal HTTPS, and CalDAV HTTPS. Each path logs a labeled error; startup returns an error before binding if a check fails or panics. - Added a server-wiring integration test with a local test CA, TLS IMAP and STARTTLS servers, and an HTTPS feed/CalDAV fixture. The isolated integration test passed. - Added `calternal-server selfcheck tls` and wired `deploy/deploy-cloud.sh` to run a credential-free TLS handshake to `imap.fastmail.com:993` inside the container after health. - Added a workspace Rust-source guard against implicit `ClientConfig::builder()` calls. ### Files `Cargo.lock`; `crates/calternal-plugin/Cargo.toml`; `crates/calternal-plugin/src/lib.rs`; `crates/calternal-plugin/src/tls.rs`; `crates/plugins/mail/Cargo.toml`; `crates/plugins/mail/src/imap.rs`; `crates/plugins/mail/src/lib.rs`; `crates/plugins/calendar/Cargo.toml`; `crates/plugins/calendar/src/client/mod.rs`; `crates/plugins/calendar/src/feeds/mod.rs`; `crates/plugins/calendar/src/feeds/subscriptions.rs`; `crates/plugins/calendar/src/lib.rs`; `crates/calternal-server/Cargo.toml`; `crates/calternal-server/src/main.rs`; `crates/calternal-server/src/tls.rs`; `crates/calternal-server/src/wire.rs`; `crates/calternal-server/src/wire/tls_integration.rs`; `deploy/deploy-cloud.sh`. ### Decisions and known gaps - SMTP endpoint validation currently performs DNS and address-policy checks only; it has no SMTP TLS transport. I preserved that behavior. Startup builds the shared TLS config under the SMTP validation label, and the Connected Accounts integration request exercises SMTP endpoint validation. - The shared helper uses bundled Mozilla WebPKI roots, matching the existing IMAP trust roots; Calendar clients now use the same roots. - The public Fastmail handshake was not run against a deployment. The deploy smoke is wired and `bash -n deploy/deploy-cloud.sh` passed. No deploy was performed. - The required `origin/dev` fetch and merge was done once before final gates at merge commit `e963219d1`. The tracking ref advanced during the job; the branch is now 7 commits behind it. I did not fetch or merge again, per the one-merge rule. ### Gates (verbatim result lines) ```text cargo fmt --check (no output; exit 0) cargo clippy -p calternal-plugin --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 29s cargo test -p calternal-plugin Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 43s test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 23.07s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-plugin-calendar --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 01s cargo test -p calternal-plugin-calendar Finished `test` profile [unoptimized + debuginfo] target(s) in 22m 17s test result: ok. 83 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.75s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 45s cargo test -p calternal-plugin-mail Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 50s test result: ok. 36 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.62s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 42s cargo test -p calternal-server Finished `test` profile [unoptimized + debuginfo] target(s) in 15m 27s test result: ok. 108 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 14.04s wire::tests::tls_integration::outbound_tls_routes_work_with_test_ca --exact --ignored --test-threads=1 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 111 filtered out; finished in 1.33s bash -n deploy/deploy-cloud.sh (no output; exit 0) git diff --check (no output; exit 0) cargo clean Removed 18322 files, 9.2GiB total ``` The worktree is clean. No push, deploy, or merge was performed after the required `origin/dev` merge.

kayg referenced this issue

2026-10-02 13:11:51 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:12:42 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-03 12:51:32 +00:00

Issue hygiene sweep 2026-10-03 #1008