Mail SELECT is refused after sustained concurrent use in the combined round #1067

Closed
opened 2026-10-04 18:29:14 +00:00 by kayg · 8 comments
Owner

The combined #1038 Mail round still refuses ordinary SELECT after sustained concurrent use. Source: job/mailround-1038, scheduling fix 1cc8383dc, separately pinned mail-test-provider server SHA-256 6bddc427158176be4f507fc4ff27be9ccd0256c2e22c38b5b956277d3b813f05. This is a real local TLS Dovecot run with generated test data.

Reproduce with the combined production web build and pinned server:

CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_SCREENSHOTS=1 node tests/adversarial/mail_stress.mjs

The first phase requests 1,800 seconds and 12 concurrent sessions. All clients stopped after refused SELECT; summary duration was 872.41 seconds. It completed 8,084 SELECT, 8,064 FETCH, SEARCH, IDLE and EXPUNGE commands. It accepted 20 APPEND, STORE, COPY and MOVE sequences. All 20 exact local receipts and upstream receipts passed their audits. There were 12 SELECT:NO and 12 client assertion failures. The server remained ready and the web sessions stayed authenticated. The second phase continues separately; final #1038 evidence will add its result.

The refusal reason was not captured by the original probe. It must not be labelled SLOW-only without evidence. The probe now reports only fixed safe refusal categories (cache unavailable, snapshot limit, metadata limit and epoch changed), while keeping arbitrary wire text private. This needs a fresh classified reproduction, a fix if the refusal is not expected, and a regression. Preserve all existing status and receipt assertions. Do not increase production limits or change expectations to hide the failure.

The IDLE pool starvation found in the same round is fixed separately: IDLE has a bounded Worker while ordinary background work retains one slot. That fix makes all ordinary write sequences complete, but does not establish why SELECT later fails. Keep this finding open; #1038 is not ready for staging.

The combined #1038 Mail round still refuses ordinary SELECT after sustained concurrent use. Source: job/mailround-1038, scheduling fix `1cc8383dc`, separately pinned mail-test-provider server SHA-256 `6bddc427158176be4f507fc4ff27be9ccd0256c2e22c38b5b956277d3b813f05`. This is a real local TLS Dovecot run with generated test data. Reproduce with the combined production web build and pinned server: ```sh CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_SCREENSHOTS=1 node tests/adversarial/mail_stress.mjs ``` The first phase requests 1,800 seconds and 12 concurrent sessions. All clients stopped after refused SELECT; summary duration was 872.41 seconds. It completed 8,084 SELECT, 8,064 FETCH, SEARCH, IDLE and EXPUNGE commands. It accepted 20 APPEND, STORE, COPY and MOVE sequences. All 20 exact local receipts and upstream receipts passed their audits. There were 12 `SELECT:NO` and 12 client assertion failures. The server remained ready and the web sessions stayed authenticated. The second phase continues separately; final #1038 evidence will add its result. The refusal reason was not captured by the original probe. It must not be labelled SLOW-only without evidence. The probe now reports only fixed safe refusal categories (cache unavailable, snapshot limit, metadata limit and epoch changed), while keeping arbitrary wire text private. This needs a fresh classified reproduction, a fix if the refusal is not expected, and a regression. Preserve all existing status and receipt assertions. Do not increase production limits or change expectations to hide the failure. The IDLE pool starvation found in the same round is fixed separately: IDLE has a bounded Worker while ordinary background work retains one slot. That fix makes all ordinary write sequences complete, but does not establish why SELECT later fails. Keep this finding open; #1038 is not ready for staging.
Author
Owner

The pinned-binary three-account phase passed its complete requested duration:

{"scenario":"three-accounts","result":"PASS","web_sessions":"PASS"}

Its summary is 1,808.70 seconds, 12 clients, three Users with one Connected Account each, 17,293 SELECT OK, 17,273 SEARCH/FETCH/IDLE/EXPUNGE OK, and 20 accepted APPEND/STORE/COPY/MOVE sequences. All 20 exact local and upstream receipt audits passed; no client failure was recorded. RSS peaked at 309.00 MiB and open descriptors at 155 in this phase. The same binary failed the one-User/one-account phase at 872.41 seconds with 12 SELECT NO. Both results remain separate. This points to higher per-User concurrency in #1067; it does not prove the cause.

The job is continuing the real production screenshot replay, a diagnostic 50-client pass and the requested fresh-fixture restart/fault cases. The 50-client diagnostic is short and cannot replace the requested long run.

The pinned-binary three-account phase passed its complete requested duration: ```json {"scenario":"three-accounts","result":"PASS","web_sessions":"PASS"} ``` Its summary is 1,808.70 seconds, 12 clients, three Users with one Connected Account each, 17,293 SELECT OK, 17,273 SEARCH/FETCH/IDLE/EXPUNGE OK, and 20 accepted APPEND/STORE/COPY/MOVE sequences. All 20 exact local and upstream receipt audits passed; no client failure was recorded. RSS peaked at 309.00 MiB and open descriptors at 155 in this phase. The same binary failed the one-User/one-account phase at 872.41 seconds with 12 SELECT NO. Both results remain separate. This points to higher per-User concurrency in #1067; it does not prove the cause. The job is continuing the real production screenshot replay, a diagnostic 50-client pass and the requested fresh-fixture restart/fault cases. The 50-client diagnostic is short and cannot replace the requested long run.
Author
Owner

Round 2 started on job/mailround-1038, prior head 0a3bf7a8eb5056730f9d7547ef65df562ed1b076. Merged origin/dev at 9fb9a4bfb2488152c83d50c55441ff5f43b572b2 cleanly; merge head b6269034026d7a0f3471b9740ab63bed69d4fc4b.

Investigating sustained single-account SELECT refusals (#1067), resolving three failing Mail regressions against DESIGN §§45/53/59, checking production migration upgrade, and rerunning the requested gates and 30-minute single-account phase. No push or deploy.

Round 2 started on `job/mailround-1038`, prior head `0a3bf7a8eb5056730f9d7547ef65df562ed1b076`. Merged `origin/dev` at `9fb9a4bfb2488152c83d50c55441ff5f43b572b2` cleanly; merge head `b6269034026d7a0f3471b9740ab63bed69d4fc4b`. Investigating sustained single-account SELECT refusals (#1067), resolving three failing Mail regressions against DESIGN §§45/53/59, checking production migration upgrade, and rerunning the requested gates and 30-minute single-account phase. No push or deploy.
Author
Owner

The current-dev single-account 30-minute phase passes against pinned feature executable SHA-256 9e046864e71e355412016b3bc73baa54f0faae9b2ad6206ef56211d17dc98bee.

Result: 1,808.72 seconds, 12 clients / one User / one Connected Account, 15,953 SELECT OK and 15,933 SEARCH/FETCH/IDLE/EXPUNGE OK. It accepts 20 APPEND/STORE/COPY/MOVE sequences; all 20 exact upstream and local receipt audits pass. No SELECT NO, client failure, receipt failure or snapshot SQL failure is recorded. Peak RSS is 298,568 KiB (291.57 MiB); peak descriptors 153. Retained web sessions and readiness pass.

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

This is a complete phase, not a short smoke run. It does not yet establish why the earlier revision failed. A historical-head comparison (0a3bf7a8e with only the content-free snapshot diagnostic patch) is now running with the same fixture and assertions. The earlier historical setup was deliberately stopped before its soak to keep one browser active; that interrupted setup is excluded from acceptance evidence. No production limits or required status/receipt assertions were changed, and #1067 is not labelled SLOW-only or resolved without a classified cause.

The current-dev single-account 30-minute phase passes against pinned feature executable SHA-256 `9e046864e71e355412016b3bc73baa54f0faae9b2ad6206ef56211d17dc98bee`. Result: 1,808.72 seconds, 12 clients / one User / one Connected Account, 15,953 SELECT OK and 15,933 SEARCH/FETCH/IDLE/EXPUNGE OK. It accepts 20 APPEND/STORE/COPY/MOVE sequences; all 20 exact upstream and local receipt audits pass. No SELECT NO, client failure, receipt failure or snapshot SQL failure is recorded. Peak RSS is 298,568 KiB (291.57 MiB); peak descriptors 153. Retained web sessions and readiness pass. ```json {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` This is a complete phase, not a short smoke run. It does not yet establish why the earlier revision failed. A historical-head comparison (`0a3bf7a8e` with only the content-free snapshot diagnostic patch) is now running with the same fixture and assertions. The earlier historical setup was deliberately stopped before its soak to keep one browser active; that interrupted setup is excluded from acceptance evidence. No production limits or required status/receipt assertions were changed, and #1067 is not labelled SLOW-only or resolved without a classified cause.
Author
Owner

#1067: cached SELECT writer contention reproduced and repaired

The snapshot path checked out the single writer for every SELECT, including a fully cached, unchanged mailbox. A background transaction could therefore hold ordinary SELECT behind the command deadline. The bounded expunge cursor and split IDLE workers do not remove that dependency.

Added a deterministic regression that primes INBOX, holds an uncommitted background edit on the writer, and requires SELECT to finish while that transaction remains open. Before the repair:

test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... FAILED
unchanged SELECT must not check out the occupied writer: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.93s

After the repair:

test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.43s

Unchanged snapshots now use one consistent WAL reader transaction. The same SQL, ownership/service visibility joins, metadata bounds, pending-intent projection and fingerprint checks serve both passes. New UIDs, flags, visibility or view changes release the reader and use the single writer. Readers never advance durable clocks. No timeout or resource limit was increased.

The existing bench profile now holds the writer for serial cached SELECT samples and a 12-client burst, including its 100,000-message case. Full Mail/server gates and the fixed-build 30-minute phase are next.

The original 862-second run did not record the SQL error class, so this deterministic defect does not identify every refusal in that older run. The unmodified merged build already completed one 30-minute diagnostic phase; fixed-build acceptance will be reported separately. Historical comparison was stopped after Cargo reused incompatible artifacts from an archived source tree. Those stopped runs are excluded; no historical result is claimed.

#1067: cached SELECT writer contention reproduced and repaired The snapshot path checked out the single writer for every SELECT, including a fully cached, unchanged mailbox. A background transaction could therefore hold ordinary SELECT behind the command deadline. The bounded expunge cursor and split IDLE workers do not remove that dependency. Added a deterministic regression that primes INBOX, holds an uncommitted background edit on the writer, and requires SELECT to finish while that transaction remains open. Before the repair: ``` test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... FAILED unchanged SELECT must not check out the occupied writer: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.93s ``` After the repair: ``` test proxy::tests::unchanged_select_does_not_wait_for_the_background_writer ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.43s ``` Unchanged snapshots now use one consistent WAL reader transaction. The same SQL, ownership/service visibility joins, metadata bounds, pending-intent projection and fingerprint checks serve both passes. New UIDs, flags, visibility or view changes release the reader and use the single writer. Readers never advance durable clocks. No timeout or resource limit was increased. The existing bench profile now holds the writer for serial cached SELECT samples and a 12-client burst, including its 100,000-message case. Full Mail/server gates and the fixed-build 30-minute phase are next. The original 862-second run did not record the SQL error class, so this deterministic defect does not identify every refusal in that older run. The unmodified merged build already completed one 30-minute diagnostic phase; fixed-build acceptance will be reported separately. Historical comparison was stopped after Cargo reused incompatible artifacts from an archived source tree. Those stopped runs are excluded; no historical result is claimed.
Author
Owner

#1067: the 7b write topology is now covered by a before/after regression.

Compared mailstress-b (6ea7361a3) with the combined Mail head (0a3bf7a8e). SELECT's snapshot transaction was inherited unchanged. calternal-db changed from one application write connection to separate ordinary WAL/NORMAL and Security WAL/FULL connections plus checkpoint maintenance. The separate Security connection can hold SQLite's write lock without occupying the ordinary pool. Cached SELECT still tried to write, so it depended on both locks.

Extended the regression to hold the Security writer first and the Index writer second. Ran that same test against the pre-repair snapshot source in the current merged worktree, then restored the repair. The copied soak binary is independent and its checksum stayed unchanged.

Before repair, with the Security transaction held:

test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... FAILED
unchanged SELECT must not wait for Index or Security writes: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.24s

After repair, both held-writer cases pass inside the full Mail gate:

test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... ok
test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s

Clippy and fmt pass. The product repair is committed at 9c8d1e763; the topology regression and its instructions are committed separately. No durability policy, deadline or resource bound was weakened. This reproduces and removes an ordinary-read dependency exposed by the 7b topology. The older sustained refusal had no error class, so no specific SQLite code is assigned to that historical run. The fixed-build 1,800-second phase is still running.

#1067: the 7b write topology is now covered by a before/after regression. Compared `mailstress-b` (`6ea7361a3`) with the combined Mail head (`0a3bf7a8e`). SELECT's snapshot transaction was inherited unchanged. `calternal-db` changed from one application write connection to separate ordinary WAL/NORMAL and Security WAL/FULL connections plus checkpoint maintenance. The separate Security connection can hold SQLite's write lock without occupying the ordinary pool. Cached SELECT still tried to write, so it depended on both locks. Extended the regression to hold the Security writer first and the Index writer second. Ran that same test against the pre-repair snapshot source in the current merged worktree, then restored the repair. The copied soak binary is independent and its checksum stayed unchanged. Before repair, with the Security transaction held: ``` test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... FAILED unchanged SELECT must not wait for Index or Security writes: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.24s ``` After repair, both held-writer cases pass inside the full Mail gate: ``` test proxy::tests::unchanged_select_does_not_wait_for_index_or_security_writer ... ok test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s ``` Clippy and fmt pass. The product repair is committed at `9c8d1e763`; the topology regression and its instructions are committed separately. No durability policy, deadline or resource bound was weakened. This reproduces and removes an ordinary-read dependency exposed by the 7b topology. The older sustained refusal had no error class, so no specific SQLite code is assigned to that historical run. The fixed-build 1,800-second phase is still running.
Author
Owner

#1067: fixed-build single-account 30-minute phase PASS

Product source 9c8d1e7630aafae04c43036cba7b80fd64e2b932, copied feature executable SHA-256 d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2. The later commits add regression coverage and instructions only. The checksum stayed unchanged during gates and the controlled pre-repair regression.

Command: CALTERNAL_SERVER_BIN=<copied-feature-executable> CALTERNAL_MAIL_STRESS_IMAGE=localhost/mailstress-1038:latest CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_PHASE=one-account CALTERNAL_MAIL_STRESS_CLIENTS=12 node tests/adversarial/mail_stress.mjs.

One User, one Connected Account, 12 clients, full 1,800-second command phase plus receipt audits: 1,808.52 seconds total. 18,402 SELECT OK; 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each APPEND, STORE, COPY and MOVE succeed. All 20 upstream and 20 local exact-MIME/flags/folder receipts pass. Expected pending-source/destination NO responses were retried only after explicit refusal; accepted writes were never replayed.

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

The mixed-soak result has failures: {} and failure_reasons: {}. Peak RSS is 280,340 KiB (273.77 MiB), peak FDs 171. After client/receipt sockets close: 145,172 KiB RSS and 127 FDs. The descriptor series plateaus and then falls. No crash, hang or receipt discrepancy was found in this phase. These are local fixture/debug resource samples, not release performance baseline numbers.

Evidence: artifacts/round2-one-account-fixed.jsonl. The required #1067 continuation is now PASS. The larger #1038 acceptance gaps and startup SLOW-only gate remain recorded separately. A perf-VM profile and the final report are in progress; no staging deployment is authorized or performed.

#1067: fixed-build single-account 30-minute phase PASS Product source `9c8d1e7630aafae04c43036cba7b80fd64e2b932`, copied feature executable SHA-256 `d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2`. The later commits add regression coverage and instructions only. The checksum stayed unchanged during gates and the controlled pre-repair regression. Command: `CALTERNAL_SERVER_BIN=<copied-feature-executable> CALTERNAL_MAIL_STRESS_IMAGE=localhost/mailstress-1038:latest CALTERNAL_MAIL_STRESS_BOUNDED_FIXTURE=1 CALTERNAL_MAIL_STRESS_PHASE=one-account CALTERNAL_MAIL_STRESS_CLIENTS=12 node tests/adversarial/mail_stress.mjs`. One User, one Connected Account, 12 clients, full 1,800-second command phase plus receipt audits: 1,808.52 seconds total. 18,402 SELECT OK; 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each APPEND, STORE, COPY and MOVE succeed. All 20 upstream and 20 local exact-MIME/flags/folder receipts pass. Expected pending-source/destination NO responses were retried only after explicit refusal; accepted writes were never replayed. ``` {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` The mixed-soak result has `failures: {}` and `failure_reasons: {}`. Peak RSS is 280,340 KiB (273.77 MiB), peak FDs 171. After client/receipt sockets close: 145,172 KiB RSS and 127 FDs. The descriptor series plateaus and then falls. No crash, hang or receipt discrepancy was found in this phase. These are local fixture/debug resource samples, not release performance baseline numbers. Evidence: `artifacts/round2-one-account-fixed.jsonl`. The required #1067 continuation is now PASS. The larger #1038 acceptance gaps and startup SLOW-only gate remain recorded separately. A perf-VM profile and the final report are in progress; no staging deployment is authorized or performed.
Author
Owner

Final mailround2-1038 report.

Head: 8091d5788ad86aaee1da831801031110462c6f6d on job/mailround-1038. Worktree clean. No push, deploy or merge beyond the authorized origin/dev integration.

Mail round 2: #1038 and #1067

READY FOR STAGING: no.

Built

Merged remote origin/dev at 9fb9a4bfb2488152c83d50c55441ff5f43b572b2
into job/mailround-1038. The merge includes #1062, #1065 and Files/open
parity. There was no push, deploy or issue closure.

Kept the deployed migration prefix: Auth 13, DB 15, Files 24, Mail 11 and
Notes 32. Mail 12–17 and Auth 14 follow that prefix. Extended the existing
production-copy upgrade test. It now checks both the older production schema
and the deployed 7b prefix. Old receipts, Mail rows, Security state and Notes
cursors survive. Only seven proxy/Auth additions are pending after 7b. A
second startup makes no backup.

Repaired cached SELECT writer contention. The pre-7b Mail snapshot used a
writer transaction for every SELECT. 7b added a separate Security write
connection. An unchanged SELECT still needed the ordinary pool and SQLite's
write lock. It could therefore wait behind Index or Security work. The
snapshot now checks the complete committed view in one WAL reader
transaction. A change releases the reader and uses the single writer. Both
passes use the same SQL, ownership checks and limits. Readers never advance
durable clocks. The regression holds each writer separately. The old snapshot
fails; the repaired snapshot completes before either writer is released.

Added fixed-code diagnostics for SQL errors and every SELECT resource refusal.
The harness reports no arbitrary wire, SQL, credential or provider text.
Extended the existing bench profile with held-writer SELECT samples and a
12-client burst. The sampler also has a 100,000-message case.

Repaired the three stale Mail tests against the current contracts:

  • SSE: require the initial canonical folder refresh, the exact owner event
    and no foreign event. DESIGN §§53, 59.
  • Worker pools: pin sync, IDLE, mutation and transfer limits. A stateful
    barrier keeps all three IDLE waits open while ordinary jobs finish. §53.
  • FETCH recovery: require all 80 UIDs and exactly one bounded
    UID SEARCH UID 1:80. Keep membership validation. §§45, 59.

Formatted the imported Notes reconciliation test so the workspace fmt gate
passes. Its behavior and assertions stay unchanged.

Evidence

The product repair is 9c8d1e7630aafae04c43036cba7b80fd64e2b932.
The expanded topology regression is a6e0ff6a5. The acceptance executable
was copied before the server test build. Its SHA-256 is
d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2.
The later regression commit changes tests and instructions only.

The fixed-product single-account phase passes: one User, one Connected
Account, 12 clients, 1,808.52 seconds including receipt audits. It reports
18,402 SELECT OK and 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each
APPEND, STORE, COPY and MOVE succeed. All 20 exact upstream and 20 local
receipts pass. Required SELECT has no refusal. Expected pending-transfer NO
responses are retried only after explicit refusal.

{"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}}

There are no mixed-soak failures or failure reasons. Peak RSS is 280,340 KiB
(273.77 MiB), peak FDs 171. After client and receipt sockets close: 145,172
KiB RSS and 127 FDs. Readiness and retained web sessions pass. These are local
fixture/debug observations. Evidence is
artifacts/round2-one-account-fixed.jsonl.

The unmodified merged product also completed a diagnostic 1,808.72-second
phase before the repair: 15,953 SELECT OK, no failures, and 20 exact upstream
and local receipts. This is separate from fixed-product acceptance.

The old snapshot failed the held-Security-writer regression in 2.24 seconds.
The fixed full Mail suite passes that same test with both writers held in turn.
The older 872.41-second refusal did not record an SQL class. Do not assign a
specific SQLite error code to that historical run.

Startup was measured on the idle host at 10.69.69.63, with
/root/perf.lock. Load average was 0.20 / 0.17 / 0.08. The existing test
passed in 3.34 seconds; total wall time was 3.40 seconds. No build ran on
that host. Two initial attempts lacked debug frontend paths and failed
before measuring startup. They are excluded.

The first full local server suite passed: 219 passed, no failures. The later
fixed-product run, concurrent with the soak on the busy host, hit only the
existing startup deadline at 15.15 seconds. This is a SLOW-only result. The
15-second limit stays unchanged.

Release profile

The updated profile completes on the perf VM under /root/perf.lock. The
Mail test executable was built locally in release mode. The VM compiled no
code. Cached SELECT samples and each 12-client burst hold the writer occupied.

Profile Median per-run SELECT p50 / p95 Mean worker CPU Mean / peak worker RSS Largest 12-client SELECT duration
10,001 messages, 3 serial runs 134.51 / 176.76 ms 147.54% 111.03 / 153.00 MiB 1,696.68 ms
100,000 messages 1,953.88 / 5,540.95 ms 163.04% 848.05 / 1,205.99 MiB 18,724.86 ms
3 workers, each with 100,000 messages and its own Index 1,790.76 / 4,051.31 ms 113.24% 891.19 / 1,205.86 MiB 24,210.30 ms

CPU and RSS include seeding, metadata and queue phases. They are not
SELECT-only resource figures. CPU can exceed 100% because SQLite uses
multiple threads. The load averages inside the lock are
6.125 / 4.5923 / 2.1152 before and 7.5220 / 5.5894 / 3.3701 after. The
starting load includes the preceding debug diagnostic run. That incomplete
debug run was stopped and is excluded. Do not treat this as a cold quiet-host
baseline.

docs/perf/baseline.json has no comparable cached SELECT profile.
mail.accounts measures HTTP account listing (p50 1.3 ms / p95 3.8 ms).
It cannot establish a SELECT regression. The existing large-snapshot cost
remains visible and is filed as
#1070. This profile is not
real-provider 100,000-message sync or 50-client acceptance. Evidence is
artifacts/round2-select-perf-vm-release.log.

Files

Round-2 edits, excluding the incoming merge:

  • crates/plugins/mail/src/proxy.rs and proxy_tests.rs.
  • crates/plugins/mail/src/routes.rs and sync.rs.
  • crates/calternal-server/src/upgrade_tests.rs.
  • crates/plugins/notes/src/lib.rs (format only).
  • tests/adversarial/mail_proxy.py, test_mail_proxy.py, mail_stress.mjs
    and mail-sync.md.
  • bench/mail-sync.py and this report.

UX gaps closed

Unchanged cached Mail views can be read while Index or Security writes are
active. UID, revision and message identity checks remain intact. This round
adds no UI. The earlier job's production screenshots remain its visual
evidence; this round makes no new visual claim.

UX gaps left and known gaps

  • #1068 remains: phone Settings sheet title and Copy link checks.
  • The latest busy-host server gate has one startup SLOW-only failure.
  • #1038 still has no complete 50-client long run on both account groups,
    100,000-message initial-sync acceptance, 50 MB/zero-byte/10,000-folder
    matrix or download storm on the final product. Earlier short diagnostics
    do not replace these cases. These gaps prevent a full #1038 acceptance
    claim and READY FOR STAGING remains no.
  • The older SELECT refusal was unclassified. The held-writer defect is
    reproduced and repaired; no exact error code is inferred for that run.
  • Historical comparison attempts were stopped when Cargo reused artifacts
    from an archived source tree. They are excluded. The controlled regression
    uses the old snapshot source in the current worktree and is valid.
  • #1070 tracks large cached SELECT latency and memory. There is no
    comparable baseline to establish a regression.
  • Four existing Svelte warnings remain. There are no Svelte errors.

Decisions

Use a checked WAL reader pass for unchanged snapshots. Keep all changes on
the single writer. This is an implementation choice consistent with
DESIGN §§2, 53, 59. Do not change durability, deadlines or cache bounds.

Use the canonical SSE hint, four bounded worker pools and bounded UID search
already implemented on dev. Update the stale test contracts while keeping or
strengthening what they prove.

Keep the startup product and 15-second test limit. The idle measurement does
not justify a limit increase. Report the later busy-host failure verbatim.

Measure the existing component profile in release mode on the perf VM. Keep
its cache timings separate from the local debug protocol phase. File the
large-snapshot performance gap as #1070. Do not call it a baseline regression
without comparable endpoint, data and build measurements.

For the merge round

Run the combined web suite with cd apps/web && bun run test and the combined
startup gate with cargo test -p calternal-server -- --test-threads=4. The
startup test must show that upgrade backfills do not delay HTTP readiness.
Recheck #1068. These do not replace the completed requested single-account
phase. The original #1038 large-data and 50-client gaps remain listed above.

Gate output, verbatim excerpts

cargo fmt --check exits 0 with no output. Cargo commands use
CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0,
CARGO_BUILD_JOBS=4 and the worktree target/tmp. The preset target directory
was not changed. Tests use --test-threads=4 unless a focused test says 1.

calternal-imap

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 21s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 35.52s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.35s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

calternal-plugin-mail

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.80s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 16.17s
test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 42.71s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 00s
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out; finished in 15.15s
test result: FAILED. 218 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 106.38s

calternal-auth

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 26s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 43s
test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 133.11s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-db

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.23s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 36.98s
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.44s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.26s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.44s
test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.25s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-notes

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 47s
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 59s
test result: ok. 264 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 509.86s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.69s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Web check

perf-lint: parse Rust product sources
perf-lint: parse browser product sources
perf-lint: validate coverage and architecture
perf-lint: PASS; 0 violations; 19340 scoped exceptions
svelte-check found 0 errors and 4 warnings in 3 files

Focused Vitest

 Test Files  4 passed (4)
      Tests  18 passed (18)

Probe diagnostics

python3 -m unittest discover -s tests/adversarial -p test_mail_proxy.py

......................
----------------------------------------------------------------------
Ran 22 tests in 0.216s

OK

Cleanup

The owned local and perf-VM executables, temporary fixtures and web build
output are removed. Review artifacts remain in the ignored artifacts/
directory. No screenshot or binary is committed. cargo clean exits 0:

     Removed 26944 files, 20.4GiB total
Final mailround2-1038 report. Head: `8091d5788ad86aaee1da831801031110462c6f6d` on `job/mailround-1038`. Worktree clean. No push, deploy or merge beyond the authorized origin/dev integration. # Mail round 2: #1038 and #1067 READY FOR STAGING: no. ## Built Merged remote `origin/dev` at `9fb9a4bfb2488152c83d50c55441ff5f43b572b2` into `job/mailround-1038`. The merge includes #1062, #1065 and Files/open parity. There was no push, deploy or issue closure. Kept the deployed migration prefix: Auth 13, DB 15, Files 24, Mail 11 and Notes 32. Mail 12–17 and Auth 14 follow that prefix. Extended the existing production-copy upgrade test. It now checks both the older production schema and the deployed 7b prefix. Old receipts, Mail rows, Security state and Notes cursors survive. Only seven proxy/Auth additions are pending after 7b. A second startup makes no backup. Repaired cached SELECT writer contention. The pre-7b Mail snapshot used a writer transaction for every SELECT. 7b added a separate Security write connection. An unchanged SELECT still needed the ordinary pool and SQLite's write lock. It could therefore wait behind Index or Security work. The snapshot now checks the complete committed view in one WAL reader transaction. A change releases the reader and uses the single writer. Both passes use the same SQL, ownership checks and limits. Readers never advance durable clocks. The regression holds each writer separately. The old snapshot fails; the repaired snapshot completes before either writer is released. Added fixed-code diagnostics for SQL errors and every SELECT resource refusal. The harness reports no arbitrary wire, SQL, credential or provider text. Extended the existing bench profile with held-writer SELECT samples and a 12-client burst. The sampler also has a 100,000-message case. Repaired the three stale Mail tests against the current contracts: - SSE: require the initial canonical folder refresh, the exact owner event and no foreign event. DESIGN §§53, 59. - Worker pools: pin sync, IDLE, mutation and transfer limits. A stateful barrier keeps all three IDLE waits open while ordinary jobs finish. §53. - FETCH recovery: require all 80 UIDs and exactly one bounded `UID SEARCH UID 1:80`. Keep membership validation. §§45, 59. Formatted the imported Notes reconciliation test so the workspace fmt gate passes. Its behavior and assertions stay unchanged. ## Evidence The product repair is `9c8d1e7630aafae04c43036cba7b80fd64e2b932`. The expanded topology regression is `a6e0ff6a5`. The acceptance executable was copied before the server test build. Its SHA-256 is `d889727108238ec7ee37fa01e5ad1876ff5681d299cedf17fba194b2d6a78eb2`. The later regression commit changes tests and instructions only. The fixed-product single-account phase passes: one User, one Connected Account, 12 clients, 1,808.52 seconds including receipt audits. It reports 18,402 SELECT OK and 18,382 each SEARCH/FETCH/IDLE/EXPUNGE OK. Twenty each APPEND, STORE, COPY and MOVE succeed. All 20 exact upstream and 20 local receipts pass. Required SELECT has no refusal. Expected pending-transfer NO responses are retried only after explicit refusal. ```text {"scenario":"one-account","result":"PASS","web_sessions":"PASS","snapshot_failures":{}} ``` There are no mixed-soak failures or failure reasons. Peak RSS is 280,340 KiB (273.77 MiB), peak FDs 171. After client and receipt sockets close: 145,172 KiB RSS and 127 FDs. Readiness and retained web sessions pass. These are local fixture/debug observations. Evidence is `artifacts/round2-one-account-fixed.jsonl`. The unmodified merged product also completed a diagnostic 1,808.72-second phase before the repair: 15,953 SELECT OK, no failures, and 20 exact upstream and local receipts. This is separate from fixed-product acceptance. The old snapshot failed the held-Security-writer regression in 2.24 seconds. The fixed full Mail suite passes that same test with both writers held in turn. The older 872.41-second refusal did not record an SQL class. Do not assign a specific SQLite error code to that historical run. Startup was measured on the idle host at `10.69.69.63`, with `/root/perf.lock`. Load average was 0.20 / 0.17 / 0.08. The existing test passed in 3.34 seconds; total wall time was 3.40 seconds. No build ran on that host. Two initial attempts lacked debug frontend paths and failed before measuring startup. They are excluded. The first full local server suite passed: 219 passed, no failures. The later fixed-product run, concurrent with the soak on the busy host, hit only the existing startup deadline at 15.15 seconds. This is a SLOW-only result. The 15-second limit stays unchanged. ## Release profile The updated profile completes on the perf VM under `/root/perf.lock`. The Mail test executable was built locally in release mode. The VM compiled no code. Cached SELECT samples and each 12-client burst hold the writer occupied. | Profile | Median per-run SELECT p50 / p95 | Mean worker CPU | Mean / peak worker RSS | Largest 12-client SELECT duration | |---|---|---|---|---| | 10,001 messages, 3 serial runs | 134.51 / 176.76 ms | 147.54% | 111.03 / 153.00 MiB | 1,696.68 ms | | 100,000 messages | 1,953.88 / 5,540.95 ms | 163.04% | 848.05 / 1,205.99 MiB | 18,724.86 ms | | 3 workers, each with 100,000 messages and its own Index | 1,790.76 / 4,051.31 ms | 113.24% | 891.19 / 1,205.86 MiB | 24,210.30 ms | CPU and RSS include seeding, metadata and queue phases. They are not SELECT-only resource figures. CPU can exceed 100% because SQLite uses multiple threads. The load averages inside the lock are 6.125 / 4.5923 / 2.1152 before and 7.5220 / 5.5894 / 3.3701 after. The starting load includes the preceding debug diagnostic run. That incomplete debug run was stopped and is excluded. Do not treat this as a cold quiet-host baseline. `docs/perf/baseline.json` has no comparable cached SELECT profile. `mail.accounts` measures HTTP account listing (p50 1.3 ms / p95 3.8 ms). It cannot establish a SELECT regression. The existing large-snapshot cost remains visible and is filed as [#1070](https://git.kayg.org/kayg/calternal/issues/1070). This profile is not real-provider 100,000-message sync or 50-client acceptance. Evidence is `artifacts/round2-select-perf-vm-release.log`. ## Files Round-2 edits, excluding the incoming merge: - `crates/plugins/mail/src/proxy.rs` and `proxy_tests.rs`. - `crates/plugins/mail/src/routes.rs` and `sync.rs`. - `crates/calternal-server/src/upgrade_tests.rs`. - `crates/plugins/notes/src/lib.rs` (format only). - `tests/adversarial/mail_proxy.py`, `test_mail_proxy.py`, `mail_stress.mjs` and `mail-sync.md`. - `bench/mail-sync.py` and this report. ## UX gaps closed Unchanged cached Mail views can be read while Index or Security writes are active. UID, revision and message identity checks remain intact. This round adds no UI. The earlier job's production screenshots remain its visual evidence; this round makes no new visual claim. ## UX gaps left and known gaps - #1068 remains: phone Settings sheet title and Copy link checks. - The latest busy-host server gate has one startup SLOW-only failure. - #1038 still has no complete 50-client long run on both account groups, 100,000-message initial-sync acceptance, 50 MB/zero-byte/10,000-folder matrix or download storm on the final product. Earlier short diagnostics do not replace these cases. These gaps prevent a full #1038 acceptance claim and READY FOR STAGING remains no. - The older SELECT refusal was unclassified. The held-writer defect is reproduced and repaired; no exact error code is inferred for that run. - Historical comparison attempts were stopped when Cargo reused artifacts from an archived source tree. They are excluded. The controlled regression uses the old snapshot source in the current worktree and is valid. - #1070 tracks large cached SELECT latency and memory. There is no comparable baseline to establish a regression. - Four existing Svelte warnings remain. There are no Svelte errors. ## Decisions Use a checked WAL reader pass for unchanged snapshots. Keep all changes on the single writer. This is an implementation choice consistent with DESIGN §§2, 53, 59. Do not change durability, deadlines or cache bounds. Use the canonical SSE hint, four bounded worker pools and bounded UID search already implemented on dev. Update the stale test contracts while keeping or strengthening what they prove. Keep the startup product and 15-second test limit. The idle measurement does not justify a limit increase. Report the later busy-host failure verbatim. Measure the existing component profile in release mode on the perf VM. Keep its cache timings separate from the local debug protocol phase. File the large-snapshot performance gap as #1070. Do not call it a baseline regression without comparable endpoint, data and build measurements. ## For the merge round Run the combined web suite with `cd apps/web && bun run test` and the combined startup gate with `cargo test -p calternal-server -- --test-threads=4`. The startup test must show that upgrade backfills do not delay HTTP readiness. Recheck #1068. These do not replace the completed requested single-account phase. The original #1038 large-data and 50-client gaps remain listed above. ## Gate output, verbatim excerpts `cargo fmt --check` exits 0 with no output. Cargo commands use `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4` and the worktree `target/tmp`. The preset target directory was not changed. Tests use `--test-threads=4` unless a focused test says 1. ### calternal-imap ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 21s Finished `test` profile [unoptimized + debuginfo] target(s) in 35.52s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.35s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s ``` ### calternal-plugin-mail ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.80s Finished `test` profile [unoptimized + debuginfo] target(s) in 16.17s test result: ok. 107 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 20.36s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-server ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 42.71s Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 00s test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out; finished in 15.15s test result: FAILED. 218 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out; finished in 106.38s ``` ### calternal-auth ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 26s Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 43s test result: ok. 120 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 133.11s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-db ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 35.23s Finished `test` profile [unoptimized + debuginfo] target(s) in 36.98s test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.44s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.26s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.44s test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.25s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### calternal-plugin-notes ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 47s Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 59s test result: ok. 264 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 509.86s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.69s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ### Web check ```text perf-lint: parse Rust product sources perf-lint: parse browser product sources perf-lint: validate coverage and architecture perf-lint: PASS; 0 violations; 19340 scoped exceptions svelte-check found 0 errors and 4 warnings in 3 files ``` ### Focused Vitest ```text Test Files 4 passed (4) Tests 18 passed (18) ``` ### Probe diagnostics `python3 -m unittest discover -s tests/adversarial -p test_mail_proxy.py` ```text ...................... ---------------------------------------------------------------------- Ran 22 tests in 0.216s OK ``` ## Cleanup The owned local and perf-VM executables, temporary fixtures and web build output are removed. Review artifacts remain in the ignored `artifacts/` directory. No screenshot or binary is committed. `cargo clean` exits 0: ```text Removed 26944 files, 20.4GiB total ```
Author
Owner

Deployed to production 2026-10-05 ~04:40 CEST in round 9 (269b1b51b). Includes the mail proxy (CalternalDAV, real Apple Mail acceptance PASS on the Mac VM), provider sync fixes, the stress-round fixes, #1067, #1068, #1078 and the Files upload identity repair. Staging healthy first; production healthy in 18 s; Auth 14 and Mail 17 migrations applied; change events 0/30 s; no expired leases.

Deployed to production 2026-10-05 ~04:40 CEST in round 9 (269b1b51b). Includes the mail proxy (CalternalDAV, real Apple Mail acceptance PASS on the Mac VM), provider sync fixes, the stress-round fixes, #1067, #1068, #1078 and the Files upload identity repair. Staging healthy first; production healthy in 18 s; Auth 14 and Mail 17 migrations applied; change events 0/30 s; no expired leases.
kayg closed this issue 2026-10-05 03:08:50 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#1067
No description provided.