PERF: Reminder create burst exceeds 180 s with a 10k-Task Note #602

New issue

Open

opened 2026-10-01 09:05:32 +00:00 by kayg · 0 comments

kayg commented

2026-10-01 09:05:32 +00:00

Owner

Found during #585. Related scale investigation: #573.

On head 960f930ff (source behavior from 9282f570c), the new bench/reminders_create.py profile starts a disposable real local debug server, creates a Daily note Log anchor, measures 17 serial Reminder creates plus conditional retries, adds one Note with 10,000 inline Tasks, then sends 17 simultaneous fresh-UID creates plus conditional retries.

Local debug host, not perf VM; load average began 26.73 / 32.02 / 31.20 and ended 33.78 / 33.64 / 32.28. The bounded perf VM preflight found its lock occupied. docs/perf/baseline.json has no matching Reminder create baseline, so these are diagnostics, not a baseline regression verdict.

Serial: 17 creates and 17 retries passed (201/412); create p50 1,737.78 ms, p95 5,714.18 ms; CPU 22.79 s / 26.39% mean; RSS mean 323,666,217 bytes, peak 393,699,328 bytes.
10,000-Task Note setup: HTTP 201 in 30.719 s.
Burst: one create and its retry returned 201/412; sixteen attempts timed out at the 180 s per-request bound. The one completed create took 172,502.28 ms. This is one observation, not a representative burst p95. Whole create-plus-retry attempt p95 was 296,047.52 ms; wall time 296.056 s; server CPU 181.17 s / 61.19% mean; RSS mean 660,167,645 bytes, peak 794,935,296 bytes.
At the bounded readback, the Index had 10,019 Tasks: the corpus, 17 serial creates and two admitted burst creates. Timed-out requests cannot be counted as seventeen successful creates.
Server log pool timeout count: 0. A first 60-second-bound run also exceeded its burst bound with zero pool timeouts.

#585 removes quadratic source hashing and completion lookup from the writer transaction. Remaining code evidence: tasks_store::reconcile_user unconditionally reindexes every Note on reconciliation, and index_source still executes per-Task SQL while holding the sole writer. The Notes event listener invokes whole-User reconciliation on file events. Profile those paths and the per-User DAV admission queue before choosing batching or an unchanged-source skip. Reproduce under the perf VM lock with the release build.

These local observations are SLOW-only. No 5xx or SQLite pool timeout was recorded in the profile. They are separate from #585's verified cancellation duplicate regression and the unresolved R8 distinct-UID native duplication.

Found during #585. Related scale investigation: #573. On head 960f930ff (source behavior from 9282f570c), the new `bench/reminders_create.py` profile starts a disposable real local debug server, creates a Daily note Log anchor, measures 17 serial Reminder creates plus conditional retries, adds one Note with 10,000 inline Tasks, then sends 17 simultaneous fresh-UID creates plus conditional retries. Local debug host, not perf VM; load average began 26.73 / 32.02 / 31.20 and ended 33.78 / 33.64 / 32.28. The bounded perf VM preflight found its lock occupied. `docs/perf/baseline.json` has no matching Reminder create baseline, so these are diagnostics, not a baseline regression verdict. - Serial: 17 creates and 17 retries passed (201/412); create p50 1,737.78 ms, p95 5,714.18 ms; CPU 22.79 s / 26.39% mean; RSS mean 323,666,217 bytes, peak 393,699,328 bytes. - 10,000-Task Note setup: HTTP 201 in 30.719 s. - Burst: one create and its retry returned 201/412; sixteen attempts timed out at the 180 s per-request bound. The one completed create took 172,502.28 ms. This is one observation, not a representative burst p95. Whole create-plus-retry attempt p95 was 296,047.52 ms; wall time 296.056 s; server CPU 181.17 s / 61.19% mean; RSS mean 660,167,645 bytes, peak 794,935,296 bytes. - At the bounded readback, the Index had 10,019 Tasks: the corpus, 17 serial creates and two admitted burst creates. Timed-out requests cannot be counted as seventeen successful creates. - Server log pool timeout count: 0. A first 60-second-bound run also exceeded its burst bound with zero pool timeouts. #585 removes quadratic source hashing and completion lookup from the writer transaction. Remaining code evidence: `tasks_store::reconcile_user` unconditionally reindexes every Note on reconciliation, and `index_source` still executes per-Task SQL while holding the sole writer. The Notes event listener invokes whole-User reconciliation on file events. Profile those paths and the per-User DAV admission queue before choosing batching or an unchanged-source skip. Reproduce under the perf VM lock with the release build. These local observations are SLOW-only. No 5xx or SQLite pool timeout was recorded in the profile. They are separate from #585's verified cancellation duplicate regression and the unresolved R8 distinct-UID native duplication.

kayg referenced this issue

2026-10-01 09:10:09 +00:00

SYNC: Reminders.app creates are duplicated (17 Reminders → 34 Tasks); SQLite pool timeouts during the burst #585

kayg referenced this issue

2026-10-01 09:34:00 +00:00

SYNC: Reminders.app creates are duplicated (17 Reminders → 34 Tasks); SQLite pool timeouts during the burst #585

kayg referenced this issue

2026-10-01 10:25:08 +00:00

SYNC: Reminders.app creates are duplicated (17 Reminders → 34 Tasks); SQLite pool timeouts during the burst #585

kayg referenced this issue

2026-10-01 11:16:52 +00:00

SYNC: Reminders.app creates are duplicated (17 Reminders → 34 Tasks); SQLite pool timeouts during the burst #585