Local server gate: Calendar write p95 exceeds budget during startup Search rebuild #1173

Open
opened 2026-10-06 02:49:06 +00:00 by kayg · 0 comments
Owner

Found during merge27j verification for #1109 at 5e683712155e6f93941f6e8331a5f62d46112d5e on the shared build host.

The unchanged #965 live startup Search reconciliation test uses 20,000 files and 32 concurrent Calendar writes. All writes returned HTTP 201 within the 20-second client deadline. Its latency assertion failed:

search_reconcile_calendar_write_profile files=20000 writes=32 p50_ms=14233 p95_ms=14586 p95_budget_ms=10000
Calendar write p95 14.586218978s exceeded 10s

Command:

CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 cargo test -p calternal-server -- --test-threads=1

Result:

test result: FAILED. 262 passed; 1 failed; 10 ignored; 0 measured; 0 filtered out; finished in 123.98s

The sole failing parent is wire::tests::live_apps_run_in_separate_processes. Its failing child is startup_search_reconciliation_keeps_calendar_writes_responsive_on_large_home. The previous startup-lock child passed in this serial run. Load average immediately after the suite: 7.92, 15.77, 21.27. This is a local debug observation; it does not establish a release performance regression. Do not weaken either latency budget. The owner rule treats SLOW-only findings on this shared host as load; the CLI gate still exits 101 and must be reported that way.

The required full server gate is not green in this merge round. Investigate the shared SQLite writer and reproduce on the locked perf VM before changing behavior or budgets. No secret data or production Home was used.

Found during merge27j verification for #1109 at `5e683712155e6f93941f6e8331a5f62d46112d5e` on the shared build host. The unchanged #965 live startup Search reconciliation test uses 20,000 files and 32 concurrent Calendar writes. All writes returned HTTP 201 within the 20-second client deadline. Its latency assertion failed: ```text search_reconcile_calendar_write_profile files=20000 writes=32 p50_ms=14233 p95_ms=14586 p95_budget_ms=10000 Calendar write p95 14.586218978s exceeded 10s ``` Command: ```sh CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 cargo test -p calternal-server -- --test-threads=1 ``` Result: ```text test result: FAILED. 262 passed; 1 failed; 10 ignored; 0 measured; 0 filtered out; finished in 123.98s ``` The sole failing parent is `wire::tests::live_apps_run_in_separate_processes`. Its failing child is `startup_search_reconciliation_keeps_calendar_writes_responsive_on_large_home`. The previous startup-lock child passed in this serial run. Load average immediately after the suite: 7.92, 15.77, 21.27. This is a local debug observation; it does not establish a release performance regression. Do not weaken either latency budget. The owner rule treats SLOW-only findings on this shared host as load; the CLI gate still exits 101 and must be reported that way. The required full server gate is not green in this merge round. Investigate the shared SQLite writer and reproduce on the locked perf VM before changing behavior or budgets. No secret data or production Home was used.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#1173
No description provided.