PERF: Profile and reduce Calendar Event FTS latency in Search #433

New issue

Open

opened 2026-09-29 12:32:39 +00:00 by kayg · 2 comments

kayg commented

2026-09-29 12:32:39 +00:00

Owner

Parent: #367, Phase 2 split. This issue tracks the Calendar Event FTS latency seen through global Search and hybrid Search.

Evidence

The 2026-09-28 100k mixed profile ran on the shared build host at load about 10–24, so these numbers are diagnostic and are not a quiet-host baseline. In 100 samples per keyword case, p50/p95/p99 latency was:

Plain: 526.03 / 894.62 / 1,552.53 ms
Tag: 450.26 / 713.11 / 775.44 ms
Type filter: 366.63 / 600.83 / 647.18 ms
Folder scope: 407.97 / 639.44 / 739.73 ms

The target at 100k is p50 ≤ 10 ms, p95 ≤ 30 ms, and p99 ≤ 80 ms. A separate 200-query profile logged 161 slow calendar_events_fts statements from 1.027 to 10.009 seconds. The report says the same query path appeared in the 100-sample profile. Warm hybrid Search measured p50/p95/p99 of 820.66 / 2,666.51 / 3,920.00 ms across 25 samples; model cold start was 4.53 seconds.

Work

Use the existing Search scale profile and server traces to measure Calendar Event FTS call count and time for keyword and hybrid requests. Inspect the query plan and any pool or lock waits. Then reduce the measured FTS contribution and record before/after numbers on the quiet perf-test VM. Keep the 2026-09-28 figures labelled as busy-host diagnostics until that remeasurement is complete. Mail Search belongs to #396.

Parent: #367, Phase 2 split. This issue tracks the Calendar Event FTS latency seen through global Search and hybrid Search. ## Evidence The 2026-09-28 100k mixed profile ran on the shared build host at load about 10–24, so these numbers are diagnostic and are not a quiet-host baseline. In 100 samples per keyword case, p50/p95/p99 latency was: - Plain: 526.03 / 894.62 / 1,552.53 ms - Tag: 450.26 / 713.11 / 775.44 ms - Type filter: 366.63 / 600.83 / 647.18 ms - Folder scope: 407.97 / 639.44 / 739.73 ms The target at 100k is p50 ≤ 10 ms, p95 ≤ 30 ms, and p99 ≤ 80 ms. A separate 200-query profile logged 161 slow `calendar_events_fts` statements from 1.027 to 10.009 seconds. The report says the same query path appeared in the 100-sample profile. Warm hybrid Search measured p50/p95/p99 of 820.66 / 2,666.51 / 3,920.00 ms across 25 samples; model cold start was 4.53 seconds. ## Work Use the existing Search scale profile and server traces to measure Calendar Event FTS call count and time for keyword and hybrid requests. Inspect the query plan and any pool or lock waits. Then reduce the measured FTS contribution and record before/after numbers on the quiet perf-test VM. Keep the 2026-09-28 figures labelled as busy-host diagnostics until that remeasurement is complete. Mail Search belongs to #396.

kayg referenced this issue

2026-09-29 12:32:53 +00:00

PERF: Reduce Search palette first-result latency and typing jank #434

kayg referenced this issue

2026-09-29 12:38:30 +00:00

PERF ROUND: benchmark + optimise average and worst-case CPU/memory (after current merges) #367

kayg referenced this issue

2026-09-29 13:09:14 +00:00

PERF ROUND: benchmark + optimise average and worst-case CPU/memory (after current merges) #367

kayg commented

2026-09-29 19:44:24 +00:00

Author

Owner

Round 2 update (2026-09-29): this run collected no quiet-VM Search or hybrid measurements. The current release server rebuild did not finish before the four-hour job limit, so no profile ran under /root/perf.lock. Keep the 2026-09-28 busy-host values above marked as diagnostic; this update adds no new latency numbers. Parent issue #367 records the exact gates and remaining work.

Round 2 update (2026-09-29): this run collected no quiet-VM Search or hybrid measurements. The current release server rebuild did not finish before the four-hour job limit, so no profile ran under `/root/perf.lock`. Keep the 2026-09-28 busy-host values above marked as diagnostic; this update adds no new latency numbers. Parent issue #367 records the exact gates and remaining work.

kayg referenced this issue

2026-09-29 19:44:27 +00:00

PERF ROUND: benchmark + optimise average and worst-case CPU/memory (after current merges) #367

kayg commented

2026-09-30 08:04:51 +00:00

Author

Owner

#367 quiet-VM remeasure (perf-test VM, 4 vCPU; release 369ab6a2f; tests/perf/search_scale.py, 200 queries per case; load < 1 at start under /root/perf.lock).

Keyword Search now meets the budget. The 526 ms p50 from the busy build host does not reproduce:

Corpus	Case	p50	p95	p99
100k mixed (files, Photos, Notes, Events, Tasks)	plain	6.22 ms	18.08 ms	23.50 ms
	tag	4.99	16.77	23.03
	type filter	4.87	13.62	24.44
	folder scope	4.95	13.44	21.85
1M files	plain	16.73	27.44	32.13
	tag	1.28	2.62	4.64

Hybrid/semantic is still a problem. Warm hybrid query p50 / p95 / p99 = 203.1 / 213.2 / 214.7 ms (first 201.4 ms) with 70,000 / 70,000 semantic paths indexed. The single traced request spent keyword_provider 9.9 ms and search_route 200.2 ms and returned 0 hits (128-byte response) for "Where can I find guidance for planning routes on a journey?". So every hybrid query waits for the full 200 ms semantic deadline and the semantic side contributes nothing on this 4-vCPU host. Model cold start 7.42 s; model reload 44.6 s.

Other Search numbers: initial 100k index 22.65 s (peak RSS 478.6 MiB); full reindex 39.8 s with peak RSS 1,133 MiB and queries at p50 203.7 ms during it; 1M initial index 283.6 s with peak RSS 1,895 MiB (#496).

I suggest narrowing this issue to the hybrid deadline/zero-hit behaviour. Raw: docs/perf/runs/2026-09-29T224501Z-369ab6a2/raw/search-{mixed,stress}.json.

#367 quiet-VM remeasure (perf-test VM, 4 vCPU; release `369ab6a2f`; `tests/perf/search_scale.py`, 200 queries per case; load < 1 at start under `/root/perf.lock`). **Keyword Search now meets the budget.** The 526 ms p50 from the busy build host does not reproduce: | Corpus | Case | p50 | p95 | p99 | |---|---|---:|---:|---:| | 100k mixed (files, Photos, Notes, Events, Tasks) | plain | 6.22 ms | 18.08 ms | 23.50 ms | | | tag | 4.99 | 16.77 | 23.03 | | | type filter | 4.87 | 13.62 | 24.44 | | | folder scope | 4.95 | 13.44 | 21.85 | | 1M files | plain | 16.73 | 27.44 | 32.13 | | | tag | 1.28 | 2.62 | 4.64 | **Hybrid/semantic is still a problem.** Warm hybrid query p50 / p95 / p99 = 203.1 / 213.2 / 214.7 ms (first 201.4 ms) with 70,000 / 70,000 semantic paths indexed. The single traced request spent `keyword_provider` 9.9 ms and `search_route` 200.2 ms and returned **0 hits** (128-byte response) for "Where can I find guidance for planning routes on a journey?". So every hybrid query waits for the full 200 ms semantic deadline and the semantic side contributes nothing on this 4-vCPU host. Model cold start 7.42 s; model reload 44.6 s. Other Search numbers: initial 100k index 22.65 s (peak RSS 478.6 MiB); full reindex 39.8 s with peak RSS 1,133 MiB and queries at p50 203.7 ms during it; 1M initial index 283.6 s with peak RSS 1,895 MiB (#496). I suggest narrowing this issue to the hybrid deadline/zero-hit behaviour. Raw: `docs/perf/runs/2026-09-29T224501Z-369ab6a2/raw/search-{mixed,stress}.json`.

kayg referenced this issue

2026-09-30 08:04:52 +00:00

PERF: Reduce Search palette first-result latency and typing jank #434

kayg referenced this issue

2026-09-30 08:46:05 +00:00

PERF ROUND: benchmark + optimise average and worst-case CPU/memory (after current merges) #367

kayg referenced this issue

2026-10-03 12:51:31 +00:00

Issue hygiene sweep 2026-10-03 #1008