kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 803 Pull requests Projects Releases Packages Wiki Activity Actions 472

PERF: Search indexing peaks at 1.9 GB RSS for 1M files and 1.1 GB during a 100k rebuild (caps cover only the writer) #496

New issue
Open
opened 2026-09-30 08:04:40 +00:00 by kayg · 6 comments
kayg commented 2026-09-30 08:04:40 +00:00
Owner
Copy link

Problem

Search indexing has a memory ceiling well above its configured caps. On a 7.7 GiB host, a 1M-file first index peaks at 1.9 GiB RSS and a 100k full rebuild at 1.1 GiB, while the configured Tantivy writer heap is 61 MiB and the pending batch cap is 32 MiB / 1,024 items.

Found by the #367 perf round (perf-test VM, 4 vCPU; release 369ab6a2f; tests/perf/search_scale.py; load < 1 at start under /root/perf.lock).

Measurements

Workload Duration Peak RSS Mean RSS Peak CPU
1M-file first index (search-stress) 283.6 s 1,987,190,784 B (1.85 GiB) — 223.7%
100k mixed first index 22.65 s 501,870,592 B — 387.5%
100k full rebuild while querying 39.8 s 1,188,155,392 B (1.11 GiB) 1,110,614,400 B 399.5%
100k semantic embedding backfill — 916,357,120 B 881,016,111 B 346.1%
Idle server, 10 Users 600 s 162.4 MB 154.7 MiB 4.0%

Thread CPU during the 1M first index: calternal-searc 56.9 s, merge_thread_0 26.3 s, notify-rs inoti 20.4 s, sqlite 9.9 s, tokio 10.2 s.

Queries stay available during the rebuild (0 failures), but at p50 / p95 / p99 = 203.7 / 265.4 / 385.0 ms versus 6.2 / 18.1 / 23.5 ms at rest.

Why (to confirm)

The caps bound the Tantivy writer and the pending update batch only. The remaining peak is outside them: candidates are the startup/rebuild path collecting the whole manifest or path list in memory, merge threads holding several segments, and the notify watcher state for 1M paths. A heap profile (heaptrack or dhat) of search_scale.py --profile stress --items 200000 would attribute it.

Suggested direction (not decided)

Stream the rebuild/manifest diff in bounded pages and make every remaining buffer a named config value with a documented default (the #367 rule). Related: #338 (1M Home 503s).

Raw: docs/perf/runs/2026-09-29T224501Z-369ab6a2/raw/search-{mixed,stress}.json.

## Problem Search indexing has a memory ceiling well above its configured caps. On a 7.7 GiB host, a 1M-file first index peaks at 1.9 GiB RSS and a 100k full rebuild at 1.1 GiB, while the configured Tantivy writer heap is 61 MiB and the pending batch cap is 32 MiB / 1,024 items. Found by the #367 perf round (perf-test VM, 4 vCPU; release `369ab6a2f`; `tests/perf/search_scale.py`; load < 1 at start under `/root/perf.lock`). ## Measurements | Workload | Duration | Peak RSS | Mean RSS | Peak CPU | |---|---:|---:|---:|---:| | 1M-file first index (`search-stress`) | 283.6 s | **1,987,190,784 B (1.85 GiB)** | — | 223.7% | | 100k mixed first index | 22.65 s | 501,870,592 B | — | 387.5% | | 100k full rebuild while querying | 39.8 s | **1,188,155,392 B (1.11 GiB)** | 1,110,614,400 B | 399.5% | | 100k semantic embedding backfill | — | 916,357,120 B | 881,016,111 B | 346.1% | | Idle server, 10 Users | 600 s | 162.4 MB | 154.7 MiB | 4.0% | Thread CPU during the 1M first index: `calternal-searc` 56.9 s, `merge_thread_0` 26.3 s, `notify-rs inoti` 20.4 s, sqlite 9.9 s, tokio 10.2 s. Queries stay available during the rebuild (0 failures), but at p50 / p95 / p99 = 203.7 / 265.4 / 385.0 ms versus 6.2 / 18.1 / 23.5 ms at rest. ## Why (to confirm) The caps bound the Tantivy writer and the pending update batch only. The remaining peak is outside them: candidates are the startup/rebuild path collecting the whole manifest or path list in memory, merge threads holding several segments, and the `notify` watcher state for 1M paths. A heap profile (heaptrack or dhat) of `search_scale.py --profile stress --items 200000` would attribute it. ## Suggested direction (not decided) Stream the rebuild/manifest diff in bounded pages and make every remaining buffer a named config value with a documented default (the #367 rule). Related: #338 (1M Home 503s). Raw: `docs/perf/runs/2026-09-29T224501Z-369ab6a2/raw/search-{mixed,stress}.json`.
kayg commented 2026-09-30 08:44:36 +00:00
Author
Owner
Copy link

heaptrack attribution (30% large Home, 300k files; details in #503): the Search rebuild path holds whole-Home maps at the same time. BTreeMap<String, ManifestEntry> 127.4 MB (flush_upserts / scan_tree / reconcile_and_report), String::clone 175.4 MB (file_result_ids, scan_tree, manifest_entry), hashbrown tables 126.5 MB (file_result_ids, SearchIndex::documents_by_path) and Vec growth 50.9 MB in documents_by_path (rebuild_all). That is about 0.48 GB at 300k files and grows linearly, which accounts for most of the 1.9 GB at 1M files. None of these structures is covered by the writer-heap or pending-batch caps.

heaptrack attribution (30% large Home, 300k files; details in #503): the Search rebuild path holds whole-Home maps at the same time. `BTreeMap<String, ManifestEntry>` 127.4 MB (`flush_upserts` / `scan_tree` / `reconcile_and_report`), `String::clone` 175.4 MB (`file_result_ids`, `scan_tree`, `manifest_entry`), `hashbrown` tables 126.5 MB (`file_result_ids`, `SearchIndex::documents_by_path`) and `Vec` growth 50.9 MB in `documents_by_path` (`rebuild_all`). That is about 0.48 GB at 300k files and grows linearly, which accounts for most of the 1.9 GB at 1M files. None of these structures is covered by the writer-heap or pending-batch caps.
kayg commented 2026-09-30 21:17:52 +00:00
Author
Owner
Copy link

Starting #496 on branch job/perf-496, based on 6c87f5ff9442cd658572139bc536d018fd5222a4 (merge-base with origin/dev). I am tracing the full indexing memory path, then will add bounded scan/rebuild memory with a regression test and compare the issue workload.

Starting #496 on branch `job/perf-496`, based on `6c87f5ff9442cd658572139bc536d018fd5222a4` (merge-base with `origin/dev`). I am tracing the full indexing memory path, then will add bounded scan/rebuild memory with a regression test and compare the issue workload.
kayg commented 2026-09-30 21:25:03 +00:00
Author
Owner
Copy link

I traced the reported heap profile to the current implementation: load_manifest materializes all search_manifest rows in a BTreeMap; documents_by_path builds a second whole-Index map; and reconcile_and_report adds whole-Home expected and seen maps. Root::list_page already provides a directory-handle-relative bounded page API. I am using it with disk-backed reconciliation rows and bounded batches so path count no longer sets the actor's working-set size.

I traced the reported heap profile to the current implementation: `load_manifest` materializes all `search_manifest` rows in a `BTreeMap`; `documents_by_path` builds a second whole-Index map; and `reconcile_and_report` adds whole-Home `expected` and `seen` maps. `Root::list_page` already provides a directory-handle-relative bounded page API. I am using it with disk-backed reconciliation rows and bounded batches so path count no longer sets the actor's working-set size.
kayg commented 2026-09-30 23:38:44 +00:00
Author
Owner
Copy link

I am handling related issue #503. I found the committed Search memory fix (0072a577c, fa52fa1d7) on job/perf-496; it is not yet present in origin/dev. To avoid duplicate Search implementation, I will integrate those existing commits into #503 after aligning with current origin/dev, then add the separate ONNX and SQLite/buffer work. Please flag if you are already coordinating a different integration.

I am handling related issue #503. I found the committed Search memory fix (`0072a577c`, `fa52fa1d7`) on `job/perf-496`; it is not yet present in `origin/dev`. To avoid duplicate Search implementation, I will integrate those existing commits into #503 after aligning with current `origin/dev`, then add the separate ONNX and SQLite/buffer work. Please flag if you are already coordinating a different integration.
kayg commented 2026-10-01 01:25:53 +00:00
Author
Owner
Copy link

The first 1M-file profile completed and passed integrity and keyword-hit verification. Local results: 1,000,000 files / 1,001,003 filesystem paths; 6,629.89s initial index; peak RSS 303,697,920 B (289.5 MiB); peak CPU 151.63%; worker budget 268,435,456 B, writer cap 67,108,864 B, batch cap 33,554,432 B. The issue baseline was 1,987,190,784 B (1.85 GiB), so measured peak RSS fell about 85%. Host load was 26.24/32.01/28.84 at start and 12.80/15.96/19.36 at end; the perf VM lock was held, so this is a noisy local timing and not a controlled latency comparison.

The 6,629.89s runtime is unexpectedly high. The new scan currently performs one SQLite manifest lookup per visited path. I am replacing those lookups with bounded per-directory-page queries, then I will rerun the 1M profile to check that the memory reduction does not retain a per-item database round trip.

The first 1M-file profile completed and passed integrity and keyword-hit verification. Local results: 1,000,000 files / 1,001,003 filesystem paths; 6,629.89s initial index; peak RSS 303,697,920 B (289.5 MiB); peak CPU 151.63%; worker budget 268,435,456 B, writer cap 67,108,864 B, batch cap 33,554,432 B. The issue baseline was 1,987,190,784 B (1.85 GiB), so measured peak RSS fell about 85%. Host load was 26.24/32.01/28.84 at start and 12.80/15.96/19.36 at end; the perf VM lock was held, so this is a noisy local timing and not a controlled latency comparison. The 6,629.89s runtime is unexpectedly high. The new scan currently performs one SQLite manifest lookup per visited path. I am replacing those lookups with bounded per-directory-page queries, then I will rerun the 1M profile to check that the memory reduction does not retain a per-item database round trip.
kayg commented 2026-10-01 02:35:45 +00:00
Author
Owner
Copy link

perf-496 report

Head: 50a4bbf049030e84e8397fa3864e606bdb8e5772 (job/perf-496). No push or merge was made.

Built

  • Replaced whole-Home manifest and reconcile maps with SQLite-backed staging and audit tables, and bounded scan, event, and Tantivy update buffers.
  • Batched manifest reads by directory page. Each SQL query binds at most 900 paths to remain below SQLite's older 999-parameter limit.
  • Added tests for budget scaling, disk-backed audit flushing, bounded directory pagination, and page lookup chunking across the SQL parameter boundary.
  • Added search profile reporting for worker, writer, page, batch, and watcher caps.

Files: crates/calternal-search/migrations/0004_bounded_reconcile.sql, crates/calternal-search/src/index.rs, crates/calternal-search/src/indexer.rs, crates/calternal-search/src/lib.rs, crates/calternal-search/tests/indexer.rs, bench/record.py, bench/test_record.py, tests/perf/search_scale.py.

Validation

cargo fmt --all --check: exit 0, no output.

cargo clippy -p calternal-search --all-targets -- -D warnings output:

    Checking calternal-search v0.0.1 (/home/kayg/Developer/calternal-wt/perf-496/crates/calternal-search)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.57s

cargo test -p calternal-search result lines:

test result: ok. 37 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 5.61s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.67s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.67s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 2.69s
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

python3 -m unittest bench.test_record: 6 tests passed. cargo build --release -p calternal-server completed. cargo clean output: Removed 14420 files, 6.2GiB total. Web build output was deleted; no web source changed.

Performance

Issue baseline, perf VM: 1M-file first index 283.6s, peak RSS 1,987,190,784 B (1.85 GiB), peak CPU 223.7%.

Completed local 1M profile before the final page-batching optimization: 1,000,000 files / 1,001,003 paths; 6,629.89s; peak RSS 303,697,920 B (289.5 MiB), peak CPU 151.63%. The measured peak was about 85% lower than baseline. Integrity coverage and the live keyword probe passed. Worker budget was 268,435,456 B; writer cap 67,108,864 B; pending batch cap 33,554,432 B; page cap 256; watcher queue 8,192. Host load average was 26.24/32.01/28.84 at start and 12.80/15.96/19.36 at end. The perf VM lock was held, so this noisy local timing is not comparable to the isolated baseline.

I then added per-page manifest reads to remove the per-file SQL round trip. The repeated 1M run reached 430,876/1,001,003 paths when the four-hour job time box required stopping it. It has no completed peak or timing result. Its start/end host loads were 18.73/17.31/16.71 and 37.91/41.06/35.25. The partial corpus was removed.

Gaps and decisions

  • The final page-batched revision still needs a complete 1M profile and a 100k mixed rebuild profile on the perf VM under /root/perf.lock. The one time-boxed adversarial probe also remains to run.
  • I chose a 256 MiB default worker budget and kept storage-dependent reconciliation state in SQLite. The design did not specify the exact budget or query page size; I capped SQL batches at 900 parameters for compatibility with SQLite builds using the older limit.
# perf-496 report Head: `50a4bbf049030e84e8397fa3864e606bdb8e5772` (`job/perf-496`). No push or merge was made. ## Built - Replaced whole-Home manifest and reconcile maps with SQLite-backed staging and audit tables, and bounded scan, event, and Tantivy update buffers. - Batched manifest reads by directory page. Each SQL query binds at most 900 paths to remain below SQLite's older 999-parameter limit. - Added tests for budget scaling, disk-backed audit flushing, bounded directory pagination, and page lookup chunking across the SQL parameter boundary. - Added search profile reporting for worker, writer, page, batch, and watcher caps. Files: `crates/calternal-search/migrations/0004_bounded_reconcile.sql`, `crates/calternal-search/src/index.rs`, `crates/calternal-search/src/indexer.rs`, `crates/calternal-search/src/lib.rs`, `crates/calternal-search/tests/indexer.rs`, `bench/record.py`, `bench/test_record.py`, `tests/perf/search_scale.py`. ## Validation `cargo fmt --all --check`: exit 0, no output. `cargo clippy -p calternal-search --all-targets -- -D warnings` output: ```text Checking calternal-search v0.0.1 (/home/kayg/Developer/calternal-wt/perf-496/crates/calternal-search) Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.57s ``` `cargo test -p calternal-search` result lines: ```text test result: ok. 37 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 5.61s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.67s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.67s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 2.69s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `python3 -m unittest bench.test_record`: 6 tests passed. `cargo build --release -p calternal-server` completed. `cargo clean` output: `Removed 14420 files, 6.2GiB total`. Web build output was deleted; no web source changed. ## Performance Issue baseline, perf VM: 1M-file first index 283.6s, peak RSS 1,987,190,784 B (1.85 GiB), peak CPU 223.7%. Completed local 1M profile before the final page-batching optimization: 1,000,000 files / 1,001,003 paths; 6,629.89s; peak RSS 303,697,920 B (289.5 MiB), peak CPU 151.63%. The measured peak was about 85% lower than baseline. Integrity coverage and the live keyword probe passed. Worker budget was 268,435,456 B; writer cap 67,108,864 B; pending batch cap 33,554,432 B; page cap 256; watcher queue 8,192. Host load average was 26.24/32.01/28.84 at start and 12.80/15.96/19.36 at end. The perf VM lock was held, so this noisy local timing is not comparable to the isolated baseline. I then added per-page manifest reads to remove the per-file SQL round trip. The repeated 1M run reached 430,876/1,001,003 paths when the four-hour job time box required stopping it. It has no completed peak or timing result. Its start/end host loads were 18.73/17.31/16.71 and 37.91/41.06/35.25. The partial corpus was removed. ## Gaps and decisions - The final page-batched revision still needs a complete 1M profile and a 100k mixed rebuild profile on the perf VM under `/root/perf.lock`. The one time-boxed adversarial probe also remains to run. - I chose a 256 MiB default worker budget and kept storage-dependent reconciliation state in SQLite. The design did not specify the exact budget or query page size; I capped SQL batches at 900 parameters for compatibility with SQLite builds using the older limit.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
wip/rev2-money-ident
wip/restyle-notes
wip/previewcard-1098
wip/palette2-1123
wip/palette-1093
wip/onboard2-1141
wip/onboard-1141.aborted-early
wip/onboard-1141
wip/nlpchip-1127
wip/morph-1104
wip/merge-round-7c5
wip/merge-round-7c4
job/notifloop-1194
wip/merge-round-7c3
wip/merge-round-7c2
wip/merge-round-7c
wip/mchrome-1084
wip/mailghost2-1094
wip/mailghost-1094
wip/kbpreview2-1118
wip/kbpreview-1118
wip/kanban-1092
wip/importhang-1121
wip/hiderev-1153
wip/hide4-1153
wip/hide3-1153
wip/hide2-1153
wip/hide-1153
wip/editreg-1132
wip/editorrail3-1113
wip/editorrail2-1113
wip/editorrail-1113
wip/e2e-b2-1071
wip/e2e-b-1071
wip/draw4-1101
wip/draw3-1101
wip/draw2-1101
wip/draw-1101
wip/directory-1199-r
wip/directory-1199
wip/delete-1119
wip/collabrev-1197
wip/collabloss-1197
wip/cards2-1083
wip/cards-1083
wip/canvas-visual
wip/canvasvis2-976
wip/calhdr-1112
wip/calcards-1115
wip/browserfix
wip/blocks-1125
wip/allday-1107
wip/agenda-decks
wip/agenda-1086
wip/adv7c-1105
wip/txentry-1198
wip/trayicons2-1095
wip/trayicons-1095
job/onboard-1141
wip/sidebar3-1094
wip/rev2-webperf
job/collabloss-1197
job/hide-1153
job/perf-1124
job/perf2-1124
job/tocrail-1191
job/restyle-settings
wip/restyle-settings
job/segmented-1200
wip/notifloop-1194
job/tagperf-1186
wip/tagperf-1186
wip/segmented-1200
job/restyle-files
job/tagdnd-1187
job/merge30
job/cards-1179
wip/cards2-1179
wip/cards-1179
wip/tocrail-1191
wip/tagdnd-1187
wip/restyle-files
wip/perf-1124
wip/merge30j
job/restyle-notes
job/wizchoices-1140
job/adv-1202
wip/wizchoices-1140
wip/restyle-1190
job/moneyfmt-1180
job/txentry-1198
wip/moneyfmt2-1180
wip/moneyfmt-1180-r
wip/moneyfmt-1180
job/pillglass-1189
job/flags-1181
wip/flags-1181
job/restyle-1190
job/restyle-mailmoney
job/restyle-search
job/settingsreg-1195
job/wizard-1140
site/website
wip/wizardrev2-1140
wip/wizardrev-1140
wip/wizard5-1140
wip/wizard4-1140
wip/wizard3-1140
wip/wizard2-1140
wip/wizard-1140
wip/pillglass-1189
wip/settingsreg-1195
job/merge29
job/fu-1171
wip/merge29j
wip/fu-1171
job/fu-1166
job/directory-1199
job/proflog-1204
job/txresearch-1188
wip/fu-1166
job/merge28
job/search-1066
wip/search-1066
wip/merge28j
job/gateslot-1182
job/bulkimport-1157
job/mailnet-1160
wip/mailnetrev-1160
wip/mailnet-1160
wip/bulkrev-1157
wip/bulkimport-1157
job/startup-1161
wip/startup-1161
job/merge27
job/linkcards-1151
wip/linkcards3-1151
wip/linkcards2-1151
wip/linkcards-1151
job/traydate-1144
wip/traydate3-1144
wip/traydate2-1144
wip/traydate-1144
job/draw-1101
wip/merge27j
job/blockpill-1152
wip/blockpill3-1152
wip/blockpill2-1152
wip/blockpill-1152
job/minihover-1149
wip/minihover2-1149
wip/minihover-1149
job/merge25
wip/merge25-r
wip/merge25b
wip/merge25
job/inspector-1129
job/tags-1110
wip/inspector3-1129
wip/inspector2-1129
wip/inspector-1129
wip/tagsrev-1110
wip/tags2-1110
wip/tags-1110
job/dates-1148
wip/datesrev-1148
wip/dates2-1148
wip/dates-1148
job/licence-1145
wip/licence2-1145
wip/licence-1145
job/selfhost-1156
job/merge23
wip/merge23
job/tagfilter-1109
wip/tagfilter2-1109
wip/tagfilter-1109
job/kbd-1134
wip/kbd2-1134
wip/kbd-1134
job/palfoot-1137
wip/selfhost-1156
wip/palfoot2-1137
wip/palfoot-1137
job/toggle-1158
wip/toggle-1158
job/kbpreview-1118
job/docratchet-1155
job/perflint-1133
job/devtests-1159
wip/docratchet-1155
wip/devtests-1159
job/segv-1136
wip/toast-1142
wip/segv-1136
job/toast-1142
job/blockreload-1147
wip/blockreload-1147
job/font-1150
wip/font-1150
job/importui-1120
job/minimonth-1149
wip/importui-1120
wip/minimonth-1149
job/depcheck-1146
wip/perflint-1133
wip/depcheck-1146
job/calcards-1115
job/blocks-1125
job/plus-1128
job/shift-1138
wip/plus2-1128
wip/plus-1128
wip/shift-1138
job/moneyfid-1130
job/editorrail-1113
wip/moneyrev-1130
wip/moneyfid-1130
job/noext-851
wip/noext-851
wip/noext3-851
wip/noext2-851
job/week-1135
wip/week-1135
job/editreg-1132
job/smoke-1122
wip/smoke-1122
job/docs-1143
job/palette2-1123
job/calhdr-1112
job/nlpchip-1127
job/mailghost-1094
job/reconnect-1131
wip/reconnect-1131
job/trayicons-1095
job/delete-1119
job/importhang-1121
job/cards-1083
job/palette-1093
job/mchrome-1084
job/e2e-a-1071
job/canvas-visual
job/previewcard-1098
job/allday-1107
wip/e2e-a2-1071
wip/e2e-a-1071
job/e2e-b-1071
job/adv7c-1105
job/kanban-1092
job/agenda-1086
job/merge-round-7c
job/morph-1104
wip/surfaces-p2
job/merge-round-9
wip/merge-round-9
job/7cfix-small
wip/7cfix-small
job/mailui-1078
job/merge-round-8
wip/merge-round-8
wip/mailui-1078
job/mailround-1038
job/applemail-accept
wip/settitle-1068
wip/mailround2-1038
wip/mailround-1038
wip/e2e-7b
job/crash-1069
wip/crash-1069
job/searchlost-1066
wip/searchlost-1066
job/7b-reconcile
job/flake-1065
wip/flake-1065
wip/merge-round-7b7
wip/merge-round-7b6
wip/merge-round-7b5
wip/merge-round-7b4
wip/7b-reconcile
job/appupdate-1059
job/nfd-1044
wip/appupdate-1059
job/e2e-7b
job/loop-1062
wip/loop-1062
job/pdfprev-1045
job/invtoggle-1053
wip/pdfprev-1045
wip/nfd-1044
wip/invtoggle-1053
job/7bfix-e2e
job/mailstress-b
wip/7bfix-e2e
wip/mailstress-b
job/7bfix-adv
wip/7bfix-adv
job/mailstress-a
job/stack-1054
wip/stack-1054
wip/mailstress-a
job/mailstress-1038
wip/mailstress-1038
job/upload500-1051
wip/upload500-1051
job/share-1034
wip/share-1034
job/syncerr-1037
job/7bfix-photos
wip/7bfix-photos
job/paste-1036
job/setside-1039
wip/setside-1039
wip/paste-1036
job/lease-1042
wip/syncerr-1037
wip/lease-1042
job/7bfix-data
job/passkeybind-1043
wip/apprevoke-1041
job/invite-1035
wip/invite-1035
job/merge-round-7b2
wip/merge-round-7b2
job/mailproxy-486
job/apprevoke-1041
job/rebuild-1033
job/pillborder-1029
wip/pillborder-1029
wip/mailproxy-486
wip/applemail-486
job/headless-998
wip/headless-998
job/groups-1028
wip/groups-1028
job/rebuildwarn-1016
wip/rebuildwarn-1016
job/startup-1011
wip/startup-1011
job/monthpill-1009
job/bgthumb-1025
job/sharetitle-1012
wip/monthpill-1009
wip/bgthumb-1025
wip/sharetitle-1012
job/canvas-cards-977
wip/canvas-cards-977
job/canvas-pencil-978
job/canvas-sketch-990
wip/canvas-sketch-990
wip/canvas-pencil-978
job/canvas-files-989
wip/canvas-files-989
job/canvas-collab-991
wip/canvas-collab-991
job/weekscroll-1018
wip/weekscroll-1018
wip/canvas-core-976
job/canvas-core-976
job/round-drag
wip/round-drag
job/round-settings
job/browserfix
wip/oapi-974
job/oapi-974
job/hist2-integrate
job/mailhtml-726
wip/mailhtml-726
wip/hist2-integrate
job/moneyfu-984
job/drag-1015
wip/drag-1015
job/rename-1017
wip/rename-1017
job/hist2-api
wip/hist2-api
job/oneacct-1014
wip/oneacct-1014
wip/moneyfu-984
job/hist2-bench
job/hist2-restore
wip/hist2-bench
job/hist2-write
job/hotfix-724
wip/hotfix-724
wip/hist2-write
wip/hist2-restore
job/hist2-store
job/hist2-ui
wip/hist2-ui
wip/hist2-store
job/searchstarve-965
job/shutdown-963
wip/shutdown-963
wip/pubedit-981
job/pubedit-981
job/analytics-973
wip/searchstarve-965
job/authflash-850
job/weeklane-969
job/pvtitle-1004
job/hist-975
wip/authflash-850
job/voicepill-617
wip/pvtitle-1004
job/headring-1003
wip/weeklane-969
wip/voicepill-617
wip/headring-1003
wip/analytics-973
job/agentscope-980
wip/thumbsandbox-988
job/thumbsandbox-988
wip/hist-975
job/links-856
wip/links-856
job/davetag-966
wip/davetag-966
job/filesstorm-1000
job/hoverpad-725
wip/filesstorm-1000
job/ffmpegblas-993
job/merge-round-7a
wip/hoverpad-725
wip/ffmpegblas-993
job/nowdot-1002
wip/verify-7a
job/noteid-857
wip/nowdot-1002
wip/noteid-857
wip/merge-round-7a
wip/agentscope-980
job/imapedge
job/a11yfix2
wip/imapedge-941
wip/imapedge
wip/a11yfix2
job/notetask-986
job/logheading
wip/logheading-998
job/textthumb-652
job/photolive-987
wip/photolive-987
job/davactive-983
job/savefix-985
job/tabicons-607
wip/davactive-983
wip/tabicons-607
wip/notetask-986
wip/savefix-985
job/dirid-627
job/buildspeed-1007
wip/dirid-627
job/agenda-decks
job/perfguards-impl
job/undo-a11y
wip/undo-a11y
job/mailperf
job/wal-824
wip/settings-50
job/settings-50
job/notesfilter-606
wip/notesfilter-606
job/surfaces-p2
wip/wal-824
job/maillayouts
wip/mailperf
wip/maillayouts
job/taskmeta-659
job/money-ident
wip/money-ident
wip/taskmeta-659
job/errstates
wip/perfguards-impl
job/headings-881
wip/headings-881
wip/errstates
job/voice-619
job/gaps-827
job/notesperf
wip/notesperf
wip/voice-619
job/hddsql-549
job/perf-stream-668
wip/perf-stream-668
wip/deeplinks-fix
job/deeplinks-fix
job/authfix
job/docsfix-rust
wip/docsfix-rust
job/webperf
job/docsfix-web
job/datafix2
job/webdav-lock-476
job/copyfix
wip/copyfix
wip/webperf
job/focus-658
wip/protofix
job/mediafix
job/protofix
wip/mediafix
job/agentfix
job/hhmm-724
wip/agentfix
job/undo-722
job/reuse
wip/webdav-lock-476
wip/reuse
job/scopefix
job/datafix
wip/hhmm-724
wip/undo-722
job/surfaces-p1
wip/hddsql-549
job/voicememos-618
wip/datafix2
wip/surfaces-p1
job/fix-940
wip/fix-940
job/blaze-surfaces
wip/datafix
wip/blaze-surfaces
job/taskday-655
job/linknav-639
wip/linknav-639
wip/gaps-827
job/isolation-707
job/audiophotos-720
wip/audiophotos-720
job/advfind-664
wip/voicememos-618
wip/taskday-655
wip/isolation-707
wip/advfind-664
wip/scopefix
wip/focus-658
job/testgaps
wip/testgaps
job/overscroll-718
wip/authfix
job/deps
wip/overscroll-718
job/rev2-agentfix
job/rev2-money-ident
job/rev2-mailperf
wip/deps
job/hardening-728
wip/hardening-728
job/searchgen-832
wip/searchgen-832
job/photopw-849
job/mailsql-825
wip/photopw-849
job/sharefix
wip/sharefix
job/rev2-mailhtml-726
job/rev2-perfguards
job/copyval-723
job/lightglass-r2
wip/lightglass-r2
wip/docsfix-web
job/copy-audit
job/macinterop-staging-r2
job/design-sync
job/rev2-taskmeta-659
job/rev2-webperf
job/docs-audit
job/rev2-advfind-664
job/rev2-mailproxy-486
job/states-audit
job/rev2-datafix
job/design-drift
job/test-gaps
job/rev2-voicememos-618
job/rev2-mediafix
job/rev2-deps
job/rev2-datafix2
job/licence-audit
job/issue-hygiene
job/rev2-protofix
job/rev2-voice-619
job/rev2-isolation-707
job/rev2-surfaces-p1
job/deeplink-audit2
job/rev2-audiophotos-720
wip/test-gaps
job/rev2-overscroll-718
job/rev2-undo-722
wip/states-audit
job/rev2-dropmd-719
job/rev2-linknav-639
job/merge-7b-plan
wip/merge-7b-plan
job/rev2-taskday-655
wip/mailsql-825
job/rev2-webdav-lock-476
job/rev2-browserfix
wip/design-drift
job/rev2-hddsql-549
wip/deeplink-audit2
job/rev2-scopefix
job/rev2-authfix
job/rev2-hardening-728
job/rev2-wal-824
job/rev2-sharefix
job/calsidebar-638
job/chrome-audit
job/ioperf
wip/ioperf
wip/chrome-audit
wip/calsidebar-638
job/dropmd-719
wip/dropmd-719
job/ocr-build
wip/ocr-build
job/blaze-settings
wip/copyval-723
job/toastring-721
wip/toastring-721
job/deployfix-732
wip/deployfix-732
wip/blaze-settings
job/money-import-recheck
job/rev-a11y
job/perf-arch-db
job/rev-7b-data
wip/textthumb-652
wip/perf-arch-db
job/sec-protocols
job/sidehdr-660
job/rev-7b-security
job/research-surfaces
job/rev-design-gaps
job/rev-mcp-api
wip/sidehdr-660
job/perf-arch-memory
wip/sec-protocols
job/perf-arch-bundle
job/snapedge-714
wip/rev-mcp-api
job/sec-supplychain
wip/research-surfaces
job/perf-arch-sync
job/rev-consistency
job/perf-arch-server
wip/perf-arch-server
wip/perf-arch-memory
job/perf-arch-io
job/perf-arch-client
job/sec-fs
job/sec-mcp-scopes
job/sec-sharing
job/perf-guards
job/sec-browser
job/sec-admin-deploy
job/sec-auth
wip/snapedge-714
job/bgpicker-717
wip/perf-arch-bundle
wip/money-import-recheck
job/advsetup-654
wip/bgpicker-717
wip/advsetup-654
job/burst-709
job/kbdcaps-710
job/app-pw-chooser
wip/burst-709
wip/app-pw-chooser
job/imaptest-625
wip/kbdcaps-710
job/fix-499
wip/fix-499
job/perf-mut-667
job/calimg-589
job/perf-snap-666
wip/calimg-589
wip/perf-snap-666
wip/perf-mut-667
job/perf-cache-665
wip/perf-cache-665
job/voicefiles-620
wip/voicefiles-620
job/admin-burst-705
wip/admin-burst-705
job/voicememos-review
wip/voicememos-review
wip/ryw-653
job/ryw-653
job/writeonopen-661
job/instant-663
wip/writeonopen-661
job/money-import-review
wip/money-import-review
wip/importjs-610
review/integrations-407-round6
wip/integrations-review
job/dragghost-612
wip/dragghost-612
job/integrations
wip/integrations
job/decider-656
job/merge-round-6
job/perf-rerun
wip/merge-round-6
job/integrations-review-round5
job/selalign-576
wip/selalign-576
job/mcp-events-491
job/files-631
job/cal-e2e-569
wip/cal-e2e-569
job/reload-423
wip/reload-423
wip/mcp-events-491
wip/files-631
job/notesbridge-644
wip/notesbridge-644
job/editor-series
job/calcard-series
wip/calcard-series
job/mcp-events-review-491
wip/mcp-events-review
wip/editor-series
job/quirks-546
job/integrations-recheck
job/tocrail-636
wip/tocrail-636
wip/quirks-546
wip/reminders-643
job/reminders-643
wip/davscale-573
job/davscale-573
job/integrations-review
wip/ocr-eval-584
job/ocr-eval-584
job/esc-537
wip/esc-537
job/toastname-586
wip/toastname-586
job/submenu-579
wip/submenu-579
job/tasks-mode
wip/tasks-mode
job/agentdocs-630
job/dupwrite-634
wip/agentdocs-630
wip/dupwrite-634
job/lightglass-588
wip/lightglass-588
job/tabswitch-549
job/ghosttask-623
wip/ghosttask-623
job/toaststack-616
job/weekstate-609
job/mailsync-613
wip/mailsync-613
wip/weekstate-609
job/maildup-626
wip/tabswitch-549
wip/maildup-626
wip/toaststack-616
job/motion-611
wip/motion-611
job/tlstest-601
wip/tlstest-601
job/perf-495
job/floating-sheet
wip/floating-sheet
job/remdup-585
wip/remdup-585
job/fix-502
wip/fix-502
job/attachplay-622
job/perf-batch
wip/perf-batch-563
wip/perf-495
hotfix/mail-sync-diag
job/mail-m3
wip/mail-m3
job/attach-poof-603
job/calhover-608
job/editorbar-604
job/mentions-605
job/merge-round-4
job/allday-514
wip/merge-round-4
wip/allday-514
job/merge-round-4a
wip/merge-round-4a
job/sharestack-580
job/fix-501
wip/sharestack-580
wip/fix-501
job/perf-batch-563
job/apw-cache-review
wip/apw-cache-review
job/probe-520
wip/probe-520
job/mac-393
wip/mac-393
job/header-571
job/flake-513
wip/flake-513
job/docs-thumb-547
wip/header-571
job/webcal-572
wip/webcal-572
wip/shortcuts-542
job/shortcuts-542
wip/docs-thumb-547
job/caldav-stress
wip/caldav-stress
wip/sweep-478
job/apw-cache-512
wip/apw-cache-512
job/money-empty-540
wip/restart-505
wip/money-empty-540
wip/fix-510
job/restart-505
job/fix-503
job/perf-496
wip/perf-496
job/fix-498
wip/fix-498
job/info-inspector-465
wip/info-inspector-465
job/fix-510
job/fix-507
wip/fix-507
wip/fix-503
job/fix-493
job/money-kinds
wip/money-kinds
job/hygiene-548
job/merge-round-3
wip/fix-493
job/drag-snap-536
wip/merge-round-3
wip/merge-round-0930
wip/drag-snap-536
job/align-538
wip/align-538
job/bg-flash
wip/bg-flash
job/money-import
job/search-count-544
wip/search-count-544
wip/money-import
job/settings-key-541
wip/settings-key-541
job/toast-539
job/preview-421
wip/preview-421
wip/toast-539
job/tasks-500-531
job/title-plain-526
wip/title-plain-526
wip/tasks-500-531
job/notes-bridge
wip/parity-484
job/parity-484
job/files-slow
job/crash-525
wip/notes-bridge
wip/files-slow
wip/crash-525
job/kbd-motion-527
wip/bg-422
job/analytics-504
wip/analytics-504
wip/kbd-motion-527
job/upload-pill-523
wip/upload-pill-523
wip/tray-order
job/tray-order
wip/overflow-mid
wip/merge-round-2
job/perf-494
wip/perf-494
wip/mcp-fast-492
wip/motion-477
wip/asr-ab-489
wip/theme-variants-506
wip/overflow-511
wip/week-header-508
wip/attach-427
job/dav-delete-471
job/iso-435
wip/iso-435
wip/files-sel-keys
wip/dav-delete-471
job/align-253
job/siwc-490
wip/siwc-490
job/money-kinds-review
wip/align-253
wip/money-kinds-review
job/small-bugs-3
wip/overlay-title-487
wip/multiget-500
wip/hidden-420
wip/webcal-ui
wip/webcal-431
job/perf-367
job/location
wip/small-bugs-3
wip/location
wip/perf-367
wip/admin-deny-483
job/tag-unicode-473
wip/tag-unicode-473
job/blur-436
wip/photos-470
wip/blur-436
wip/small-bugs-4
wip/hunt-20260930
wip/settings-hdr-482
wip/chips-416
job/dedup-375
wip/dedup-375
job/doc-stack
wip/doc-stack
job/tokens-literals
wip/tokens-literals
job/jobs-leftovers
wip/send-fast
wip/paste-467
wip/money-numbers
job/money-plugin
wip/money-plugin
job/break-dav
wip/merge-batch
wip/crossday-469
wip/mac-verify
wip/mail-m2
wip/break-dav
wip/money-review2
job/money-md
job/modes-424
wip/money-md
wip/jobs-leftovers
job/agenda-413
wip/agenda-413
wip/modes-424
job/recog-417
wip/recog-417
wip/bounce-425
wip/ab-384-luna
job/webdav-perf
wip/webdav-perf
job/toast-ring
wip/toast-ring
job/money-review
wip/money-review
wip/micro-motion
wip/settings-card
wip/minical
job/notes-imap-428
job/least-priv
wip/ui-small-2
wip/flaky-426
wip/drag-end-418
job/jank
wip/jank
wip/least-priv
wip/docs-site
job/agenda
job/sec-batch
wip/sec-batch
wip/per-user-index
job/area-calendars
wip/area-calendars
job/parity
wip/parity
job/documents-research
wip/documents-research
job/test-infra
job/reminders-sync
wip/small-bugs-2
wip/reminders-sync
wip/gestures
job/google-oauth
wip/tags-merge
wip/tags
job/e2e-theme
wip/e2e-theme
job/icon-align
wip/test-infra
wip/select-align
wip/editor-385
job/voice
wip/webdav
job/webdav
job/app-pw-ui
job/editor-integrity
wip/editor-integrity
wip/voice
wip/quota
wip/cal-followups
wip/icon-align
job/composer-scale
wip/composer-scale
job/jobs-page
wip/jobs-page
job/hig-type
wip/hig-type
wip/app-pw-ui
job/motion-spring
job/mcp
wip/motion-spring
wip/mcp
job/small-bugs
wip/push-hosts
job/profile-sign
wip/touch-369
wip/profile-sign
job/mobile-focus
wip/mobile-focus
wip/ui-polish-354
wip/small-bugs
wip/dup-task
job/toast-polish
job/app-pw-scopes
wip/toast-polish
wip/app-pw-scopes
wip/cli-agent
wip/selection-pills
job/preview-attach
wip/preview-attach
job/dav-proppatch
wip/dav-proppatch
wip/cal-switcher
job/atomic-race
wip/atomic-race
job/photos-shared
wip/photos-shared
wip/cal-grid
wip/note-rewrite
wip/search-rebuild
job/mail-m1
job/paperless-import
wip/paperless-import
wip/mail-m1
wip/hidden-activity
wip/search-d
wip/pricing-research
wip/cursors
wip/auto-scheme
job/single-pills
wip/single-pills
wip/xuser-matrix
wip/money-format
wip/app-pw-setup
wip/purge-dos
wip/vault-health
wip/caldav-apple
wip/xuser-audit
wip/e2e-green
wip/tabbar
wip/adv-harness
wip/maple-mono
job/search-fix
wip/search-fix
wip/search-perf-c
job/adv-harness
wip/sidebar-headers
job/glass
wip/temp-index
job/polish
wip/polish
wip/file-protocols
wip/money-research
wip/glass
wip/voice-models
wip/collab-redo
job/voice-research
wip/hunt-20260928
wip/notes-actions-research
wip/search-pad
wip/search-perf
wip/search-sticky
wip/editor-undo
wip/chrome-rules
wip/motion
wip/appearance-research
wip/appearance
wip/audit-bugs
wip/cal-glass
wip/block-actions
wip/authz-order
wip/event-stripes
wip/chrome-sidebar
wip/auth-flaky
wip/robust-2
wip/gate-fix
wip/menu-blur
wip/import-calternaljs
wip/tray-fix
job/import-calternaljs
wip/index-order
wip/audit-fixes
wip/search-chevrons
research/mail
wip/phone-chrome
wip/dedup-break
wip/csp
wip/ui-audit
wip/select-toast
wip/perf
wip/flat-layout
wip/fonts
wip/event-tint
wip/sync-converge
wip/data-split
wip/glass-audit
wip/robustness
wip/sync-chaos
wip/search-thumbs
wip/fuzz
wip/menu-icons
wip/search-pill
wip/sync-changing
wip/heading-links
wip/date-formats
wip/a11y
wip/break-editor
wip/e2e-fix
wip/settings-sections
wip/sync-root-guard
wip/search-palette
wip/share-edit
job/toasts
wip/toasts
wip/cont-analytics
wip/authz-review
wip/popovers
wip/overlay-glass
wip/change-feed
wip/editor-modes
wip/composer-align
wip/cont-agenda
wip/agenda-merge
job/agent-conventions
wip/agent-conventions
wip/backend-misc
job/route-audit
wip/route-audit
wip/ui-batch
wip/heif-hardening
wip/grid-resize
wip/ask-page
wip/webmcp
job/deeplink-audit
wip/deeplinks
wip/shortcuts
wip/cont-tz-days
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#496
No description provided.