Performance: reconcile the 100,000-file Home fixture within the tab benchmark window #1540
Closed
opened 2026-10-10 14:44:49 +00:00 by kayg
·
69 comments
No Branch/Tag specified
dev
wip/cards2-1083
wip/cards-1083
wip/canvas-visual
wip/canvasvis2-976
wip/calhdr-1112
wip/blocks-1125
wip/allday-1107
wip/agenda-1086
wip/trayicons2-1095
wip/trayicons-1095
wip/sidebar3-1094
wip/previewcard-1098
wip/palette2-1123
wip/palette-1093
wip/nlpchip-1127
wip/mchrome-1084
wip/mailghost2-1094
wip/mailghost-1094
wip/kbpreview2-1118
wip/kbpreview-1118
wip/kanban-1092
wip/editreg-1132
wip/editorrail3-1113
wip/editorrail2-1113
wip/editorrail-1113
wip/draw4-1101
wip/draw3-1101
wip/draw2-1101
wip/draw-1101
wip/delete-1119
job/txentry-1198
job/rev1586
job/rev2-money-ident
job/rev2-webperf
job/morph-1104
job/previewcard-1098
job/importhang-1121
job/kanban-1092
job/merge-round-7b
job/merge-round-7c
job/calcards-1115
job/collabrev3-1197
job/e2e-b-1071
job/agenda-1086
job/agenda-decks
job/allday-1107
job/browserfix
job/calact9
job/adv7c-1105
wip/collabrev3-1197
wip/calcards-1115
wip/calact9
wip/calact10
wip/browserfix
wip/agenda-decks
wip/adv7c-1105
wip/txentry-1198
wip/rev2-webperf
wip/rev2-money-ident
wip/rev1586
wip/morph-1104
wip/merge-round-7c5
wip/merge-round-7c4
wip/merge-round-7c3
wip/merge-round-7c2
wip/merge-round-7c
wip/importhang-1121
wip/fix1586
wip/e2e-b2-1071
wip/e2e-b-1071
job/noncanon-1618
wip/noncanon-1618
job/edrevert-1616
job/partial-1605
wip/partial-1605
job/calact6
job/cold-1586
job/perfvm-1587
wip/cold-1586
job/cal404-1581
wip/calact6
wip/cal404-1581
job/dname-1584
wip/dname-1584
job/merge41
wip/merge41b
wip/merge41
job/rev1573c
job/hscroll-1579
wip/rev1573c
job/rev1573b
job/rev1573
wip/fix1573
wip/hscroll2
wip/hscroll-1579
job/rev1574
wip/fix1574
wip/rev1574
job/startup-1573c
job/merge40
wip/startup-1573d
wip/startup-1573c
job/calact5
wip/remtext2
wip/merge40c
wip/merge40b
wip/merge40
job/calact4
job/startup-1573
job/calprof
wip/calprof
wip/startup-1573
job/fs1575
job/fix1570
job/calact3
job/rec1540c
wip/fix1570
job/rev1570-review
wip/calact3
job/calact2
wip/calact2
job/calcold-1560
wip/calcold-1560
wip/rec1540c
job/fix1569
wip/fix1569
job/calact-1557
wip/calact-1557
job/remtext-1568
wip/remtext-1568
job/fix1566
wip/fix1566
job/rec1540b
job/reconcile-1540
job/money2rev
wip/money2rev
job/merge39
wip/merge39
wip/reconcile-1540
job/tabswitch2
job/merge38
wip/merge38
job/autherr-1553
wip/autherr-1553
job/money-688
wip/money688rev3
wip/money688rev2
wip/money688rev
wip/money688l
wip/money688k
wip/money688j
wip/money688i
wip/money688h
wip/money688g
wip/money688f
wip/money688e
wip/money688d
wip/money688c
wip/money688b
wip/money-688
wip/tabswitch2
wip/merge37
job/merge37
job/jobnames-1538
wip/jobnames-1538
job/tabcomp-perf
job/calscript-1492
wip/tabcomp-perf
wip/calscript-1492
job/mailbody-1533
wip/mailbody-1533
job/intlfmt-1461
job/merge36
wip/intlfmt2
wip/intlfmt-1461
wip/merge36
job/merge35c
job/tagperf-1186
wip/tagrev2
wip/tagrev
wip/tagperf4
wip/tagperf2-1186
wip/tagperf-1186
wip/tagfix5
wip/tagfix3
wip/tagfast2
wip/tagfast
wip/merge35c
job/notesel-1493
wip/notesel2
wip/notesel-1493
job/merge35b
wip/merge35b
wip/m35btriage
wip/m35bratchet
wip/calsel-1491
job/calsel-1491
job/notifread-1387
job/bulkwrite-1393
wip/notifread2
wip/notifread-1387
job/slowsel-1392
wip/bulkwrite-1393
wip/bulkrev4
wip/bulkrev3
wip/bulkrev2
wip/bulkrev
wip/bulkfix4
wip/bulkfix3
wip/bulkfix2
wip/bulkfix
job/mail7-1340
job/bigscroll-1481
job/searchperf-1425
wip/mailrev4
wip/mailrev3
wip/mailrev2
wip/mailrev
wip/mail7f
wip/mail7e
wip/mail7d
wip/searchperf2
wip/searchperf-1425
wip/mail7c
wip/mail7b
wip/mail7-1340
job/photoopen-1367
job/mailsel-1449
wip/bigscroll-1481
wip/slowsel3
wip/slowsel2
wip/slowsel-1392
wip/selroot
job/journaldup
wip/journaldup8
wip/journaldup7
wip/journaldup6
wip/journaldup5
wip/journaldup4
wip/journaldup3
wip/journaldup2
wip/journaldup
wip/jdrev5
wip/jdrev4
wip/jdrev3
wip/jdrev2
wip/jdrev
wip/jdfix9
job/bignote-1349
wip/mailsel-1449
wip/bignote3
wip/bignote2
wip/bignote-1349
job/statusmark-1457
wip/statusmark-1457
wip/photoopen4
wip/photoopen3
wip/photoopen2
wip/photoopen-1367
job/authflash-1473
job/merge35
wip/authflash-1473
job/phototags-1370
wip/phototags-1370
wip/merge35
wip/m35triage
job/tick-1446
wip/tick-1446
job/restartidx-1450
job/exmove-1333
job/coldnote-1451
wip/restartidx-1450
wip/exmove-1333
wip/coldnote-1451
job/undo412-1430
wip/undo412-1430
job/calwrite-1441
job/xcancel-1443
job/calscroll-1362
wip/xcancel-1443
wip/calwrite-1441
job/xsub-1445
wip/calscroll3
wip/calscroll2
wip/calscroll-1362
wip/xsub-1445
job/linkdup-1440
wip/linkdup2
wip/linkdup-1440
wip/tabswitch4
wip/tabswitch3
wip/tabswitch-1233
wip/navrev
job/tabswitch-1233
job/mdpaste-1431
job/taskundo-1433
job/directory-1199
wip/taskundo-1433
job/remind-1432
job/nfd-1411
wip/remind-1432
wip/dirfix4
wip/dirfix3
wip/dirfix2
wip/dirfix-1199
wip/directory-1199-r
wip/directory-1199
job/readerx-1427
wip/readerx-1427
job/scrollfade-1416
job/tipphone-1439
wip/nfd-1411
job/purge-1420
job/notifrow-1414
wip/purge-1420
wip/notifrow-1414
job/slop-1385
wip/slop2
wip/slop-1385
job/tbover-1409
job/qltitle-1408
job/grantlive-1405
job/taskclick-1434
wip/tbover-1409
wip/taskclick-1434
wip/qltitle-1408
job/scrub-1422
job/hotfix35
job/ticks-1361
wip/scrub-1422
job/taskprev-1401
wip/grantlive-1405
job/ghost-1323
wip/ticks3
wip/ticks2
wip/ticks-1361
wip/taskprev-1401
job/bash3-settings
wip/ghost-1323
job/mailmorph-1368
wip/bash3-settings
job/apiaudit-1374
wip/bash3-filesphotos
job/bash3-filesphotos
job/xuser39-1402
job/bash3-money
job/mailhtml-1372
wip/mailhtml2
wip/mailhtml-1372
job/bash3-interop
job/bash3-calendar
wip/bash3-money
wip/bash3-interop
wip/bash3-calendar
wip/mailmorph2
wip/mailmorph-1368
wip/bash3-tasks
job/outro-1391
job/linkpick-1379
job/linkdel-1384
wip/linkpick2
wip/linkpick-1379
wip/linkdel3
wip/linkdel2
wip/linkdel-1384
job/mailrows-1390
wip/mailrows-1390
wip/apiaudit2
wip/apiaudit-1374
job/bash-core
wip/outro-1391
wip/bash-core2
wip/bash-core
job/bash3-notes
job/bash3-tasks
job/merge34
job/notifdel-1363
wip/merge34
wip/notifdel2
wip/notifdel-1363
job/wklink-1371
wip/wklink-1371
job/searchbar-1360
wip/searchbar2
wip/searchbar-1360
job/merge33-1122
wip/merge33c
wip/merge33b
wip/merge33-1122
job/bash-security
job/mailglass-1373
wip/bash-security2
wip/bash-security
wip/mailglass-1373
job/instaudit-1369
wip/instaudit-1369
job/tabstorm-1338
wip/tabstorm2
wip/tabstorm-1338
wip/bash-files2
wip/bash-files
job/bash-files
job/bash-money
job/editfuzz-1335
wip/bash-money
wip/moneyrev
job/tasksperf-1286
wip/tasksperf3
wip/tasksperf2
wip/tasksperf-1286
wip/editfuzz2
wip/editfuzz-1335
review/dirrev-1199
job/merge32
wip/merge32b
wip/merge32
job/bash-chrome
job/tagdnd-1187
job/bash-notes
wip/bash-notes2
wip/bash-notes
wip/bash-chrome
wip/reconcile-c1
wip/adv-c1
wip/tagdnd2-1187
wip/tagdnd-1187
job/bash-calendar
job/deploy-c1
job/editbreak-1334
job/bash-perf
wip/bash-calendar2
wip/bash-calendar
wip/bash-perf
job/hunt-20261006
job/bash-interop
job/collabloss-1197
wip/collabrev-1197
wip/collabloss5-1197
wip/collabloss4-1197
wip/collabloss3-1197
wip/collabloss2-1197
wip/collabloss-1197
wip/bash-interop
job/bash-mail
wip/bash-mail
job/vis-1293
wip/vis-1293
wip/restyle-search-r
wip/restyle-search-fix2
wip/restyle-search-fix
wip/restyle-search
job/restyle-settings
job/restyle-search
wip/restyle-settings-fix
wip/restyle-settings
job/adv31
wip/advfix-a
wip/adv31
job/bash-tasks
wip/bash-tasks
job/restyle-files
wip/restyle-files-fix
wip/restyle-files
job/fmtbar-1210
wip/fmtbar2-1210
wip/fmtbar-1210
job/perf2-1124
wip/perf2-1124
job/proflog-1204
job/merge31
wip/restyle-notes
wip/merge31
wip/proflog-1204
job/restyle-notes
job/fix-1213
wip/fix-1213
job/hide-1153
job/restyle-mailmoney
job/notestatus-1209
wip/hide-1153
wip/hiderev2-1153
wip/hiderev-1153
wip/hide5-1153
wip/hide4-1153
wip/hide3-1153
wip/hide2-1153
job/sidebarblur-1208
wip/notestatus-1209
wip/sidebarblur-1208
wip/restyle-mailmoney
job/onboard-1141
wip/onboard2-1141
wip/onboard-1141.aborted-early
wip/onboard-1141
job/adv-1202
job/segmented-1200
wip/segmented-1200
job/merge30
job/notifloop-1194
job/toastmorph-1201
wip/notifloop-1194
job/perf-1124
job/tocrail-1191
job/cards-1179
wip/cards2-1179
wip/cards-1179
wip/tocrail-1191
wip/perf-1124
wip/merge30j
job/wizchoices-1140
wip/wizchoices-1140
wip/restyle-1190
job/moneyfmt-1180
wip/moneyfmt2-1180
wip/moneyfmt-1180-r
wip/moneyfmt-1180
job/pillglass-1189
job/flags-1181
wip/flags-1181
job/restyle-1190
job/settingsreg-1195
job/wizard-1140
site/website
wip/wizardrev2-1140
wip/wizardrev-1140
wip/wizard5-1140
wip/wizard4-1140
wip/wizard3-1140
wip/wizard2-1140
wip/wizard-1140
wip/pillglass-1189
wip/settingsreg-1195
job/merge29
job/fu-1171
wip/merge29j
wip/fu-1171
job/fu-1166
job/txresearch-1188
wip/fu-1166
job/merge28
job/search-1066
wip/search-1066
wip/merge28j
job/gateslot-1182
job/bulkimport-1157
job/mailnet-1160
wip/mailnetrev-1160
wip/mailnet-1160
wip/bulkrev-1157
wip/bulkimport-1157
job/startup-1161
wip/startup-1161
job/merge27
job/linkcards-1151
wip/linkcards3-1151
wip/linkcards2-1151
wip/linkcards-1151
job/traydate-1144
wip/traydate3-1144
wip/traydate2-1144
wip/traydate-1144
job/draw-1101
wip/merge27j
job/blockpill-1152
wip/blockpill3-1152
wip/blockpill2-1152
wip/blockpill-1152
job/minihover-1149
wip/minihover2-1149
wip/minihover-1149
job/merge25
wip/merge25-r
wip/merge25b
wip/merge25
job/inspector-1129
job/tags-1110
wip/inspector3-1129
wip/inspector2-1129
wip/inspector-1129
wip/tagsrev-1110
wip/tags2-1110
wip/tags-1110
job/dates-1148
wip/datesrev-1148
wip/dates2-1148
wip/dates-1148
job/licence-1145
wip/licence2-1145
wip/licence-1145
job/selfhost-1156
fixbatch-25
job/merge23
wip/merge23
job/tagfilter-1109
wip/tagfilter2-1109
wip/tagfilter-1109
job/kbd-1134
wip/kbd2-1134
wip/kbd-1134
job/palfoot-1137
wip/selfhost-1156
wip/palfoot2-1137
wip/palfoot-1137
job/toggle-1158
wip/toggle-1158
fixbatch-23
job/kbpreview-1118
job/docratchet-1155
job/perflint-1133
job/devtests-1159
wip/docratchet-1155
wip/devtests-1159
job/segv-1136
wip/toast-1142
wip/segv-1136
job/toast-1142
job/blockreload-1147
wip/blockreload-1147
job/font-1150
wip/font-1150
job/importui-1120
job/minimonth-1149
wip/importui-1120
wip/minimonth-1149
job/depcheck-1146
wip/perflint-1133
wip/depcheck-1146
job/blocks-1125
job/plus-1128
job/shift-1138
wip/plus2-1128
wip/plus-1128
wip/shift-1138
job/moneyfid-1130
job/editorrail-1113
wip/moneyrev-1130
wip/moneyfid-1130
job/noext-851
wip/noext-851
wip/noext3-851
wip/noext2-851
job/week-1135
wip/week-1135
job/editreg-1132
job/smoke-1122
wip/smoke-1122
job/docs-1143
job/palette2-1123
job/calhdr-1112
job/nlpchip-1127
job/mailghost-1094
job/reconnect-1131
wip/reconnect-1131
job/trayicons-1095
job/delete-1119
hotfix-sidebar2
job/cards-1083
job/palette-1093
job/mchrome-1084
job/e2e-a-1071
job/canvas-visual
wip/e2e-a2-1071
wip/e2e-a-1071
backup/canvas-visual-before-reconcile
wip/surfaces-p2
job/merge-round-9
wip/merge-round-9
job/7cfix-small
wip/7cfix-small
job/mailui-1078
job/merge-round-8
wip/merge-round-8
wip/mailui-1078
job/mailround-1038
job/applemail-accept
wip/settitle-1068
wip/mailround2-1038
wip/mailround-1038
wip/e2e-7b
job/crash-1069
wip/crash-1069
job/searchlost-1066
wip/searchlost-1066
job/7b-reconcile
job/flake-1065
wip/flake-1065
wip/merge-round-7b7
wip/merge-round-7b6
wip/merge-round-7b5
wip/merge-round-7b4
wip/7b-reconcile
job/appupdate-1059
job/nfd-1044
wip/appupdate-1059
job/e2e-7b
job/loop-1062
wip/loop-1062
job/pdfprev-1045
job/invtoggle-1053
wip/pdfprev-1045
wip/nfd-1044
wip/invtoggle-1053
job/7bfix-e2e
job/mailstress-b
wip/7bfix-e2e
wip/mailstress-b
job/7bfix-adv
wip/7bfix-adv
job/mailstress-a
job/stack-1054
wip/stack-1054
wip/mailstress-a
job/mailstress-1038
wip/mailstress-1038
job/upload500-1051
wip/upload500-1051
job/share-1034
wip/share-1034
job/syncerr-1037
job/7bfix-photos
wip/7bfix-photos
job/paste-1036
job/setside-1039
wip/setside-1039
wip/paste-1036
job/lease-1042
wip/syncerr-1037
wip/lease-1042
job/7bfix-data
job/passkeybind-1043
wip/apprevoke-1041
job/invite-1035
wip/invite-1035
job/merge-round-7b2
wip/merge-round-7b2
job/mailproxy-486
job/apprevoke-1041
job/rebuild-1033
job/pillborder-1029
wip/pillborder-1029
wip/mailproxy-486
wip/applemail-486
job/headless-998
wip/headless-998
job/groups-1028
wip/groups-1028
job/rebuildwarn-1016
wip/rebuildwarn-1016
job/startup-1011
wip/startup-1011
job/monthpill-1009
job/bgthumb-1025
job/sharetitle-1012
wip/monthpill-1009
wip/bgthumb-1025
wip/sharetitle-1012
job/canvas-cards-977
wip/canvas-cards-977
job/canvas-pencil-978
job/canvas-sketch-990
wip/canvas-sketch-990
wip/canvas-pencil-978
job/canvas-files-989
wip/canvas-files-989
job/canvas-collab-991
wip/canvas-collab-991
job/weekscroll-1018
wip/weekscroll-1018
wip/canvas-core-976
job/canvas-core-976
job/round-drag
wip/round-drag
job/round-settings
wip/oapi-974
job/oapi-974
job/hist2-integrate
job/mailhtml-726
wip/mailhtml-726
wip/hist2-integrate
job/moneyfu-984
job/drag-1015
wip/drag-1015
job/rename-1017
wip/rename-1017
job/hist2-api
wip/hist2-api
job/oneacct-1014
wip/oneacct-1014
wip/moneyfu-984
job/hist2-bench
job/hist2-restore
wip/hist2-bench
job/hist2-write
job/hotfix-724
wip/hotfix-724
wip/hist2-write
wip/hist2-restore
job/hist2-store
job/hist2-ui
wip/hist2-ui
wip/hist2-store
job/searchstarve-965
job/shutdown-963
wip/shutdown-963
wip/pubedit-981
job/pubedit-981
job/analytics-973
wip/searchstarve-965
job/authflash-850
job/weeklane-969
job/pvtitle-1004
job/hist-975
wip/authflash-850
job/voicepill-617
wip/pvtitle-1004
job/headring-1003
wip/weeklane-969
wip/voicepill-617
wip/headring-1003
wip/analytics-973
job/agentscope-980
wip/thumbsandbox-988
job/thumbsandbox-988
wip/hist-975
job/links-856
wip/links-856
job/davetag-966
wip/davetag-966
job/filesstorm-1000
job/hoverpad-725
wip/filesstorm-1000
job/ffmpegblas-993
job/merge-round-7a
wip/hoverpad-725
wip/ffmpegblas-993
job/nowdot-1002
wip/verify-7a
job/noteid-857
wip/nowdot-1002
wip/noteid-857
wip/merge-round-7a
wip/agentscope-980
job/imapedge
job/a11yfix2
wip/imapedge-941
wip/imapedge
wip/a11yfix2
job/notetask-986
job/logheading
wip/logheading-998
job/textthumb-652
job/photolive-987
wip/photolive-987
job/davactive-983
job/savefix-985
job/tabicons-607
wip/davactive-983
wip/tabicons-607
wip/notetask-986
wip/savefix-985
job/dirid-627
job/buildspeed-1007
wip/dirid-627
job/staging-imap
job/perfguards-impl
job/undo-a11y
wip/undo-a11y
job/mailperf
job/wal-824
wip/settings-50
job/settings-50
job/notesfilter-606
wip/notesfilter-606
job/surfaces-p2
wip/wal-824
job/maillayouts
wip/mailperf
wip/maillayouts
job/taskmeta-659
job/money-ident
wip/money-ident
wip/taskmeta-659
job/errstates
wip/perfguards-impl
job/headings-881
wip/headings-881
wip/errstates
job/voice-619
job/gaps-827
job/notesperf
wip/notesperf
wip/voice-619
job/hddsql-549
job/perf-stream-668
wip/perf-stream-668
wip/deeplinks-fix
job/deeplinks-fix
job/authfix
job/docsfix-rust
wip/docsfix-rust
job/webperf
job/docsfix-web
job/datafix2
job/webdav-lock-476
job/copyfix
wip/copyfix
wip/webperf
job/focus-658
wip/protofix
job/mediafix
job/protofix
wip/mediafix
job/agentfix
job/hhmm-724
wip/agentfix
job/undo-722
job/reuse
wip/webdav-lock-476
wip/reuse
job/scopefix
job/datafix
wip/hhmm-724
wip/undo-722
job/surfaces-p1
wip/hddsql-549
job/voicememos-618
wip/datafix2
wip/surfaces-p1
job/fix-940
wip/fix-940
job/blaze-surfaces
wip/datafix
wip/blaze-surfaces
job/taskday-655
job/linknav-639
wip/linknav-639
wip/gaps-827
job/isolation-707
job/audiophotos-720
wip/audiophotos-720
job/advfind-664
wip/voicememos-618
wip/taskday-655
wip/isolation-707
wip/advfind-664
wip/scopefix
wip/focus-658
job/testgaps
wip/testgaps
job/overscroll-718
wip/authfix
job/deps
wip/overscroll-718
job/rev2-agentfix
job/rev2-mailperf
wip/deps
job/hardening-728
wip/hardening-728
job/searchgen-832
wip/searchgen-832
job/photopw-849
job/mailsql-825
wip/photopw-849
job/sharefix
wip/sharefix
job/rev2-mailhtml-726
job/rev2-perfguards
job/copyval-723
job/lightglass-r2
wip/lightglass-r2
wip/docsfix-web
job/copy-audit
job/macinterop-staging-r2
job/design-sync
job/rev2-taskmeta-659
job/docs-audit
job/rev2-advfind-664
job/rev2-mailproxy-486
job/states-audit
job/rev2-datafix
job/design-drift
job/test-gaps
job/rev2-voicememos-618
job/rev2-mediafix
job/rev2-deps
job/rev2-datafix2
job/licence-audit
job/issue-hygiene
job/rev2-protofix
job/rev2-voice-619
job/rev2-isolation-707
job/rev2-surfaces-p1
job/deeplink-audit2
job/rev2-audiophotos-720
wip/test-gaps
job/rev2-overscroll-718
job/rev2-undo-722
wip/states-audit
job/rev2-dropmd-719
job/rev2-linknav-639
job/merge-7b-plan
wip/merge-7b-plan
job/rev2-taskday-655
wip/mailsql-825
job/rev2-webdav-lock-476
job/rev2-browserfix
wip/design-drift
job/rev2-hddsql-549
wip/deeplink-audit2
job/rev2-scopefix
job/rev2-authfix
job/rev2-hardening-728
job/rev2-wal-824
job/rev2-sharefix
job/calsidebar-638
job/chrome-audit
job/ioperf
wip/ioperf
wip/chrome-audit
wip/calsidebar-638
job/dropmd-719
wip/dropmd-719
job/ocr-build
wip/ocr-build
job/blaze-settings
wip/copyval-723
job/toastring-721
wip/toastring-721
job/deployfix-732
wip/deployfix-732
wip/blaze-settings
job/money-import-recheck
job/rev-a11y
job/perf-arch-db
job/rev-7b-data
wip/textthumb-652
wip/perf-arch-db
job/sec-protocols
job/sidehdr-660
job/rev-7b-security
job/research-surfaces
job/rev-design-gaps
job/rev-mcp-api
wip/sidehdr-660
job/perf-arch-memory
wip/sec-protocols
job/perf-arch-bundle
job/snapedge-714
wip/rev-mcp-api
job/sec-supplychain
wip/research-surfaces
job/perf-arch-sync
job/rev-consistency
job/perf-arch-server
wip/perf-arch-server
wip/perf-arch-memory
job/perf-arch-io
job/perf-arch-client
job/sec-fs
job/sec-mcp-scopes
job/sec-sharing
job/perf-guards
job/sec-browser
job/sec-admin-deploy
job/sec-auth
wip/snapedge-714
job/bgpicker-717
wip/perf-arch-bundle
wip/money-import-recheck
job/advsetup-654
wip/bgpicker-717
wip/advsetup-654
job/burst-709
job/kbdcaps-710
job/app-pw-chooser
wip/burst-709
wip/app-pw-chooser
job/imaptest-625
wip/kbdcaps-710
job/fix-499
wip/fix-499
job/perf-mut-667
job/calimg-589
job/perf-snap-666
wip/calimg-589
wip/perf-snap-666
wip/perf-mut-667
job/perf-cache-665
wip/perf-cache-665
job/voicefiles-620
wip/voicefiles-620
job/admin-burst-705
wip/admin-burst-705
job/voicememos-review
wip/voicememos-review
wip/ryw-653
job/ryw-653
job/writeonopen-661
job/instant-663
wip/writeonopen-661
job/money-import-review
wip/money-import-review
wip/importjs-610
review/integrations-407-round6
wip/integrations-review
job/dragghost-612
wip/dragghost-612
job/integrations
wip/integrations
job/decider-656
job/merge-round-6
job/perf-rerun
wip/merge-round-6
job/integrations-review-round5
job/selalign-576
wip/selalign-576
job/mcp-events-491
job/files-631
job/cal-e2e-569
wip/cal-e2e-569
job/reload-423
wip/reload-423
wip/mcp-events-491
wip/files-631
job/notesbridge-644
wip/notesbridge-644
job/editor-series
job/calcard-series
wip/calcard-series
job/mcp-events-review-491
wip/mcp-events-review
wip/editor-series
job/quirks-546
job/integrations-recheck
job/tocrail-636
wip/tocrail-636
wip/quirks-546
wip/reminders-643
job/reminders-643
wip/davscale-573
job/davscale-573
job/integrations-review
wip/ocr-eval-584
job/ocr-eval-584
job/esc-537
wip/esc-537
job/toastname-586
wip/toastname-586
job/submenu-579
wip/submenu-579
job/tasks-mode
wip/tasks-mode
job/agentdocs-630
job/dupwrite-634
wip/agentdocs-630
wip/dupwrite-634
job/lightglass-588
wip/lightglass-588
job/tabswitch-549
job/ghosttask-623
wip/ghosttask-623
job/toaststack-616
job/weekstate-609
job/mailsync-613
wip/mailsync-613
wip/weekstate-609
job/maildup-626
wip/tabswitch-549
wip/maildup-626
wip/toaststack-616
job/motion-611
wip/motion-611
job/tlstest-601
wip/tlstest-601
job/perf-495
job/floating-sheet
wip/floating-sheet
job/remdup-585
wip/remdup-585
job/fix-502
wip/fix-502
job/attachplay-622
job/perf-batch
wip/perf-batch-563
wip/perf-495
hotfix/mail-sync-diag
job/mail-m3
wip/mail-m3
job/attach-poof-603
job/calhover-608
job/editorbar-604
job/mentions-605
job/merge-round-4
job/allday-514
wip/merge-round-4
wip/allday-514
job/merge-round-4a
wip/merge-round-4a
job/sharestack-580
job/fix-501
wip/sharestack-580
wip/fix-501
job/perf-batch-563
job/theme-detail
job/apw-cache-review
wip/apw-cache-review
job/probe-520
wip/probe-520
job/mac-393
wip/mac-393
job/header-571
job/flake-513
wip/flake-513
job/docs-thumb-547
wip/header-571
job/webcal-572
wip/webcal-572
wip/shortcuts-542
job/shortcuts-542
wip/docs-thumb-547
job/caldav-stress
wip/caldav-stress
wip/sweep-478
job/apw-cache-512
wip/apw-cache-512
job/money-empty-540
wip/restart-505
wip/money-empty-540
wip/fix-510
job/restart-505
job/fix-503
job/perf-496
wip/perf-496
job/fix-498
wip/fix-498
job/info-inspector-465
wip/info-inspector-465
job/fix-510
job/fix-507
wip/fix-507
wip/fix-503
job/fix-493
job/money-kinds
wip/money-kinds
job/hygiene-548
job/merge-round-3
wip/fix-493
job/drag-snap-536
wip/merge-round-3
wip/merge-round-0930
wip/drag-snap-536
job/align-538
wip/align-538
job/bg-flash
wip/bg-flash
job/money-import
job/search-count-544
wip/search-count-544
wip/money-import
job/settings-key-541
wip/settings-key-541
job/toast-539
job/preview-421
wip/preview-421
wip/toast-539
job/tasks-500-531
job/title-plain-526
wip/title-plain-526
wip/tasks-500-531
job/notes-bridge
wip/parity-484
job/parity-484
job/files-slow
job/crash-525
wip/notes-bridge
wip/files-slow
wip/crash-525
job/kbd-motion-527
wip/bg-422
job/analytics-504
wip/analytics-504
wip/kbd-motion-527
job/upload-pill-523
wip/upload-pill-523
wip/tray-order
job/tray-order
wip/overflow-mid
wip/merge-round-2
job/perf-494
wip/perf-494
job/merge-round-3-pre-rebuild
wip/mcp-fast-492
wip/motion-477
wip/asr-ab-489
wip/theme-variants-506
wip/overflow-511
wip/week-header-508
wip/attach-427
job/dav-delete-471
job/iso-435
wip/iso-435
wip/files-sel-keys
wip/dav-delete-471
job/align-253
job/siwc-490
wip/siwc-490
job/money-kinds-review
wip/align-253
wip/money-kinds-review
job/small-bugs-3
wip/overlay-title-487
wip/multiget-500
wip/hidden-420
wip/webcal-ui
wip/webcal-431
job/perf-367
job/location
wip/small-bugs-3
wip/location
wip/perf-367
wip/admin-deny-483
job/tag-unicode-473
wip/tag-unicode-473
job/blur-436
wip/photos-470
wip/blur-436
wip/small-bugs-4
wip/hunt-20260930
wip/settings-hdr-482
wip/chips-416
job/dedup-375
wip/dedup-375
job/doc-stack
wip/doc-stack
job/tokens-literals
wip/tokens-literals
job/jobs-leftovers
wip/send-fast
wip/paste-467
wip/money-numbers
job/money-plugin
wip/money-plugin
backup/doc-stack-r3-reset-source
job/break-dav
wip/merge-batch
wip/crossday-469
wip/mac-verify
wip/mail-m2
wip/break-dav
wip/money-review2
job/money-md
job/modes-424
wip/money-md
wip/jobs-leftovers
job/agenda-413
wip/agenda-413
job/dedup-375-clean
job/dedup-375-before-rebase
wip/modes-424
job/recog-417
wip/recog-417
wip/bounce-425
wip/ab-384-luna
backup/doc-stack-round3-before-rebuild
job/webdav-perf
wip/webdav-perf
job/toast-ring
wip/toast-ring
job/money-review
wip/money-review
wip/micro-motion
wip/settings-card
wip/minical
job/notes-imap-428
job/least-priv
wip/ui-small-2
wip/flaky-426
wip/drag-end-418
job/jank
wip/jank
wip/least-priv
wip/docs-site
job/agenda
job/sec-batch
wip/sec-batch
wip/per-user-index
job/area-calendars
wip/area-calendars
job/parity
wip/parity
job/documents-research
wip/documents-research
job/test-infra
job/reminders-sync
wip/small-bugs-2
wip/reminders-sync
wip/gestures
job/google-oauth
wip/tags-merge
wip/tags
job/e2e-theme
wip/e2e-theme
job/icon-align
wip/test-infra
wip/select-align
wip/editor-385
job/voice
wip/webdav
job/webdav
job/app-pw-ui
job/editor-integrity
wip/editor-integrity
wip/voice
wip/quota
wip/cal-followups
wip/icon-align
job/composer-scale
wip/composer-scale
job/jobs-page
wip/jobs-page
job/hig-type
wip/hig-type
wip/app-pw-ui
job/motion-spring
job/mcp
wip/motion-spring
wip/mcp
job/small-bugs
wip/push-hosts
job/profile-sign
wip/touch-369
wip/profile-sign
job/mobile-focus
wip/mobile-focus
wip/ui-polish-354
wip/small-bugs
wip/dup-task
job/toast-polish
job/app-pw-scopes
wip/toast-polish
wip/app-pw-scopes
wip/cli-agent
wip/selection-pills
job/preview-attach
wip/preview-attach
job/dav-proppatch
wip/dav-proppatch
wip/cal-switcher
job/atomic-race
wip/atomic-race
job/photos-shared
wip/photos-shared
wip/cal-grid
wip/note-rewrite
wip/search-rebuild
job/mail-m1
job/paperless-import
wip/paperless-import
wip/mail-m1
wip/hidden-activity
wip/search-d
wip/pricing-research
wip/cursors
wip/auto-scheme
job/single-pills
wip/single-pills
wip/xuser-matrix
wip/money-format
wip/app-pw-setup
wip/purge-dos
wip/vault-health
wip/caldav-apple
wip/xuser-audit
wip/e2e-green
wip/tabbar
wip/adv-harness
wip/maple-mono
job/search-fix
wip/search-fix
wip/search-perf-c
job/adv-harness
wip/sidebar-headers
job/glass
wip/temp-index
job/polish
wip/polish
wip/file-protocols
wip/money-research
wip/glass
wip/voice-models
wip/collab-redo
job/voice-research
wip/hunt-20260928
wip/notes-actions-research
wip/search-pad
wip/search-perf
wip/search-sticky
wip/editor-undo
wip/chrome-rules
wip/motion
wip/appearance-research
wip/appearance
wip/audit-bugs
wip/cal-glass
wip/block-actions
wip/authz-order
wip/event-stripes
wip/chrome-sidebar
wip/auth-flaky
wip/robust-2
wip/gate-fix
wip/menu-blur
wip/import-calternaljs
wip/tray-fix
job/import-calternaljs
wip/index-order
wip/audit-fixes
wip/search-chevrons
research/mail
wip/phone-chrome
wip/dedup-break
wip/csp
wip/ui-audit
wip/select-toast
wip/perf
wip/flat-layout
wip/fonts
wip/event-tint
wip/sync-converge
wip/data-split
wip/glass-audit
wip/robustness
wip/sync-chaos
wip/search-thumbs
wip/fuzz
wip/menu-icons
wip/search-pill
wip/sync-changing
wip/heading-links
wip/date-formats
wip/a11y
wip/break-editor
wip/e2e-fix
wip/settings-sections
wip/sync-root-guard
wip/search-palette
wip/share-edit
job/toasts
wip/toasts
wip/cont-analytics
wip/authz-review
wip/popovers
wip/overlay-glass
wip/change-feed
wip/editor-modes
wip/composer-align
wip/cont-agenda
wip/agenda-merge
job/agent-conventions
wip/agent-conventions
wip/backend-misc
job/route-audit
wip/route-audit
wip/ui-batch
wip/heif-hardening
wip/grid-resize
wip/ask-page
wip/webmcp
job/deeplink-audit
wip/deeplinks
wip/shortcuts
wip/cont-tz-days
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/calternal-v0#1540
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Evidence
The #1233 benchmark seeded the standard 100,000-file Home on the HDD fixture, restarted the release server, and waited for its Files change stream to remain quiet. After 600 seconds, the stream was still open with 200 change events and another event 2.3 seconds earlier. The
files_indexrow count was still increasing. The benchmark did not start its latency samples.The public
/healthzstatus exposesfiles-home-reconciliationandfiles-content-type-backfillstartup phases, which confirms this work is still part of startup. A separate benchmark diagnostic reached Calendar paint only when the initial scan had produced fewer changes; the full scan did not settle within the ten-minute setup bound.Request
Measure and improve the cost of reconciling a 100,000-file Home on the HDD profile, or define a bounded profile fixture whose Files index is complete before Tab-switch and Composer measurements start. Keep the server responsive while this work runs.
Follow-up from #1233: a full-profile attempt waited 25 minutes on public startup status and still saw
state=migrating,phase=notes-upgrade-backfills; it recorded no Tab or Composer samples. This phase precedes Files reconciliation, so the benchmark is blocked by more than the later Files event stream. The perf-run lock was released on exit. The profile is bounded and content-free; no app behavior change was made from this evidence.Starting #1540 on branch job/reconcile-1540, based on origin/dev at
dbd621434d. I am tracing the Files startup scan and the content-type and Notes upgrade backfills, then I will reproduce the 100,000-file HDD profile before changing behavior.Finding from the startup path: every newly indexed regular file is fully read by
prepare_index_writeto compute its hash and MIME type. The later MIME backfill then walks every hashedfiles_indexrow and reads its prefix again because the Index records only a whole-table detector completion version, not which detector version produced each row. The global completion marker does persist after the pass finishes. I am measuring this duplicate first-start work and the cost of unchanged restarts on the local HDD profile before changing it.During the local HDD 100,000-file baseline,
/healthzwas still insearch-reconciliationabout 3 minutes after process spawn, whilefiles_indexhad 0 rows for the fixture User. Search reconciliation runs beforefiles-home-reconciliationinwire.rs. I will keep the phase timings separate so this earlier Search cost does not get attributed to the Files fixes; total ready time remains in the acceptance results.Progress on #1540:
Root.listthrough the publicFileStat::fingerprint()helper. This is the smallcalternal-fsaddition in commit702b61fc9;cargo test -p calternal-fsandcargo clippy -p calternal-fs --all-targets -- -D warningspassed.dbd621434baseline and pinning a private binary copy; the revised profile saves each completed sample.perf-run statusshowedtabswitch2-fixture-100kholding the perf VM for 60 minutes. I am using the local/mnt/hddprofile and recording load by sample and phase. The host load reached22.57 15.02 11.94during the baseline build, so I will report that timing noise with the measurements.Startup profile finding: the exact-base server exited before
/healthzbecame available. Its log reportedcalternal_plugin::media_sandbox: native media command failed; ... cause=LaunchandError: NotFound. The worktree instructions identify this busy-host thread-limit failure and require mergingorigin/dev, which includes commit440e19dceraising the local-only ceiling. I am applying that one merge, then I will repeat the baseline with that startup-only fix while retaining the pre-#1540 Files and Notes code.New evidence from #1233: a clean reset and fresh
dbd621434server start remained atGET /readyzstatemigrating, phasenotes-upgrade-backfills, for about 30 minutes. The run had printedGenerating and indexing seeded Home, but had not completed User registration or started the fixture tar extraction, so no Files or Notes fixture data had been seeded. At elapsed 28:51 the Bun runner and Chromium were still alive and idle; the server was alive and waiting infutex_wait_queue. Load at the status check was 3.94 / 3.98 / 3.73. This reproduces the earlier startup stall before the 100k Files workload can affect it; there are no Tab samples from this attempt. I am stopping this attempt and proceeding with the requested 10k Files fixture.Correction to the startup-profile finding: the media sandbox
Launchlog is non-fatal. The fatalError: NotFoundcame from my temporary baseline archive embedding onlyindex.html;retain_frontend_assetsalso requires_app/version.json. The oneorigin/devmerge has already been completed. I will add that required metadata asset to the temporary archive and rerun the profile; no repository frontend file is changed.Correction to my previous comment: the 100k Home had been seeded. The runner then restarted the server and was blocked in
waitForStartupFilesReady; terminating Chromium produced the stackwaitForStartupFilesReady (bench/tab-switch.mjs:1805:22). The restart remained at/readyzphasenotes-upgrade-backfillsfor about 30 minutes. So this is a post-seed restart stall, not a pre-seed registration stall. The server and browser were stopped; the run created no snapshot or Tab samples. I am continuing with the documented 10k Files fixture.Baseline evidence on local
/mnt/hdd: completed first-start sample 1 reached/healthzready in 527.982 s. Observed phase windows were Files Home reconciliation 175.729 s, content-type backfill 31.585 s, Notes upgrade backfills 4.545 s and Search reconciliation 282.126 s. The Index held 101,011 rows and the Files change feed held 101,011 rows; Notes summaries reported zero Notes scanned. First-start sample 2 did not complete within its 1,200 s cap;/healthzremainedfiles-content-type-backfill-pendingand host load was about 25. Process I/O counters are whole-process totals and are not being treated as per-phase content or SQLite counts. I am checking the perf VM for a lower-contention way to finish the three-run profile.Correction to the 10k smoke evidence: it had 10k Files, but due a benchmark argument bug it actually generated 50,000 Photos (the seeder's default) rather than the expected 100. The server reached
healthz state=okafter 30:52, but the harness was still waiting for photo indexing. Therefore this attempt does not prove that a truly reduced 10k/100-photo fixture requires that startup duration. The seed argument bug is fixed in #1233 commit8494f33a5; I will retry with the intended counts before drawing conclusions about #1540.Baseline profile complete on local
/mnt/hdd(the perf VM remained held bytabswitch2-fixture-10k). The 100,000-file fixture contains 4,400,000 source bytes and 1,011 directories. Host load varied from about 8 to 36.Completed spawn-to-ready first starts: 527.982, 537.464, 770.203 and 884.286 s (four completed starts because one extra start was needed to seed the restart set; median 653.833 s). Two other first-start attempts hit their 1,200 s and 1,800 s caps at MIME-backfill-pending. Across the four completed starts, median phase windows were Search 261.539 s, Files Home reconciliation 284.983 s, content-type backfill 44.084 s, and Notes upgrade backfills 6.079 s.
Clean restarts: 100.956, 80.165 and 79.054 s (median 80.165 s). Files reconciliation took 56.303, 43.945 and 44.197 s (median 44.197 s). SIGKILL restarts: 87.490, 92.989 and 85.901 s (median 87.490 s); Files took 50.263, 55.152 and 49.446 s (median 50.263 s). Clean and killed restarts kept 101,011 Index rows and 101,011 change-feed rows; MIME backfill did not run again.
/healthzrequest p95 medians were 17.080 ms on clean restarts and 16.416 ms after SIGKILL.Evidence for the unchanged-Home question: the base code’s stored fingerprint skips content hashing on clean and killed restarts, and the completed MIME version skips backfill. The repeated work is a full metadata traversal plus one
files_upload_pendingSELECT for each of 101,011 entries and 1,012 folder-reconciled UPSERTs. On a fresh Index, code-path counts are 100,000 full-file hash reads / 4.4 MB in Files reconciliation, then 100,000 prefix reads / 4.4 MB in MIME backfill; all fixture files are 44-byte text files. Files writes one folder transaction per directory (1,012 transactions, 2,136 batched Index statements at the base 48-row batch), produces 101,011 feed rows, and creates 1,012 folder rows. MIME uses 782 page transactions plus one finish transaction; MIME values were already correct, so it changed zero Index rows and emitted zero change-feed rows. Notes logged zero Notes scanned; the fresh database had five Notes metadata markers, one daily-log marker, and one projection job. The baseline server did not expose file-read counters, so these per-phase content and SQL operation counts are derived from its code and the deterministic fixture; process/procI/O totals are retained separately in the profile.Compile finding during
cargo clippy -p calternal-plugin-files --all-targets -- -D warnings: the folder-batchedquery_scalar(...).fetch_all(...)returnsVec<String>, but the result was assigned directly toHashSet<String>(index.rs:953, E0308). I am converting the returned vector into the set and rerunning the gate.The first full
cargo test -p calternal-plugin-filesrun finished with 281 passed, 4 ignored and 2 failed.dev_files_schema_upgrades_through_share_log_and_sidecar_migrationsexpected latest migration 33 and observed 34 after adding #1540.review_background_batches_resume_and_keep_job_visiblefailed its persisted Voice Memo item-ID assertion (actual05eef4c4-9fe1-4bbb-8e97-080737f6b040, expected8e92d000-02e3-40e6-a570-94cc37bd8243). I will update the migration assertion to 34 and investigate the identity mismatch without changing its expected identity.Evidence from #1233 cold-restart profiling on 2026-10-10: the pinned
dbd621434server had seeded and indexed the reduced Home (10,000 Files, 100 Photos). After a clean server stop, restoring the unchanged snapshot, and starting the server for a cold sample,/healthzreportedstate=migrating,phase=search-reconciliationat 18:59 UTC. This shows startup reconciliation runs again on the restored Home. I am continuing with the 10k fixture and enforcing the 60-second readiness gate; I will add whether this restart passed the gate and its elapsed time.Update to the #1233 evidence above: the clean restore-and-start did not reach readiness. After the full 60-second gate,
/healthzstill returned HTTP 200 withstate=migrating,phase=search-reconciliation; the benchmark kept zero cold samples. The attempt used the unchanged 10k Files/100 Photos snapshot and pinneddbd621434binary. I am reducing the fixture further and will retry before concluding whether cold coverage is possible within the gate.Further evidence from the same #1233 run: I restarted fixture creation with only 100 Files and 100 Photos, plus 20 regular Notes, 20 Tasks and 100 Transactions. At about three minutes after server start,
/healthzstill reportedmigrating / notes-upgrade-backfills. The smoke seed has 366 Daily notes and 10,980 Log entries (30 per day), so this phase is independent of the Files count; that is a likely contributor, not yet a confirmed cause. I am waiting for the initial seed readiness result before deciding whether to reduce Log volume further.Evidence from focused regression run on the existing partial Voice Memos migration test: moving three of twenty identical recordings and reopening Root changed ctime for the untouched recordings that shared the inode, while device, inode, size, mtime and bytes stayed the same. Startup rehashed them but also replaced their Item IDs. The reconcile path now uses ctime to force content verification, then keeps the Item ID when the saved fingerprint and verified digest match.
cargo test -p calternal-plugin-files tests::voice_memos_review::review_background_batches_resume_and_keep_job_visible -- --exactpasses (1 passed, 0 failed).Update: the 100-File/100-Photo fixture reached startup-complete after 1,236,269 ms. After a clean stop and snapshot, its unchanged restart reached
/healthz state=okin 20,055 ms. This is the first settled reduced snapshot; the next cold sample will still test the separate 60-second restart gate after dropping Linux caches.The full
cargo test -p calternal-serverexposed expected release-inventory assertions that still ended at Files migration 0033 (counts 38, 30, and 41). I updated those assertions and their module notes to include migration 0034;wire::upgrade_tests::production_round_9_schema_upgrades_once_with_original_receiptsnow passes (1 passed, 0 failed). The worktree had noapps/web/build, and the web build dependencies were not installed, so server tests use a gitignored local RustEmbed shell fixture for the required HTML, CSP inline-script hash, version marker and immutable chunk. No web source or lockfile changed.The ctime experiment found a hard-link regression: renaming one of 20 identical recordings changed ctime for untouched hard-linked siblings. The scan then rehashed those siblings and changed their item IDs. I removed ctime from the persisted token. Reconciliation now skips on matching device/inode/size/mtime and verifies that token again before commit. The offline-change regression replaces a same-size file through the atomic calternal-fs write path; the new inode forces a rehash. This keeps that data-safety case while avoiding hard-link false positives. A rewrite that keeps the same inode, size and mtime is not detected; calternal-fs writes replace files atomically.
Update from #1233: the fixture now settles once, then the unchanged 100-file Home + database snapshot restores within the 60-second cold-start gate. Seven cache-dropped clean restarts reached
/healthz state=okin 48.142, 48.155, 45.278, 47.649, 38.704, 49.695 and 49.293 seconds (p50 48.142 s, max 49.695 s). This fixture has 100 Files, 100 Photos, 20 Notes, 20 Tasks, 100 Transactions, 366 Daily notes and 10,980 Log entries. Its initial seed still required 1,236,269 ms for readiness. The 10,000-file cache-dropped fixture did not reach ready within 60 seconds and remains unsuitable for cold samples. The 100-file snapshot is bounded and documented; it is not a substitute for the 100,000-file target. No product-code fix was made in this benchmark.Progress: the changed MIME reader now counts every successful content read, including files whose stored MIME value was already correct. It only writes rows whose MIME projection changed. I committed this as
9a7e6ef2c. The reproducible local HDD profile is in bench/reconcile-1540.py (commit1ab75ab10).The perf VM was busy, so I am using the allowed local HDD at /mnt/hdd. Existing baseline evidence has four first-start samples, three clean restarts, and three SIGKILL restarts from the same baseline binary. I am collecting after-change samples next.
Measurement note: the local HDD profile completed two post-change cold starts (336.079s and 326.939s). Its third cold start timed out at the configured 1,800s limit with 69,076/100,000 files indexed; the live server remained responsive in
files-home-reconciliation, and the profile stopped the process and removed its disposable Home. Host load had risen to 37.42 at the timeout. This is an incomplete, contended measurement, not a product crash. I am collecting one replacement cold-start sample with a longer timeout plus the three restart samples.Measurement finding: the completed local HDD profile (100,000 files, 4.4 MB; host load varied from 7.78 to 37.42) gives clean restart 52.179s, 44.678s, 41.056s, median 44.678s. The baseline median is 80.165s, so this is 44.3% faster and misses the ≥50% acceptance threshold by ~4.6s. SIGKILL restart is 40.936s, 42.011s, 45.618s, median 42.011s versus 87.490s baseline, a 52.0% improvement. Clean/SIGKILL Files reconciliation reads 0 files/bytes, writes 0 index rows/SQLite index statements, and adds 0 feed rows; completion event count is 1.
Files Home still takes a median 9.697s across 1,012 folders, while the other startup phases take ~35s. I am reducing redundant per-folder generation and sidecar work, then I will run the profile and gates once on the final code. The replacement first-start outlier was 998.399s under load 25.80/28.09/22.61; most of this was search reconciliation at 617.609s. The three post-change cold starts including the two earlier samples have a 336.079s median versus 653.833s baseline, so cold-start acceptance passes despite the outlier.
Finding and fix: the final-code cold-start profile reached
files-content-type-backfill-pendingwithdetector_version=0,target_version=1, and a durable cursor atBench-file-010220.txt; no ready sample could be completed. The page write, progress reset, and completion transactions did not use the shared SQLite busy retry. I added full-transaction retries for these writes, kept thumbnail marker removal idempotent before page commit, and added a regression that holds a separate SQLite writer lock through the 5-second busy timeout. The test verifies the MIME row and detector version complete after the lock clears.Verification: focused retry test passed (
1 passed; 0 failed); fullcargo test -p calternal-plugin-filespassed (284 passed; 0 failed; 4 ignored). Commits:d4bd38c38(retry and test),44c5cc5de(profile captures worker warnings).Progress finding from the final binary
a6d7e8c64617166f9670857674bf4369685b425c5d9d7df749bcbf56f2b46c8e, profiled on local/mnt/hddbecause the shared perf VM was busy. The first two of three 100,000-file cold starts took 1,297.470s and 1,004.957s to/healthzready; the four baseline cold starts had a 653.833s median. Files Home took 731.191s and 551.055s in these two runs (baseline Files phase: 175.729s, 224.224s, 475.421s, 345.743s). Sample load averages were 5.03/8.43/10.22 and 7.99/10.70/12.01; load during the Files phase was 8.97/8.51/10.46 in the second run. The Files counters confirm these were full first scans (100,000 content reads, 4,400,000 bytes, 101,011 rows/feed rows, 1,021 index transactions); MIME read all 100,000 files and saved detector version 1. The third cold start is still running. These results fail the first-start +10% acceptance limit if they hold; I will finish the requested samples and report the phase-level evidence.The three first-start samples on final binary
a6d7e8c64617166f9670857674bf4369685b425c5d9d7df749bcbf56f2b46c8eare complete on local/mnt/hdd: 1,297.470s, 1,004.957s, and 1,150.593s (median 1,150.593s). Baseline first-start median is 653.833s, so this HDD round is 76% slower and exceeds the +10% acceptance limit. Files Home phases were 731.191s, 551.055s, 610.500s (median 610.500s; baseline median 345.733s). Search reconciliation was also higher in these samples (390.320s, 336.410s, 420.500s). Load averages before samples were 5.03/8.43/10.22, 7.99/10.70/12.01, and 7.96/8.76/9.56; the first Files phase sampled 17.14/13.57/11.65. Each cold start scanned 100,000 files, read 4,400,000 content bytes during Files and again in MIME, wrote 101,011 Index rows/feed rows in 1,021 transactions, and emitted one Files completion event. The last sample had 610.500s Files Home and 84.300s MIME. The clean and SIGKILL restart samples are running now. This is a real failure of the acceptance limit in the local HDD profile; the host disk is shared, so the environment is a likely contributor, not proof of a code regression.The three final-code clean restarts are complete: 48.349s, 36.234s, and 37.708s (median 37.708s). Against the baseline clean-restart median of 80.165s, this is a 53.0% improvement and meets the ≥50% acceptance threshold. Each pass enumerated the 100,000 files, classified all 100,000 as unchanged, read 0 file contents / 0 bytes, wrote 0 Index rows / 0 SQLite Index transactions, and added 0 change-feed rows; it emitted one completion event. The MIME detector marker stayed at version 1, so the MIME backfill did not run again. SIGKILL restart samples are in progress.
Correction to the Files-phase baseline median in my previous profile comment: the four baseline Files Home phase values (175.729s, 224.224s, 345.743s, 475.421s) have a median of 284.984s. The final-code median is 610.509s, which is 114.2% slower. The whole first-start medians remain 653.833s baseline and 1,150.593s final (76.0% slower). The stated per-sample values and restart results are unchanged.
Finished
Branch:
job/reconcile-1540Head:
6a11b68e1cca410bce6fe71587e82c7bb852c9afBuilt
(device, inode, size, mtime seconds, mtime nanoseconds)token. Unchanged files skip hashing and MIME detection. A same-size atomic replacement made while the server is down changes the inode and is reindexed.bench/reconcile-1540.pyfor a disposable 100,000-file HDD Home. It records phase time, load, health request latency, process I/O, Files/MIME counters, Notes counters, Index/feed rows, and completion events.Profile
All final samples used the local rotational HDD at
/mnt/hdd; the shared perf VM was initially reported busy. The final cold-start profile is noisy and fails the first-start limit. No paired baseline run was made on the perf VM.The first-start Files phase median was 610.509s versus 284.984s at baseline. Search reconciliation was also slower in these local runs. The shared
/dev/sdareached 75% I/O busy in a five-second sample during the profile, so the local timing does not isolate code from disk contention. The counters are stable: each cold start read 100,000 files / 4,400,000 bytes in Files and again in MIME; wrote 101,011 Index rows/feed rows in 1,021 transactions and 3,148 Index statements; and emitted one Files completion event. MIME completed 782 pages in 783 progress transactions, changed 0 rows, and emitted 0 extra events. Notes scanned 0 items.Every clean and SIGKILL restart classified all 100,000 files as unchanged: 0 file-content reads, 0 bytes read, 0 Index rows or Index transactions written, 0 feed rows, and one Files completion event. The detector version remained 1; MIME backfill did not repeat.
/healthzp95 during those restart samples was 6.1–8.5ms.Gates
Captured output:
cargo test -p calternal-fscompleted. The captured output includestest result: ok. 108 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 12.76sand two passing integration groups (1 test each). The wrapper clipped the final storage-group summary and did not retain the overall exit line; no failure appeared in the captured output.cargo fmt --checkpassed again after the documentation-only commit.Decisions and gaps
DESIGN.mddoes not pick an identity token. The implementation uses device, inode, size, and mtime; it excludes ctime because a hard-link rename changed sibling ctime and caused unchanged siblings to be rehashed. Same-inode, same-size, same-mtime in-place rewrites can therefore evade the token. Normal calternal-fs writes are atomic; the regression test covers an atomic replacement while the server is down.Starting the paired first-start comparison on worktree branch
job/rec1540b. The source under test isjob/reconcile-1540at6a11b68e1cca410bce6fe71587e82c7bb852c9af(also this worktree's starting HEAD); baseline source isorigin/devcommit4ebeba0b8. I read the issue and latest comments.perf-run statusis free. I will run three interleaved A/B pairs on the perf VM, each with a fresh 100,000-file Home and page cache dropped immediately before startup.Build-prep finding:
cargo build --release -p calternal-server --lockedstopped because the ignoredapps/web/build/directory is absent;RustEmbedthen reports thatFrontend::getis unavailable. The prior #1540 history records the same local build requirement. I will create the temporary shell fixture required by the embed macro in both source worktrees. This will not change tracked web files or either source revision.Build-prep finding: the exact
4ebeba0b8baseline build failed with unresolved Money/Notes APIs (migrations,recover_register_writes,configure_with_owned_paths,backfill_after_serving_with_owned_paths). Source inspection shows those functions are public in that checkout. The candidate and baseline builds share this job's Cargo target, so I suspect stale cross-worktree artifacts; I have already pinned and verified the candidate binary. I will clear only this job's target and rebuild the baseline from scratch before measuring.Build-prep update: the exact baseline source compiled successfully after clearing this job's shared Cargo target (
Finished release profilein 15m29s). This confirms the prior missing-API errors came from reusing target artifacts across the two worktrees. Both source revisions now have release builds; I am waiting for the current perf-run lock to release before staging the baseline and measuring.Comparison-scope finding:
4ebeba0b8is not an ancestor of6a11b68e1. The exact commit diff spans 75 paths (5,735 insertions and 21,137 deletions), including the #1540 Files work and changes brought in by the branch'sorigin/devmerge, such as Money and Notes startup code. I will report total first-start time plus every phase; the Files reconciliation phase will isolate the Files-specific cost. This keeps the requested source revisions unchanged.Source finding pending measurement: the fixture has 1,012 folders. The candidate calls
recover_movesonce before each folder scan, so an emptyfiles_pending_movequeue adds about 1,012 SELECTs during the pass; the baseline calls it before the scan, not per folder. In the same diff, the candidate replaces per-file upload-intent checks with one folder query and reuses the directory listing's FileStat. The phase timings will show the net cost before I change anything.Perf VM launch finding: the first A1 attempt seeded 100,000 files but produced no timing sample. Both release binaries exited before health readiness because
libsherpa-onnx-c-api.sowas absent from the VM loader path (exit status 127). I staged the two shared libraries from the exact local release build under/root/reconcile1540b/lib;lddnow resolves all dependencies for both hashes. I will rerun A1 withLD_LIBRARY_PATHset. No benchmark result from the failed attempt is included.Perf VM finding: baseline A1 seeded the fresh 100,000-file Home, but did not complete a valid first-start sample. Startup reached
files-content-type-backfill-pending;wait_for_content_type_backfillreturned false, and the harness correctly rejected the run. The captured server log tail contains repeatedsystem cron failederrors: SQLite pool timed out waiting for an open connection (timestamps 01:42:30–01:44:34 UTC). The harness removed the disposable Home andperf-runreleased the lock. This is recorded as a failed attempt, not as a timing; I am checking whether baseline MIME rebuild itself failed before retrying.Perf VM finding: candidate B1 reached
/healthzstartup readiness only to fall back tofiles-content-type-backfill-pending; the captured log tail reports initial Files reconciliation failed withIndex is busy; retry shortly. Candidate B2 reached readiness, but the branch profile attempted shutdown before saving the first-start sample and then failed its shutdown check. The valid timing/counters were therefore lost. I updated only the temporary measurement copy to save each first-start sample before shutdown, retain phase logs on shutdown failure, and set the restart-run count to zero because this comparison measures first starts only; the server code and restart correctness are untouched. The shared perf lock is now held bycalact-1557-ab(60-minute reservation from 02:17 UTC), so I am waiting before any further measured run.Candidate B2 repeat finding (perf-run lock held; load at launch 3.58, 3.99, 3.09): after seeding 100,000 files and dropping page cache, startup returned to MIME-pending because initial Files reconciliation failed. Targeted content-free diagnostics identify the failure at
crates/plugins/files/src/index.rs:2349:cause="sqlite_busy", followed byIndex is busy; retry shortly. The server log also records a periodic search integrity check failing with SQLite code 5. The profile rejected the run and saved no first-start sample. This is a startup failure under the measured workload, not a timing; I will run baseline A2 with identical diagnostics to check whether it also occurs there.Baseline A2 reached ready with a valid saved sample: first-start 433.035 s; load before start 2.16, 2.31, 2.57. The phase times were Search reconciliation 304.788 s, Files Home reconciliation 76.582 s, Files content-type backfill 22.002 s, Notes upgrade backfills 2.792 s, and all remaining phases together 26.871 s. This fresh fixture had 100,000 seeded files / 4,400,000 source bytes, 101,011
files_indexrows, 101,011 change-feed rows, and 1,012 folder markers. The baseline revision has no Files phase-total logs, so file content-read counts and Index write statements are not directly instrumented in this sample; its code hashes each of the 100,000 files and its MIME pass reads each row's prefix. The profile saved the sample before shutdown; the server then failed the profile's stop check. The current patched profile hasRECONCILE_RUNS=0, so it measures first starts only.Baseline A3 reached ready at 467.278 s (load before start 2.14, 3.34, 3.13). Phases: Search reconciliation 304.735 s, Files Home reconciliation 112.447 s, MIME backfill 21.126 s. It indexed 101,011 rows / 101,011 change-feed rows from the 100,000-file fixture. The two successful baseline samples so far have a median of 450.1565 s; median Files reconciliation is 94.5145 s, MIME backfill 21.564 s, and Search reconciliation 304.7615 s. A1 was rejected because the MIME projection remained pending. These are cold-cache perf-VM runs; no candidate timing is available yet because candidate startup did not complete Files reconciliation.
Candidate B3 (cold-cache 100,000-file Home; load before start 2.66, 2.84, 2.89) failed before
/healthzready. The profile rejected startup after MIME backfill stayed pending. Its captured logs include a Filessqlite_busywarning at 03:05:59 UTC, then system cron failed with SQLite code 5 (database is locked) at 03:06:14 UTC, and initial tag reconciliation failed at 03:06:19 UTC. This is the third candidate first-start attempt without a valid timing; the earlier B1/B2-repeat attempts also failed with FilesIndex is busy. It shows a repeatable startup reliability problem under this workload, but there is no candidate median to apply the >10% timing rule. I will make one final candidate attempt while the perf VM is free; I am not changing the feature code without comparable completed timings.Candidate B4 repeat reproduced the startup failure under the same cold-cache 100,000-file workload: at 03:26:34 UTC the Files warning reported
stage=crates/plugins/files/src/index.rs:2349 cause="sqlite_busy", followed by initial Files reconciliation failing withIndex is busy; retry shortly. System cron reported SQLite code 5 at 03:26:19 and 03:26:39 UTC; initial tag reconciliation then failed at 03:26:54. No candidate timing was saved. This is the fourth candidate startup failure. I will run the paired A4 baseline next, then stop within the four-hour cap.Final report — measurement only
Branch
job/rec1540b, HEAD6a11b68e1cca410bce6fe71587e82c7bb852c9af. No tracked files changed and no commit was made.Result
Three baseline first starts reached ready: A2 433.035 s, A3 467.278 s, A4 356.155 s. Baseline median: 433.035 s. One candidate first start reached ready: B3 repeat 425.699 s, which is 1.7% faster than that baseline median; compared with its paired A3 sample, it is 8.9% faster. The reported first-start slowdown was not reproduced in the completed sample.
The three-pair median rule remains inconclusive: only one candidate sample completed. Four other candidate attempts failed before readiness with SQLite busy/locked errors during initial Files/tag reconciliation. Baseline A1 also remained MIME-pending; three other baseline starts completed. The repeated candidate startup failures are recorded above in this issue and need follow-up. I made no source fix because the requested >10%-slower candidate median was not established and this is a measurement-only job.
Completed sample details
perf-run; the lock is free now.Method and gaps
I built both exact revisions and used the branch profile with a cold-cache drop after fixture seeding. The temporary profile copy persists the first-start sample before shutdown and runs first-start only (
RECONCILE_RUNS=0); this avoids losing the sample when the server stop check fails and omits restart measurements. It does not change server code or restart behavior. The first staged launch also required the two build-matched Sherpa/ONNX shared libraries because they were absent from the perf VM loader path; both binary hashes were verified.The requested six valid starts were not completed: repeated candidate SQLite contention prevented four candidate attempts from reaching ready, and the four-hour window ended before another complete three-pair set was possible. The completed data do not show a first-start slowdown, but they are insufficient for a candidate median. No change was made to restart correctness; this run did not measure restart reads.
Gates and cleanup
No source code changed, so Rust/web gates and adversarial tests were not run.
cargo cleanoutput:Removed 9260 files, 3.4GiB totalThe temporary embed fixture, baseline worktree, binaries, scripts, JSON results, and perf-VM staging directory were removed.
git status --shortis clean.Starting rec1540c on branch job/rec1540c, base/head
6a11b68e1c. I will reproduce the SQLite startup failure, trace write-pool contention, fix it with a real-file concurrent startup regression, and run the requested paired profile and gates. No pushes, deployments, issue closure, or macOS VM use.Local HDD baseline
4ebeba0b8, two fresh 10,000-file Homes: spawn-to-ready 64.445 s and 54.414 s, zero startup failures. Host load is high, so these are reproduction checks rather than the paired decision sample. Added profile failure retention and optional content-free phase tracing (f11ebb27d). Candidate release build is still running with four build jobs. Diagnostic additions will record the failing Files statement, actual busy_timeout, authority checkout state, and checkpoint phase boundaries. The perf VM is currently reserved by calact2-1557-smoke.Diagnostic candidate (
6a11b68e1behavior plus content-free lock logging) completed three fresh 10,000-file local HDD starts: 58.933, 53.970, 53.351 s; and three fresh 30,000-file starts: 114.334, 99.071, 85.534 s. Zero failures in these six runs. Files reconciliation times at 30k: 29.243, 16.652, 20.183 s. A checkpoint pass took 2.444 s; no Files BUSY diagnostic fired. This does not reproduce the reported 100k cold-cache failure. The perf VM remains actually locked (nonblocking flock check exit 1), so I am running one local 100k diagnostic start. Added a 25 ms scheduling budget between bounded UPSERT statements; the regression verifies committing a prefix, releasing the writer, and resuming without replay. The concurrent startup fixture uses fixed offline import files so test setup does not pay per-upload quota admission walks. No production filesystem boundary changed.Cold perf-VM diagnostic reproduced SQLite contention and completed at 528.636 s. Failing statement: files_index UPSERT on the ordinary writer; PRAGMA busy_timeout reported 5000 ms; authority pool had its connection idle. At 04:23:30.462 UTC maintenance began TRUNCATE, Files BUSY appeared at 04:23:35.483, and maintenance finished at 04:23:36.352. Another TRUNCATE began at 04:23:52.199, Files BUSY appeared at 04:23:57.211, and maintenance finished at 04:24:08.024. That pass lasted 21.438 s including its preceding PASSIVE work. The existing three retries happened to recover this run, but checkpoint I/O can exceed that finite budget. Zero busy_timeout on maintenance bounds lock acquisition, not I/O while it owns SQLite’s writer.
Fix in progress: keep foreground retries bounded, repeat resumable background retry cycles after BUSY/LOCKED/checkout contention, release namespace guards between passes, reserve Index writes with BEGIN IMMEDIATE, and keep the committed 25 ms / 512-row scheduling bounds. Saved fingerprints and the MIME cursor preserve restart reuse. The controlled real-file regression uses a FULL writer held for 4 s with a 20 ms ordinary timeout, then runs Notes backfills alongside resumed Files/MIME workers. It passes with the fix (5.94 s); I am checking that it fails with the previous implementation. Decisions: background retries continue until cancellation, pool closure, or a non-transient error; checkpoint durability and reclamation policy remain unchanged.
Root-cause fix committed as
98f243baf; origin/dev merged once asf32133b48. The merge conflict was only contracts/perf/ratchet.json; resolved from the exact combined exception ledger (22125 total; 16201 io.unresolved-call), with no raised ceiling.The real SQLite regression holds a FULL writer for four seconds with ordinary busy_timeout=20 ms, then overlaps resumed reconciliation and MIME work with Notes backfills. Before the fix it fails with Failure(503, ServiceUnavailable, "Index is busy; retry shortly") (exit=101). With the fix:
Focused clippy for calternal-db and calternal-plugin-files also passed. Background retries reuse the shared bounded retry helper, releasing namespace guards between failed passes and resuming durable MIME cursors. Multi-statement ordinary writes reserve BEGIN IMMEDIATE before reads. Reconciliation retains its 512-row/25 ms soft transaction budget. Foreground retry policy and checkpoint policy remain unchanged.
The first locked cold baseline sample is running now. Final per-crate gates and the candidate release build are in progress. The bench now reuses tests/perf/upload_scale.py's ProcessSampler for CPU/RSS rather than adding duplicate sampling logic.
Cold perf-VM baseline A1 completed under
perf-run calternal rec1540c-A1 15: fresh 100,000-file Home, exact baseline4ebeba0b8prebuilt binary, cache dropped after seed under the lock.The lock was released after A1. Candidate B1 follows when the post-origin/dev release build finishes. All outputs are retained in this worktree under target/tmp and the own perf-VM results directory.
Startup-path audit found that initial Tag reconciliation directly returned SQLite errors. Earlier #1540 evidence included "initial tag reconciliation failed" after the same code-5 contention. The server now applies the shared resumable background retry only to this startup call; ordinary Tag operations retain their existing policy. No Tag crate behavior or API contract changed.
Follow-up commit:
0a5fdc44c. Real SQLite regression:The test first proves the unwrapped Tag pass returns a BUSY error while a FULL writer is held, then proves startup resumes beyond the bounded retry cycle and a closed pool stops retries. Final clippy passed:
The earlier full crate gates all passed; the updated server is being verified again and its release binary rebuilt. Benchmark samples now retain startup diagnostics and reject failed initial Files/Search/Notes/Tag phases even when /healthz reports ready. Supplemental baseline A1 (303.841s) predates this diagnostic retention and will not enter the final paired median. The complete final round will use the updated profile for both binaries. Perf VM currently belongs to calcold-1560; local load5 is above the owner's measurement limit, so no local comparison will substitute for it.
Decision: startup-only Tag retries stop on permanent errors or pool closure; foreground policy and Tag projection design are unchanged.
Final Rust gate evidence for rec1540c, head
9dd286c8eda1f330ff2c250737fe48bdfab4a26f. The benchmark-only last commit does not change Rust source. All gates use CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and the worktree target/tmp for TMPDIR. Test threads are limited to four. Output below is verbatim.cargo fmt --check; echo exit=$?cargo clippy -p calternal-fs -p calternal-plugin-files -p calternal-server --all-targets -- -D warnings; echo exit=$?cargo test -p calternal-plugin-files -p calternal-fs -p calternal-server; echo exit=$?cargo clippy -p calternal-db --all-targets -- -D warnings; echo exit=$?cargo test -p calternal-db; echo exit=$?First two final cold-cache pairs completed under the rec1540c-pairs perf-run lock. Each uses a fresh 100,000-file Home and the same updated profile.
4ebeba0b89dd286c8ePair 3 and the candidate restart samples are still running. There is no final median or READY decision yet.
Each completed sample has one shared-environment diagnostic: native-media startup reports
cause=Launch. Read-only verification foundcalternal-media-sandboxabsent from PATH. This non-SLOW finding is filed separately as #1572. No environment change is made in the middle of the paired phase. It does not abort these text-Home startup phases.Candidate RSS varies: B1 mean/peak 350978546/525017088 bytes, B2 249630110/432001024 bytes. All resource samples will be reported, including this spread.
The first requested 100,000-file paired round at
9dd286c8ecompleted with a merge blocker: candidate B3 failed its initial Search scan. B1/B2 passed; all three baseline starts passed. The profile now checks initial-phase errors after readiness and retained this failure instead of counting a false success.Exact error:
Search's actor converted the typed database error into its public string reply before the server could classify contention. I am adding a startup-only resumable Search hook with a typed retry hint, using the same shared background retry helper. The existing ordinary maintenance entry point keeps its bounded behavior. This minimal public hook in calternal-search is necessary for the server to resume the initial scan; saved fingerprints retain committed progress. A real SQLite WAL regression holds another connection's writer lock beyond multiple bounded retry cycles and checks completion and one Search document after release.
The old round is NOT READY. Its failure will remain in the report. A new three-pair round and three clean restarts must pass on the fixed head.
Committed
1e707bbc0: the server now uses a startup-only Search hook that retains typed SQLite BUSY/LOCKED and pool-checkout contention across the actor reply. It retries through calternal_db::retry_when_background. Each retry is a fresh actor command; ordinary maintenance retains its previous bounded contract. The actor can process queued work during shared backoff. Permanent errors and stopped actors remain terminal.The real WAL regression passed: another connection held BEGIN IMMEDIATE beyond multiple retry cycles with a 20 ms ordinary busy_timeout; the initial non-resumable call failed with database is locked, the startup hook completed after release, the Note manifest row existed once, and its Search result existed once. Existing test expectations were not changed. Search/server clippy passed. The fresh release build and final per-crate gates are running; a new three-pair perf round must pass before READY.
Earlier paired round at
9dd286c8e(superseded; NOT READY)New verification round is running at committed head
1e707bbc03. Candidate binary SHA-256: 4d0a76aecf8eed3ad8004f3b09a55d786f78096a7277eb5f0ed4da7568cd8dc2. The VM lock is held by perf-run calternal rec1540c-searchfix; entry load average was 0.08, 0.13, 0.77. Cold first starts drop page caches only inside this lock. No VM build or Mac access.A1 baseline
4ebeba0b8passed: 277.912 s; mean CPU 51.02%; mean RSS 258679688 bytes; peak RSS 451133440 bytes; failures 0; graceful SIGTERM exit 0. Candidate B1 is preparing its fresh Home. The six first starts and three clean candidate restarts must all complete before READY.The final format and clippy gates passed for fs, Files, server and Search. Full per-crate tests are in progress. Web check passed with zero errors and warnings. The previous post-merge full web test passed 318 files / 2316 tests; no web source changed afterward.
Final Rust gates at
1e707bbc03. All exits are 0. No existing expectations changed. Existing ignored tests retain their explicit-profile/live-app reasons. Full verbatim output follows.cargo fmt --check; echo exit=$?cargo clippy -p calternal-fs -p calternal-plugin-files -p calternal-server -p calternal-search --all-targets -- -D warnings; echo exit=$?cargo test -p calternal-plugin-files -p calternal-fs -p calternal-server -p calternal-search; echo exit=$?Two completed pairs on
1e707bbc0; all four first starts passed and stopped with SIGTERM exit 0. The third pair and three clean restarts remain.All six first starts on the fixed head passed with no startup failures and graceful SIGTERM exit 0. Clean restarts are running.
4ebeba0b8seconds1e707bbc0secondsMedians: baseline 283.989 s; candidate 241.823 s (-14.85%). This passes the first-start timing rule. READY still requires all three clean restarts and zero unchanged-file content reads / Index writes.
Final verification:
1e707bbc03Three interleaved cold first-start pairs ran on the perf VM under
perf-run calternal rec1540c-searchfix 50. Entry load average: 0.08, 0.13, 0.77. The lock covered cache drops and every measured run. Candidate SHA-256:4d0a76aecf8eed3ad8004f3b09a55d786f78096a7277eb5f0ed4da7568cd8dc2. Baseline4ebeba0b8SHA-256:58ca0cefe0ffe215a31f0ec2844a4c2979806fd2cc4574925959c61be0d84e99. Each fresh Home has 100,000 files / 4,400,000 source bytes. No build ran on the VM. No macOS VM was used.First-start medians (p50): baseline 283.989 s; candidate 241.823 s; candidate 14.85% faster. With three samples, nearest-rank p95 is the largest sample: baseline 296.526 s; candidate 246.793 s. All six first starts ended with graceful SIGTERM exit 0. No startup phase failed.
Every restart checked 100,000 unchanged files and recorded zero source bytes read, Index transactions, Index statements and change-feed rows. Each clean restart followed a proven SIGTERM exit 0. Clean restart 3 was then intentionally SIGKILLed to begin the crash-restart series. The three extra SIGKILL restarts also passed. The profile exited 0 with
failed_profile_runs=0.The worst candidate peak RSS was 490.84 MiB, versus 439.92 MiB baseline (+11.58%, below the 15% peak-RSS threshold). Mean startup RSS averaged across the three samples was slightly lower on the candidate. Median health p95 was 5.698 ms candidate / 6.112 ms baseline. Startup CPU, RSS and latency are recorded per sample above.
docs/perf/baseline.jsonhas no directly comparable #1540 cold-start entry; the required paired4ebeba0b8binary is the comparison baseline.Earlier attempts and failures
The superseded
9dd286c8eround is retained below. B3 failed its initial Search scan with SQLite code 5. It was not counted as a successful startup, and no restarts ran on that failed attempt. The Search hook and real SQLite regression were added in1e707bbc0before the new round.Earlier paired round at
9dd286c8e(superseded; NOT READY)Earlier local reproductions, on the busy build host and not used for the READY comparison: baseline 10,000 files: 64.445 / 54.414 s; diagnostic candidate 10,000 files: 58.933 / 53.970 / 53.351 s; diagnostic candidate 30,000 files: 114.334 / 99.071 / 85.534 s; diagnostic candidate 100,000 files: 382.880 s. Those completed without terminal startup failure. The perf diagnostic 100,000-file run completed in 528.636 s and captured the checkpoint/Files contention. The earlier supplemental baseline was 303.841 s; it predates the initial-phase error guard and is excluded from the comparison. The pre-fix focused real-SQLite concurrency regression failed with Files 503; after the fix it passed. The new Search regression deliberately verifies the old bounded entry point returns database-is-locked before the resumable entry point completes.
READY
1e707bbc03Built
Fixed startup contention on branch job/rec1540c. Head:
1e707bbc0382bb0999663c77a36ab7be209833d8.Root cause and evidence
The maintenance connection's TRUNCATE checkpoint held SQLite's exclusive writer lock through cold-disk I/O. Nonwaiting acquisition did not bound the time after acquisition. The ordinary writer had busy_timeout=5000 ms; the authority pool was idle. The diagnostic Files statement was
files_index_upsert(INSERT INTO files_index ... ON CONFLICT ...). The checkpoint took 7,824 ms and 21,438 ms in the captured passes. The corresponding Files contention warnings occurred at 04:23:35.483500 UTC and 04:23:57.211766 UTC. Debug phase boundaries identify the maintenance connection without SQL values or User paths.The first paired round exposed a second path that lost the typed error across the Search actor reply:
That failed
9dd286c8erun remains in the evidence. It led to the Search hook in the final head; no failed startup is counted as a success.Measurement
Three new interleaved 100,000-file pairs passed with zero startup failures:
4ebeba0b8sMedians: baseline 283.989 s, candidate 241.823 s, 14.85% faster. All first-start shutdowns were SIGTERM exit 0.
Clean restarts: 30.672 / 29.773 / 29.829 s. Each followed SIGTERM exit 0 and checked 100,000 unchanged files with zero content reads, source bytes, Index rows, Index transactions, Index statements and change-feed rows. Extra SIGKILL restarts: 29.733 / 29.709 / 30.314 s, also with zero reads/writes and zero startup failures. The complete profile exited 0 with
failed_profile_runs=0.Per-sample CPU, mean/peak RSS and health p50/p95, plus every earlier attempt and failure, are in the preceding measurements comment and
artifacts/rec1540c/measurements.md. Worst peak RSS increased 11.57%, below the 15% threshold; median health p95 fell from 6.112 to 5.698 ms. The required paired binary is the baseline because docs/perf/baseline.json has no comparable #1540 cold-start entry.Expected work: first start is O(files + folders), with one content read per changed file and bounded Index transactions; restart is O(metadata entries), with zero content reads and zero unchanged-file Index writes. The measured fixture contains 100,000 files and 4,400,000 source bytes.
Files
bench/reconcile-1540.pycrates/calternal-db/src/checkpoint.rscrates/calternal-db/src/lib.rscrates/calternal-db/src/sqlite.rscrates/calternal-search/src/indexer.rscrates/calternal-server/src/wire.rscrates/plugins/files/src/index.rscrates/plugins/files/src/lib.rscontracts/perf/adoption-1058.jsoncontracts/perf/exceptions.jsoncontracts/perf/ratchet.jsonThe three perf ledgers contain exact live pins/counts. No rule was weakened. No dependency or migration was added. The required origin/dev integration ran once before final gates.
Gates (verbatim terminal summaries)
Full final Rust output was posted verbatim in the earlier gates comment. Raw output is retained in artifacts/rec1540c. All commands used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and the worktree TMPDIR. Tests used RUST_TEST_THREADS=4.
cargo fmt --check; echo exit=$?cargo clippy -p calternal-fs -p calternal-plugin-files -p calternal-server -p calternal-search --all-targets -- -D warnings; echo exit=$?cargo test -p calternal-plugin-files -p calternal-fs -p calternal-server -p calternal-search; echo exit=$?The added calternal-db helper also passed its own clippy/test gates (full raw logs retained):
bun run --cwd apps/web check; echo exit=$?Post-merge full web suite (web source unchanged afterward),
cd apps/web && bun run test --maxWorkers=2; echo exit=$?:Frozen Bun install, production web build and final server release build also exited 0. The focused Search contention regression passed. No existing test expectation changed. Existing ignored tests retain their explicit-profile/live-app reasons; live apps ran in their parent regression.
Decisions
Known gaps and cleanup
The perf VM has no native-media launcher. Both baseline and candidate recorded its Launch error; it did not abort this text-file startup profile. Filed separately as #1572. This fixture does not measure media work.
The writer budget is a scheduling budget, not a hard limit on SQLite or checkpoint I/O. Resumable retries cover longer contention; cancellation and permanent errors remain terminal.
cargo cleanexited 0 and reported:Web build output and local candidate ELF copies were removed. The working tree is clean. The measurement lock was released. A new job then took the perf lock; the bounded cleanup attempt exited 124, so
/root/rec1540cremains with its verified cleanup script and copied binaries. The cleanup wait has ended; all raw results are already retained locally. No macOS VM was used.READY
1e707bbc03Starting #1540 follow-up on branch
job/rec1540cat1e707bbc0382bb0999663c77a36ab7be209833d8; merge base withorigin/dev:0c712b98c164b6734d54f63485012bd1a11005b4. I will resolve the 23 newio.unresolved-callfindings in Files and Money without changing retry or batching behavior, then verify that the ratchet returns to at most the merge-base values.Finding: the committed ratchet is 9 over the merge-base ceiling in both total entries and
io.unresolved-call(22,137 vs 22,128; 16,213 vs 16,204). The 21 Files additions are seven in-memory calls (RecordAttempt::Upsert,Box::new,FileStat::fingerprint,Instant::now,len,is_some_and, andelapsed) repeated across three API surfaces. The two Money additions are the samemoney.find_foldercall in two operations; it is the #1554 cold admission check, bounded to at most 200 Budget headers and covered bycold_register_admission_rejects_foreign_and_missing_budgets. The initial guard run passes against the committed ratchet with 22,137 scoped exceptions, confirming the raised ceiling hides these new exact scopes.Verification finding: the first
cargo clippy -p calternal-db --all-targets -- -D warningsstopped while compiling a dependency because the shared sccache process tried to create its temporary directory under/home/kayg/Developer/calternal-wt/rev1570/target/tmp, which is outside this worktree and does not exist. This is an environment failure before crate linting. I am rerunning with the Rust compiler wrapper bypassed and this worktree'starget/tmpset explicitly.Finished
Branch:
job/rec1540cHEAD:
c570131960887d67b642346070acecfed06a555bMerge base:
0c712b98c164b6734d54f63485012bd1a11005bChange
Removed the 23 newly adopted exception entries and re-pinned the affected source hashes with the existing adoption tooling. The ratchet is below merge-base values: total
22128 -> 22114(-14);io.unresolved-call16204 -> 16190(-14). No ratchet values were raised by hand.Added exact-site
purebindings for in-memory calls in Files indexing and a test-pinnedbounded-owner-metadata-readbinding for the Money cold ownership check. The retry and batching code and its behavior were not changed. The Money binding remains limited to the existing test-covered lookup: at most 200 Budget headers and no Month history.Adoption output:
Adopted 9908 exact #1058 sites; ordinary future ratchet remains activePerf guard output:
perf-lint: PASS; 0 violations; 22114 scoped exceptionsFiles
contracts/perf/adoption-1058.jsoncontracts/perf/exceptions.jsoncontracts/perf/ratchet.jsoncontracts/perf/registry.jsonscripts/perf_guards/rules.pyscripts/perf_guards/test_rules.pyGate output (verbatim summaries)
cargo fmt --check: no stdout or stderr; exit status 0.cargo clippy -p calternal-db --all-targets -- -D warnings:cargo test -p calternal-db:cargo clippy -p calternal-plugin-files --all-targets -- -D warnings:cargo test -p calternal-plugin-files:cargo clippy -p calternal-search --all-targets -- -D warnings:cargo test -p calternal-search:cargo clippy -p calternal-server --all-targets -- -D warnings:cargo test -p calternal-server:python3 -m pytest scripts/perf_guards:Decisions, gaps, cleanup
The design docs do not name a capability for this Money ownership check. I added a narrowly scoped, exact-site capability with its ownership regression test because this checked lookup is bounded and excludes Month history; it does not generalize to other source reads. No measured code path changed, so I did not re-measure, as directed. No remaining gaps for this ratchet fix.
Ran
cargo clean(Removed 22207 files, 17.8GiB total) and removedapps/web/buildandapps/web/.svelte-kitafter the Server tests.Deployed to calternal.cloud in merge round 40 (build
c0f8da461, 01:38 IST 2026-10-12). Health state ok, /readyz 200, served-page check passed. Pre-deploy backup taken.