Server: does every restart re-index the whole Home? (510 s first index)

kayg commented

2026-10-08 05:46:21 +00:00

Owner

Problem

API audit #1374 on the perf VM: after seeding a Home, startup indexing took 510 s before routes were ready. It is unknown whether every restart repeats this work or only the first start on new data. A deploy restarts production; long re-indexing would make every deploy slow and burn CPU.

Work

On the perf VM with the shared release server: seed the #1374 fixture Home, start, wait until indexing is done, stop, start again. Measure for each start: time to /healthz ready, time until Search/Tasks/Notes routes return complete data, CPU seconds, disk read bytes. Do the same with a Home that changed by 10 files between starts. Expected: a restart over unchanged data does no full re-index (index state and file fingerprints persist; only changed files are reprocessed), and routes serve while background work continues (#1161). Fix what does not meet that, with a regression test that restarts over unchanged data and asserts no full re-index (count of files reprocessed = 0). Gates: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test for touched crates; quote verbatim.

## Problem API audit #1374 on the perf VM: after seeding a Home, startup indexing took 510 s before routes were ready. It is unknown whether every restart repeats this work or only the first start on new data. A deploy restarts production; long re-indexing would make every deploy slow and burn CPU. ## Work On the perf VM with the shared release server: seed the #1374 fixture Home, start, wait until indexing is done, stop, start again. Measure for each start: time to /healthz ready, time until Search/Tasks/Notes routes return complete data, CPU seconds, disk read bytes. Do the same with a Home that changed by 10 files between starts. Expected: a restart over unchanged data does no full re-index (index state and file fingerprints persist; only changed files are reprocessed), and routes serve while background work continues (#1161). Fix what does not meet that, with a regression test that restarts over unchanged data and asserts no full re-index (count of files reprocessed = 0). Gates: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test for touched crates; quote verbatim.

kayg commented

2026-10-08 13:01:52 +00:00

Author

Owner

Started work on job/restartidx-1450, based on d395b6e25da20d8c67b8b9fc1a593b813939c8a7 (current origin/dev was af202e1514955ddf0f8b4b3d2503c780d807ed84 at inspection). I am tracing startup indexing and persisted fingerprints, then will add a restart regression and measure the required scenarios.

Started work on `job/restartidx-1450`, based on `d395b6e25da20d8c67b8b9fc1a593b813939c8a7` (current `origin/dev` was `af202e1514955ddf0f8b4b3d2503c780d807ed84` at inspection). I am tracing startup indexing and persisted fingerprints, then will add a restart regression and measure the required scenarios.

kayg commented

2026-10-08 13:05:30 +00:00

Author

Owner

Source finding: reconcile_at_start() currently routes through check_and_repair_inner(). That full integrity pass visits every Home file, calls indexed_file_for_actor() to read/hash/parse it, and only then compares the result with search_manifest (crates/calternal-search/src/indexer.rs, initial_reconcile, process_scan_visit). Existing Search rows avoid writes, but not the full content pass. The #1374 rerun measured 510,067 ms from process start to complete Index counts on 2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 Files and 24 Photos. I am preserving the explicit deep integrity check while changing ordinary restart reconciliation to use persisted file fingerprints.

Source finding: `reconcile_at_start()` currently routes through `check_and_repair_inner()`. That full integrity pass visits every Home file, calls `indexed_file_for_actor()` to read/hash/parse it, and only then compares the result with `search_manifest` (`crates/calternal-search/src/indexer.rs`, `initial_reconcile`, `process_scan_visit`). Existing Search rows avoid writes, but not the full content pass. The #1374 rerun measured 510,067 ms from process start to complete Index counts on 2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 Files and 24 Photos. I am preserving the explicit deep integrity check while changing ordinary restart reconciliation to use persisted file fingerprints.

kayg referenced this issue from a commit

2026-10-08 14:03:51 +00:00

Skip unchanged Search sources on restart (#1450)

kayg referenced this issue from a commit

2026-10-08 14:03:51 +00:00

Add persisted-index restart benchmark (#1450)

kayg referenced this issue from a commit

2026-10-08 14:03:51 +00:00

Keep restart benchmark scratch paths fixed (#1450)

kayg referenced this issue from a commit

2026-10-08 14:03:51 +00:00

Separate Search route readiness from indexing completion (#1450)

kayg referenced this issue from a commit

2026-10-08 14:03:51 +00:00

Assert restart source work in benchmark (#1450)

kayg referenced this issue from a commit

2026-10-08 14:03:51 +00:00

Fix restart benchmark phase assertions (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Document Search restart reconciliation decisions (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Fix restart fixture generator arguments (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Encode fixture emoji safely for Bun Python strings (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Correct Python fixture argument slice (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Detach perf-run launcher from SSH session (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Avoid queuing restart phases behind another benchmark (#1450)

kayg referenced this issue from a commit

2026-10-08 14:31:41 +00:00

Check perf availability before fixture transfers (#1450)

kayg commented

2026-10-08 14:44:35 +00:00

Author

Owner

Baseline first-start measurement for #1450 on shared release af202e151 (server SHA f1a597f4681e4f4e08d383c9f065333e16f9293607645c74b1d80637ba239d80), with 2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 text Files, and 24 Photos:

/healthz: 1.521 s; /readyz: 4.104 s.
Notes and Tasks returned the expected pages with HTTP 200 in 19.308 s.
Search returned its one expected result with HTTP 200 in 70.169 s while indexing continued; Search indexing completed in 531.071 s.
Server CPU was 201.02 s, peak sample RSS was 713,547,776 bytes, read_bytes was 0, and rchar was 3,705,476,222 bytes. The zero physical reads with high rchar is consistent with a warm page cache after fixture seeding.

This confirms that the routes can return data while background Search indexing continues. This pre-fix shared binary does not emit the reprocessed-file counter. I am waiting for a free perf lock before the unchanged-restart and ten-file-change phases.

Baseline first-start measurement for #1450 on shared release `af202e151` (server SHA `f1a597f4681e4f4e08d383c9f065333e16f9293607645c74b1d80637ba239d80`), with 2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 text Files, and 24 Photos: - `/healthz`: 1.521 s; `/readyz`: 4.104 s. - Notes and Tasks returned the expected pages with HTTP 200 in 19.308 s. - Search returned its one expected result with HTTP 200 in 70.169 s while indexing continued; Search indexing completed in 531.071 s. - Server CPU was 201.02 s, peak sample RSS was 713,547,776 bytes, `read_bytes` was 0, and `rchar` was 3,705,476,222 bytes. The zero physical reads with high `rchar` is consistent with a warm page cache after fixture seeding. This confirms that the routes can return data while background Search indexing continues. This pre-fix shared binary does not emit the reprocessed-file counter. I am waiting for a free perf lock before the unchanged-restart and ten-file-change phases.

kayg referenced this issue from a commit

2026-10-08 14:59:44 +00:00

Wait between restart phases without queuing (#1450)

kayg commented

2026-10-08 16:17:09 +00:00

Author

Owner

Measured the seeded #1374-shaped Home (2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 Files, 24 Photos) on the shared pre-fix release server (af202e151). The first start reached /healthz in 1.515 s and /readyz in 6.646 s; Notes and Tasks were complete by 21.861 s, Search returned its expected hit at 77.724 s, and Search indexing completed at 523.773 s. CPU was 213.78 s; physical reads were 4,096 bytes and logical reads were 3.78 GB.

The unchanged restart reached /healthz in 1.852 s and /readyz in 4.426 s; Search, Notes and Tasks were complete in 4.426 s and indexing finished in 9.623 s. CPU was 7.82 s, RSS 243,421,184 bytes and physical reads 143,015,936 bytes. This confirms the 523.773-second first pass does not repeat on unchanged data; routes are complete before the restart scan ends. The candidate regression test independently asserts zero source files reprocessed on unchanged restart and one after changing one file. The 10-file-change perf phase is still pending the shared VM lock.

Measured the seeded #1374-shaped Home (2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 Files, 24 Photos) on the shared pre-fix release server (`af202e151`). The first start reached `/healthz` in 1.515 s and `/readyz` in 6.646 s; Notes and Tasks were complete by 21.861 s, Search returned its expected hit at 77.724 s, and Search indexing completed at 523.773 s. CPU was 213.78 s; physical reads were 4,096 bytes and logical reads were 3.78 GB. The unchanged restart reached `/healthz` in 1.852 s and `/readyz` in 4.426 s; Search, Notes and Tasks were complete in 4.426 s and indexing finished in 9.623 s. CPU was 7.82 s, RSS 243,421,184 bytes and physical reads 143,015,936 bytes. This confirms the 523.773-second first pass does not repeat on unchanged data; routes are complete before the restart scan ends. The candidate regression test independently asserts zero source files reprocessed on unchanged restart and one after changing one file. The 10-file-change perf phase is still pending the shared VM lock.

kayg commented

2026-10-08 17:17:36 +00:00

Author

Owner

Built

Search startup now reconciles persisted Files fingerprints against Search manifest rows. An unchanged source avoids opening and hashing its content. Missing or legacy fingerprint data still falls back to indexing; fresh-index, writer-recovery and explicit integrity paths keep their deep checks.
Added restart_uses_saved_file_fingerprints_and_refreshes_only_changed_source. It restarts a persistent index, asserts 0 source files reprocessed when unchanged, then changes one Note and asserts exactly 1 reprocessed.
Added bench/restart-index-1450.mjs for the #1374 Home shape. It records health/readiness, Search/Notes/Tasks completeness, indexing completion, CPU, RSS, disk reads and load average under separate perf-run locks.

Perf VM measurements

Both runs used the shared pre-fix release binary from af202e151 (SHA-256 f1a597f4681e4f4e08d383c9f065333e16f9293607645c74b1d80637ba239d80) with 2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 Files and 24 Photos.

First start: /healthz 1.515 s; /readyz 6.646 s; Notes 16.813 s; Tasks 21.861 s; Search result 77.724 s; Search indexing complete 523.773 s; CPU 213.78 s; RSS 701,366,272 bytes; physical reads 4,096 bytes; logical reads 3,781,688,704 bytes.
Unchanged restart: /healthz 1.852 s; /readyz 4.426 s; all routes complete 4.426 s; Search indexing complete 9.623 s; CPU 7.82 s; RSS 243,421,184 bytes; physical reads 143,015,936 bytes; logical reads 281,633,739 bytes.
The unchanged run confirms the 523.773-second first pass did not repeat. The pre-fix binary did not emit the new reprocessed-file counter, so those benchmark counts are null; the candidate regression test verifies 0 and 1 directly.
The 10-file-change phase did not run. After 60 minutes without an available perf slot, the profile recorded perf-vm-still-busy-after-60m-changed. No lock was held by this job during its wait. The candidate release build was not available in this job, so these are baseline measurements, not candidate performance numbers.

Expected reconciliation work is O(searchable paths) metadata traversal and bounded manifest lookup; source bytes and projection work are limited to changed or unverified files.

Gates

cargo fmt --check
(exit 0; no output)

cargo clippy -p calternal-search --all-targets -- -D warnings
    Checking calternal-search v0.0.1 (/home/kayg/Developer/calternal-wt/restartidx-1450/crates/calternal-search)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.86s

cargo test -p calternal-search
 test result: ok. 61 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 36.08s
 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.95s
 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.45s
 test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 611.14s
 test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
 test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 4.15s
 test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

node --check bench/restart-index-1450.mjs and git diff --check also exited 0 with no output. cargo clean: Removed 8225 files, 4.0GiB total.

Decisions and gaps

DESIGN does not prescribe the ordinary-startup fingerprint comparison sequence. I used persisted Files metadata plus Search manifest metadata; deep verification remains for missing metadata and integrity/recovery paths.
UX gaps: not applicable; this is a backend indexing change.
The changed-10-files perf phase and candidate-release perf numbers remain unmeasured because the shared VM stayed locked. After a matching release binary is available, rerun bun bench/restart-index-1450.mjs with CALTERNAL_SERVER_RELEASE_BIN, RESTARTIDX_SERVER_REV, and the media runtime set.

Files: crates/calternal-search/src/indexer.rs, bench/restart-index-1450.mjs.
Head: 45c9cf134345f26fd40b9c64fa7b8e2b783376b2.

## Built - Search startup now reconciles persisted Files fingerprints against Search manifest rows. An unchanged source avoids opening and hashing its content. Missing or legacy fingerprint data still falls back to indexing; fresh-index, writer-recovery and explicit integrity paths keep their deep checks. - Added `restart_uses_saved_file_fingerprints_and_refreshes_only_changed_source`. It restarts a persistent index, asserts 0 source files reprocessed when unchanged, then changes one Note and asserts exactly 1 reprocessed. - Added `bench/restart-index-1450.mjs` for the #1374 Home shape. It records health/readiness, Search/Notes/Tasks completeness, indexing completion, CPU, RSS, disk reads and load average under separate `perf-run` locks. ## Perf VM measurements Both runs used the shared pre-fix release binary from `af202e151` (SHA-256 `f1a597f4681e4f4e08d383c9f065333e16f9293607645c74b1d80637ba239d80`) with 2,003 Notes, 1,000 Tasks, 12 Logs, 5,000 Files and 24 Photos. - First start: `/healthz` 1.515 s; `/readyz` 6.646 s; Notes 16.813 s; Tasks 21.861 s; Search result 77.724 s; Search indexing complete 523.773 s; CPU 213.78 s; RSS 701,366,272 bytes; physical reads 4,096 bytes; logical reads 3,781,688,704 bytes. - Unchanged restart: `/healthz` 1.852 s; `/readyz` 4.426 s; all routes complete 4.426 s; Search indexing complete 9.623 s; CPU 7.82 s; RSS 243,421,184 bytes; physical reads 143,015,936 bytes; logical reads 281,633,739 bytes. - The unchanged run confirms the 523.773-second first pass did not repeat. The pre-fix binary did not emit the new reprocessed-file counter, so those benchmark counts are `null`; the candidate regression test verifies 0 and 1 directly. - The 10-file-change phase did not run. After 60 minutes without an available perf slot, the profile recorded `perf-vm-still-busy-after-60m-changed`. No lock was held by this job during its wait. The candidate release build was not available in this job, so these are baseline measurements, not candidate performance numbers. Expected reconciliation work is O(searchable paths) metadata traversal and bounded manifest lookup; source bytes and projection work are limited to changed or unverified files. ## Gates ```text cargo fmt --check (exit 0; no output) cargo clippy -p calternal-search --all-targets -- -D warnings Checking calternal-search v0.0.1 (/home/kayg/Developer/calternal-wt/restartidx-1450/crates/calternal-search) Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.86s cargo test -p calternal-search test result: ok. 61 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 36.08s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.95s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.45s test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 611.14s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 4.15s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `node --check bench/restart-index-1450.mjs` and `git diff --check` also exited 0 with no output. `cargo clean`: `Removed 8225 files, 4.0GiB total`. ## Decisions and gaps - DESIGN does not prescribe the ordinary-startup fingerprint comparison sequence. I used persisted Files metadata plus Search manifest metadata; deep verification remains for missing metadata and integrity/recovery paths. - UX gaps: not applicable; this is a backend indexing change. - The changed-10-files perf phase and candidate-release perf numbers remain unmeasured because the shared VM stayed locked. After a matching release binary is available, rerun `bun bench/restart-index-1450.mjs` with `CALTERNAL_SERVER_RELEASE_BIN`, `RESTARTIDX_SERVER_REV`, and the media runtime set. Files: `crates/calternal-search/src/indexer.rs`, `bench/restart-index-1450.mjs`. Head: `45c9cf134345f26fd40b9c64fa7b8e2b783376b2`.

kayg referenced this issue

2026-10-09 05:37:24 +00:00

Merge round 35c #1500

kayg referenced this issue from a commit

2026-10-09 11:11:17 +00:00

Merge job/restartidx-1450 (#1450)

kayg commented

2026-10-09 11:11:36 +00:00

Author

Owner

Deployed to calternal.cloud in merge round 35c (build 5a2c02c13, live 2026-10-09 ~16:35 IST). Closing.

Deployed to calternal.cloud in merge round 35c (build 5a2c02c13, live 2026-10-09 ~16:35 IST). Closing.