Notes: fast Home open at any size, nothing 'not found' mid-scan (#52)

kayg commented

2026-10-13 19:34:18 +00:00

Owner

Closes #52.

What changed

Index writes are batched. Everything queued goes out in one write and one fsync. A cold open used to open and fsync ids.jsonl once per note.
Warm opens validate by stat. After each scan, the size and modification time of every note file are saved in notes/scan-state.tsv. On the next open, files are only stated and only the changed ones are read. A first scan, with no saved state, takes already-indexed files as they are, so upgrading doesn't announce every note as changed.
Cold opens identify in batches. Unknown files have their heads read in parallel and are indexed in one batch. Journal events go out a thousand at a time.
Nothing is "not found" mid-scan.
- notes.list returns incomplete: true until the startup scan finishes.
- get, room join and explicit-id create look up a note the index doesn't know yet: a day by its file name, anything else by waiting for the scan (60 s cap).
- So they never answer "not found" and never create a second note for the same day.
Homes open eagerly. Every Home starts opening in the background when the server starts, and /healthz never waits for it.
Fixed on the way: paths reported in the same moment as a rescan were dropped, so an outside edit landing in that window was missed. They're always checked now.
Fixed on the way (data safety, caught by CI's kill -9 test on #54): after a crash before a note's first save, its typing is in history but the file was never written. The scan treated "indexed, but no file" as deleted and dropped the note. The old code had the same flaw, hidden only because the 150 ms debounce let a request open the room first. Such a note is now reopened, so its room rebuilds it and writes the file. A note nobody typed in is still forgotten. Both cases have regression tests, and the first one fails without the fix.
Fixed on the way (caught by CI on #56): the startup scan and a request could both reach the same file, and the second one took it for a copy and gave it a second id (4,001 entries for 4,000 files). Both paths now skip a file that's already indexed. A direct test produces 5 entries for 3 files without the guard and passes with it.

New code is in separate files (scan.rs, tests/opening.rs), with small edits to lib.rs, home.rs, index.rs, watch.rs, actions.rs, main.rs and search.rs. It's rebased onto dev after search (#53). The notes search provider now lists notes only once the startup scan has settled, so search never drops notes it would re-add moments later. Any future summary should return the same incomplete flag, via Notes::incomplete(user).

Numbers

Measured on calternal-dev's SSD. The perf VM was busy with the storage benchmark (load 32), so these aren't perf-VM numbers. Homes come from the #44 fixture generator, seed 1. Old is dev at d95b8e8, new is this branch. Cold means the index was removed and the page cache dropped.

notes	cold, old: index complete	cold, new: index complete	cold: late note asked right after start	warm, new: healthz / first list / scan done
1,000	10.0 s	0.26 s	old `not_found`, new found (142 ms)	16 ms / 11 ms (all 1,000) / 0.10 s
10,000	119.6 s	0.76 s	old `not_found`, new found (530 ms)	11 ms / 112 ms (all 10,000) / 0.35 s
100,000	~20 min (extrapolated)	6.8 s	new found (5.1 s, waits for the scan)	7 ms / 0.70 s (all 100,000) / 1.9-3.1 s

Warm 100k background work:

Old: re-reads and re-hashes every file after opening, 141 MB read in the first 8 s and still going, out of 422 MB of notes. It also announces every migrated note as changed.
New: 12.6 MB read in total (the id index plus the scan state).

Still open: the 100k list itself costs about 0.3 s to serialize per call. notes.list needs paging, which is outside #52.

Tests

7 new tests: the 4 below, plus crash-before-first-save, never-typed-is-forgotten and already-indexed-is-not-a-copy. The workspace passes 210 tests after the rebase, 3 runs in a row.
- a cold open indexes every file once with its own id (one index line per note)
- asking mid-scan for a 4,000-note Home never answers "not found" and never duplicates a day
- a warm open announces exactly the edit, the delete and the add made while closed
- an upgrade with no saved scan announces nothing
The notes suite passes 3 times in a row (56 tests), and the workspace passes 182 tests. Clippy -D warnings is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Closes #52. ## What changed - **Index writes are batched.** Everything queued goes out in one write and one fsync. A cold open used to open and fsync `ids.jsonl` once per note. - **Warm opens validate by stat.** After each scan, the size and modification time of every note file are saved in `notes/scan-state.tsv`. On the next open, files are only `stat`ed and only the changed ones are read. A first scan, with no saved state, takes already-indexed files as they are, so upgrading doesn't announce every note as changed. - **Cold opens identify in batches.** Unknown files have their heads read in parallel and are indexed in one batch. Journal events go out a thousand at a time. - **Nothing is "not found" mid-scan.** - `notes.list` returns `incomplete: true` until the startup scan finishes. - `get`, room join and explicit-id `create` look up a note the index doesn't know yet: a day by its file name, anything else by waiting for the scan (60 s cap). - So they never answer "not found" and never create a second note for the same day. - **Homes open eagerly.** Every Home starts opening in the background when the server starts, and `/healthz` never waits for it. - **Fixed on the way:** paths reported in the same moment as a rescan were dropped, so an outside edit landing in that window was missed. They're always checked now. - **Fixed on the way (data safety, caught by CI's kill -9 test on #54):** after a crash before a note's first save, its typing is in history but the file was never written. The scan treated "indexed, but no file" as deleted and dropped the note. The old code had the same flaw, hidden only because the 150 ms debounce let a request open the room first. Such a note is now reopened, so its room rebuilds it and writes the file. A note nobody typed in is still forgotten. Both cases have regression tests, and the first one fails without the fix. - **Fixed on the way (caught by CI on #56):** the startup scan and a request could both reach the same file, and the second one took it for a copy and gave it a second id (4,001 entries for 4,000 files). Both paths now skip a file that's already indexed. A direct test produces 5 entries for 3 files without the guard and passes with it. New code is in separate files (`scan.rs`, `tests/opening.rs`), with small edits to `lib.rs`, `home.rs`, `index.rs`, `watch.rs`, `actions.rs`, `main.rs` and `search.rs`. It's rebased onto `dev` after search (#53). The notes search provider now lists notes only once the startup scan has settled, so search never drops notes it would re-add moments later. Any future summary should return the same `incomplete` flag, via `Notes::incomplete(user)`. ## Numbers Measured on calternal-dev's SSD. The perf VM was busy with the storage benchmark (load 32), so these aren't perf-VM numbers. Homes come from the #44 fixture generator, seed 1. Old is `dev` at d95b8e8, new is this branch. Cold means the index was removed and the page cache dropped. | notes | cold, old: index complete | cold, new: index complete | cold: late note asked right after start | warm, new: healthz / first list / scan done | |---|---|---|---|---| | 1,000 | 10.0 s | **0.26 s** | old `not_found`, new found (142 ms) | 16 ms / 11 ms (all 1,000) / 0.10 s | | 10,000 | 119.6 s | **0.76 s** | old `not_found`, new found (530 ms) | 11 ms / 112 ms (all 10,000) / 0.35 s | | 100,000 | ~20 min (extrapolated) | **6.8 s** | new found (5.1 s, waits for the scan) | 7 ms / 0.70 s (all 100,000) / 1.9-3.1 s | **Warm 100k background work:** - **Old:** re-reads and re-hashes every file after opening, 141 MB read in the first 8 s and still going, out of 422 MB of notes. It also announces every migrated note as changed. - **New:** 12.6 MB read in total (the id index plus the scan state). **Still open:** the 100k list itself costs about 0.3 s to serialize per call. `notes.list` needs paging, which is outside #52. ## Tests - 7 new tests: the 4 below, plus crash-before-first-save, never-typed-is-forgotten and already-indexed-is-not-a-copy. The workspace passes 210 tests after the rebase, 3 runs in a row. - a cold open indexes every file once with its own id (one index line per note) - asking mid-scan for a 4,000-note Home never answers "not found" and never duplicates a day - a warm open announces exactly the edit, the delete and the add made while closed - an upgrade with no saved scan announces nothing - The notes suite passes 3 times in a row (56 tests), and the workspace passes 182 tests. Clippy `-D warnings` is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 4 commits

2026-10-13 19:34:18 +00:00

Notes: open a Home fast at any size, never "not found" mid-scan (#52 ) a5e45d6d36

- The id index flushes everything queued in one write and one fsync
  (cold opens did one open+fsync per note).
- A warm open stats every file and compares it with the size and mtime
  saved after the last scan (notes/scan-state.tsv); only changed files
  are read. A first scan (no saved state) takes indexed files as they
  are, so upgrades don't announce every note as changed.
- A cold open reads unknown files' heads in parallel, indexes them in one
  batch and records journal events a thousand at a time.
- While the startup scan runs, notes.list says `incomplete: true`; get,
  room join and explicit-id create find a not-yet-indexed note first (a
  day by its file name, anything else by waiting for the scan), so they
  never answer "not found" and never make a second day.
- Every Home opens in the background at start; /healthz never waits.
- Fix found on the way: paths reported in the same moment as a rescan
  were dropped; they are always checked now.

Part of #52

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: never forget a note typed but not yet saved when a restart finds no file d9b900880f

Found by CI on #54 (kill -9 test): after a crash before a note's first
save, history holds the typing but the file was never written. The scan
treated "indexed, no file" as deleted and dropped the note. The old code
had the same flaw, hidden by the 150 ms debounce letting a request open
the room first. Now a note whose file was never written but whose
history has changes is reopened (its room rebuilds and writes it); a note
never typed in is still forgotten. Regression tests for both, and the
first one fails without the fix.

Part of #52

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: a file reached by both the scan and a request is never a copy of itself 85262878ea

Found by CI on #56: a day asked for mid-scan is indexed by the request,
then the batch scan reached the same file, saw its id already in use and
took it for a copy, giving it a second id (4,001 entries for 4,000
files). The batch and the single-file paths now skip a file that is
already indexed. A direct test fails without the guard (5 entries for 3
files) and passes with it.

Part of #52

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes search: list notes only once the startup scan has settled

CI / rust (pull_request) Successful in 44s

Details

CI / web (pull_request) Successful in 45s

Details

44a9f66ea5

A cold open's index is partial until the scan finishes; listing then
would make search drop notes it re-adds moments later.

Part of #52

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg commented

2026-10-13 19:35:55 +00:00

Author

Owner

Reviewed by Claude: cold open 10k 119.6 s → 0.76 s, 100k ~20 min → 6.8 s; warm open stat-only; never not_found mid-scan; three real bugs fixed with regression tests, incl. a data-safety one (crash before first save dropping a note whose typing was in history). CI green. Merging. Follow-up: notes.list paging (0.3 s per call at 100k).

kayg commented

2026-10-15 08:35:28 +00:00

Author

Owner

Closing: this branch is already fully contained in dev (it landed through a merge train). Branch cleanup, 2026-10-15.

Closing: this branch is already fully contained in `dev` (it landed through a merge train). Branch cleanup, 2026-10-15.

kayg closed this pull request

2026-10-15 08:35:29 +00:00

CI / rust (pull_request) Successful in 44s

Details

CI / web (pull_request) Successful in 45s

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Rows
Columns

Notes: fast Home open at any size, nothing 'not found' mid-scan (#52) #58

What changed

Numbers

Tests

Pull request closed