Notes: a cold Home fills its list, then makes the list's indexes #384

Closed

kayg wants to merge 0 commits from issue/44-cold-list-2 into dev

kayg commented

2026-10-15 00:15:34 +00:00

Owner

Part of #44 (cold.index at 100k well under 50 s). Based on dev; independent of the other #44 PRs. Replaces #372, which is the same change plus a second commit: the fill keeps notes_updated.

Found: on the perf VM, a cold Home of 100,000 notes writes 3.5 GB (cold.disk_written) for a 240 MB list store. The note list has eight secondary indexes with random keys (ids, paths, link targets, tags). Filling an empty store 4,096 notes per transaction rewrites most of their pages on every batch, and WAL writes everything twice.

Change: when 10,000 notes or more go into an empty list store:

begin_fresh drops seven of the eight secondary indexes. It keeps notes_updated: the app pages through the list newest first while a cold Home fills, and without that index every page would sort the whole store. Keeping it costs about a tenth more writing;
rows go in with nothing to clear first (there are no old tags or links);
end_fresh makes the indexes once, by running the schema again.

Safety:

A page asked for during the fill doesn't name an index (INDEXED BY), so SQLite picks its own plan. The list is marked incomplete then anyway.
A note stored meanwhile by someone else (an edit during the fill) is cleared and rewritten as usual. It is never doubled.
A crash halfway is harmless: opening the store runs the schema, which makes any missing index.

Measured with a replay of the 100k fill: the real rows from a 100k meta.db, shuffled, 4,096 per transaction, WAL on disk:

	written	time
before	3,966 MB	18–28 s
after, every index dropped	1,274 MB	8–11 s
after, `notes_updated` kept (this PR)	1,409 MB	10–13 s

Times are from the shared, busy dev host, so only the ratio is meaningful. Perf VM, round 15, with every index dropped (#372): cold.index at 100k 58.8 → 23.3 s, cold.disk_written 3,470 → 1,010 MB (see #44).

New test a_cold_fill_ends_with_every_index_and_right_answers: 400 notes, tag totals, every index present, one link each, backlinks. Tests use the fresh path from 200 notes, so every big-Home test goes through it. cargo test -p calternal-notes (170) and clippy are clean.

🤖 Generated with Claude Code

Part of #44 (`cold.index` at 100k well under 50 s). Based on dev; independent of the other #44 PRs. **Replaces #372**, which is the same change plus a second commit: the fill keeps `notes_updated`. **Found:** on the perf VM, a cold Home of 100,000 notes writes **3.5 GB** (`cold.disk_written`) for a 240 MB list store. The note list has eight secondary indexes with random keys (ids, paths, link targets, tags). Filling an empty store 4,096 notes per transaction rewrites most of their pages on every batch, and WAL writes everything twice. **Change:** when 10,000 notes or more go into an *empty* list store: - `begin_fresh` drops seven of the eight secondary indexes. It keeps `notes_updated`: the app pages through the list newest first while a cold Home fills, and without that index every page would sort the whole store. Keeping it costs about a tenth more writing; - rows go in with nothing to clear first (there are no old tags or links); - `end_fresh` makes the indexes once, by running the schema again. Safety: - A page asked for during the fill doesn't name an index (`INDEXED BY`), so SQLite picks its own plan. The list is marked `incomplete` then anyway. - A note stored meanwhile by someone else (an edit during the fill) is cleared and rewritten as usual. It is never doubled. - A crash halfway is harmless: opening the store runs the schema, which makes any missing index. **Measured** with a replay of the 100k fill: the real rows from a 100k `meta.db`, shuffled, 4,096 per transaction, WAL on disk: | | written | time | |---|---|---| | before | 3,966 MB | 18–28 s | | after, every index dropped | 1,274 MB | 8–11 s | | after, `notes_updated` kept (this PR) | 1,409 MB | 10–13 s | Times are from the shared, busy dev host, so only the ratio is meaningful. Perf VM, round 15, with every index dropped (#372): `cold.index` at 100k 58.8 → **23.3 s**, `cold.disk_written` 3,470 → **1,010 MB** (see #44). New test `a_cold_fill_ends_with_every_index_and_right_answers`: 400 notes, tag totals, every index present, one link each, backlinks. Tests use the fresh path from 200 notes, so every big-Home test goes through it. `cargo test -p calternal-notes` (170) and clippy are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kayg added 2 commits

2026-10-15 00:15:34 +00:00

Notes: a cold Home fills its list, then makes the list's indexes

CI / server (pull_request) Successful in 33s

Details

CI / rust (pull_request) Successful in 2m36s

Details

CI / web (pull_request) Failing after 2m12s

Details

a956a83510

The note list keeps eight secondary indexes with random keys (ids,
paths, link targets, tags). Filling an empty list store 4,096 notes per
transaction rewrote most of their pages every batch: a cold Home of
100,000 notes wrote 3.5 GB for a 240 MB store (cold.disk_written on the
perf VM).

Filling an empty store with 10,000 notes or more now drops those
indexes, adds the rows with nothing to clear first, and makes the
indexes once at the end. A replay of the 100k fill: 3,966 -> 1,274 MB
written, about a third of the time. Pages asked for meanwhile don't name
an index; an edit stored during the fill is cleared as usual; a stop
halfway is harmless, since opening the store makes any missing index.
Tests take the fresh way from 200 notes. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: a fresh fill keeps the list's newest-first index

CI / server (pull_request) Successful in 19s

Details

CI / rust (pull_request) Successful in 1m37s

Details

CI / web (pull_request) Failing after 2m8s

Details

470e971fc8

The app pages through the note list newest change first while a cold
Home fills; without notes_updated every page sorted the whole store.
Kept, it costs a tenth more writing (a replay of the 100k fill: 1,274 ->
1,409 MB, against 3,966 MB with every index kept up). (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-15 00:15:39 +00:00

Notes: a cold Home fills its list, then makes the list's indexes #372

kayg referenced this pull request

2026-10-15 01:27:56 +00:00

Perf harness: large-Home fixtures, edit storms, perf-VM gate #44

kayg referenced this pull request

2026-10-15 02:22:38 +00:00

Merge train 2026-10-15j: server speed at 100k (#44), bar-stays test (#387) #391

kayg commented

2026-10-15 02:23:02 +00:00

Author