kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Notes: a cold Home fills its list, then makes the list's indexes #372

Closed
kayg wants to merge 0 commits from issue/44-cold-list into dev
pull from: issue/44-cold-list
merge into: kayg:dev
Conversation 1 Commits - Files changed - +155 -10
kayg commented 2026-10-14 21:05:47 +00:00
Owner
Copy link

Part of #44 (cold.index at 100k well under 50 s). Based on dev; independent of the other #44 PRs.

Found: on the perf VM, a cold Home of 100,000 notes writes 3.5 GB (cold.disk_written) for a 240 MB list store. The note list has eight secondary indexes with random keys (ids, paths, link targets, tags). Filling an empty store 4,096 notes per transaction rewrites most of their pages on every batch, and WAL writes everything twice.

Change: when 10,000 notes or more go into an empty list store:

  • begin_fresh drops the eight secondary indexes;
  • rows go in with nothing to clear first (there are no old tags or links);
  • end_fresh makes the indexes once, by running the schema again.

Safety:

  • A page asked for during the fill doesn't name an index (INDEXED BY), so SQLite picks its own plan. The list is marked incomplete then anyway.
  • A note stored meanwhile by someone else (an edit during the fill) is cleared and rewritten as usual. It is never doubled.
  • A crash halfway is harmless: opening the store runs the schema, which makes any missing index.

Measured with a replay of the 100k fill: the real rows from a 100k meta.db, shuffled, 4,096 per transaction, WAL on disk:

written time
before 3,966 MB 18–28 s
after 1,274 MB 8–11 s

Times are from the shared, busy dev host, so only the ratio is meaningful. Perf-VM cold.index / cold.disk_written will be posted on #44 (round 15).

New test a_cold_fill_ends_with_every_index_and_right_answers: 400 notes, tag totals, every index present, one link each, backlinks. Tests use the fresh path from 200 notes, so every big-Home test goes through it. cargo test -p calternal-notes (170) and clippy are clean.

🤖 Generated with Claude Code

Part of #44 (`cold.index` at 100k well under 50 s). Based on dev; independent of the other #44 PRs. **Found:** on the perf VM, a cold Home of 100,000 notes writes **3.5 GB** (`cold.disk_written`) for a 240 MB list store. The note list has eight secondary indexes with random keys (ids, paths, link targets, tags). Filling an empty store 4,096 notes per transaction rewrites most of their pages on every batch, and WAL writes everything twice. **Change:** when 10,000 notes or more go into an *empty* list store: - `begin_fresh` drops the eight secondary indexes; - rows go in with nothing to clear first (there are no old tags or links); - `end_fresh` makes the indexes once, by running the schema again. Safety: - A page asked for during the fill doesn't name an index (`INDEXED BY`), so SQLite picks its own plan. The list is marked `incomplete` then anyway. - A note stored meanwhile by someone else (an edit during the fill) is cleared and rewritten as usual. It is never doubled. - A crash halfway is harmless: opening the store runs the schema, which makes any missing index. **Measured** with a replay of the 100k fill: the real rows from a 100k `meta.db`, shuffled, 4,096 per transaction, WAL on disk: | | written | time | |---|---|---| | before | 3,966 MB | 18–28 s | | after | 1,274 MB | 8–11 s | Times are from the shared, busy dev host, so only the ratio is meaningful. Perf-VM `cold.index` / `cold.disk_written` will be posted on #44 (round 15). New test `a_cold_fill_ends_with_every_index_and_right_answers`: 400 notes, tag totals, every index present, one link each, backlinks. Tests use the fresh path from 200 notes, so every big-Home test goes through it. `cargo test -p calternal-notes` (170) and clippy are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Notes: a cold Home fills its list, then makes the list's indexes
Some checks failed
CI / server (pull_request) Successful in 33s
Details
CI / rust (pull_request) Successful in 2m36s
Details
CI / web (pull_request) Failing after 2m12s
Details
a956a83510 
The note list keeps eight secondary indexes with random keys (ids,
paths, link targets, tags). Filling an empty list store 4,096 notes per
transaction rewrote most of their pages every batch: a cold Home of
100,000 notes wrote 3.5 GB for a 240 MB store (cold.disk_written on the
perf VM).

Filling an empty store with 10,000 notes or more now drops those
indexes, adds the rows with nothing to clear first, and makes the
indexes once at the end. A replay of the 100k fill: 3,966 -> 1,274 MB
written, about a third of the time. Pages asked for meanwhile don't name
an index; an edit stored during the fill is cleared as usual; a stop
halfway is harmless, since opening the store makes any missing index.
Tests take the fresh way from 200 notes. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-15 00:15:39 +00:00
Author
Owner
Copy link

Replaced by #384: the same change, plus the fill keeps notes_updated so the app's newest-first pages stay indexed during a cold fill.

Replaced by #384: the same change, plus the fill keeps `notes_updated` so the app's newest-first pages stay indexed during a cold fill.
kayg closed this pull request 2026-10-15 00:15:39 +00:00
Some checks failed
CI / server (pull_request) Successful in 33s
Details
CI / rust (pull_request) Successful in 2m36s
Details
CI / web (pull_request) Failing after 2m12s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!372
No description provided.