kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Notes: a compact id index #189

Closed
kayg wants to merge 18 commits from issue/44-r2-index into dev
pull from: issue/44-r2-index
merge into: kayg:dev
Conversation 1 Commits 18 Files changed 57 +4177 -416
kayg commented 2026-10-14 07:28:11 +00:00
Owner
Copy link

Replaces #144: rebased onto dev at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR.

Part of #44. Stacked on #188.

The id index held every note's id and path in two maps of owned strings: ~25 MB at 100k notes, most of what was left of the heap at rest. Each note's id and path are now stored once in a slot; lookups by id and by folded path map a hash to slot numbers and check the slot on every hit (collisions keep a short list, newest first, matching a map's overwrite). A model test runs 5,000 random puts/renames/case changes/removes against two plain maps.

At rest at 100k notes: RSS 52-54 MB, was 60.

🤖 Generated with Claude Code

Replaces #144: rebased onto `dev` at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR. Part of #44. Stacked on #188. The id index held every note's id and path in two maps of owned strings: ~25 MB at 100k notes, most of what was left of the heap at rest. Each note's id and path are now stored once in a slot; lookups by id and by folded path map a hash to slot numbers and check the slot on every hit (collisions keep a short list, newest first, matching a map's overwrite). A model test runs 5,000 random puts/renames/case changes/removes against two plain maps. At rest at 100k notes: RSS 52-54 MB, was 60. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kayg added 18 commits 2026-10-14 07:28:11 +00:00 
`calternal-fixtures` makes realistic Homes (1k/10k/100k notes) in the real
on-disk format, byte-identical for a seed: daily notes with log entries,
regular notes with frontmatter (+05:30 times), links `[t](<path.md>)`,
tags, tasks, attachment refs, 1 in 2,000 notes at 50,000 words, some in a
subfolder, some without a `calternal-id`.

`calternal-perf` runs the real server binary on them: first-start indexing
(cold), then fresh servers on the indexed Home measuring startup to
/healthz, notes.list, journal catch-up, opening notes (room join and
notes.get), sync round trip and save ack, and an edit storm (4 clients,
10k keystrokes with save checks, plus an outside app rewriting the file)
that checks nothing was lost or doubled. RSS and disk bytes written come
from /proc. Median of runs per p50/p95 with a noise figure; budgets in
perf/budgets.json, --gate exits 1 on a breach.

perf/run-vm.sh runs it on the perf machine; perf/browser.sh wraps the
existing @perf keystroke-to-paint browser test.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
The storm kept typing before the first "\nOutside line", but a keystroke
right after that newline glued onto the outside line, moved the boundary
past it, and later keystrokes edited it. The check then reported the line
missing, though the server had kept every line exactly once (verified in
the saved files). Typing now stops before the newline ahead of the first
outside line. run-vm.sh copies with tar (no rsync needed) and keeps the
build outside the copied tree.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
docs/perf/budgets.md says what each metric measures, how (perf VM, no host
cache, page cache dropped, median of 5 runs, noise), the gated budget and
the target for each, the first baseline at 1k/10k/100k notes, and the
problems it shows: a fake outside change journaled per unopened note on
every start, slow first-start indexing, one fsync per keystroke per room,
and heavy whole-Home lists at 100k. perf/budgets.json holds the same
limits for the harness; targets the code doesn't meet yet don't gate.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
dev now pages `notes.list` and filters by tag, so the harness measures
what the app asks for: a page of 50 newest first, A to Z, by tag, and 20
pages on through a cursor. A cold run is done when a one-note page counts
every note and the Home says its scan finished (`incomplete: false`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
The request that opens the Home is now the first page of the note list
(`home.first_page`, gated). The old whole-Home list stays measured
(`home.first_list`, `home.list`) but only as a target: it is deprecated
and costs seconds at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
dev gained the batched startup scan, paged lists and fan-out before the
disk while this was measured, which fixed four of the problems the first
baseline found. The page now holds the baseline at e2f4928 (5 warm and 3
cold runs at 1k, 10k and 100k notes), budgets set just above it, and the
problems still open: memory at 100k, the first request waiting for the
Home at 100k, a flat 51 ms first open, keystroke throughput.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Perf: idle CPU, wake-ups and memory; CPU while one person types
Some checks failed
CI / rust (pull_request) Failing after 24s
Details
CI / web (pull_request) Failing after 10m31s
Details
b148e7a1a0 
Once per Home size the harness now waits until the Home settled and every
room closed, then measures 5 idle minutes from /proc: CPU ticks, context
switches of every thread (wake-ups), RSS and threads. It also measures the
server's CPU and wake-ups while one person types five characters a second
with a second client watching. Budgets: idle CPU <= 0.1 %, wake-ups <=
0.2/s, typing CPU <= 2 % (gated); RSS at 100k <= 100 MB (target until
met). The server's stdout (its log) now goes to server.log.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Server: TCP_NODELAY on every connection
Some checks failed
CI / rust (pull_request) Failing after 22s
Details
CI / web (pull_request) Failing after 10m29s
Details
bc53c03781 
A reply the server writes in pieces (a room's "joined", then the note's
state) waited for the client's delayed ACK, about 40 ms, so opening any
note in the editor took at least that long; the perf harness showed a
flat ~51 ms first open at every Home size (#44). Every accepted
connection now sets TCP_NODELAY. Budget: open.note p95 <= 50 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Notes: a first open doesn't wait for the disk
Some checks failed
CI / rust (pull_request) Failing after 25s
Details
CI / web (pull_request) Failing after 6m53s
Details
7a39ead68e 
A note opened for the first time waited for its seed (the file's text and
its history row) to be fsynced before the editor got it: 10 to 35 ms on
the perf VM, the rest of the first-open time once TCP_NODELAY is on. The
seed is now queued like typing (#32): the note opens at once, "Saved" and
anything reading a version wait until it is durable, and a crash before
it lands seeds again under a new lineage, which a client holding the old
copy is told about (ADR 0020). Tests for both. Budget: open.note p95 <=
15 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Notes: the first page never waits for the Home to open
Some checks failed
CI / rust (pull_request) Failing after 24s
Details
CI / web (pull_request) Failing after 7m35s
Details
3265d4a8a5 
At 100,000 notes the first request after a restart waited 1 to 5 s on the
perf VM: it opened the Home (the id index, and the file watcher walking
the whole folder) before answering, then counted every row of the list.

- Pages come from the note list store (`meta.db`, opened on its own in a
  few ms) while the Home is still opening, marked `incomplete`.
- The file watcher is set up off the opening path; the startup scan is
  asked for once it is in place, so no change slips between the two.
- The list keeps its note count with every change (`stats`), so an
  unfiltered page doesn't count 100,000 rows from a cold disk; pages read
  on their own connection, never behind the scan's writes; opening the
  store no longer reads the whole table to check it.

Tests: a page right after a restart doesn't open the Home and says it may
be incomplete; the kept count follows adds, deletes and edits. Budget:
home.first_page p95 <= 100 ms at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
At 100,000 notes the server held 400 to 700 MB, most of it the backlink
map (heaptrack: 310 of 346 MB of heap at rest), which was also rebuilt by
reading every note file on every start.

- Links are stored in `meta.db` (table `links`: source, target fold key,
  line), written with the note's row whenever the note is described for
  the list, so they survive restarts and cost no memory. A note open
  right now takes its links from its live text. Backlinks, rename's
  "who links here" and titles are SQL lookups; the line shown next to a
  backlink is read from the linking note when asked. The list store's
  schema moves to 4: it is rebuilt from the files once.
- The cold scan keeps only each file's claimed id, not its first 8 KB,
  and reads in bounded chunks, storing the list part by part.
- The whole-Home list (deprecated) makes one copy fewer.

At rest after a restart at 100k notes: heap 30 MB (the id index 25),
RSS 82 MB (was 403). Budget: idle.rss <= 100 MB at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Notes: the list store writes fast again
Some checks failed
CI / rust (pull_request) Successful in 2m10s
Details
CI / web (pull_request) Failing after 6m51s
Details
cd7cbdaa44 
Storing links in the list store made the first index of 100,000 notes
six times slower on the perf VM (54 s -> 310 s): every statement was
prepared again for every note, and during the startup scan the list was
described twice at once (once by the scan, once by the worker following
the scan's "created" events), the two fighting over one connection.

- Every statement of a list write is a cached prepared statement
  (3x faster per note).
- Many notes go in a few thousand per transaction, with room in the page
  cache for the duration, then the memory is handed back.
- During the startup scan the worker waits for the scan, then redoes
  only the notes whose file changed after the scan looked; a note just
  made in the app is still listed at once.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
The palette's search (`search.query`) had no number yet: the harness now
asks it a few words, a tag, a phrase and a query typed a letter at a time
(`search.query` p95 <= 100 ms), and records memory afterwards. It runs
before the idle stretch, so background work that search starts and that
never goes back to sleep shows up there: it does today, about 4 wake-ups
a second at rest (the search writer polls every 250 ms).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Perf: count wake-ups per thread
Some checks failed
CI / rust (pull_request) Successful in 1m42s
Details
CI / web (pull_request) Failing after 6m13s
Details
836ebf903f 
The wake-up count summed every thread's context switches at two moments,
so a thread exiting in between (a finished background read) made the
difference go negative and read as 0. It now compares thread by thread.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Search: the writer sleeps until there is work
Some checks failed
CI / rust (pull_request) Successful in 1m56s
Details
CI / web (pull_request) Failing after 5m46s
Details
8f4e292ecd 
The search index's writer thread woke every 250 ms forever to see whether
the log needed a checkpoint: about 4 wake-ups a second on an idle server
once anyone had searched (the perf harness's idle stretch, #44). It now
waits for a job without a timeout; only after writes does it wait a quiet
250 ms, checkpoint once and go back to sleep. Idle wake-ups: 3.9/s -> 0.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Search: 2 MiB page cache per connection; a first build holds less
Some checks failed
CI / rust (pull_request) Successful in 1m48s
Details
CI / web (pull_request) Failing after 6m17s
Details
6c7aa75fb9 
Each search connection (the writer and up to four readers, per person)
kept up to 16 MiB of SQLite pages; heaptrack at rest after a few searches
at 100,000 notes showed 19 MB of SQLite cache. They now keep at most
2 MiB each. The first index build (minutes at 100k) also kept every
item and stamp twice for its whole run; it now drops them once it knows
what is stale.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Server: mimalloc
Some checks failed
CI / rust (pull_request) Successful in 1m59s
Details
CI / web (pull_request) Failing after 7m27s
Details
344cc8a194 
glibc keeps a heap per thread and rarely hands freed memory back, so
every burst (a search reading the index, the startup scan, a big list)
left the server holding tens of megabytes it no longer used. At rest at
100,000 notes after a few searches: 156 MB RSS with glibc, 89 to 101 MB
with mimalloc; settled before searching, 82 vs 60 MB (#44). The live heap
is about 30 MB either way.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Notes: a compact id index
Some checks failed
CI / rust (pull_request) Failing after 1m33s
Details
CI / web (pull_request) Failing after 7m26s
Details
f351f4da56 
The id index held every note's id and path twice over, in two maps of
owned strings: about 25 MB at 100,000 notes, most of what was left of the
heap at rest (#44). Each note's id and path are now stored once, in a
slot; the lookups by id and by folded path map a hash to slot numbers and
check the slot on every hit. A model test runs thousands of puts,
renames, case changes and removes against two plain maps.

At rest at 100k notes (mimalloc): RSS 52-54 MB, was 60.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-14 08:05:27 +00:00
Author
Owner
Copy link

Superseded by #200 (dev failed cargo fmt; the stack now starts with that fix).

Superseded by #200 (dev failed cargo fmt; the stack now starts with that fix).
kayg closed this pull request 2026-10-14 08:05:27 +00:00
Some checks failed
CI / rust (pull_request) Failing after 1m33s
Details
CI / web (pull_request) Failing after 7m26s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!189
No description provided.