Notes: links live in the list store, not in memory

kayg commented

2026-10-14 04:17:26 +00:00

Owner

Part of #44. Stacked on #119 (first page) (merge that first).

At rest at 100,000 notes the server held ~400 MB RSS (VM: 600-900 MB after use); heaptrack at rest showed 310 of 346 MB of heap in the backlink map, which was also rebuilt by reading every note file on every start.

Links now live in meta.db (links: source, target fold key, line), written with the note's row whenever the note is described for the list. Nothing is read again at startup and they cost no memory. Open notes take their links from live text. Backlinks, rename's "who links here" and titles are SQL lookups; the line shown next to a backlink is read from the linking note when asked. The list store schema goes to 4 (rebuilt from the files once).
The cold scan keeps only each file's claimed id (not its first 8 KB) and reads in bounded chunks, storing the list part by part.
The deprecated whole-Home list makes one copy fewer.

At rest at 100k (local, system allocator): heap 30 MB (the id index ~25 MB), RSS 82 MB, was 403. Budget: idle.rss <= 100 MB at 100k (gated). Perf VM numbers will follow as a comment.

🤖 Generated with Claude Code

Part of #44. Stacked on #119 (first page) (merge that first). At rest at 100,000 notes the server held ~400 MB RSS (VM: 600-900 MB after use); heaptrack at rest showed 310 of 346 MB of heap in the backlink map, which was also rebuilt by reading every note file on every start. - Links now live in `meta.db` (`links`: source, target fold key, line), written with the note's row whenever the note is described for the list. Nothing is read again at startup and they cost no memory. Open notes take their links from live text. Backlinks, rename's "who links here" and titles are SQL lookups; the line shown next to a backlink is read from the linking note when asked. The list store schema goes to 4 (rebuilt from the files once). - The cold scan keeps only each file's claimed id (not its first 8 KB) and reads in bounded chunks, storing the list part by part. - The deprecated whole-Home list makes one copy fewer. At rest at 100k (local, system allocator): heap 30 MB (the id index ~25 MB), RSS 82 MB, was 403. Budget: `idle.rss` <= 100 MB at 100k (gated). Perf VM numbers will follow as a comment. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kayg added 11 commits

2026-10-14 04:17:26 +00:00

Perf harness: fixture Homes, server scenarios, budgets 80f457b6bb

`calternal-fixtures` makes realistic Homes (1k/10k/100k notes) in the real
on-disk format, byte-identical for a seed: daily notes with log entries,
regular notes with frontmatter (+05:30 times), links `[t](<path.md>)`,
tags, tasks, attachment refs, 1 in 2,000 notes at 50,000 words, some in a
subfolder, some without a `calternal-id`.

`calternal-perf` runs the real server binary on them: first-start indexing
(cold), then fresh servers on the indexed Home measuring startup to
/healthz, notes.list, journal catch-up, opening notes (room join and
notes.get), sync round trip and save ack, and an edit storm (4 clients,
10k keystrokes with save checks, plus an outside app rewriting the file)
that checks nothing was lost or doubled. RSS and disk bytes written come
from /proc. Median of runs per p50/p95 with a noise figure; budgets in
perf/budgets.json, --gate exits 1 on a breach.

perf/run-vm.sh runs it on the perf machine; perf/browser.sh wraps the
existing @perf keystroke-to-paint browser test.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf storm: typing never glues onto an outside line f18aaeea88

The storm kept typing before the first "\nOutside line", but a keystroke
right after that newline glued onto the outside line, moved the boundary
past it, and later keystrokes edited it. The check then reported the line
missing, though the server had kept every line exactly once (verified in
the saved files). Typing now stops before the newline ahead of the first
outside line. run-vm.sh copies with tar (no rsync needed) and keeps the
build outside the copied tree.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf budgets: one page, baseline from the perf VM b7e52efb38

docs/perf/budgets.md says what each metric measures, how (perf VM, no host
cache, page cache dropped, median of 5 runs, noise), the gated budget and
the target for each, the first baseline at 1k/10k/100k notes, and the
problems it shows: a fake outside change journaled per unopened note on
every start, slow first-start indexing, one fsync per keystroke per room,
and heavy whole-Home lists at 100k. perf/budgets.json holds the same
limits for the harness; targets the code doesn't meet yet don't gate.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: the paged note list, by change, title and tag d13595285e

dev now pages `notes.list` and filters by tag, so the harness measures
what the app asks for: a page of 50 newest first, A to Z, by tag, and 20
pages on through a cursor. A cold run is done when a one-note page counts
every note and the Home says its scan finished (`incomplete: false`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: the first request asks for a page, like the app 2ddcb95bbf

The request that opens the Home is now the first page of the note list
(`home.first_page`, gated). The old whole-Home list stays measured
(`home.first_list`, `home.list`) but only as a target: it is deprecated
and costs seconds at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf budgets: baseline on today's dev, budgets from it fddce15425

dev gained the batched startup scan, paged lists and fan-out before the
disk while this was measured, which fixed four of the problems the first
baseline found. The page now holds the baseline at e2f4928 (5 warm and 3
cold runs at 1k, 10k and 100k notes), budgets set just above it, and the
problems still open: memory at 100k, the first request waiting for the
Home at 100k, a flat 51 ms first open, keystroke throughput.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: idle CPU, wake-ups and memory; CPU while one person types

CI / rust (pull_request) Successful in 2m18s

Details

CI / web (pull_request) Failing after 7m20s

Details

ab43a84f0b

Once per Home size the harness now waits until the Home settled and every
room closed, then measures 5 idle minutes from /proc: CPU ticks, context
switches of every thread (wake-ups), RSS and threads. It also measures the
server's CPU and wake-ups while one person types five characters a second
with a second client watching. Budgets: idle CPU <= 0.1 %, wake-ups <=
0.2/s, typing CPU <= 2 % (gated); RSS at 100k <= 100 MB (target until
met). The server's stdout (its log) now goes to server.log.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Server: TCP_NODELAY on every connection

CI / rust (pull_request) Successful in 2m30s

Details

CI / web (pull_request) Failing after 3m18s

Details

205e43cd08

A reply the server writes in pieces (a room's "joined", then the note's
state) waited for the client's delayed ACK, about 40 ms, so opening any
note in the editor took at least that long; the perf harness showed a
flat ~51 ms first open at every Home size (#44). Every accepted
connection now sets TCP_NODELAY. Budget: open.note p95 <= 50 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: a first open doesn't wait for the disk

CI / rust (pull_request) Failing after 2m7s

Details

CI / web (pull_request) Failing after 3m52s

Details

fb51448d71

A note opened for the first time waited for its seed (the file's text and
its history row) to be fsynced before the editor got it: 10 to 35 ms on
the perf VM, the rest of the first-open time once TCP_NODELAY is on. The
seed is now queued like typing (#32): the note opens at once, "Saved" and
anything reading a version wait until it is durable, and a crash before
it lands seeds again under a new lineage, which a client holding the old
copy is told about (ADR 0020). Tests for both. Budget: open.note p95 <=
15 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: the first page never waits for the Home to open

CI / rust (pull_request) Successful in 3m54s

Details

CI / web (pull_request) Failing after 3m13s

Details

4e2ae40582

At 100,000 notes the first request after a restart waited 1 to 5 s on the
perf VM: it opened the Home (the id index, and the file watcher walking
the whole folder) before answering, then counted every row of the list.

- Pages come from the note list store (`meta.db`, opened on its own in a
  few ms) while the Home is still opening, marked `incomplete`.
- The file watcher is set up off the opening path; the startup scan is
  asked for once it is in place, so no change slips between the two.
- The list keeps its note count with every change (`stats`), so an
  unfiltered page doesn't count 100,000 rows from a cold disk; pages read
  on their own connection, never behind the scan's writes; opening the
  store no longer reads the whole table to check it.

Tests: a page right after a restart doesn't open the Home and says it may
be incomplete; the kept count follows adds, deletes and edits. Budget:
home.first_page p95 <= 100 ms at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: links live in the list store, not in memory

CI / rust (pull_request) Failing after 1m53s

Details

CI / web (pull_request) Failing after 2m52s

Details

95717e1920

At 100,000 notes the server held 400 to 700 MB, most of it the backlink
map (heaptrack: 310 of 346 MB of heap at rest), which was also rebuilt by
reading every note file on every start.

- Links are stored in `meta.db` (table `links`: source, target fold key,
  line), written with the note's row whenever the note is described for
  the list, so they survive restarts and cost no memory. A note open
  right now takes its links from its live text. Backlinks, rename's
  "who links here" and titles are SQL lookups; the line shown next to a
  backlink is read from the linking note when asked. The list store's
  schema moves to 4: it is rebuilt from the files once.
- The cold scan keeps only each file's claimed id, not its first 8 KB,
  and reads in bounded chunks, storing the list part by part.
- The whole-Home list (deprecated) makes one copy fewer.

At rest after a restart at 100k notes: heap 30 MB (the id index 25),
RSS 82 MB (was 403). Budget: idle.rss <= 100 MB at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-14 05:39:14 +00:00

Perf: measure search, and search before idling #140

kayg referenced this pull request

2026-10-14 07:28:03 +00:00

Notes: links live in the list store, not in memory #184

kayg commented

2026-10-14 07:28:42 +00:00

Author

Owner

Superseded by #184 (rebased onto current dev).

kayg closed this pull request