Server: mimalloc

kayg commented

2026-10-14 08:05:10 +00:00

Owner

Replaces #188 (and earlier): rebased onto dev at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR.

Part of #44. Stacked on #198. A decision to confirm: a new dependency (mimalloc, C, builds in seconds).

glibc keeps a heap per thread and rarely hands freed memory back, so every burst (a search reading the index, the startup scan, a big list) left the server holding memory it no longer used. At rest at 100k notes after a few searches: 156 MB RSS with glibc, 89-101 MB with mimalloc; settled before searching 82 vs 60 MB. The live heap is ~30 MB either way (heaptrack). Cost: small Homes start a little higher (idle ~20 MB instead of ~12).

🤖 Generated with Claude Code

Replaces #188 (and earlier): rebased onto `dev` at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR. Part of #44. Stacked on #198. A decision to confirm: a new dependency (`mimalloc`, C, builds in seconds). glibc keeps a heap per thread and rarely hands freed memory back, so every burst (a search reading the index, the startup scan, a big list) left the server holding memory it no longer used. At rest at 100k notes after a few searches: 156 MB RSS with glibc, 89-101 MB with mimalloc; settled before searching 82 vs 60 MB. The live heap is ~30 MB either way (heaptrack). Cost: small Homes start a little higher (idle ~20 MB instead of ~12). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kayg added 18 commits

2026-10-14 08:05:10 +00:00

Notes tests: modules in rustfmt's order e3e8bed1f6

dev failed `cargo fmt --check` (trash and writers in tests.rs), which
fails the rust job of every PR on top of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf harness: fixture Homes, server scenarios, budgets 6b0c32d9fd

`calternal-fixtures` makes realistic Homes (1k/10k/100k notes) in the real
on-disk format, byte-identical for a seed: daily notes with log entries,
regular notes with frontmatter (+05:30 times), links `[t](<path.md>)`,
tags, tasks, attachment refs, 1 in 2,000 notes at 50,000 words, some in a
subfolder, some without a `calternal-id`.

`calternal-perf` runs the real server binary on them: first-start indexing
(cold), then fresh servers on the indexed Home measuring startup to
/healthz, notes.list, journal catch-up, opening notes (room join and
notes.get), sync round trip and save ack, and an edit storm (4 clients,
10k keystrokes with save checks, plus an outside app rewriting the file)
that checks nothing was lost or doubled. RSS and disk bytes written come
from /proc. Median of runs per p50/p95 with a noise figure; budgets in
perf/budgets.json, --gate exits 1 on a breach.

perf/run-vm.sh runs it on the perf machine; perf/browser.sh wraps the
existing @perf keystroke-to-paint browser test.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf storm: typing never glues onto an outside line 4cff5afd56

The storm kept typing before the first "\nOutside line", but a keystroke
right after that newline glued onto the outside line, moved the boundary
past it, and later keystrokes edited it. The check then reported the line
missing, though the server had kept every line exactly once (verified in
the saved files). Typing now stops before the newline ahead of the first
outside line. run-vm.sh copies with tar (no rsync needed) and keeps the
build outside the copied tree.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf budgets: one page, baseline from the perf VM 7229efd754

docs/perf/budgets.md says what each metric measures, how (perf VM, no host
cache, page cache dropped, median of 5 runs, noise), the gated budget and
the target for each, the first baseline at 1k/10k/100k notes, and the
problems it shows: a fake outside change journaled per unopened note on
every start, slow first-start indexing, one fsync per keystroke per room,
and heavy whole-Home lists at 100k. perf/budgets.json holds the same
limits for the harness; targets the code doesn't meet yet don't gate.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: the paged note list, by change, title and tag f1c75eba0e

dev now pages `notes.list` and filters by tag, so the harness measures
what the app asks for: a page of 50 newest first, A to Z, by tag, and 20
pages on through a cursor. A cold run is done when a one-note page counts
every note and the Home says its scan finished (`incomplete: false`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: the first request asks for a page, like the app 9b6aa4f2c7

The request that opens the Home is now the first page of the note list
(`home.first_page`, gated). The old whole-Home list stays measured
(`home.first_list`, `home.list`) but only as a target: it is deprecated
and costs seconds at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf budgets: baseline on today's dev, budgets from it 122436c7f6

dev gained the batched startup scan, paged lists and fan-out before the
disk while this was measured, which fixed four of the problems the first
baseline found. The page now holds the baseline at e2f4928 (5 warm and 3
cold runs at 1k, 10k and 100k notes), budgets set just above it, and the
problems still open: memory at 100k, the first request waiting for the
Home at 100k, a flat 51 ms first open, keystroke throughput.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: idle CPU, wake-ups and memory; CPU while one person types

CI / rust (pull_request) Successful in 1m59s

Details

CI / web (pull_request) Failing after 8m17s

Details

cad5d3cd32

Once per Home size the harness now waits until the Home settled and every
room closed, then measures 5 idle minutes from /proc: CPU ticks, context
switches of every thread (wake-ups), RSS and threads. It also measures the
server's CPU and wake-ups while one person types five characters a second
with a second client watching. Budgets: idle CPU <= 0.1 %, wake-ups <=
0.2/s, typing CPU <= 2 % (gated); RSS at 100k <= 100 MB (target until
met). The server's stdout (its log) now goes to server.log.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Server: TCP_NODELAY on every connection

CI / rust (pull_request) Failing after 1m31s

Details

CI / web (pull_request) Failing after 2m8s

Details

6d93844bae

A reply the server writes in pieces (a room's "joined", then the note's
state) waited for the client's delayed ACK, about 40 ms, so opening any
note in the editor took at least that long; the perf harness showed a
flat ~51 ms first open at every Home size (#44). Every accepted
connection now sets TCP_NODELAY. Budget: open.note p95 <= 50 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: a first open doesn't wait for the disk

CI / rust (pull_request) Failing after 1m46s

Details

CI / web (pull_request) Failing after 8m3s

Details

118f953fd8

A note opened for the first time waited for its seed (the file's text and
its history row) to be fsynced before the editor got it: 10 to 35 ms on
the perf VM, the rest of the first-open time once TCP_NODELAY is on. The
seed is now queued like typing (#32): the note opens at once, "Saved" and
anything reading a version wait until it is durable, and a crash before
it lands seeds again under a new lineage, which a client holding the old
copy is told about (ADR 0020). Tests for both. Budget: open.note p95 <=
15 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: the first page never waits for the Home to open

CI / rust (pull_request) Successful in 2m19s

Details

CI / web (pull_request) Failing after 7m47s

Details

69aa0cdb14

At 100,000 notes the first request after a restart waited 1 to 5 s on the
perf VM: it opened the Home (the id index, and the file watcher walking
the whole folder) before answering, then counted every row of the list.

- Pages come from the note list store (`meta.db`, opened on its own in a
  few ms) while the Home is still opening, marked `incomplete`.
- The file watcher is set up off the opening path; the startup scan is
  asked for once it is in place, so no change slips between the two.
- The list keeps its note count with every change (`stats`), so an
  unfiltered page doesn't count 100,000 rows from a cold disk; pages read
  on their own connection, never behind the scan's writes; opening the
  store no longer reads the whole table to check it.

Tests: a page right after a restart doesn't open the Home and says it may
be incomplete; the kept count follows adds, deletes and edits. Budget:
home.first_page p95 <= 100 ms at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: links live in the list store, not in memory 193ede9722

At 100,000 notes the server held 400 to 700 MB, most of it the backlink
map (heaptrack: 310 of 346 MB of heap at rest), which was also rebuilt by
reading every note file on every start.

- Links are stored in `meta.db` (table `links`: source, target fold key,
  line), written with the note's row whenever the note is described for
  the list, so they survive restarts and cost no memory. A note open
  right now takes its links from its live text. Backlinks, rename's
  "who links here" and titles are SQL lookups; the line shown next to a
  backlink is read from the linking note when asked. The list store's
  schema moves to 4: it is rebuilt from the files once.
- The cold scan keeps only each file's claimed id, not its first 8 KB,
  and reads in bounded chunks, storing the list part by part.
- The whole-Home list (deprecated) makes one copy fewer.

At rest after a restart at 100k notes: heap 30 MB (the id index 25),
RSS 82 MB (was 403). Budget: idle.rss <= 100 MB at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: the list store writes fast again

CI / rust (pull_request) Successful in 2m2s

Details

CI / web (pull_request) Failing after 7m30s

Details

b9fac643d3

Storing links in the list store made the first index of 100,000 notes
six times slower on the perf VM (54 s -> 310 s): every statement was
prepared again for every note, and during the startup scan the list was
described twice at once (once by the scan, once by the worker following
the scan's "created" events), the two fighting over one connection.

- Every statement of a list write is a cached prepared statement
  (3x faster per note).
- Many notes go in a few thousand per transaction, with room in the page
  cache for the duration, then the memory is handed back.
- During the startup scan the worker waits for the scan, then redoes
  only the notes whose file changed after the scan looked; a note just
  made in the app is still listed at once.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: measure search, and search before idling 67b706541b

The palette's search (`search.query`) had no number yet: the harness now
asks it a few words, a tag, a phrase and a query typed a letter at a time
(`search.query` p95 <= 100 ms), and records memory afterwards. It runs
before the idle stretch, so background work that search starts and that
never goes back to sleep shows up there: it does today, about 4 wake-ups
a second at rest (the search writer polls every 250 ms).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: count wake-ups per thread

CI / web (pull_request) Failing after 5m25s

Details

CI / rust (pull_request) Failing after 43m24s

Details

eaa56c86b6

The wake-up count summed every thread's context switches at two moments,
so a thread exiting in between (a finished background read) made the
difference go negative and read as 0. It now compares thread by thread.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: the writer sleeps until there is work

CI / rust (pull_request) Successful in 33m54s

Details

CI / web (pull_request) Failing after 8m39s

Details

f479ef0343

The search index's writer thread woke every 250 ms forever to see whether
the log needed a checkpoint: about 4 wake-ups a second on an idle server
once anyone had searched (the perf harness's idle stretch, #44). It now
waits for a job without a timeout; only after writes does it wait a quiet
250 ms, checkpoint once and go back to sleep. Idle wake-ups: 3.9/s -> 0.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: 2 MiB page cache per connection; a first build holds less

CI / rust (pull_request) Successful in 2m43s

Details

CI / web (pull_request) Failing after 7m30s

Details

f4caee6fee

Each search connection (the writer and up to four readers, per person)
kept up to 16 MiB of SQLite pages; heaptrack at rest after a few searches
at 100,000 notes showed 19 MB of SQLite cache. They now keep at most
2 MiB each. The first index build (minutes at 100k) also kept every
item and stamp twice for its whole run; it now drops them once it knows
what is stale.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Server: mimalloc

CI / rust (pull_request) Successful in 2m3s

Details

CI / web (pull_request) Failing after 7m39s

Details

30e125e3b0

glibc keeps a heap per thread and rarely hands freed memory back, so
every burst (a search reading the index, the startup scan, a big list)
left the server holding tens of megabytes it no longer used. At rest at
100,000 notes after a few searches: 156 MB RSS with glibc, 89 to 101 MB
with mimalloc; settled before searching, 82 vs 60 MB (#44). The live heap
is about 30 MB either way.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-14 08:05:12 +00:00

Notes: a compact id index #200

kayg referenced this pull request

2026-10-14 08:05:25 +00:00

Server: mimalloc #188

kayg referenced this pull request

2026-10-14 08:53:45 +00:00

Perf harness: large-Home fixtures, edit storms, perf-VM gate #44

kayg referenced this pull request

2026-10-14 09:23:57 +00:00

Search: 2 MiB page cache per connection; a first build holds less #198