Perf harness: fixture Homes, server scenarios, budgets

kayg commented

2026-10-14 07:27:57 +00:00

Owner

Replaces #106: rebased onto dev at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR.

Part of #44 (closes it once the budgets are agreed). Builds the harness every feature is measured with (ADR 0005).

What is here

crates/fixtures (calternal-fixtures --notes 10k --seed 1 --out DIR): realistic Homes of 1k / 10k / 100k notes in the real on-disk format, byte-identical for a seed. ~30% daily notes (YYYY-MM-DD.md, log entries like - 09:12 - 09:48 text #tag, some written loosely), regular Notes/<Title>.md with frontmatter (calternal-id, title, date, last edited at +05:30, tags), H1, lists, tasks, quotes, code, [t](<path.md>) links, #tags, attachment refs; 1 in 2,000 notes has 50,000 words; 1% in Archive/, 2% without an id. Writes fixture.json with sample notes for the runner.
crates/perf (calternal-perf --bin target/release/calternal --work DIR --sizes 1k,10k,100k --runs 5): runs the real server binary.
- cold.*: first start on a never-indexed Home: startup, first list, time until every note is indexed, disk written, peak RSS.
- warm runs (fresh server each, page cache dropped with --drop-caches): startup.healthz, home.first_page (the request that opens the Home), home.page / page_title / page_tag / page_next, the deprecated whole-Home list, journal.catch_up, open.note (room join until synced), open.api_get, open.long_note (50k words), sync.round_trip and sync.save_ack, the edit storm (4 clients, 10k keystrokes each with a save check, plus an outside app rewriting the file every 500 ms; checks all copies converge, every outside line exactly once, file == note after save), mem.* (RSS / HWM from /proc), storm.disk_written (/proc/pid/io), shutdown.
- idle.* (once per size): after the Home settled and rooms closed, 5 idle minutes: CPU ticks, context switches of all threads, RSS. load.typing_*: one person typing 5 chars/s with a second client watching.
- Report: median over runs of each run's p50 / p95, noise = spread of run p50s; markdown + JSON; --budgets perf/budgets.json --gate exits 1 on a breach; "gate": false marks targets the code doesn't meet yet.
perf/run-vm.sh: copy, build and run on the perf VM, refusing to run while another benchmark is running. perf/browser.sh: wraps the existing @perf keystroke-to-paint Playwright test (phase two: feed it a big Home and the same report).
docs/perf/budgets.md: method, budgets and targets per metric, baseline, known problems.

Baseline from the perf VM on dev at e2f4928: see the latest comment on #44. It caught problems that #78 and #83 then fixed (indexing 35x faster, fake outside-change events gone, 10x keystroke throughput); still open: ~0.8-0.9 GB RSS at 100k notes, the first request at 100k waits 1.1-1.4 s, a flat ~51 ms first open of any note. Those are next, on their own branch.

🤖 Generated with Claude Code

Replaces #106: rebased onto `dev` at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR. Part of #44 (closes it once the budgets are agreed). Builds the harness every feature is measured with (ADR 0005). ## What is here - **`crates/fixtures`** (`calternal-fixtures --notes 10k --seed 1 --out DIR`): realistic Homes of 1k / 10k / 100k notes in the real on-disk format, byte-identical for a seed. ~30% daily notes (`YYYY-MM-DD.md`, log entries like `- 09:12 - 09:48 text #tag`, some written loosely), regular `Notes/<Title>.md` with frontmatter (`calternal-id`, `title`, `date`, `last edited` at +05:30, `tags`), H1, lists, tasks, quotes, code, `[t](<path.md>)` links, `#tags`, attachment refs; 1 in 2,000 notes has 50,000 words; 1% in `Archive/`, 2% without an id. Writes `fixture.json` with sample notes for the runner. - **`crates/perf`** (`calternal-perf --bin target/release/calternal --work DIR --sizes 1k,10k,100k --runs 5`): runs the real server binary. - `cold.*`: first start on a never-indexed Home: startup, first list, time until every note is indexed, disk written, peak RSS. - warm runs (fresh server each, page cache dropped with `--drop-caches`): `startup.healthz`, `home.first_page` (the request that opens the Home), `home.page` / `page_title` / `page_tag` / `page_next`, the deprecated whole-Home list, `journal.catch_up`, `open.note` (room join until synced), `open.api_get`, `open.long_note` (50k words), `sync.round_trip` and `sync.save_ack`, the **edit storm** (4 clients, 10k keystrokes each with a save check, plus an outside app rewriting the file every 500 ms; checks all copies converge, every outside line exactly once, file == note after save), `mem.*` (RSS / HWM from `/proc`), `storm.disk_written` (`/proc/pid/io`), `shutdown`. - `idle.*` (once per size): after the Home settled and rooms closed, 5 idle minutes: CPU ticks, context switches of all threads, RSS. `load.typing_*`: one person typing 5 chars/s with a second client watching. - Report: median over runs of each run's p50 / p95, noise = spread of run p50s; markdown + JSON; `--budgets perf/budgets.json --gate` exits 1 on a breach; `"gate": false` marks targets the code doesn't meet yet. - `perf/run-vm.sh`: copy, build and run on the perf VM, refusing to run while another benchmark is running. `perf/browser.sh`: wraps the existing `@perf` keystroke-to-paint Playwright test (phase two: feed it a big Home and the same report). - `docs/perf/budgets.md`: method, budgets and targets per metric, baseline, known problems. Baseline from the perf VM on `dev` at e2f4928: see the latest comment on #44. It caught problems that #78 and #83 then fixed (indexing 35x faster, fake outside-change events gone, 10x keystroke throughput); still open: ~0.8-0.9 GB RSS at 100k notes, the first request at 100k waits 1.1-1.4 s, a flat ~51 ms first open of any note. Those are next, on their own branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kayg added 7 commits

2026-10-14 07:27:57 +00:00

Perf harness: fixture Homes, server scenarios, budgets 9c57eff20a

`calternal-fixtures` makes realistic Homes (1k/10k/100k notes) in the real
on-disk format, byte-identical for a seed: daily notes with log entries,
regular notes with frontmatter (+05:30 times), links `[t](<path.md>)`,
tags, tasks, attachment refs, 1 in 2,000 notes at 50,000 words, some in a
subfolder, some without a `calternal-id`.

`calternal-perf` runs the real server binary on them: first-start indexing
(cold), then fresh servers on the indexed Home measuring startup to
/healthz, notes.list, journal catch-up, opening notes (room join and
notes.get), sync round trip and save ack, and an edit storm (4 clients,
10k keystrokes with save checks, plus an outside app rewriting the file)
that checks nothing was lost or doubled. RSS and disk bytes written come
from /proc. Median of runs per p50/p95 with a noise figure; budgets in
perf/budgets.json, --gate exits 1 on a breach.

perf/run-vm.sh runs it on the perf machine; perf/browser.sh wraps the
existing @perf keystroke-to-paint browser test.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf storm: typing never glues onto an outside line 7aa1d0c836

The storm kept typing before the first "\nOutside line", but a keystroke
right after that newline glued onto the outside line, moved the boundary
past it, and later keystrokes edited it. The check then reported the line
missing, though the server had kept every line exactly once (verified in
the saved files). Typing now stops before the newline ahead of the first
outside line. run-vm.sh copies with tar (no rsync needed) and keeps the
build outside the copied tree.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf budgets: one page, baseline from the perf VM 9247e596d4

docs/perf/budgets.md says what each metric measures, how (perf VM, no host
cache, page cache dropped, median of 5 runs, noise), the gated budget and
the target for each, the first baseline at 1k/10k/100k notes, and the
problems it shows: a fake outside change journaled per unopened note on
every start, slow first-start indexing, one fsync per keystroke per room,
and heavy whole-Home lists at 100k. perf/budgets.json holds the same
limits for the harness; targets the code doesn't meet yet don't gate.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: the paged note list, by change, title and tag 1b398b0f48

dev now pages `notes.list` and filters by tag, so the harness measures
what the app asks for: a page of 50 newest first, A to Z, by tag, and 20
pages on through a cursor. A cold run is done when a one-note page counts
every note and the Home says its scan finished (`incomplete: false`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: the first request asks for a page, like the app c12dd7cb32

The request that opens the Home is now the first page of the note list
(`home.first_page`, gated). The old whole-Home list stays measured
(`home.first_list`, `home.list`) but only as a target: it is deprecated
and costs seconds at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf budgets: baseline on today's dev, budgets from it 16760e2367

dev gained the batched startup scan, paged lists and fan-out before the
disk while this was measured, which fixed four of the problems the first
baseline found. The page now holds the baseline at e2f4928 (5 warm and 3
cold runs at 1k, 10k and 100k notes), budgets set just above it, and the
problems still open: memory at 100k, the first request waiting for the
Home at 100k, a flat 51 ms first open, keystroke throughput.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: idle CPU, wake-ups and memory; CPU while one person types

CI / rust (pull_request) Failing after 24s

Details

CI / web (pull_request) Failing after 10m31s

Details

b148e7a1a0

Once per Home size the harness now waits until the Home settled and every
room closed, then measures 5 idle minutes from /proc: CPU ticks, context
switches of every thread (wake-ups), RSS and threads. It also measures the
server's CPU and wake-ups while one person types five characters a second
with a second client watching. Budgets: idle CPU <= 0.1 %, wake-ups <=
0.2/s, typing CPU <= 2 % (gated); RSS at 100k <= 100 MB (target until
met). The server's stdout (its log) now goes to server.log.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-14 07:27:58 +00:00

Server: TCP_NODELAY on every connection #181

kayg referenced this pull request

2026-10-14 07:28:33 +00:00

Perf harness: fixture Homes, server scenarios, budgets #106

kayg referenced this pull request

2026-10-14 08:04:50 +00:00

Perf harness: fixture Homes, server scenarios, budgets #191

kayg commented

2026-10-14 08:05:13 +00:00

Author