kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Notes: the first page never waits for the Home to open #183

Closed
kayg wants to merge 10 commits from issue/44-r2-first-page into dev
pull from: issue/44-r2-first-page
merge into: kayg:dev
Conversation 1 Commits 10 Files changed 39 +3436 -77
kayg commented 2026-10-14 07:28:02 +00:00
Owner
Copy link

Replaces #119: rebased onto dev at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR.

Part of #44. Stacked on #182.

At 100,000 notes the first request after a restart waited 1 to 5 s on the perf VM: it opened the Home (id index, plus the file watcher walking the whole folder), then counted every row of the list from a cold disk.

  • Pages come from the note list store (meta.db, opened on its own in a few ms) while the Home is still opening, marked incomplete.
  • The file watcher is set up off the opening path; the startup scan is asked for once it is in place.
  • The list keeps its note count with every change (stats table), pages read on their own (query-only) connection, and opening the store no longer reads the whole table to check it.

Tests: a page right after a restart doesn't open the Home and says incomplete; the kept count follows adds, deletes and edits. Budget: home.first_page p95 <= 100 ms at 100k.

🤖 Generated with Claude Code

Replaces #119: rebased onto `dev` at 2a2478c (after #112, #123, #130 and the rest), since the forge ignores pushes to an open PR. Part of #44. Stacked on #182. At 100,000 notes the first request after a restart waited 1 to 5 s on the perf VM: it opened the Home (id index, plus the file watcher walking the whole folder), then counted every row of the list from a cold disk. - Pages come from the note list store (`meta.db`, opened on its own in a few ms) while the Home is still opening, marked `incomplete`. - The file watcher is set up off the opening path; the startup scan is asked for once it is in place. - The list keeps its note count with every change (`stats` table), pages read on their own (query-only) connection, and opening the store no longer reads the whole table to check it. Tests: a page right after a restart doesn't open the Home and says `incomplete`; the kept count follows adds, deletes and edits. Budget: `home.first_page` p95 <= 100 ms at 100k. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kayg added 10 commits 2026-10-14 07:28:02 +00:00 
`calternal-fixtures` makes realistic Homes (1k/10k/100k notes) in the real
on-disk format, byte-identical for a seed: daily notes with log entries,
regular notes with frontmatter (+05:30 times), links `[t](<path.md>)`,
tags, tasks, attachment refs, 1 in 2,000 notes at 50,000 words, some in a
subfolder, some without a `calternal-id`.

`calternal-perf` runs the real server binary on them: first-start indexing
(cold), then fresh servers on the indexed Home measuring startup to
/healthz, notes.list, journal catch-up, opening notes (room join and
notes.get), sync round trip and save ack, and an edit storm (4 clients,
10k keystrokes with save checks, plus an outside app rewriting the file)
that checks nothing was lost or doubled. RSS and disk bytes written come
from /proc. Median of runs per p50/p95 with a noise figure; budgets in
perf/budgets.json, --gate exits 1 on a breach.

perf/run-vm.sh runs it on the perf machine; perf/browser.sh wraps the
existing @perf keystroke-to-paint browser test.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
The storm kept typing before the first "\nOutside line", but a keystroke
right after that newline glued onto the outside line, moved the boundary
past it, and later keystrokes edited it. The check then reported the line
missing, though the server had kept every line exactly once (verified in
the saved files). Typing now stops before the newline ahead of the first
outside line. run-vm.sh copies with tar (no rsync needed) and keeps the
build outside the copied tree.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
docs/perf/budgets.md says what each metric measures, how (perf VM, no host
cache, page cache dropped, median of 5 runs, noise), the gated budget and
the target for each, the first baseline at 1k/10k/100k notes, and the
problems it shows: a fake outside change journaled per unopened note on
every start, slow first-start indexing, one fsync per keystroke per room,
and heavy whole-Home lists at 100k. perf/budgets.json holds the same
limits for the harness; targets the code doesn't meet yet don't gate.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
dev now pages `notes.list` and filters by tag, so the harness measures
what the app asks for: a page of 50 newest first, A to Z, by tag, and 20
pages on through a cursor. A cold run is done when a one-note page counts
every note and the Home says its scan finished (`incomplete: false`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
The request that opens the Home is now the first page of the note list
(`home.first_page`, gated). The old whole-Home list stays measured
(`home.first_list`, `home.list`) but only as a target: it is deprecated
and costs seconds at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
dev gained the batched startup scan, paged lists and fan-out before the
disk while this was measured, which fixed four of the problems the first
baseline found. The page now holds the baseline at e2f4928 (5 warm and 3
cold runs at 1k, 10k and 100k notes), budgets set just above it, and the
problems still open: memory at 100k, the first request waiting for the
Home at 100k, a flat 51 ms first open, keystroke throughput.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Perf: idle CPU, wake-ups and memory; CPU while one person types
Some checks failed
CI / rust (pull_request) Failing after 24s
Details
CI / web (pull_request) Failing after 10m31s
Details
b148e7a1a0 
Once per Home size the harness now waits until the Home settled and every
room closed, then measures 5 idle minutes from /proc: CPU ticks, context
switches of every thread (wake-ups), RSS and threads. It also measures the
server's CPU and wake-ups while one person types five characters a second
with a second client watching. Budgets: idle CPU <= 0.1 %, wake-ups <=
0.2/s, typing CPU <= 2 % (gated); RSS at 100k <= 100 MB (target until
met). The server's stdout (its log) now goes to server.log.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Server: TCP_NODELAY on every connection
Some checks failed
CI / rust (pull_request) Failing after 22s
Details
CI / web (pull_request) Failing after 10m29s
Details
bc53c03781 
A reply the server writes in pieces (a room's "joined", then the note's
state) waited for the client's delayed ACK, about 40 ms, so opening any
note in the editor took at least that long; the perf harness showed a
flat ~51 ms first open at every Home size (#44). Every accepted
connection now sets TCP_NODELAY. Budget: open.note p95 <= 50 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Notes: a first open doesn't wait for the disk
Some checks failed
CI / rust (pull_request) Failing after 25s
Details
CI / web (pull_request) Failing after 6m53s
Details
7a39ead68e 
A note opened for the first time waited for its seed (the file's text and
its history row) to be fsynced before the editor got it: 10 to 35 ms on
the perf VM, the rest of the first-open time once TCP_NODELAY is on. The
seed is now queued like typing (#32): the note opens at once, "Saved" and
anything reading a version wait until it is durable, and a crash before
it lands seeds again under a new lineage, which a client holding the old
copy is told about (ADR 0020). Tests for both. Budget: open.note p95 <=
15 ms.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Notes: the first page never waits for the Home to open
Some checks failed
CI / rust (pull_request) Failing after 24s
Details
CI / web (pull_request) Failing after 7m35s
Details
3265d4a8a5 
At 100,000 notes the first request after a restart waited 1 to 5 s on the
perf VM: it opened the Home (the id index, and the file watcher walking
the whole folder) before answering, then counted every row of the list.

- Pages come from the note list store (`meta.db`, opened on its own in a
  few ms) while the Home is still opening, marked `incomplete`.
- The file watcher is set up off the opening path; the startup scan is
  asked for once it is in place, so no change slips between the two.
- The list keeps its note count with every change (`stats`), so an
  unfiltered page doesn't count 100,000 rows from a cold disk; pages read
  on their own connection, never behind the scan's writes; opening the
  store no longer reads the whole table to check it.

Tests: a page right after a restart doesn't open the Home and says it may
be incomplete; the kept count follows adds, deletes and edits. Budget:
home.first_page p95 <= 100 ms at 100k notes.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-14 08:05:18 +00:00
Author
Owner
Copy link

Superseded by #194 (dev failed cargo fmt; the stack now starts with that fix).

Superseded by #194 (dev failed cargo fmt; the stack now starts with that fix).
kayg closed this pull request 2026-10-14 08:05:19 +00:00
Some checks failed
CI / rust (pull_request) Failing after 24s
Details
CI / web (pull_request) Failing after 7m35s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!183
No description provided.