Memory: remove the whole-Home notes.list (#398) #444

Merged

kayg merged 19 commits from train/2026-10-15m into dev 2026-10-15 16:19:44 +00:00

Conversation 0 Commits 19 Files changed 39 +927 -176

kayg commented 2026-10-15 16:19:37 +00:00

Owner

Copy link

Part of #398 (approved by kayg on 2026-10-15, Q5 on #174).

What

• notes.list no longer has a whole-Home form. With no input it now returns the first page of 50, the same as {}. Before, it built every note of the Home into one answer: 100,000 objects at 100k notes. The app only asks for pages (limit: 500), and the form was marked "deprecated; for one release".
• Item and Notes::list stay inside the server. The search provider and renames still use them, and they read the id index, not the list store.
• The perf harness no longer calls the whole-Home form. home.first_list, home.list and mem.after_list are gone, and mem.after_first_page is new. The cold run opens the Home with a first page, as the app does, and reports it as cold.first_page.
• One daily test now checks the id index. It used to read the whole list through the action. A page of the list catches up a moment after a note is made, and a page-based version of the test failed on a slower machine.

The contract (contracts/actions.json) doesn't change, because the input schema is the same.

Numbers

These are from the perf VM at 100k notes, #44 round 21. All8 was run twice: as is, and with the harness's 21 whole-Home calls left out. That comparison is what this PR does to a real run. Values are p50 / p95.

metric	with whole-Home calls	without
mem.peak MB	269 / 275	111 / 120
idle.rss MB	78.9	65.6
mem.after_opens MB	442 / 466 (all5, r13)	48.6 / 48.7 (all5, r13)
search.query ms	6.27 / 28.5	4.74 / 38.8
sync.save_ack ms	8.93 / 34.7	6.10 / 15.0

On this branch, the #398 runs on the perf VM (100k, dropped caches) gave these numbers:

• glibc defaults: mem.peak 142 / 180 MB.
• With the two-heap tunable (the next PR): 108 / 131 MB.
• Each warm run's peak now comes from the opens, the search and the storm, not from a list.

Checks: cargo fmt, clippy -D warnings and the workspace tests are clean (Rust 1.99.0).

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

(Replaces #417, merged with current dev. Review on #417.)

Part of #398 (approved by kayg on 2026-10-15, Q5 on #174). ## What - **`notes.list` no longer has a whole-Home form.** With no input it now returns the first page of 50, the same as `{}`. Before, it built every note of the Home into one answer: 100,000 objects at 100k notes. The app only asks for pages (`limit: 500`), and the form was marked "deprecated; for one release". - **`Item` and `Notes::list` stay** inside the server. The search provider and renames still use them, and they read the id index, not the list store. - **The perf harness no longer calls the whole-Home form.** `home.first_list`, `home.list` and `mem.after_list` are gone, and `mem.after_first_page` is new. The cold run opens the Home with a first page, as the app does, and reports it as `cold.first_page`. - **One daily test now checks the id index.** It used to read the whole list through the action. A page of the list catches up a moment after a note is made, and a page-based version of the test failed on a slower machine. The contract (`contracts/actions.json`) doesn't change, because the input schema is the same. ## Numbers These are from the perf VM at 100k notes, #44 round 21. All8 was run twice: as is, and with the harness's 21 whole-Home calls left out. That comparison is what this PR does to a real run. Values are p50 / p95. | metric | with whole-Home calls | without | |---|---|---| | mem.peak MB | 269 / 275 | **111 / 120** | | idle.rss MB | 78.9 | **65.6** | | mem.after_opens MB | 442 / 466 (all5, r13) | **48.6 / 48.7** (all5, r13) | | search.query ms | 6.27 / 28.5 | 4.74 / 38.8 | | sync.save_ack ms | 8.93 / 34.7 | 6.10 / 15.0 | On this branch, the #398 runs on the perf VM (100k, dropped caches) gave these numbers: - glibc defaults: `mem.peak` 142 / 180 MB. - With the two-heap tunable (the next PR): 108 / 131 MB. - Each warm run's peak now comes from the opens, the search and the storm, not from a list. Checks: `cargo fmt`, `clippy -D warnings` and the workspace tests are clean (Rust 1.99.0). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T (Replaces #417, merged with current dev. Review on #417.)

kayg added 19 commits 2026-10-15 16:19:37 +00:00

Deploy: two allocator heaps via GLIBC_TUNABLES (#398) ... dc3f66b738

`glibc.malloc.arena_max=2` in the next and cloud quadlets, instead of a
`mallopt` call in the server (#386). With a heap per busy thread, what a
burst freed sat in many half-empty heaps; two heaps took the idle server
at 100,000 notes from 102 to 80 MB and the peak from 451 to 263 MB on
the perf VM (#44 round 19), with no request slower.

Checked on calternal-staging: `quadlet -dryrun` turns both units into
`--env GLIBC_TUNABLES=glibc.malloc.arena_max=2` (exit 0), and glibc in
a container started that way lists `glibc.malloc.arena_max: 0x2`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: start the server with the deploys' allocator settings (#398) ... 6076b15249

The harness runs the binary itself, not the container, so it now passes
the same `GLIBC_TUNABLES` as the quadlets (two heaps), unless the run
sets its own (empty for glibc's defaults). Perf numbers then describe
what is deployed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Memory: remove the whole-Home notes.list (#398) ... 44306033b1

`notes.list` without input built every note of the Home into one answer
(100,000 objects at 100k notes, hundreds of megabytes at once). The app
only asks for pages, and the form was deprecated "for one release".
Without input it now returns the first page of 50, like `{}`.

The perf harness no longer calls it: `home.first_list`, `home.list` and
`mem.after_list` are gone (`mem.after_first_page` instead), and the cold
run opens the Home with a first page (`cold.first_page`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Memory: a compact id index (#398) ... e60f693439

The id index holds every note, so it is most of what an idle server
keeps at 100,000 notes. Its two lookups were maps from a hash to a slot
bucket (40 bytes a note each), and each note's id and path were two
allocations. Now the lookups are hash tables of slot numbers only (4
bytes a note), found by hashing what the slot holds, and id and path
share one allocation. A path that two notes claim still answers with
the newest one, and with the older one again once the newer one goes.

A heap profile of an idle server at 100,000 notes (staging, heaptrack,
same steps): live heap 28.7 MB -> 21.3 MB; the index 17.5 MB -> 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Memory: hand freed memory back after bursts ... 401d55ed06

glibc keeps what a burst of work freed (the startup scan, the list store
catching up, a search index build or catch-up, the whole-Home list)
inside its heaps, a heap per thread, and rarely returns it: at 100,000
notes the server sat at 150 MB or more with about 30 MB in use (#44).

A new small crate, `calternal-memory`, calls glibc's `malloc_trim` two
seconds after the last burst (a burst arms one wait on a thread of its
own: nothing runs on a timer, an idle server never wakes up for it). It
is the one crate allowed an `unsafe` call, for that one call; the
workspace still forbids `unsafe` everywhere else. Hooked after the
startup scan, big list refreshes, search catch-up and updates, and the
whole-Home list.

Local, 100k notes, system allocator: settled 82 -> 60 MB, after searches
152-160 -> 84 MB.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Memory: let go of SQLite page caches after whole-table walks (#398) ... f076915206

Three connections kept full page caches on an idle server, filled by
work that never needs those pages again: the search writer after a
catch-up (it now runs `PRAGMA shrink_memory` with its quiet-moment
checkpoint), the reader that scans the whole index for the typo
vocabulary, and the list store's writer after it reads every stamp at
start.

A heap profile of an idle server at 100,000 notes (staging, heaptrack,
same steps, on top of the compact id index): live heap 21.3 MB ->
11.3 MB; SQLite 10.7 MB -> 0.7 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Memory: no transparent huge pages ... d5cebeb198

With transparent huge pages on `always` (Debian's default, and the perf
VM's), the kernel's background thread merges heap regions into whole
2 MB pages even when most of the region is free. A trimmed heap is
mostly free pages, so an idle server at 100,000 notes grew from 126 to
155 MB while doing nothing, 84 MB of it in huge pages that trimming
can't hand back.

The server now asks the kernel at start (`prctl(PR_SET_THP_DISABLE)`)
to keep it on normal pages, in calternal-memory, the one crate allowed
an `unsafe` call. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Memory: an 8 MiB cache for the cold fill, not 64 (#398) ... e30a9b861a

A big batch into the list store and making its indexes after a fresh
fill ran with a 64 MiB page cache, and SQLite's sorter for `CREATE
INDEX` takes as much again: 137 MB of a cold 100,000-note open's 203 MB
heap peak (heaptrack on staging). With 8 MiB the cold open on the perf
VM peaks at 196 MB instead of 295, and indexing takes as long (25.6 s
against 24.9).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: a leak soak scenario (#398) ... c0dbe6ba9e

`--soak-secs N` adds a long stretch on one server, once per Home size:
two cycles, each a third of the time of steady use (pages, a search,
notes opened in the editor and through the API, keystrokes typed and
taken out again with a save check, a page of the journal, round after
round) and a sixth idle. Memory is sampled every 10 s into
`soak-<notes>.tsv`. A leak ratchets the idle level up from one cycle to
the next (`soak.idle_ratchet`); `soak.load_slope` and `soak.idle_slope`
show growth under load and while idle. `--soak-secs 1800` is the
30-minute soak.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Serve the app precompressed, compress big JSON answers, bigger list pages (#422) ... 761af690f2

- Build: a vite plugin writes .br (brotli q11) and .gz (gzip -9) next to every
  text file of dist/, once, so the Containerfile build gets them too.
- Server: ServeDir/ServeFile send the precompressed file that matches
  Accept-Encoding, with Vary: Accept-Encoding; the plain file is the fallback.
  Caching rules (immutable for hashed files, no-cache for the page and sw.js)
  are unchanged and apply to the compressed copy.
- API: tower-http compression (br/gzip level 4) on the request routes only,
  for answers over 1 KB. Never the live socket.
- Notes list: the server accepts pages up to 5000; the app's background fill
  asks for 2000 after the first 500.
- Tests: precompressed serving and the compression layer; the faked deploy in
  pwa-update.spec writes sw.js.br/.gz too; served.spec checks encodings.
- Perf: home.page_big (five pages of 2,000).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: a read-only soak (#398) ... 0873a7e060

`--soak-read-only` runs the soak without typing: no note, history or
search index grows, so a step in the idle level between the two cycles
is memory kept, not data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Compression level note: measured numbers ... db2c62dabb

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Merge remote-tracking branch 'origin/issue/398-no-whole-list' into train/2026-10-15m e34c4a4ba5

Merge remote-tracking branch 'origin/issue/398-tunables' into train/2026-10-15m b7fabab5ad

Merge remote-tracking branch 'origin/issue/398-floor' into train/2026-10-15m ea62a3f44a

Merge remote-tracking branch 'origin/issue/398-cold-cache' into train/2026-10-15m 97bcd46e81

Merge remote-tracking branch 'origin/issue/398-soak' into train/2026-10-15m fb65af87a3

Merge remote-tracking branch 'origin/issue/398-glibc-trim' into train/2026-10-15m ... 2a190d1ad1

# Conflicts:
#	crates/search/src/indexer.rs

Merge remote-tracking branch 'origin/issue/422-compress' into train/2026-10-15m ... 150b875547

# Conflicts:
#	docs/perf/budgets.md

kayg merged commit 499d1f8e44 into dev 2026-10-15 16:19:44 +00:00

kayg deleted branch train/2026-10-15m 2026-10-15 16:19:44 +00:00

kayg referenced this pull request from a commit 2026-10-15 16:19:45 +00:00

Merge pull request 'Memory: remove the whole-Home notes.list (#398)' (#444) from train/2026-10-15m into dev

kayg referenced this pull request 2026-10-15 22:40:26 +00:00

Memory: a cold open holds less at once (#398) #462

kayg referenced this pull request 2026-10-15 22:40:47 +00:00

Memory: a cold open holds less at once (#398) #459

kayg referenced this pull request 2026-10-15 22:40:49 +00:00

Memory: an idle Home lets go of its id index (#398, prototype) #460

kayg referenced this pull request 2026-10-16 01:30:49 +00:00

Memory: allocator A/B (mimalloc), tunables, drop whole-Home list, lower the floor #398

kayg referenced this pull request 2026-10-16 03:17:15 +00:00

Memory: a cold open holds less at once (#398) #467

kayg referenced this pull request 2026-10-26 08:45:58 +00:00

Perf harness: large-Home fixtures, edit storms, perf-VM gate #44

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Rust server

  

design

Mockups, UI and TASTE

  

docs

ADRs, CONTEXT, TASTE

  

idea

Parked idea, never worked until promoted

  

infra

Deploy, CI, VMs

  

perf

Performance and budgets

  

reliability

Data integrity, corruption traps

  

web

Web client

No labels

backend

design

docs

idea

infra

perf

reliability

web

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal!444

No description provided.