kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Memory: two allocator heaps for the whole server #386

Closed
kayg wants to merge 3 commits from issue/44-arenas into dev
pull from: issue/44-arenas
merge into: kayg:dev
Conversation 2 Commits 3 Files changed 14 +225 -5
kayg commented 2026-10-15 00:33:29 +00:00
Owner
Copy link

Part of #44 (idle RSS under 100 MB and peak under 150 MB at 100k, system allocator). On top of #358 (itself on #314): merge those first. This PR's own change is the last commit (e175704).

glibc gives each busy thread its own heap ("arena"), up to eight per core. Memory freed after a burst then sits half-used across many heaps, where neither reuse nor malloc_trim gets much of it back.

Measured on the perf VM, 100k notes, 3 runs, round 19 vs round 15. Same binary (all6 = every open #44 PR), with GLIBC_TUNABLES=glibc.malloc.arena_max=2 vs without:

metric all6 all6, 2 heaps all6, no whole list all6, no whole list, 2 heaps
idle.rss MB 102 80.4 92.4 66.8
mem.peak MB 451 / 473 263 / 273 140 / 150 105 / 117
mem.after_search MB 138 / 143 103 / 103 120 / 140 96.7 / 97.3
search.query ms 5.70 / 36.3 4.98 / 43.9 5.62 / 19.5 4.98 / 19.5
open.note ms 3.73 / 5.30 1.44 / 3.36 3.06 / 9.17 1.52 / 3.19
home.first_page ms 49 / 88 31 / 115 45 / 54 18 / 40
sync.save_ack ms 14.1 / 30.3 12.3 / 23.0 7.86 / 17.9 7.26 / 13.8

Change:

  • calternal_memory::few_heaps() calls mallopt(M_ARENA_MAX, 2). That is a third unsafe call in calternal-memory, still the only crate allowed one; it needs your nod.
  • It has to run before any thread allocates. So main now builds the tokio runtime itself (same multi-thread, enable_all runtime as #[tokio::main]) and calls few_heaps() and no_huge_pages() first.
  • It has no effect off Linux/glibc.

Checked: cargo test -p calternal-memory, cargo test -p calternal and clippy (memory, calternal) are clean; calternal --version runs.

🤖 Generated with Claude Code

Part of #44 (idle RSS under 100 MB and peak under 150 MB at 100k, system allocator). **On top of #358** (itself on #314): merge those first. This PR's own change is the last commit (e175704). glibc gives each busy thread its own heap ("arena"), up to eight per core. Memory freed after a burst then sits half-used across many heaps, where neither reuse nor `malloc_trim` gets much of it back. **Measured** on the perf VM, 100k notes, 3 runs, round 19 vs round 15. Same binary (all6 = every open #44 PR), with `GLIBC_TUNABLES=glibc.malloc.arena_max=2` vs without: | metric | all6 | all6, 2 heaps | all6, no whole list | all6, no whole list, 2 heaps | |---|---|---|---|---| | idle.rss MB | 102 | **80.4** | 92.4 | **66.8** | | mem.peak MB | 451 / 473 | **263 / 273** | 140 / 150 | **105 / 117** | | mem.after_search MB | 138 / 143 | 103 / 103 | 120 / 140 | 96.7 / 97.3 | | search.query ms | 5.70 / 36.3 | 4.98 / 43.9 | 5.62 / 19.5 | 4.98 / 19.5 | | open.note ms | 3.73 / 5.30 | 1.44 / 3.36 | 3.06 / 9.17 | 1.52 / 3.19 | | home.first_page ms | 49 / 88 | 31 / 115 | 45 / 54 | 18 / 40 | | sync.save_ack ms | 14.1 / 30.3 | 12.3 / 23.0 | 7.86 / 17.9 | 7.26 / 13.8 | **Change:** - `calternal_memory::few_heaps()` calls `mallopt(M_ARENA_MAX, 2)`. That is a third `unsafe` call in calternal-memory, still the only crate allowed one; **it needs your nod**. - It has to run before any thread allocates. So `main` now builds the tokio runtime itself (same multi-thread, `enable_all` runtime as `#[tokio::main]`) and calls `few_heaps()` and `no_huge_pages()` first. - It has no effect off Linux/glibc. Checked: `cargo test -p calternal-memory`, `cargo test -p calternal` and clippy (memory, calternal) are clean; `calternal --version` runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Memory: hand freed memory back after bursts
Some checks failed
CI / server (pull_request) Successful in 25s
Details
CI / rust (pull_request) Successful in 2m18s
Details
CI / web (pull_request) Failing after 3m3s
Details
fdbaeee3f3 
glibc keeps what a burst of work freed (the startup scan, the list store
catching up, a search index build or catch-up, the whole-Home list)
inside its heaps, a heap per thread, and rarely returns it: at 100,000
notes the server sat at 150 MB or more with about 30 MB in use (#44).

A new small crate, `calternal-memory`, calls glibc's `malloc_trim` two
seconds after the last burst (a burst arms one wait on a thread of its
own: nothing runs on a timer, an idle server never wakes up for it). It
is the one crate allowed an `unsafe` call, for that one call; the
workspace still forbids `unsafe` everywhere else. Hooked after the
startup scan, big list refreshes, search catch-up and updates, and the
whole-Home list.

Local, 100k notes, system allocator: settled 82 -> 60 MB, after searches
152-160 -> 84 MB.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Memory: no transparent huge pages
Some checks failed
CI / server (pull_request) Successful in 22s
Details
CI / rust (pull_request) Successful in 1m38s
Details
CI / web (pull_request) Failing after 2m19s
Details
ca952790c3 
With transparent huge pages on `always` (Debian's default, and the perf
VM's), the kernel's background thread merges heap regions into whole
2 MB pages even when most of the region is free. A trimmed heap is
mostly free pages, so an idle server at 100,000 notes grew from 126 to
155 MB while doing nothing, 84 MB of it in huge pages that trimming
can't hand back.

The server now asks the kernel at start (`prctl(PR_SET_THP_DISABLE)`)
to keep it on normal pages, in calternal-memory, the one crate allowed
an `unsafe` call. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Memory: two allocator heaps for the whole server
Some checks failed
CI / server (pull_request) Successful in 24s
Details
CI / rust (pull_request) Successful in 1m51s
Details
CI / web (pull_request) Failing after 2m16s
Details
e175704d6d 
glibc gives each busy thread a heap of its own (up to eight per core),
and what a burst freed then sat half-used in many of them. On the perf
VM at 100,000 notes (glibc tunable, round 19), two heaps took the idle
server from 102 to 80 MB and the peak from 451 to 263 MB; without the
deprecated whole-Home list, idle 92 -> 67 MB and peak 140 -> 105 MB. No
request got slower (search.query, open.note, first page, saves).

calternal-memory calls mallopt(M_ARENA_MAX, 2), first thing in main,
before the runtime's threads start: main now builds the tokio runtime
itself instead of #[tokio::main]. Huge pages are turned off there too.
(#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-15 02:00:22 +00:00
Author
Owner
Copy link

Latency A/B on the perf VM (round 23). It interleaves all7 and all8, where all8 = all7 + this PR, so this PR is the only difference. 1k and 10k, 5 warm runs each, done twice. Numbers are p50 / p95, in the order all7 → all8.

metric 1k, pass 1 1k, pass 2 10k, pass 1 10k, pass 2
home.first_page 8.4/32.9 → 8.2/8.8 35/104 → 14/42 49/128 → 18/61 11/89 → 11.5/12.5
open.note 1.25/3.05 → 1.14/2.67 2.31/5.47 → 1.90/4.57 1.51/2.84 → 1.21/2.49 1.43/3.18 → 1.30/2.82
search.query 4.2/13.6 → 3.9/7.7 4.6/11.2 → 5.5/17.3 5.0/25 → 5.5/19.2 4.5/16.4 → 4.6/11.8
sync.round_trip 0.93/2.84 → 0.59/0.95 0.76/2.72 → 0.73/1.86 0.65/2.32 → 0.72/2.53 0.59/1.40 → 0.59/1.03
sync.save_ack 10.6/33 → 8.8/24 10.3/50 → 9.5/31 10.5/35 → 11.3/38 9.5/29 → 8.8/21
mem.peak MB 40/50 → 36/40 39/54 → 36/40 63/90 → 46/56 69/94 → 45/57

No latency cost, often a little better, with a lower peak. In round 21 at 100k, all8 against round 20's all7: idle.rss 104 → 78.9 MB, peak 475 → 269, cold peak 555 → 300.

Latency A/B on the perf VM (round 23). It interleaves all7 and all8, where all8 = all7 + this PR, so this PR is the only difference. 1k and 10k, 5 warm runs each, done twice. Numbers are p50 / p95, in the order all7 → all8. | metric | 1k, pass 1 | 1k, pass 2 | 10k, pass 1 | 10k, pass 2 | |---|---|---|---|---| | home.first_page | 8.4/32.9 → 8.2/8.8 | 35/104 → 14/42 | 49/128 → 18/61 | 11/89 → 11.5/12.5 | | open.note | 1.25/3.05 → 1.14/2.67 | 2.31/5.47 → 1.90/4.57 | 1.51/2.84 → 1.21/2.49 | 1.43/3.18 → 1.30/2.82 | | search.query | 4.2/13.6 → 3.9/7.7 | 4.6/11.2 → 5.5/17.3 | 5.0/25 → 5.5/19.2 | 4.5/16.4 → 4.6/11.8 | | sync.round_trip | 0.93/2.84 → 0.59/0.95 | 0.76/2.72 → 0.73/1.86 | 0.65/2.32 → 0.72/2.53 | 0.59/1.40 → 0.59/1.03 | | sync.save_ack | 10.6/33 → 8.8/24 | 10.3/50 → 9.5/31 | 10.5/35 → 11.3/38 | 9.5/29 → 8.8/21 | | mem.peak MB | 40/50 → 36/40 | 39/54 → 36/40 | 63/90 → 46/56 | 69/94 → 45/57 | No latency cost, often a little better, with a lower peak. In round 21 at 100k, all8 against round 20's all7: idle.rss 104 → 78.9 MB, peak 475 → 269, cold peak 555 → 300.
kayg commented 2026-10-15 10:02:27 +00:00
Author
Owner
Copy link

Replaced by #418 (#398): kayg asked for the two-heap limit as a tunable, so it is now GLIBC_TUNABLES=glibc.malloc.arena_max=2 in both quadlets (checked with quadlet -dryrun on staging), and the perf harness passes the same setting. No mallopt, and no unsafe for it. Closing.

Replaced by #418 (#398): kayg asked for the two-heap limit as a tunable, so it is now `GLIBC_TUNABLES=glibc.malloc.arena_max=2` in both quadlets (checked with `quadlet -dryrun` on staging), and the perf harness passes the same setting. No `mallopt`, and no `unsafe` for it. Closing.
kayg closed this pull request 2026-10-15 10:02:28 +00:00
Some checks failed
CI / server (pull_request) Successful in 24s
Details
CI / rust (pull_request) Successful in 1m51s
Details
CI / web (pull_request) Failing after 2m16s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!386
No description provided.