kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Memory: hand freed memory back after bursts #248

Closed
kayg wants to merge 1 commit from issue/44-trim into dev
pull from: issue/44-trim
merge into: kayg:dev
Conversation 2 Commits 1 Files changed 9 +110 -2
kayg commented 2026-10-14 11:27:55 +00:00
Owner
Copy link

Part of #44. Independent of #227 (merge in any order).

glibc keeps what a burst freed (startup scan, list store catching up, a search build or catch-up, the whole-Home list) in its per-thread heaps and rarely returns it: on the perf VM at 100k notes, dev sits at 385 MB RSS at rest (483 MB peak) with ~30 MB in use.

  • New small crate calternal-memory: after_burst() calls glibc's malloc_trim(0) two seconds after the last burst. A burst arms one wait on a thread of its own: no timers, an idle server never wakes up for it (the idle.wakeups_per_s gate covers this).
  • Needs a nod: it is the one crate allowed an unsafe call, for that one call (its own [lints.rust] unsafe_code = "deny" plus an allow on the call). The workspace still forbids unsafe everywhere else.
  • Hooked after the startup scan, big list refreshes (>= 512 notes), search catch-up and updates, and the whole-Home list.

Local at 100k (system allocator): settled 82 -> 60 MB, after searches 152-160 -> 84 MB. Perf-VM numbers for 1k/10k/100k (and with glibc.malloc.arena_max=2) will be posted on #44.

🤖 Generated with Claude Code

Part of #44. Independent of #227 (merge in any order). glibc keeps what a burst freed (startup scan, list store catching up, a search build or catch-up, the whole-Home list) in its per-thread heaps and rarely returns it: on the perf VM at 100k notes, dev sits at **385 MB RSS at rest** (483 MB peak) with ~30 MB in use. - New small crate `calternal-memory`: `after_burst()` calls glibc's `malloc_trim(0)` two seconds after the last burst. A burst arms one wait on a thread of its own: no timers, an idle server never wakes up for it (the `idle.wakeups_per_s` gate covers this). - **Needs a nod:** it is the one crate allowed an `unsafe` call, for that one call (its own `[lints.rust] unsafe_code = "deny"` plus an `allow` on the call). The workspace still forbids `unsafe` everywhere else. - Hooked after the startup scan, big list refreshes (>= 512 notes), search catch-up and updates, and the whole-Home list. Local at 100k (system allocator): settled 82 -> 60 MB, after searches 152-160 -> 84 MB. Perf-VM numbers for 1k/10k/100k (and with `glibc.malloc.arena_max=2`) will be posted on #44. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Memory: hand freed memory back after bursts
Some checks failed
CI / web (pull_request) Failing after 23s
Details
CI / rust (pull_request) Successful in 1m51s
Details
cd41cbf2f4 
glibc keeps what a burst of work freed (the startup scan, the list store
catching up, a search index build or catch-up, the whole-Home list)
inside its heaps, a heap per thread, and rarely returns it: at 100,000
notes the server sat at 150 MB or more with about 30 MB in use (#44).

A new small crate, `calternal-memory`, calls glibc's `malloc_trim` two
seconds after the last burst (a burst arms one wait on a thread of its
own: nothing runs on a timer, an idle server never wakes up for it). It
is the one crate allowed an `unsafe` call, for that one call; the
workspace still forbids `unsafe` everywhere else. Hooked after the
startup scan, big list refreshes, search catch-up and updates, and the
whole-Home list.

Local, 100k notes, system allocator: settled 82 -> 60 MB, after searches
152-160 -> 84 MB.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-14 16:27:14 +00:00
Author
Owner
Copy link

Superseded by the rebased memory PR (also trims after requests and room closes).

Superseded by the rebased memory PR (also trims after requests and room closes).
kayg closed this pull request 2026-10-14 16:27:15 +00:00
kayg commented 2026-10-14 16:27:22 +00:00
Author
Owner
Copy link

-> #314

-> #314
Some checks failed
CI / web (pull_request) Failing after 23s
Details
CI / rust (pull_request) Successful in 1m51s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!248
No description provided.