Memory: a cold open holds less at once (#398)

kayg commented

2026-10-15 22:33:03 +00:00

Owner

Part of #398: the cold-open peak goes under 150 MB. Based on #417. Merge it together with #429, or before it: on a rotational disk, #429 alone makes cold indexing about 30% slower, and this PR wins that time back.

What

A cold open of 100,000 notes held three big things at once. Each is fixed here:

Parsed links: each note's list reused the buffer of the found links, three times the size a (target, line) pair needs. That is how into_iter().map().collect() behaves: it keeps the source allocation. At the peak this was 57 MB of heap (heaptrack). The pairs now get a list of their own, sized to fit.
Scan announcements: until the id index was on disk, the scan held a full event, JSON included, for every note it found. It now keeps only the kind, id and path, and builds each thousand events as they're recorded. This is the change from the closed #388, which didn't move the peak back when the 64 MiB cache dominated it.
Reads at once: the scan, the note list and search each read files on threads of their own, about 75 threads in total. They now share one limit (reads.rs) of 16 reads at a time. A permit is taken before a thread is asked for, so waiting costs no thread.

Numbers, perf VM, 100k notes

SSD (sda). 3 runs each, every column with #427, #428 and #429:

	cold.mem_peak MB	cold.index s
without this PR	185 (177–190)	25.3 (21.3–26.2)
links + events, no read limit	137	23.7
+ read limit 32	129	24.8
+ read limit 16 (this PR)	123 (121–124)	26.6 (19.0–29.8); 18.0 and 19.3 in a later pair of runs
+ read limit 8	118	23.2

Rotational disk (sdb1, ext4, ROTA=1). The Home and work folder are on the HDD; 3 to 6 runs each:

	cold.index s	cold.mem_peak MB
dev, 2 heaps (6)	41.8 (40.0–51.9)	313–328
#417 alone (3)	41.6	314
#429 alone (3)	55.2 (47.5–56.6)	210
all #398 PRs without this one (3)	52.8	187
all #398 PRs with this one (6)	44.8 (41.2–59.6)	123

The rotational-disk cost with this PR is +7% against dev, inside the 10% budget. A bigger cache only for making the indexes (32 or 64 MiB) didn't index faster on either disk and raised the peak to 175 and 230 MB, so it isn't used.

Checks: cargo fmt, clippy -D warnings and the workspace tests are clean (Rust 1.99.0).

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Part of #398: the cold-open peak goes under 150 MB. **Based on #417.** Merge it together with #429, or before it: on a rotational disk, #429 alone makes cold indexing about 30% slower, and this PR wins that time back. ## What A cold open of 100,000 notes held three big things at once. Each is fixed here: - **Parsed links:** each note's list reused the buffer of the found links, three times the size a (target, line) pair needs. That is how `into_iter().map().collect()` behaves: it keeps the source allocation. At the peak this was 57 MB of heap (heaptrack). The pairs now get a list of their own, sized to fit. - **Scan announcements:** until the id index was on disk, the scan held a full event, JSON included, for every note it found. It now keeps only the kind, id and path, and builds each thousand events as they're recorded. This is the change from the closed #388, which didn't move the peak back when the 64 MiB cache dominated it. - **Reads at once:** the scan, the note list and search each read files on threads of their own, about 75 threads in total. They now share one limit (`reads.rs`) of 16 reads at a time. A permit is taken before a thread is asked for, so waiting costs no thread. ## Numbers, perf VM, 100k notes **SSD (sda).** 3 runs each, every column with #427, #428 and #429: | | cold.mem_peak MB | cold.index s | |---|---|---| | without this PR | 185 (177–190) | 25.3 (21.3–26.2) | | links + events, no read limit | 137 | 23.7 | | + read limit 32 | 129 | 24.8 | | **+ read limit 16 (this PR)** | **123** (121–124) | 26.6 (19.0–29.8); 18.0 and 19.3 in a later pair of runs | | + read limit 8 | 118 | 23.2 | **Rotational disk** (sdb1, ext4, ROTA=1). The Home and work folder are on the HDD; 3 to 6 runs each: | | cold.index s | cold.mem_peak MB | |---|---|---| | dev, 2 heaps (6) | 41.8 (40.0–51.9) | 313–328 | | #417 alone (3) | 41.6 | 314 | | #429 alone (3) | **55.2** (47.5–56.6) | 210 | | all #398 PRs without this one (3) | 52.8 | 187 | | **all #398 PRs with this one (6)** | **44.8** (41.2–59.6) | **123** | The rotational-disk cost with this PR is +7% against dev, inside the 10% budget. A bigger cache only for making the indexes (32 or 64 MiB) didn't index faster on either disk and raised the peak to 175 and 230 MB, so it isn't used. Checks: `cargo fmt`, `clippy -D warnings` and the workspace tests are clean (Rust 1.99.0). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 1 commit

2026-10-15 22:33:03 +00:00

Memory: a cold open holds less at once (#398 )

CI / server (pull_request) Successful in 1m6s

Details

CI / rust (pull_request) Successful in 3m19s

Details

CI / web (pull_request) Failing after 42m40s

Details

2e34981d31

Three things a cold open of 100,000 notes kept at the same time:

- Each note's parsed links reused the found links' buffer, three times
  the size a (target, line) pair needs (`into_iter().map().collect()`
  keeps the allocation): 57 MB of the heap peak. They now get a list of
  their own, sized to fit.
- The scan announced every note it found with a full event (JSON
  included) held until the id index was on disk: it now keeps just the
  kind, id and path, and makes each thousand events as they're recorded
  (from #388).
- The scan, the note list and search each read files on a thread of
  their own, about 75 threads at once. They now share a limit of 16
  reads at a time; waiting costs no thread.

Perf VM, 100k notes, 3 runs each, with #427-#429: cold.mem_peak 185 ->
123 MB, cold.index 25.3 -> 18-27 s (same). On a rotational disk the
read limit also wins back what #429's smaller cache cost: cold.index
41.8 s on dev, 55.2 s with #429 alone, 44.8 s with this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-15 22:33:13 +00:00

Memory: an 8 MiB cache for the cold fill (#398) #429

kayg referenced this pull request

2026-10-15 22:33:43 +00:00

Memory: allocator A/B (mimalloc), tunables, drop whole-Home list, lower the floor #398

kayg referenced this pull request

2026-10-15 22:40:26 +00:00

Memory: a cold open holds less at once (#398) #462

kayg commented

2026-10-15 22:40:47 +00:00

Author