Merge train 2026-10-15j: server speed at 100k (#44), bar-stays test (#387) #391

Merged

kayg merged 25 commits from train/2026-10-15j into dev 2026-10-15 02:22:45 +00:00

Conversation 0 Commits 25 Files changed 15 +856 -147

kayg commented 2026-10-15 02:22:38 +00:00

Owner

Copy link

Server speed from #44, plus the #387 test fix.

Includes #227, #291, #315, #316, #338, #339, #340, #382, #384, #385, #389. Held back for kayg: #314, #358 and #386 (they add the first unsafe code; see #174).

Resolved two conflicts: in search/read.rs, #339's recent(table) now works with #340's newest(); and #384 and #385 each added a test at the end of opening.rs.

Train j results: cargo fmt and clippy (-D warnings) clean, and every workspace test passes. Browser tests (flow, notes, links, tags, trash, tree, day, picker, served, pwa, editor, switching, bar-stays and more): chromium-390 48 passed, webkit-390 47, chromium-1440 65. 0 failed.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Server speed from #44, plus the #387 test fix. Includes #227, #291, #315, #316, #338, #339, #340, #382, #384, #385, #389. Held back for kayg: #314, #358 and #386 (they add the first `unsafe` code; see #174). Resolved two conflicts: in `search/read.rs`, #339's `recent(table)` now works with #340's `newest()`; and #384 and #385 each added a test at the end of `opening.rs`. Train j results: cargo fmt and clippy (-D warnings) clean, and every workspace test passes. Browser tests (flow, notes, links, tags, trash, tree, day, picker, served, pwa, editor, switching, bar-stays and more): chromium-390 48 passed, webkit-390 47, chromium-1440 65. 0 failed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 25 commits 2026-10-15 02:22:38 +00:00

Notes: a compact id index ... 6b10db7381

The id index held every note's id and path twice over, in two maps of
owned strings: about 25 MB of heap at 100,000 notes, most of what was
left at rest (#44). Each note's id and path are now stored once, in a
slot; the lookups by id and by folded path map a hash to slot numbers and
check the slot on every hit (a collision keeps a short list, newest
first, like a map's overwrite). A model test runs thousands of puts,
renames, case changes and removes against two plain maps.

Heap at rest at 100k notes: about 8 MB less.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Perf: search measured once its index has caught up ... 34af2271df

`search.query` ran while a first search build (2 to 4 minutes at 100,000
notes on the perf VM) was still going, so one run's build decided the
p95 and the spread between runs was over 100%. The harness now times the
first query on its own (`search.first_query`), waits for the index to
catch up (`search.catch_up`), then measures `search.query`.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: a first build reads its notes side by side ... 6d7f7ea2d0

The search index's first build fetched each note one after another, a
file read each: at 100,000 notes it took two and a half minutes on the
perf VM (no disk cache). The items of a batch (64) are now fetched side by
side and stored in the order asked for, so row ids still follow it.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: answers at 100k notes without scoring every match ... e3d445e02d

Profiling search on the 100,000-note fixture (perf VM p95 ~460 ms)
found three costs, none of them the index itself:

- Typo vocabulary: it was rebuilt on the query path whenever the index
  had changed, and building it reads the whole index (1.6 s at 100k).
  It is now asked for only when exact matches are few, and a big one is
  rebuilt on a thread of its own while searches use the one they have; a
  small one (quick to build, or under 5,000 items) is still rebuilt in
  place.
- Ranking with many matches: the 4 ms budget couldn't stop it, because
  FTS5 merges a common word's whole posting list in one step that never
  reaches the progress handler: 0.3 to 0.8 s. With many matches, titles,
  recency and frecency already cover the candidates, so it is skipped.
- Frecency: `MATCH ... AND rowid IN (200 ids)` runs the whole match
  again for every listed row (0.6 to 1.6 s for 30 rows). The recently
  used items are now checked on their text, stopping at the first words
  that satisfy the query.

The search tests' settle helper waits 250 ms (was 30) for the indexer to
pick up an emitted change; on a loaded CI host 30 ms was sometimes too
short (`the_index_follows_changes_live`).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: no frecency check or title ranking it doesn't need ... eb510621bc

Phase timings of steady-state searches at 100,000 notes on the perf VM:
checking the most-used items took 25 to 45 ms (it reads their text, long
notes included), and ranking title matches about 30 ms (FTS5 scores every
note holding the words, in any column, before keeping titles).

- The most-used items are checked only when there are many matches:
  with few, the newest matches already hold all of them. And only the
  64 most recent of them.
- Title matches come newest first and the ranking that orders all
  candidates orders them too.

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: one letter looks at titles; short words aren't typo-fixed ... 29ef5cf68a

More phase timings at 100,000 notes on the perf VM:
- A one-letter query (the palette's first keystroke) took 75-100 ms: a
  one-letter prefix isn't in FTS5's prefix lists, so the body search
  merged every word with that letter. One letter now looks at titles.
- One or two letters with few matches went through typo correction,
  which has nothing to fix in them: now only words of three letters or
  more are corrected.
- Checking the most-used items read their whole text, 50,000-word notes
  included; now their first 16 KB, where what they are about usually is.

Steady state at 100k (index caught up), 110 queries: p50 58-62 ms, p95
79-85 ms (dev: p50 85-165, p95 135-560).

Part of #44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: walk the few used things, not every item, to find them ... 30bfbed85b

Finding the 64 things used most recently joined every item to the used
ones: SQLite, left to choose, walked all 100,000 items. That was 16 ms of
every search at 100,000 notes, even with nothing used yet. CROSS JOIN
walks the used things first, and an index on when they were used finds
the newest without sorting them all. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: titles get an index of their own ... 2c07d114a5

A title match asked the full index for words in the title column, which
walks every note holding the words in any column: 3 to 11 ms at 100,000
notes, and 8 ms for the palette's first letter. `titles` holds the words
of titles alone (no text kept, ~3 MB at 100,000 notes): the same lookups
take under 1 ms. An index made before it fills it on open from what it
has, in a quarter second at 100,000 notes, rather than rebuilding. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Search: whole words first, then as prefixes ... 80f7ff9cd1

A typed word as a prefix (`coffee*`) makes FTS5 gather every note holding
any word that starts that way into one list before answering: 7 to 25 ms
a word at 100,000 notes when the word is common. As a whole word, FTS5
walks its list from the newest end and stops at enough matches: under
half a millisecond. So the newest matches are first looked up with words
of four letters or more whole; only if that finds too few does the
prefix form run, so a word still being typed is found as before. Words
of two or three letters stay prefixes: the index keeps their lists
ready. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: a cold Home fills its list, then makes the list's indexes ... a956a83510

The note list keeps eight secondary indexes with random keys (ids,
paths, link targets, tags). Filling an empty list store 4,096 notes per
transaction rewrote most of their pages every batch: a cold Home of
100,000 notes wrote 3.5 GB for a 240 MB store (cold.disk_written on the
perf VM).

Filling an empty store with 10,000 notes or more now drops those
indexes, adds the rows with nothing to clear first, and makes the
indexes once at the end. A replay of the 100k fill: 3,966 -> 1,274 MB
written, about a third of the time. Pages asked for meanwhile don't name
an index; an edit stored during the fill is cleared as usual; a stop
halfway is harmless, since opening the store makes any missing index.
Tests take the fresh way from 200 notes. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: the list store's write-ahead log shrinks back ... 53d881de9b

SQLite leaves a write-ahead log at the size of the biggest transaction it
ever held: on the perf VM a 100,000-note Home kept a 127 MB meta.db-wal
beside its 240 MB list store, after a cold fill and a few storms. With
journal_size_limit it's cut back to 8 MB whenever a checkpoint resets
it. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: a fresh fill keeps the list's newest-first index ... 470e971fc8

The app pages through the note list newest change first while a cold
Home fills; without notes_updated every page sorted the whole store.
Kept, it costs a tenth more writing (a replay of the 100k fill: 1,274 ->
1,409 MB, against 3,966 MB with every index kept up). (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Notes: shutting down empties the list store's log ... a2224be561

The list stores stay open until the process ends, so their write-ahead
logs were never folded back in: after a harness run at 100,000 notes the
perf VM had a 42 MB meta.db-wal, and the next start read it all back
before the first page (opening the store took 27 ms of a 41 ms first
page on dropped caches; busier runs before it left more). Shutting down
now checkpoints each list store with TRUNCATE. (#44)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

bar-stays: wait for the link-chip tap to settle instead of a fixed 500ms (#387) ... 6a671494f4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Merge remote-tracking branch 'origin/issue/44-index-2' into train/2026-10-15j dd8f360b3e

Merge remote-tracking branch 'origin/issue/44-perf-search-steady' into train/2026-10-15j c29c9dd959

Merge remote-tracking branch 'origin/issue/44-search-speed-2' into train/2026-10-15j c80f21851f

Merge remote-tracking branch 'origin/issue/44-search-build-2' into train/2026-10-15j 51a085fad2

Merge remote-tracking branch 'origin/issue/44-search-frecent' into train/2026-10-15j f378cc062c

Merge remote-tracking branch 'origin/issue/44-search-titles' into train/2026-10-15j 26797829ac

Merge remote-tracking branch 'origin/issue/44-search-exact' into train/2026-10-15j ... ac0f249268

# Conflicts:
#	crates/search/src/read.rs

Merge remote-tracking branch 'origin/issue/44-meta-wal' into train/2026-10-15j 3f56b9d6a8

Merge remote-tracking branch 'origin/issue/44-cold-list-2' into train/2026-10-15j c04c86f261

Merge remote-tracking branch 'origin/issue/44-clean-close' into train/2026-10-15j ... 16793b914f

# Conflicts:
#	crates/notes/src/tests/opening.rs

Merge remote-tracking branch 'origin/issue/387-bar-stays-webkit' into train/2026-10-15j 8b4fbbe4f3

kayg merged commit a305ee22a1 into dev 2026-10-15 02:22:45 +00:00

kayg referenced this pull request 2026-10-15 02:22:45 +00:00

Notes: a compact id index #227

kayg referenced this pull request from a commit 2026-10-15 02:22:46 +00:00

Merge pull request 'Merge train 2026-10-15j: server speed at 100k (#44), bar-stays test (#387)' (#391) from train/2026-10-15j into dev

kayg referenced this pull request 2026-10-15 02:22:48 +00:00

Perf: search measured once its index has caught up #291

kayg referenced this pull request 2026-10-15 02:22:51 +00:00

Search: answers at 100k notes without scoring every match #315

kayg referenced this pull request 2026-10-15 02:22:53 +00:00

Search: a first build reads its notes side by side #316

kayg referenced this pull request 2026-10-15 02:22:55 +00:00

Search: walk the few used things, not every item, to find them #338

kayg referenced this pull request 2026-10-15 02:22:57 +00:00

Search: titles get an index of their own #339

kayg referenced this pull request 2026-10-15 02:22:58 +00:00

Search: whole words first, then as prefixes #340

kayg referenced this pull request 2026-10-15 02:23:00 +00:00

Notes: the list store's write-ahead log shrinks back #382

kayg referenced this pull request 2026-10-15 02:23:02 +00:00

Notes: a cold Home fills its list, then makes the list's indexes #384

kayg referenced this pull request 2026-10-15 02:23:04 +00:00

Notes: shutting down empties the list store's log #385

kayg referenced this pull request 2026-10-15 02:23:06 +00:00

bar-stays: link-chip tap waits for the bar to settle (#387) #389

kayg referenced this pull request 2026-10-15 02:25:36 +00:00

kayg's requests and decisions, 2026-10-14 (index) #174

kayg referenced this pull request 2026-10-15 02:25:37 +00:00

Perf harness: large-Home fixtures, edit storms, perf-VM gate #44

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Rust server

  

design

Mockups, UI and TASTE

  

docs

ADRs, CONTEXT, TASTE

  

idea

Parked idea, never worked until promoted

  

infra

Deploy, CI, VMs

  

perf

Performance and budgets

  

reliability

Data integrity, corruption traps

  

web

Web client

No labels

backend

design

docs

idea

infra

perf

reliability

web

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal!391

No description provided.