kayg/calternal

Watch

Fork 0

PERF: Notes rule 2 — bound lists by rows and bytes with signed keyset pages (#663) #703

New issue

Open

opened 2026-10-02 05:39:24 +00:00 by kayg · 3 comments

kayg commented

2026-10-02 05:39:24 +00:00

Owner

Context: #663 matrix and DESIGN §58 rule 2. Scope: Notes must bound lists by rows and bytes with signed keyset pages. Shared client machinery belongs only to #665–#668.

Evidence at c4a61e8cf0:

Endpoint: GET /api/v1/notes.
Source: crates/plugins/notes/src/lib.rs:3344. Numeric cursor is OFFSET; API cap is 100 but bytes are not capped. noteIndex.svelte.ts:51 drains every page into byId.
Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence.

Expected and regression tests:
At 10k items (and largest realistic plugin data), every response has ≤100 item rows and an explicit total byte cap; nested arrays count toward both. Signed cursors bind User, view/filter/sort/zone and keyset position. Edits between pages do not skip or repeat unchanged items; malformed/cross-User/replayed scope cursors cannot change authorization. No OFFSET and no startup whole-corpus drain. Client retains first page plus one lookahead and a bounded resident row/byte window. Preserve deep links, selection, Copy link and current list order. Test large values and variable row size at the byte boundary, empty pages, filter switches, deleted cursor anchor and cache eviction.

Performance test:
Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison.

Reuse/ownership:

Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code.

Acceptance:
Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input.

Representative measurement: corrected GET /api/v1/notes?limit=100 on the locked perf VM, five successful serial reads and five successful concurrent reads. Serial median/p95/max 52.2/69.6/69.6 ms; burst 124.7/125.4/125.4 ms. Response has 100 summary rows and 12,660 bytes; no HTTP ETag. Serial CPU 570 ms, RSS 422936576 bytes; load inside lock 2.7/2.43/1.81. This is a list-read measurement, not body/open/edit acceptance. The first audit used the wrong trailing-slash URI and returned 404; those samples are excluded from this result and retained on #663.

Runtime source cc25c441b7a974185622a1dee853cf38686d2b67, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9, shared release server with embedded production SPA. Source evidence above is the newer audit base; do not infer code equivalence. HDD: bench/hdd-emu.sh, direct-I/O loop/ext4/dm-delay 8 ms read/write, 200 IOPS/150 MiB/s caps, flock -w 14400 /root/perf.lock around every phase. QD1 125.0 IOPS/8.028 ms median; QD16 200.9 IOPS/100.139 ms. Fixture: 366 Daily notes, 10,980 Logs, 20 other Notes/Tasks, 100 Files/Photos, three Budgets and 100 transactions. The phase later stopped on an Admin burst transport error before a Note-body probe; no body result is claimed.

Baseline docs/perf/baseline.json Notes list p50/p95 1.3/3.1 ms uses another fixture, build and transport. No controlled regression ratio is valid. #663 records the coverage gap.

Context: #663 matrix and DESIGN §58 rule 2. Scope: Notes must bound lists by rows and bytes with signed keyset pages. Shared client machinery belongs only to #665–#668. Evidence at c4a61e8cf090170f35b1bed3350d9de20c83ecd5: - Endpoint: `GET /api/v1/notes`. - Source: `crates/plugins/notes/src/lib.rs:3344`. Numeric cursor is OFFSET; API cap is 100 but bytes are not capped. noteIndex.svelte.ts:51 drains every page into byId. - Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence. Expected and regression tests: At 10k items (and largest realistic plugin data), every response has ≤100 item rows and an explicit total byte cap; nested arrays count toward both. Signed cursors bind User, view/filter/sort/zone and keyset position. Edits between pages do not skip or repeat unchanged items; malformed/cross-User/replayed scope cursors cannot change authorization. No OFFSET and no startup whole-corpus drain. Client retains first page plus one lookahead and a bounded resident row/byte window. Preserve deep links, selection, Copy link and current list order. Test large values and variable row size at the byte boundary, empty pages, filter switches, deleted cursor anchor and cache eviction. Performance test: Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison. Reuse/ownership: Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code. Acceptance: Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input. Representative measurement: corrected `GET /api/v1/notes?limit=100` on the locked perf VM, five successful serial reads and five successful concurrent reads. Serial median/p95/max 52.2/69.6/69.6 ms; burst 124.7/125.4/125.4 ms. Response has 100 summary rows and 12,660 bytes; no HTTP ETag. Serial CPU 570 ms, RSS 422936576 bytes; load inside lock 2.7/2.43/1.81. This is a list-read measurement, not body/open/edit acceptance. The first audit used the wrong trailing-slash URI and returned 404; those samples are excluded from this result and retained on #663. Runtime source `cc25c441b7a974185622a1dee853cf38686d2b67`, binary SHA-256 `2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9`, shared release server with embedded production SPA. Source evidence above is the newer audit base; do not infer code equivalence. HDD: bench/hdd-emu.sh, direct-I/O loop/ext4/dm-delay 8 ms read/write, 200 IOPS/150 MiB/s caps, flock -w 14400 /root/perf.lock around every phase. QD1 125.0 IOPS/8.028 ms median; QD16 200.9 IOPS/100.139 ms. Fixture: 366 Daily notes, 10,980 Logs, 20 other Notes/Tasks, 100 Files/Photos, three Budgets and 100 transactions. The phase later stopped on an Admin burst transport error before a Note-body probe; no body result is claimed. Baseline docs/perf/baseline.json Notes list p50/p95 1.3/3.1 ms uses another fixture, build and transport. No controlled regression ratio is valid. #663 records the coverage gap.

kayg referenced this issue

2026-10-02 05:46:53 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:10:34 +00:00

Author

Owner

Client audit evidence for #663; keep with #701/#703/#704, no duplicate issue.

At origin/dev c4a61e8cf0 and merge-round-7a 2f4482ded0, apps/web/src/lib/notes/noteIndex.svelte.ts:52–65 walks all 100-row pages in series and keeps every summary in byId/#byPath. NotesExplorer.svelte:127 starts this on mount. This is not a body-paint gate, but at 100k Notes it requests about 1,000 pages and retains the full directory. byTitle scans that directory for each lookup. Each recent-change page also replaces summaries through #put, even when the summary did not change.

NotesExplorer.svelte:154–191 reads noteIndex.revision, rebuilds and sorts each expanded folder's rows, and renders the full built row array (:399 onward). Folder page/show-more controls bound each initial folder, but expanded/displayed rows accumulate and are not virtualised. A change to an unrelated indexed Note therefore invalidates the visible tree.

Fix: keep a bounded visible directory window and indexed on-demand identity/title resolution; use narrow per-item subscriptions, preserve unchanged summaries, and virtualise long trees with the existing collection primitives. Test a large directory while editing one offscreen Note: no whole-tree sort or new objects for unchanged visible rows; test wikilinks, expanded folders, reveal and Copy link without whole-directory preload. No browser time is claimed. New editor-only findings are #747 and #749.

Client audit evidence for #663; keep with #701/#703/#704, no duplicate issue. At origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5 and merge-round-7a 2f4482ded066d9c5d9c59130377907f7fd2916c9, apps/web/src/lib/notes/noteIndex.svelte.ts:52–65 walks all 100-row pages in series and keeps every summary in byId/#byPath. NotesExplorer.svelte:127 starts this on mount. This is not a body-paint gate, but at 100k Notes it requests about 1,000 pages and retains the full directory. byTitle scans that directory for each lookup. Each recent-change page also replaces summaries through #put, even when the summary did not change. NotesExplorer.svelte:154–191 reads noteIndex.revision, rebuilds and sorts each expanded folder's rows, and renders the full built row array (:399 onward). Folder page/show-more controls bound each initial folder, but expanded/displayed rows accumulate and are not virtualised. A change to an unrelated indexed Note therefore invalidates the visible tree. Fix: keep a bounded visible directory window and indexed on-demand identity/title resolution; use narrow per-item subscriptions, preserve unchanged summaries, and virtualise long trees with the existing collection primitives. Test a large directory while editing one offscreen Note: no whole-tree sort or new objects for unchanged visible rows; test wikilinks, expanded folders, reveal and Copy link without whole-directory preload. No browser time is claimed. New editor-only findings are #747 and #749.

kayg referenced this issue

2026-10-02 13:10:40 +00:00

PERF: Tag suggestions rebuild whole-Home object sets and Tag pages use OFFSET #784

kayg referenced this issue

2026-10-02 13:10:49 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:12:19 +00:00

Author

Owner

Additional source/SQL-plan evidence from #663, base c4a61e8cf. GET /api/v1/notes?sort=title uses title COLLATE NOCASE,id (notes/src/lib.rs:3352), but migration 0001_notes.sql:11 creates a binary title index. A scratch SQLite Index with the exact migration gives:

SEARCH note_items USING INDEX sqlite_autoindex_note_items_2 (user_id=?)
USE TEMP B-TREE FOR ORDER BY

The edited order uses note_items_recent with no temp sort. This is a plan proof, not a latency measurement. Please include a matching indexed sort key/collation in #703, preserving current title ordering and existing test expectations. Test no temporary ORDER BY tree for both orders plus stable signed keyset continuation. The pending #665 branch adds HTTP 304 but still calls store::read/view before conditional_json; it does not resolve #702 read/parse work.

Additional source/SQL-plan evidence from #663, base c4a61e8cf. GET /api/v1/notes?sort=title uses `title COLLATE NOCASE,id` (`notes/src/lib.rs:3352`), but migration `0001_notes.sql:11` creates a binary title index. A scratch SQLite Index with the exact migration gives: ```text SEARCH note_items USING INDEX sqlite_autoindex_note_items_2 (user_id=?) USE TEMP B-TREE FOR ORDER BY ``` The edited order uses `note_items_recent` with no temp sort. This is a plan proof, not a latency measurement. Please include a matching indexed sort key/collation in #703, preserving current title ordering and existing test expectations. Test no temporary ORDER BY tree for both orders plus stable signed keyset continuation. The pending #665 branch adds HTTP 304 but still calls store::read/view before conditional_json; it does not resolve #702 read/parse work.

kayg referenced this issue

2026-10-02 13:12:21 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:17:31 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:19:51 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:23:22 +00:00

Log attachments: deleted file still shown, attachments not live, batch send not live, Log order, raw camera names #427

kayg commented

2026-10-02 13:24:03 +00:00

Author

Owner

SQLite audit #663 on job/perf-arch-db. Base c4a61e8cf0; queued merge-round-7a 2f4482ded0 checked separately.

Notes database evidence: notes/src/lib.rs:3355 orders by title COLLATE NOCASE,id; 0001_notes.sql:12 indexes BINARY title. On 100k synthetic Notes the first 101-row title page uses USE TEMP B-TREE FOR ORDER BY and about 1,201,000 VM instructions. A temporary (user_id,title COLLATE NOCASE,id) index removes the sort and returns the same ordered identities in about 1,000 instructions. Recent first page is about 1,000, offset 90,000 about 361,000. A simple edited seek is below 1,000, but the production keyset must also handle equal and NULL edited values and sign/scope the cursor. Add a matching NOCASE index with the keyset work; do not change sort semantics. Test large first/deep pages, mixed-case titles and equal/NULL edited values. Tasks list/standalone retain OFFSET too (:779/:810); task created ties need a matching composite order. Queued perf-cache-665 retains these queries.

Local query-work figures use Python SQLite 3.53.3 and synthetic data; VM callbacks count 1,000-instruction units. They are not production latency, CPU/RSS or HDD budget samples. The locked Rust bundled SQLite is 3.51.3. Repeat with audit-sqlite.py --plans and confirm the production-version plan before implementing. No existing assertion or product behavior was changed. Reuse this issue instead of filing a duplicate.

SQLite audit #663 on job/perf-arch-db. Base c4a61e8cf090170f35b1bed3350d9de20c83ecd5; queued merge-round-7a 2f4482ded066d9c5d9c59130377907f7fd2916c9 checked separately. Notes database evidence: notes/src/lib.rs:3355 orders by title COLLATE NOCASE,id; 0001_notes.sql:12 indexes BINARY title. On 100k synthetic Notes the first 101-row title page uses USE TEMP B-TREE FOR ORDER BY and about 1,201,000 VM instructions. A temporary (user_id,title COLLATE NOCASE,id) index removes the sort and returns the same ordered identities in about 1,000 instructions. Recent first page is about 1,000, offset 90,000 about 361,000. A simple edited seek is below 1,000, but the production keyset must also handle equal and NULL edited values and sign/scope the cursor. Add a matching NOCASE index with the keyset work; do not change sort semantics. Test large first/deep pages, mixed-case titles and equal/NULL edited values. Tasks list/standalone retain OFFSET too (:779/:810); task created ties need a matching composite order. Queued perf-cache-665 retains these queries. Local query-work figures use Python SQLite 3.53.3 and synthetic data; VM callbacks count 1,000-instruction units. They are not production latency, CPU/RSS or HDD budget samples. The locked Rust bundled SQLite is 3.51.3. Repeat with audit-sqlite.py --plans and confirm the production-version plan before implementing. No existing assertion or product behavior was changed. Reuse this issue instead of filing a duplicate.

kayg referenced this issue

2026-10-02 13:39:12 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#703

No description provided.

Rows
Columns