PERF: Files rule 2 — bound lists by rows and bytes with signed keyset pages (#663) #680

New issue

Open

opened 2026-10-02 05:36:28 +00:00 by kayg · 1 comment

kayg commented

2026-10-02 05:36:28 +00:00

Owner

Context: #663 matrix and DESIGN §58 rule 2. Scope: Files must bound lists by rows and bytes with signed keyset pages. Shared client machinery belongs only to #665–#668.

Evidence at c4a61e8cf0:

Endpoint: GET /api/v1/files/entries.
Source: apps/web/src/lib/files/api.ts:19. Client PAGE=500, server permits 500 (lib.rs:2600); listAll drains all pages. Signed keyset cursor already exists at listing.rs:468; no total byte bound.
Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence.

Expected and regression tests:
At 10k items (and largest realistic plugin data), every response has ≤100 item rows and an explicit total byte cap; nested arrays count toward both. Signed cursors bind User, view/filter/sort/zone and keyset position. Edits between pages do not skip or repeat unchanged items; malformed/cross-User/replayed scope cursors cannot change authorization. No OFFSET and no startup whole-corpus drain. Client retains first page plus one lookahead and a bounded resident row/byte window. Preserve deep links, selection, Copy link and current list order. Test large values and variable row size at the byte boundary, empty pages, filter switches, deleted cursor anchor and cache eviction.

Performance test:
Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison.

Reuse/ownership:
This issue owns the minimum reusable signed cursor codec/bounds contract if other plugins need it. Extend listing.rs rather than build another cursor implementation. Expose only the public pieces that other adoption issues need. Preserve Files sync API limits if they have a separate protocol; the UI list bound is 100.
Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code.

Acceptance:
Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input.

Representative measurement from the audit (not a full-view budget result):
/api/v1/files/entries?limit=100, five serial requests after fixture readiness, all HTTP 200. Median/p95/max 87.8/1069.4/1069.4 ms; five-request burst median/p95/max 2033.2/2034.0/2034.0 ms. Serial window server CPU 1090 ms, RSS 443625472 bytes; no ETag on these sampled responses. Load inside lock 3.32/1.98/0.88.
Shared release server source cc25c441b7a974185622a1dee853cf38686d2b67, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9; embedded production SPA; Chromium browser on the build host through SSH/HTTPS. Server/Home/Index on perf VM HDD emulator: direct-I/O loop, 8 ms read/write dm-delay, 200 IOPS and 150 MiB/s caps. Every measured phase held flock -w 14400 /root/perf.lock. Qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty, Admin one User.
Structural source evidence above is the newer audit base, not the measured binary revision. No claim that these revisions are equivalent. The baseline in docs/perf/baseline.json uses another fixture/build/transport; no regression ratio is valid here. See #663 for matching baseline endpoint values and coverage gaps.

Context: #663 matrix and DESIGN §58 rule 2. Scope: Files must bound lists by rows and bytes with signed keyset pages. Shared client machinery belongs only to #665–#668. Evidence at c4a61e8cf090170f35b1bed3350d9de20c83ecd5: - Endpoint: `GET /api/v1/files/entries`. - Source: `apps/web/src/lib/files/api.ts:19`. Client PAGE=500, server permits 500 (lib.rs:2600); listAll drains all pages. Signed keyset cursor already exists at listing.rs:468; no total byte bound. - Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence. Expected and regression tests: At 10k items (and largest realistic plugin data), every response has ≤100 item rows and an explicit total byte cap; nested arrays count toward both. Signed cursors bind User, view/filter/sort/zone and keyset position. Edits between pages do not skip or repeat unchanged items; malformed/cross-User/replayed scope cursors cannot change authorization. No OFFSET and no startup whole-corpus drain. Client retains first page plus one lookahead and a bounded resident row/byte window. Preserve deep links, selection, Copy link and current list order. Test large values and variable row size at the byte boundary, empty pages, filter switches, deleted cursor anchor and cache eviction. Performance test: Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison. Reuse/ownership: This issue owns the minimum reusable signed cursor codec/bounds contract if other plugins need it. Extend listing.rs rather than build another cursor implementation. Expose only the public pieces that other adoption issues need. Preserve Files sync API limits if they have a separate protocol; the UI list bound is 100. Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code. Acceptance: Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input. Representative measurement from the audit (not a full-view budget result): `/api/v1/files/entries?limit=100`, five serial requests after fixture readiness, all HTTP 200. Median/p95/max 87.8/1069.4/1069.4 ms; five-request burst median/p95/max 2033.2/2034.0/2034.0 ms. Serial window server CPU 1090 ms, RSS 443625472 bytes; no ETag on these sampled responses. Load inside lock 3.32/1.98/0.88. Shared release server source `cc25c441b7a974185622a1dee853cf38686d2b67`, binary SHA-256 `2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9`; embedded production SPA; Chromium browser on the build host through SSH/HTTPS. Server/Home/Index on perf VM HDD emulator: direct-I/O loop, 8 ms read/write dm-delay, 200 IOPS and 150 MiB/s caps. Every measured phase held `flock -w 14400 /root/perf.lock`. Qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty, Admin one User. Structural source evidence above is the newer audit base, not the measured binary revision. No claim that these revisions are equivalent. The baseline in docs/perf/baseline.json uses another fixture/build/transport; no regression ratio is valid here. See #663 for matching baseline endpoint values and coverage gaps.

kayg referenced this issue

2026-10-02 05:46:53 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:10:35 +00:00

Author

Owner

Client runtime audit for #663; existing #670/#680 own this fix.

origin/dev c4a61e8cf0, unchanged load strategy on merge-round-7a 2f4482ded0: FilesBrowser.svelte:224–259 calls listAll; files/api.ts:145–179 walks all remaining pages automatically and emits the cumulative array. Each publication feeds FilesBrowser:195–211 sorting, keys, byKey and collection items over the growing folder. Rendering is virtualised, but resident rows and requests remain proportional to the entire folder. firstPageCache bounds first pages only, not mounted raw data.

Keep initial page paint, then demand-driven pages and one prefetch. Set row and byte caps on mounted data as well as snapshots. Retain selection identity when pages leave memory; do not make Select all operate only on resident rows. Add a 100k-entry folder test: opening must not fetch every page; after long scroll, resident rows/bytes stay capped and unrelated selection does not rebuild sorted item metadata. This is code evidence; no latency budget violation is claimed.

Client runtime audit for #663; existing #670/#680 own this fix. origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5, unchanged load strategy on merge-round-7a 2f4482ded066d9c5d9c59130377907f7fd2916c9: FilesBrowser.svelte:224–259 calls listAll; files/api.ts:145–179 walks all remaining pages automatically and emits the cumulative array. Each publication feeds FilesBrowser:195–211 sorting, keys, byKey and collection items over the growing folder. Rendering is virtualised, but resident rows and requests remain proportional to the entire folder. firstPageCache bounds first pages only, not mounted raw data. Keep initial page paint, then demand-driven pages and one prefetch. Set row and byte caps on mounted data as well as snapshots. Retain selection identity when pages leave memory; do not make Select all operate only on resident rows. Add a 100k-entry folder test: opening must not fetch every page; after long scroll, resident rows/bytes stay capped and unrelated selection does not rebuild sorted item metadata. This is code evidence; no latency budget violation is claimed.

kayg referenced this issue

2026-10-02 13:17:31 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:23:22 +00:00

Log attachments: deleted file still shown, attachments not live, batch send not live, Log order, raw camera names #427

kayg referenced this issue

2026-10-03 12:51:32 +00:00

Issue hygiene sweep 2026-10-03 #1008