kayg/calternal

Watch

Fork 0

PERF: Notes rule 8 — keep counts and background work off the first usable path (#663) #704

New issue

Open

opened 2026-10-02 05:39:25 +00:00 by kayg · 1 comment

kayg commented

2026-10-02 05:39:25 +00:00

Owner

Context: #663 matrix and DESIGN §58 rule 8. Scope: Notes must keep counts and background work off the first usable path. Shared client machinery belongs only to #665–#668.

Evidence at c4a61e8cf0:

Endpoint: GET /api/v1/notes.
Source: apps/web/src/lib/notes/noteIndex.svelte.ts:51. Navigator/index metadata drains all title pages. General Note GET parses on request; separate reader pool already exists for list queries.
Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence.

Expected and regression tests:
Compare first-row/first-card paint with counts/stats deliberately delayed. Interactive input and usable rows must not await counts, thumbnail jobs or non-selected sidebars. Under a held writer and indexing/backfill batches, reads use a separate read-only pool and return a complete committed answer. Bound batches, yield between them and record CPU/RSS plus per-phase latency. Keep all visible totals correct; do not remove counts or replace them with fake values. Record accepted and durable timings via #667, without content or credentials.

Performance test:
Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison.

Reuse/ownership:
General Note projections must not normalize a source file or write on open (#661). Preserve collaboration epochs and pending edits (#634 and Notes restart issues); reuse #549 Journal source projection and publish revisions atomically.
Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code.

Acceptance:
Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input.

Representative measurement: corrected GET /api/v1/notes?limit=100 on the locked perf VM, five successful serial reads and five successful concurrent reads. Serial median/p95/max 52.2/69.6/69.6 ms; burst 124.7/125.4/125.4 ms. Response has 100 summary rows and 12,660 bytes; no HTTP ETag. Serial CPU 570 ms, RSS 422936576 bytes; load inside lock 2.7/2.43/1.81. This is a list-read measurement, not body/open/edit acceptance. The first audit used the wrong trailing-slash URI and returned 404; those samples are excluded from this result and retained on #663.

Runtime source cc25c441b7a974185622a1dee853cf38686d2b67, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9, shared release server with embedded production SPA. Source evidence above is the newer audit base; do not infer code equivalence. HDD: bench/hdd-emu.sh, direct-I/O loop/ext4/dm-delay 8 ms read/write, 200 IOPS/150 MiB/s caps, flock -w 14400 /root/perf.lock around every phase. QD1 125.0 IOPS/8.028 ms median; QD16 200.9 IOPS/100.139 ms. Fixture: 366 Daily notes, 10,980 Logs, 20 other Notes/Tasks, 100 Files/Photos, three Budgets and 100 transactions. The phase later stopped on an Admin burst transport error before a Note-body probe; no body result is claimed.

Baseline docs/perf/baseline.json Notes list p50/p95 1.3/3.1 ms uses another fixture, build and transport. No controlled regression ratio is valid. #663 records the coverage gap.

Context: #663 matrix and DESIGN §58 rule 8. Scope: Notes must keep counts and background work off the first usable path. Shared client machinery belongs only to #665–#668. Evidence at c4a61e8cf090170f35b1bed3350d9de20c83ecd5: - Endpoint: `GET /api/v1/notes`. - Source: `apps/web/src/lib/notes/noteIndex.svelte.ts:51`. Navigator/index metadata drains all title pages. General Note GET parses on request; separate reader pool already exists for list queries. - Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence. Expected and regression tests: Compare first-row/first-card paint with counts/stats deliberately delayed. Interactive input and usable rows must not await counts, thumbnail jobs or non-selected sidebars. Under a held writer and indexing/backfill batches, reads use a separate read-only pool and return a complete committed answer. Bound batches, yield between them and record CPU/RSS plus per-phase latency. Keep all visible totals correct; do not remove counts or replace them with fake values. Record accepted and durable timings via #667, without content or credentials. Performance test: Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison. Reuse/ownership: General Note projections must not normalize a source file or write on open (#661). Preserve collaboration epochs and pending edits (#634 and Notes restart issues); reuse #549 Journal source projection and publish revisions atomically. Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code. Acceptance: Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input. Representative measurement: corrected `GET /api/v1/notes?limit=100` on the locked perf VM, five successful serial reads and five successful concurrent reads. Serial median/p95/max 52.2/69.6/69.6 ms; burst 124.7/125.4/125.4 ms. Response has 100 summary rows and 12,660 bytes; no HTTP ETag. Serial CPU 570 ms, RSS 422936576 bytes; load inside lock 2.7/2.43/1.81. This is a list-read measurement, not body/open/edit acceptance. The first audit used the wrong trailing-slash URI and returned 404; those samples are excluded from this result and retained on #663. Runtime source `cc25c441b7a974185622a1dee853cf38686d2b67`, binary SHA-256 `2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9`, shared release server with embedded production SPA. Source evidence above is the newer audit base; do not infer code equivalence. HDD: bench/hdd-emu.sh, direct-I/O loop/ext4/dm-delay 8 ms read/write, 200 IOPS/150 MiB/s caps, flock -w 14400 /root/perf.lock around every phase. QD1 125.0 IOPS/8.028 ms median; QD16 200.9 IOPS/100.139 ms. Fixture: 366 Daily notes, 10,980 Logs, 20 other Notes/Tasks, 100 Files/Photos, three Budgets and 100 transactions. The phase later stopped on an Admin burst transport error before a Note-body probe; no body result is claimed. Baseline docs/perf/baseline.json Notes list p50/p95 1.3/3.1 ms uses another fixture, build and transport. No controlled regression ratio is valid. #663 records the coverage gap.

kayg referenced this issue

2026-10-02 05:46:53 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:10:50 +00:00

Author

Owner

Additional disk IO evidence from perf-arch-io (#663), source origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5. This belongs to #704's background priority work; no duplicate issue is needed.

crates/plugins/notes/src/lib.rs:99-101 calls Task reconciliation and then Note reconciliation. tasks_store.rs:695 and store.rs:1932 each call the synchronous full-Home Markdown scan. store.rs:1884-1927 enumerates the entire visible Home, reads every .md file and retains all text before returning. The first pass calls store::index for every Note (tasks_store.rs:718); the second calls it again (store.rs:1935). store::index:671 rebuilds both the Note and Task projections, and index_note_projection:715 starts a writer transaction without first checking a source hash. Even unchanged files receive projection work. lib.rs:2100 holds the per-User lock over this full operation.

This path runs after the startup Files completion notice (lib.rs:2063-2064) and through the hourly notes.reconcile schedule (calternal-server/src/wire.rs:5025). Merge-round-7a retains both scans and the unconditional per-file indexing. Queued #653 changes Journal ACK publication, not these reconciliation loops.

Reasoned impact: repeated directory/stat operations and duplicate Markdown reads/parses on HDD; on host-cached data the extra parsing and writer transactions remain. Large scans run synchronous IO on a Tokio worker and delay edits under the per-User lock. #549's normal Journal snapshot reads now use WAL and should not be described as still taking that lock. This audit did not time these loops.

Fix within #704: one bounded source scan, dependency-ordered Task/Note publication, a durable source fingerprint/hash to skip unchanged projections, and bounded lock/write batches. Run file IO off Tokio. Preserve stable Log IDs, read-your-writes, Task-target dependency order and out-of-band restore repair. Do not use size/mtime alone where same-size/timestamp changes must be detected.

Tests: instrument reads and projection calls; an unchanged second reconcile must avoid duplicate source parsing/writes, while a changed Task target is indexed before its dependent Daily note. Pause a scan and prove normal Journal GET still returns the complete old committed snapshot and an unrelated edit can proceed between batches. Keep restore/delete/race tests. Extend the locked HDD startup/background profile rather than copying #549's older timings into a new benchmark result.

Additional disk IO evidence from perf-arch-io (#663), source origin/dev `c4a61e8cf090170f35b1bed3350d9de20c83ecd5`. This belongs to #704's background priority work; no duplicate issue is needed. `crates/plugins/notes/src/lib.rs:99-101` calls Task reconciliation and then Note reconciliation. `tasks_store.rs:695` and `store.rs:1932` each call the synchronous full-Home Markdown scan. `store.rs:1884-1927` enumerates the entire visible Home, reads every .md file and retains all text before returning. The first pass calls `store::index` for every Note (`tasks_store.rs:718`); the second calls it again (`store.rs:1935`). `store::index:671` rebuilds both the Note and Task projections, and `index_note_projection:715` starts a writer transaction without first checking a source hash. Even unchanged files receive projection work. `lib.rs:2100` holds the per-User lock over this full operation. This path runs after the startup Files completion notice (`lib.rs:2063-2064`) and through the hourly notes.reconcile schedule (`calternal-server/src/wire.rs:5025`). Merge-round-7a retains both scans and the unconditional per-file indexing. Queued #653 changes Journal ACK publication, not these reconciliation loops. Reasoned impact: repeated directory/stat operations and duplicate Markdown reads/parses on HDD; on host-cached data the extra parsing and writer transactions remain. Large scans run synchronous IO on a Tokio worker and delay edits under the per-User lock. #549's normal Journal snapshot reads now use WAL and should not be described as still taking that lock. This audit did not time these loops. Fix within #704: one bounded source scan, dependency-ordered Task/Note publication, a durable source fingerprint/hash to skip unchanged projections, and bounded lock/write batches. Run file IO off Tokio. Preserve stable Log IDs, read-your-writes, Task-target dependency order and out-of-band restore repair. Do not use size/mtime alone where same-size/timestamp changes must be detected. Tests: instrument reads and projection calls; an unchanged second reconcile must avoid duplicate source parsing/writes, while a changed Task target is indexed before its dependent Daily note. Pause a scan and prove normal Journal GET still returns the complete old committed snapshot and an unrelated edit can proceed between batches. Keep restore/delete/race tests. Extend the locked HDD startup/background profile rather than copying #549's older timings into a new benchmark result.

kayg referenced this issue

2026-10-02 13:18:27 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:21:20 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#704

No description provided.

Rows
Columns