#679 - PERF: Calendar rule 8 — keep counts and background work off the first usable path (#663) - kayg/calternal

kayg commented

2026-10-02 05:36:27 +00:00

Owner

Context: #663 matrix and DESIGN §58 rule 8. Scope: Calendar must keep counts and background work off the first usable path. Shared client machinery belongs only to #665–#668.

Evidence at c4a61e8cf0:

Endpoint: GET /api/v1/calendar/range.
Source: crates/plugins/calendar/src/view.rs:579. Range computes detailed aggregation and awaits thumbnail enqueue before response; read-only pools exist but counts/work are still on first usable path.
Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence.

Expected and regression tests:
Compare first-row/first-card paint with counts/stats deliberately delayed. Interactive input and usable rows must not await counts, thumbnail jobs or non-selected sidebars. Under a held writer and indexing/backfill batches, reads use a separate read-only pool and return a complete committed answer. Bound batches, yield between them and record CPU/RSS plus per-phase latency. Keep all visible totals correct; do not remove counts or replace them with fake values. Record accepted and durable timings via #667, without content or credentials.

Performance test:
Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison.

Reuse/ownership:

Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code.

Acceptance:
Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input.

Representative measurement from the audit (not a full-view budget result):
/api/v1/calendar/range?from=2026-09-18&to=2026-10-02&tz=UTC, five serial requests after fixture readiness, all HTTP 200. Median/p95/max 64.2/518.9/518.9 ms; five-request burst median/p95/max 107.6/123.7/123.7 ms. Serial window server CPU 290 ms, RSS 286105600 bytes; no ETag on these sampled responses. Load inside lock 3.1/1.9/0.84.
Shared release server source cc25c441b7a974185622a1dee853cf38686d2b67, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9; embedded production SPA; Chromium browser on the build host through SSH/HTTPS. Server/Home/Index on perf VM HDD emulator: direct-I/O loop, 8 ms read/write dm-delay, 200 IOPS and 150 MiB/s caps. Every measured phase held flock -w 14400 /root/perf.lock. Qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty, Admin one User.
Structural source evidence above is the newer audit base, not the measured binary revision. No claim that these revisions are equivalent. The baseline in docs/perf/baseline.json uses another fixture/build/transport; no regression ratio is valid here. See #663 for matching baseline endpoint values and coverage gaps.

Context: #663 matrix and DESIGN §58 rule 8. Scope: Calendar must keep counts and background work off the first usable path. Shared client machinery belongs only to #665–#668. Evidence at c4a61e8cf090170f35b1bed3350d9de20c83ecd5: - Endpoint: `GET /api/v1/calendar/range`. - Source: `crates/plugins/calendar/src/view.rs:579`. Range computes detailed aggregation and awaits thumbnail enqueue before response; read-only pools exist but counts/work are still on first usable path. - Representative endpoint numbers, server/client revisions, fixture size, lock/HDD qualification and limits are in the #663 production/HDD table. Those numbers do not prove this rule passes; the structural gap above is separate evidence. Expected and regression tests: Compare first-row/first-card paint with counts/stats deliberately delayed. Interactive input and usable rows must not await counts, thumbnail jobs or non-selected sidebars. Under a held writer and indexing/backfill batches, reads use a separate read-only pool and return a complete committed answer. Bound batches, yield between them and record CPU/RSS plus per-phase latency. Keep all visible totals correct; do not remove counts or replace them with fake values. Record accepted and durable timings via #667, without content or credentials. Performance test: Extend the existing bench profile for this hot path and the #549/#641 harness. Use production builds on root@10.69.69.63, bench/hdd-emu.sh and flock -w 14400 /root/perf.lock. Record load inside the lock; ≥5 samples, median/p95/max, average CPU/RSS and one realistic large-data/burst case. Separate warm, cold, accepted and durable boundaries. First usable 10k view ≤1.5 s; cached open/warm return ≤100 ms and accepted action ≤150 ms where applicable. Compare only a matching baseline in docs/perf/baseline.json; missing profiles require a new recorded baseline, not a made-up comparison. Reuse/ownership: Reuse #555 userStorage, #549 route caches, Files signed keysets/change feed and the existing Db reader_pool. #641 owns blaze measurements; #642 owns Settings opening; #640 owns Mail layouts; #639 owns linked Note opening. Integrate their active/completed branches before changing related code. Acceptance: Existing tests and status expectations stay intact. Run the per-crate gates (and calternal-server for route/contract changes), web gates if changed, and one time-boxed real-server regression/adversarial round for any new API contract. Preserve calternal-fs as the only filesystem interface and the server as the only writer. UI changes need pointer/touch/keyboard/screen-reader coverage and real-production captures at 390/820/1440 in both themes. Do not change shared motion for keyboard input. Representative measurement from the audit (not a full-view budget result): `/api/v1/calendar/range?from=2026-09-18&to=2026-10-02&tz=UTC`, five serial requests after fixture readiness, all HTTP 200. Median/p95/max 64.2/518.9/518.9 ms; five-request burst median/p95/max 107.6/123.7/123.7 ms. Serial window server CPU 290 ms, RSS 286105600 bytes; no ETag on these sampled responses. Load inside lock 3.1/1.9/0.84. Shared release server source `cc25c441b7a974185622a1dee853cf38686d2b67`, binary SHA-256 `2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9`; embedded production SPA; Chromium browser on the build host through SSH/HTTPS. Server/Home/Index on perf VM HDD emulator: direct-I/O loop, 8 ms read/write dm-delay, 200 IOPS and 150 MiB/s caps. Every measured phase held `flock -w 14400 /root/perf.lock`. Qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty, Admin one User. Structural source evidence above is the newer audit base, not the measured binary revision. No claim that these revisions are equivalent. The baseline in docs/perf/baseline.json uses another fixture/build/transport; no regression ratio is valid here. See #663 for matching baseline endpoint values and coverage gaps.

kayg referenced this issue

2026-10-02 05:46:53 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:10:37 +00:00

PERF: Analytics GET waits for derived cache writes and recomputes cold history #782

kayg commented

2026-10-02 13:14:19 +00:00

Author

Owner

#663 sync audit evidence for rule 8, base c4a61e8cf, checked in round-7a 2f4482ded. Keep #679 as the Calendar background-work owner.

calendar/src/routes.rs:220 refresh_all reads all enabled owners and awaits each User, then each account and each calendar serially (round-7a :257 onward). The startup refresh has no per-account durable continuation or fairness queue; one slow provider can delay later Users. The CalDAV client bounds one response to 5,000 objects and 8 MiB (client/mod.rs:27–29) and rejects more-results rather than paging it. This bounds memory but can prevent large account catch-up; test recovery, do not mark bounded rejection as successful convergence.

calendar/src/cache/store.rs:521 acquires the one writer connection before parsing metadata and recurrence bounds for each item (:546 and :550) in the full batch. WAL readers remain separate, so this does not itself prove read serialization. It does hold the shared writer while CPU parses a bounded 5,000-object response, and other interactive mutations must wait. Precompute/validate before the transaction; use a consistent staging/publish boundary or bounded commit protocol without acknowledging partial snapshots. The provider cursor and projection must commit together. Extend tests for a slow first provider and an unrelated User, and for interactive mutations during a 5,000-object batch.

Source-derived impact only. No runtime latency claim or new standalone issue; keep the source/fix/test evidence here under #679.

#663 sync audit evidence for rule 8, base c4a61e8cf, checked in round-7a 2f4482ded. Keep #679 as the Calendar background-work owner. `calendar/src/routes.rs:220` refresh_all reads all enabled owners and awaits each User, then each account and each calendar serially (round-7a :257 onward). The startup refresh has no per-account durable continuation or fairness queue; one slow provider can delay later Users. The CalDAV client bounds one response to 5,000 objects and 8 MiB (client/mod.rs:27–29) and rejects more-results rather than paging it. This bounds memory but can prevent large account catch-up; test recovery, do not mark bounded rejection as successful convergence. `calendar/src/cache/store.rs:521` acquires the one writer connection before parsing metadata and recurrence bounds for each item (:546 and :550) in the full batch. WAL readers remain separate, so this does not itself prove read serialization. It does hold the shared writer while CPU parses a bounded 5,000-object response, and other interactive mutations must wait. Precompute/validate before the transaction; use a consistent staging/publish boundary or bounded commit protocol without acknowledging partial snapshots. The provider cursor and projection must commit together. Extend tests for a slow first provider and an unrelated User, and for interactive mutations during a 5,000-object batch. Source-derived impact only. No runtime latency claim or new standalone issue; keep the source/fix/test evidence here under #679.

kayg commented

2026-10-02 13:14:39 +00:00

Author

Owner

Client audit for #663. Keep this under #669/#679; no duplicate issue.

At origin/dev c4a61e8cf0, calendar/data.ts:397–402 reads /calendar/range then awaits addTasks (/notes/tasks/day) before returning even though the Task request range is already known. readYear at :530–548 has the same sequence. Warm cache returns already show retained data and refresh in the background; keep that behavior. Independent source requests can start together, or primary data can publish before the optional Task update, with generation checks and retained identity. Tests should hold the Task reply pending and prove the primary data can become usable without losing later Tasks. No browser time is claimed. The new pointer-path finding #751 is a separate focused slice, not a whole-grid reactive-layout claim.

Client audit for #663. Keep this under #669/#679; no duplicate issue. At origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5, calendar/data.ts:397–402 reads /calendar/range then awaits addTasks (/notes/tasks/day) before returning even though the Task request range is already known. readYear at :530–548 has the same sequence. Warm cache returns already show retained data and refresh in the background; keep that behavior. Independent source requests can start together, or primary data can publish before the optional Task update, with generation checks and retained identity. Tests should hold the Task reply pending and prove the primary data can become usable without losing later Tasks. No browser time is claimed. The new pointer-path finding #751 is a separate focused slice, not a whole-grid reactive-layout claim.

kayg commented

2026-10-02 13:15:43 +00:00

Author

Owner

Additional #663 source evidence at c4a61e8cf: subscription lists (calendar/src/feeds/subscriptions.rs:336) read JSON definitions then await read_cache once per subscription; Calendar range repeats that loop at :1418, decodes each cached event vector and expands recurrence. This is one query per feed, not per Event, but range work grows with subscriptions even for a narrow day. Batch status/cache reads, precompute revision/timezone-window projections, and keep subscription JSON reads off interactive paths. Coordinate with #677; preserve explicit refresh semantics. Add query-count and zero-file-read assertions for several subscriptions. No latency claim.

Additional #663 source evidence at c4a61e8cf: subscription lists (`calendar/src/feeds/subscriptions.rs:336`) read JSON definitions then await read_cache once per subscription; Calendar range repeats that loop at :1418, decodes each cached event vector and expands recurrence. This is one query per feed, not per Event, but range work grows with subscriptions even for a narrow day. Batch status/cache reads, precompute revision/timezone-window projections, and keep subscription JSON reads off interactive paths. Coordinate with #677; preserve explicit refresh semantics. Add query-count and zero-file-read assertions for several subscriptions. No latency claim.

kayg referenced this issue

2026-10-02 13:16:50 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:17:31 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:18:14 +00:00

PERF: Calendar rule 1 — serve interactive reads from committed precomputed projections (#663) #677

kayg referenced this issue

2026-10-02 13:19:27 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:19:51 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:21:20 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:34:17 +00:00

Author

Owner

SQLite audit #663. Base c4a61e8cf0; queued round-7a 2f4482ded0.

Additional writer occupancy evidence: base cache/store.rs:522 begins apply_calendar_sync before parsing event_metadata and indexed_bounds in its change loop. Full snapshots first delete the old calendar. Round-7a retains this. Prepare metadata before writer checkout; stage large replacements and use a small final publish with bounded retirement. Preserve complete old/new snapshots and provider cursor atomicity. Test failure/restart and an unrelated writer between durable batches. WAL readers do not normally wait on the write lock; the sole writer queue and shared disk I/O are the concrete resources.

Figures are local Python SQLite 3.53.3 query work in 1,000-instruction callback units, not production latency. Rust bundles 3.51.3. Recheck production plans before implementation; no product edit or assertion change.

SQLite audit #663. Base c4a61e8cf090170f35b1bed3350d9de20c83ecd5; queued round-7a 2f4482ded066d9c5d9c59130377907f7fd2916c9. Additional writer occupancy evidence: base cache/store.rs:522 begins apply_calendar_sync before parsing event_metadata and indexed_bounds in its change loop. Full snapshots first delete the old calendar. Round-7a retains this. Prepare metadata before writer checkout; stage large replacements and use a small final publish with bounded retirement. Preserve complete old/new snapshots and provider cursor atomicity. Test failure/restart and an unrelated writer between durable batches. WAL readers do not normally wait on the write lock; the sole writer queue and shared disk I/O are the concrete resources. Figures are local Python SQLite 3.53.3 query work in 1,000-instruction callback units, not production latency. Rust bundles 3.51.3. Recheck production plans before implementation; no product edit or assertion change.

kayg referenced this issue

2026-10-02 13:39:12 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-03 12:51:32 +00:00

Issue hygiene sweep 2026-10-03 #1008

Rows
Columns

PERF: Calendar rule 8 — keep counts and background work off the first usable path (#663) #679