Performance budget: route load, INP and bundle size

kayg commented

2026-09-26 09:40:18 +00:00

Owner

Performance first (CLAUDE.md). Measure every main route on a production build (web build + release server, Playwright, CPU 4x throttling, 390 and 1440): first contentful paint, time to interactive data, INP for the common interactions (open palette, open composer, switch mode, scroll Calendar grid, open a 5k-file folder, open a 2k-photo album), JS/CSS per route. Commit the numbers (docs/perf/BASELINE.md with commands). Then cut: code-split heavy routes and editor/charts, lazy-load overlays, virtualise any list that is not, remove duplicate dependencies, avoid layout thrash. Decision rule: every change keeps a before/after number; revert changes that don't move the metric. Commit per improvement.

504 KiB

546 KiB

522 KiB

469 KiB

146 KiB

156 KiB

155 KiB

130 KiB

kayg commented

2026-09-26 09:41:23 +00:00

Author

Owner

Started #164 on branch job/perf from dev at 82d4c258bbcc6f34e8805d0018004fa8e621b9a5. Reading the product performance budgets and existing production Playwright harness. I will measure and commit the baseline before changing runtime code; the host is currently heavily loaded (load average 48.79, 79.63, 131.67), so I will record load beside timings and interleave before/after runs.

Started #164 on branch `job/perf` from `dev` at `82d4c258bbcc6f34e8805d0018004fa8e621b9a5`. Reading the product performance budgets and existing production Playwright harness. I will measure and commit the baseline before changing runtime code; the host is currently heavily loaded (load average 48.79, 79.63, 131.67), so I will record load beside timings and interleave before/after runs.

kayg commented

2026-09-26 10:01:21 +00:00

Author

Owner

Measurement finding: the release server embeds the SPA at compile time. A borrowed release binary served a different branch's UI, so I discarded those browser numbers. The matching release build is compiling now; no optimization has been applied. Host load during attempted runs was about 48–66 (1-minute average), so the final baseline will include load with each run.

kayg commented

2026-09-26 10:03:42 +00:00

Author

Owner

Baseline commit 81b7024: docs/perf/BASELINE.md records the production build (2m 59s), root layout JS 257,596 raw / 81,511 gzip bytes, root CSS 125,435 / 24,720 bytes, shared CSS 112,518 / 19,425 bytes. The matching release server is still compiling; route and interaction timings will be appended before runtime optimization. Current 1-minute load average is about 47–50.

Baseline commit `81b7024`: `docs/perf/BASELINE.md` records the production build (2m 59s), root layout JS 257,596 raw / 81,511 gzip bytes, root CSS 125,435 / 24,720 bytes, shared CSS 112,518 / 19,425 bytes. The matching release server is still compiling; route and interaction timings will be appended before runtime optimization. Current 1-minute load average is about 47–50.

kayg commented

2026-09-26 10:12:35 +00:00

Author

Owner

Build-host finding: the first matching release build failed in the shared sccache daemon, which tried to create a temporary directory at /home/kayg/Developer/calternal-wt/tz-days/target/tmp/ after that directory had been removed. The error was sccache: error: Failed to create temp dir while compiling miniz_oxide; no project source error was reported. Retrying with RUSTC_WRAPPER= and this worktree's TMPDIR.

Build-host finding: the first matching release build failed in the shared `sccache` daemon, which tried to create a temporary directory at `/home/kayg/Developer/calternal-wt/tz-days/target/tmp/` after that directory had been removed. The error was `sccache: error: Failed to create temp dir` while compiling `miniz_oxide`; no project source error was reported. Retrying with `RUSTC_WRAPPER=` and this worktree's `TMPDIR`.

kayg referenced this issue

2026-09-26 15:56:35 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 09:32:58 +00:00

Author

Owner

Owner 2026-09-27: performance review becomes a standard PERIODIC practice (not a merge gate); aim for minimal CPU and memory on average and in worst cases. Extend this issue's scope:

A reusable harness bench/run.sh (release build, fixed seeded datasets, interleaved runs, load averages recorded) that writes machine-readable results (JSON) and a human summary to docs/perf/.md, plus docs/perf/baseline.json.
Metrics, average: server idle RSS and CPU, p50/p95 latency for the main routes, web bundle size per route, first load and INP for common interactions. Worst case: peak RSS and CPU under a concurrent request storm, a large Home (10k notes, 50k files, 20k photos), a big import, a sync of thousands of files, the heaviest page renders (5k-file folder, 2k-photo album, 10k-entry Analytics year, 1 MB note).
bench/compare.py: compares a run with the baseline and prints regressions over thresholds (p95 latency +10%, peak RSS +15%, idle RSS +10%, bundle +5%, INP +20%); exit code 0 always (never a gate); an option to file one fj issue per regression.
Then do the original optimisation work: measure first, commit the baseline, one commit per improvement with before/after numbers.
Claude sets up the weekly timer once the harness lands.

Owner 2026-09-27: performance review becomes a standard PERIODIC practice (not a merge gate); aim for minimal CPU and memory on average and in worst cases. Extend this issue's scope: 1. A reusable harness bench/run.sh (release build, fixed seeded datasets, interleaved runs, load averages recorded) that writes machine-readable results (JSON) and a human summary to docs/perf/<date>.md, plus docs/perf/baseline.json. 2. Metrics, average: server idle RSS and CPU, p50/p95 latency for the main routes, web bundle size per route, first load and INP for common interactions. Worst case: peak RSS and CPU under a concurrent request storm, a large Home (10k notes, 50k files, 20k photos), a big import, a sync of thousands of files, the heaviest page renders (5k-file folder, 2k-photo album, 10k-entry Analytics year, 1 MB note). 3. bench/compare.py: compares a run with the baseline and prints regressions over thresholds (p95 latency +10%, peak RSS +15%, idle RSS +10%, bundle +5%, INP +20%); exit code 0 always (never a gate); an option to file one fj issue per regression. 4. Then do the original optimisation work: measure first, commit the baseline, one commit per improvement with before/after numbers. Claude sets up the weekly timer once the harness lands.

kayg commented

2026-09-27 09:34:45 +00:00

Author

Owner

Resuming #164 after the VM restart. Branch: job/perf; original base: dev at 82d4c258bbcc6f34e8805d0018004fa8e621b9a5; current checkpoint: a2cd32314ab54029e74f778b22b7b5d42f41a6e1. The tree is clean. The production bundle baseline is committed; route timings and the expanded periodic benchmark harness/comparer remain to finish. I will continue from this checkpoint and integrate current dev before final gates.

Resuming #164 after the VM restart. Branch: `job/perf`; original base: `dev` at `82d4c258bbcc6f34e8805d0018004fa8e621b9a5`; current checkpoint: `a2cd32314ab54029e74f778b22b7b5d42f41a6e1`. The tree is clean. The production bundle baseline is committed; route timings and the expanded periodic benchmark harness/comparer remain to finish. I will continue from this checkpoint and integrate current `dev` before final gates.

kayg commented

2026-09-27 09:53:13 +00:00

Author

Owner

Finding: the requested 2k-photo album workload is not available in this build. crates/plugins/photos/src/routes.rs exposes timeline, day, item, stack, upload and search routes; there is no album route or collection API. apps/web/src/lib/photos/PhotosSidebar.svelte also says albums await API support. I will measure the real Photos timeline with seeded libraries and record the album workload as unavailable.

Finding: the requested 2k-photo album workload is not available in this build. `crates/plugins/photos/src/routes.rs` exposes timeline, day, item, stack, upload and search routes; there is no album route or collection API. `apps/web/src/lib/photos/PhotosSidebar.svelte` also says albums await API support. I will measure the real Photos timeline with seeded libraries and record the album workload as unavailable.

kayg referenced this issue

2026-09-27 10:23:51 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 10:25:21 +00:00

Author

Owner

Finding: this build exposes no client sync protocol or sync session API (rg found no sync plugin or /api/v1/sync route). The full probe measures the large Home startup scan after fixed data is seeded while the server is stopped, plus real upload API throughput in separate throwaway Instances. This gives real reindex and import measurements, but it does not represent an active client sync session. I also retain the previously reported missing Photos album API and use the real timeline workload.

Finding: this build exposes no client sync protocol or sync session API (`rg` found no sync plugin or `/api/v1/sync` route). The full probe measures the large Home startup scan after fixed data is seeded while the server is stopped, plus real upload API throughput in separate throwaway Instances. This gives real reindex and import measurements, but it does not represent an active client sync session. I also retain the previously reported missing Photos album API and use the real timeline workload.

kayg commented

2026-09-27 10:26:29 +00:00

Author

Owner

Finding: the installed Bun 1.4.2 prints bun run help for bun --cwd apps/web run build and exits without building the SPA. I changed bench/run.sh to run the build from an apps/web subshell. The direct cd apps/web && bun run build production build is now running; browser timings will only use that matching output.

Finding: the installed Bun 1.4.2 prints `bun run` help for `bun --cwd apps/web run build` and exits without building the SPA. I changed `bench/run.sh` to run the build from an `apps/web` subshell. The direct `cd apps/web && bun run build` production build is now running; browser timings will only use that matching output.

kayg commented

2026-09-27 10:34:58 +00:00

Author

Owner

Finding: the matching production route probe now completes at 390 px and 1440 px. In this one-run smoke at 4× CPU throttle, data-ready was 2.9–8.3 s across the routes; transfer gzip per route was 270–322 kB JS and 48–53 kB CSS. The 24-client storm completed 10,976 requests with HTTP 200, p95 57.9 ms; idle RSS was 146,956,288 bytes. This smoke validates the harness and is not the interleaved baseline; the full run remains pending.

kayg commented

2026-09-27 10:37:58 +00:00

Author

Owner

Finding: the large Home probe waits for the first-owner setup token, but it launched the server with RUST_LOG=error. The server emits that one-time setup URL at warn (crates/calternal-auth/src/api.rs), so /readyz returned 200 while the probe could not read the token. I will set the probe's minimum log level to warn; the current smoke run is being stopped and will be repeated.

Finding: the large Home probe waits for the first-owner setup token, but it launched the server with `RUST_LOG=error`. The server emits that one-time setup URL at `warn` (`crates/calternal-auth/src/api.rs`), so `/readyz` returned 200 while the probe could not read the token. I will set the probe's minimum log level to `warn`; the current smoke run is being stopped and will be repeated.

kayg commented

2026-09-27 10:53:32 +00:00

Author

Owner

Finding: the reduced Home smoke indexed all fixtures and completed the three worst-case views on a production build at 4× CPU throttle. With 1,400 Photos, 5,000 files, 20 Markdown Notes and 100 Log entries: startup indexing took 24.7 s; the 5k-file folder showed aria-setsize=5000 with 27 DOM options and rendered in 19.8 s; all five Analytics charts became ready in 30.3 s (warm API p95 1.63 s); the 1 MiB Note rendered in 16.7 s. Server peak RSS during that run was 736,514,048 bytes. The host load average was 41.7 / 35.9 / 30.9, so these are smoke values, not the baseline. The full fixed dataset remains to run.

Finding: the reduced Home smoke indexed all fixtures and completed the three worst-case views on a production build at 4× CPU throttle. With 1,400 Photos, 5,000 files, 20 Markdown Notes and 100 Log entries: startup indexing took 24.7 s; the 5k-file folder showed `aria-setsize=5000` with 27 DOM options and rendered in 19.8 s; all five Analytics charts became ready in 30.3 s (warm API p95 1.63 s); the 1 MiB Note rendered in 16.7 s. Server peak RSS during that run was 736,514,048 bytes. The host load average was 41.7 / 35.9 / 30.9, so these are smoke values, not the baseline. The full fixed dataset remains to run.

kayg commented

2026-09-27 11:07:40 +00:00

Author

Owner

Finding: the three-run route probe completed at 390 px and 1440 px on the matching production build. At 4× CPU throttle, /today data-ready p95 was 7,805 ms at 390 px and 15,901 ms at 1440 px; /settings/appearance transferred 321,617 JS gzip bytes and 53,020 CSS gzip bytes. Interaction p95 was 3,608 ms for palette, 3,329 ms for Composer, and 5,447 ms for mode switch. The 24-client storm returned HTTP 200 for all 4,532 requests, with p95 152 ms. Host load was 34.40 / 32.19 / 30.73 before the run. Full workloads and baseline file are still running. These figures show a large initial route bundle and slow interaction samples on this loaded host; I will inspect the completed machine report before choosing an optimization.

Finding: the three-run route probe completed at 390 px and 1440 px on the matching production build. At 4× CPU throttle, `/today` data-ready p95 was 7,805 ms at 390 px and 15,901 ms at 1440 px; `/settings/appearance` transferred 321,617 JS gzip bytes and 53,020 CSS gzip bytes. Interaction p95 was 3,608 ms for palette, 3,329 ms for Composer, and 5,447 ms for mode switch. The 24-client storm returned HTTP 200 for all 4,532 requests, with p95 152 ms. Host load was 34.40 / 32.19 / 30.73 before the run. Full workloads and baseline file are still running. These figures show a large initial route bundle and slow interaction samples on this loaded host; I will inspect the completed machine report before choosing an optimization.

kayg commented

2026-09-27 11:09:25 +00:00

Author

Owner

Finding: the first bench/run.sh --full attempt completed all route and request-storm measurements, then stopped before Calendar setup because calendar-perf.mjs had a narrower Playwright lookup than the shared harness. I replaced its duplicate resolver with loadPlaywright() from harness.mjs (commit 63c3817d). The completed route JSON is saved under target/perf/2026-09-27T105614Z-11c881ed/routes.json; I will continue the remaining workloads from that result to avoid repeating the routes.

Finding: the first `bench/run.sh --full` attempt completed all route and request-storm measurements, then stopped before Calendar setup because `calendar-perf.mjs` had a narrower Playwright lookup than the shared harness. I replaced its duplicate resolver with `loadPlaywright()` from `harness.mjs` (commit `63c3817d`). The completed route JSON is saved under `target/perf/2026-09-27T105614Z-11c881ed/routes.json`; I will continue the remaining workloads from that result to avoid repeating the routes.

kayg referenced this issue

2026-09-27 11:13:19 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 11:13:42 +00:00

Author

Owner

Finding: Calendar probe diagnostics included captured startup output verbatim if the server exited before readiness. That output can contain the one-time setup token. No such failure occurred in this run, but I redacted the setup token in that error path (2c41f8d1) so future logs stay safe.

Finding: Calendar probe diagnostics included captured startup output verbatim if the server exited before readiness. That output can contain the one-time setup token. No such failure occurred in this run, but I redacted the setup token in that error path (`2c41f8d1`) so future logs stay safe.

kayg commented

2026-09-27 11:34:47 +00:00

Author

Owner

Finding: Calendar grid readiness was inconsistent under the shared host load. One production probe timed out after 30 seconds waiting for .tg.ready; a diagnostic run on the same seeded 200-Log week reached /calendar/week/2026-09-21 with .tg.ready, a 1,056 px grid and 142.86 px day columns. Its load averages were 32.48 / 28.92 / 30.15. I raised the harness readiness timeout to 120 seconds and added a sanitized page/layout snapshot on timeout (37788888). The release server rebuilt successfully from the merged dev (Finished release profile [optimized] target(s) in 4m 22s).

Finding: Calendar grid readiness was inconsistent under the shared host load. One production probe timed out after 30 seconds waiting for `.tg.ready`; a diagnostic run on the same seeded 200-Log week reached `/calendar/week/2026-09-21` with `.tg.ready`, a 1,056 px grid and 142.86 px day columns. Its load averages were 32.48 / 28.92 / 30.15. I raised the harness readiness timeout to 120 seconds and added a sanitized page/layout snapshot on timeout (`37788888`). The release server rebuilt successfully from the merged `dev` (`Finished `release` profile [optimized] target(s) in 4m 22s`).

kayg commented

2026-09-27 11:53:40 +00:00

Author

Owner

Finding: the phone Calendar probe did not have a product loading failure. At 390 px it reached .tg.ready with no load error, but the harness waited for more than 20 .block buttons; the actual grid had one block because narrow columns intentionally fold overlapping Log entries into “+N” controls. I changed readiness to wait for seven populated day columns, which is independent of lane width, and added populated-column / “+N” counts to timeout diagnostics (fe264ecc). The probe still needs a fresh run to validate both viewports.

Finding: the phone Calendar probe did not have a product loading failure. At 390 px it reached `.tg.ready` with no load error, but the harness waited for more than 20 `.block` buttons; the actual grid had one block because narrow columns intentionally fold overlapping Log entries into “+N” controls. I changed readiness to wait for seven populated day columns, which is independent of lane width, and added populated-column / “+N” counts to timeout diagnostics (`fe264ecc`). The probe still needs a fresh run to validate both viewports.

kayg commented

2026-09-27 12:04:43 +00:00

Author

Owner

Finding: the corrected Calendar profile completed at both sizes. The seeded phone week rendered 1 individual block and 68 “+N” piles; desktop rendered 139 blocks and 89 piles. At 4× CPU throttle, desktop horizontal fling frame p95 was 2,599.9 ms (synthetic baseline 133.4 ms), and pinch frame p95 was 683.3 ms with 96 long tasks over 50 ms. Phone horizontal fling frame p95 was 133.2 ms (synthetic baseline 16.8 ms). Host load averages for the Calendar run were 28.9 / 23.3 / 25.4. The desktop result shows extra Calendar main-thread work, but these values are affected by the heavily shared host; I will retain the matched synthetic baseline and inspect the profile before choosing an optimization.

kayg commented

2026-09-27 12:28:57 +00:00

Author

Owner

Finding in the full Home workload: the 20,000-Photo / 50,000-file fixture has been active for 22 minutes. At the latest sample the release server used about 52% of one core and 570 MB RSS; host load was 27.86 / 31.73 / 32.95. The API count probe has not returned, so I have not recorded this as a completed indexing time. I committed the measured routes and Calendar profile as a partial checkpoint (c4397d9f) while the Home and import workloads continue.

Finding in the full Home workload: the 20,000-Photo / 50,000-file fixture has been active for 22 minutes. At the latest sample the release server used about 52% of one core and 570 MB RSS; host load was 27.86 / 31.73 / 32.95. The API count probe has not returned, so I have not recorded this as a completed indexing time. I committed the measured routes and Calendar profile as a partial checkpoint (`c4397d9f`) while the Home and import workloads continue.

kayg commented

2026-09-27 12:34:35 +00:00

Author

Owner

Finding: the full Home run failed at its 25-minute server-readiness deadline; it produced no Home JSON and no /readyz success. The redacted startup log reported SQLite pool acquisition delays up to 9.16 s and repeated files_index insert statements lasting 1.08–1.42 s. The last captured SQL warning was at 12:29:52 UTC. Source tracing reaches reconcile_all → reconcile_folder_locked → one record() call per directory entry, with a reader lookup and writer upsert for each file. Host load during the run ranged around 24–40. I am checking a semantics-preserving batched startup-index write and will compare readiness time against the same fixture.

Finding: the full Home run failed at its 25-minute server-readiness deadline; it produced no Home JSON and no `/readyz` success. The redacted startup log reported SQLite pool acquisition delays up to 9.16 s and repeated `files_index` insert statements lasting 1.08–1.42 s. The last captured SQL warning was at 12:29:52 UTC. Source tracing reaches `reconcile_all` → `reconcile_folder_locked` → one `record()` call per directory entry, with a reader lookup and writer upsert for each file. Host load during the run ranged around 24–40. I am checking a semantics-preserving batched startup-index write and will compare readiness time against the same fixture.

kayg commented

2026-09-27 12:51:38 +00:00

Author

Owner

Finding and change: the full 20,000-Photo / 50,000-file Home probe reached its 25-minute readiness deadline. Startup reconciliation had made one previous-row lookup and one autocommit UPSERT per entry; captured single-row inserts took 1.08–1.42 seconds under host load. Commit 39d6f2e5 now reads prior rows once per folder and writes changed rows in multi-row UPSERTs of at most 64 rows inside one folder transaction. The existing filesystem fingerprint/hash checks and SQLite row triggers remain in place, so item IDs, folder generations and durable change-feed events retain their current rules.

The regression covers an 80-entry folder across the batch boundary, a repeated startup scan with stable IDs and no duplicate feed events, and a changed file that emits one feed event with the new hash. Files crate result: test result: ok. 102 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 56.03s.

Decision: use 64 rows per SQL statement (896 bind values, under SQLite's historical 999-variable limit) and commit each changed folder as one transaction. The matched release rerun of the full Home fixture is pending; I will report its measured result separately.

Finding and change: the full 20,000-Photo / 50,000-file Home probe reached its 25-minute readiness deadline. Startup reconciliation had made one previous-row lookup and one autocommit UPSERT per entry; captured single-row inserts took 1.08–1.42 seconds under host load. Commit `39d6f2e5` now reads prior rows once per folder and writes changed rows in multi-row UPSERTs of at most 64 rows inside one folder transaction. The existing filesystem fingerprint/hash checks and SQLite row triggers remain in place, so item IDs, folder generations and durable change-feed events retain their current rules. The regression covers an 80-entry folder across the batch boundary, a repeated startup scan with stable IDs and no duplicate feed events, and a changed file that emits one feed event with the new hash. Files crate result: `test result: ok. 102 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 56.03s`. Decision: use 64 rows per SQL statement (896 bind values, under SQLite's historical 999-variable limit) and commit each changed folder as one transaction. The matched release rerun of the full Home fixture is pending; I will report its measured result separately.

kayg commented

2026-09-27 12:56:41 +00:00

Author

Owner

Post-merge production Analytics E2E at c2ff7b40 seeded 378 Log entries and indexed them in 249 seconds. Dashboard links, chart accessibility, responsive layouts, and three period changes passed. The year-to-quarter switch recorded a 211 ms CPU / 702 ms wall task, exceeding the test's 50 ms CPU limit; quarter-to-month, month-to-week, and week-to-year recorded maxima of 45, 16, and 31 ms. Other worktrees had active browser/build jobs, so please replay the long task on a quiet host before assigning a product cause.

Post-merge production Analytics E2E at c2ff7b40 seeded 378 Log entries and indexed them in 249 seconds. Dashboard links, chart accessibility, responsive layouts, and three period changes passed. The year-to-quarter switch recorded a 211 ms CPU / 702 ms wall task, exceeding the test's 50 ms CPU limit; quarter-to-month, month-to-week, and week-to-year recorded maxima of 45, 16, and 31 ms. Other worktrees had active browser/build jobs, so please replay the long task on a quiet host before assigning a product cause.

kayg commented

2026-09-27 13:40:08 +00:00

Author

Owner

Measured result after 39d6f2e5: the same 20,000-Photo / 50,000-file / 10,000-Note / 10,000-Log Home now reaches /readyz in 640.9 seconds and indexes the full fixture in 689.3 seconds. It indexed 20,000 Photos, 5,000 folder entries, 45,000 other Files rows, 10,001 Notes, and 10,000 Log entries. The previous matched run produced no readiness success before its 25-minute deadline, so there is no numeric before-time for a percentage comparison. This run is a successful completion under that deadline.

The three-run 4×-throttled Home views recorded p95 render times of 27.44 s for the 5k-file folder, 60.56 s for the 10k-entry Analytics year, and 17.78 s for the 1 MiB Note. Large Home peak RSS was 1,175,289,856 bytes; host load averaged about 39.4 / 36.4 / 35.3. The complete route, Calendar and Home checkpoint is committed as 5854fbc3; the separate 10k-Note and 50k-file upload measurements are still running.

Measured result after `39d6f2e5`: the same 20,000-Photo / 50,000-file / 10,000-Note / 10,000-Log Home now reaches `/readyz` in 640.9 seconds and indexes the full fixture in 689.3 seconds. It indexed 20,000 Photos, 5,000 folder entries, 45,000 other Files rows, 10,001 Notes, and 10,000 Log entries. The previous matched run produced no readiness success before its 25-minute deadline, so there is no numeric before-time for a percentage comparison. This run is a successful completion under that deadline. The three-run 4×-throttled Home views recorded p95 render times of 27.44 s for the 5k-file folder, 60.56 s for the 10k-entry Analytics year, and 17.78 s for the 1 MiB Note. Large Home peak RSS was 1,175,289,856 bytes; host load averaged about 39.4 / 36.4 / 35.3. The complete route, Calendar and Home checkpoint is committed as `5854fbc3`; the separate 10k-Note and 50k-file upload measurements are still running.

kayg commented

2026-09-27 13:45:20 +00:00

Author

Owner

Baseline decision: I preserved the pre-optimization machine run from commit 37788888 as docs/perf/baseline.json. It has complete route and Calendar values, but no Home/import values: the matching large Home probe had failed its 25-minute server-readiness deadline before the batch change. The successful 10m41s Home startup on 39d6f2e5 is therefore documented as the first completed Home reference, not as a before/after percentage. The coverage limitation is recorded in docs/perf/BASELINE.md; future runs can compare the completed route and Calendar dimensions without treating post-change Home data as a pre-change baseline.

Baseline decision: I preserved the pre-optimization machine run from commit `37788888` as `docs/perf/baseline.json`. It has complete route and Calendar values, but no Home/import values: the matching large Home probe had failed its 25-minute server-readiness deadline before the batch change. The successful 10m41s Home startup on `39d6f2e5` is therefore documented as the first completed Home reference, not as a before/after percentage. The coverage limitation is recorded in `docs/perf/BASELINE.md`; future runs can compare the completed route and Calendar dimensions without treating post-change Home data as a pre-change baseline.

kayg commented

2026-09-27 13:54:34 +00:00

Author

Owner

The 10k-Note upload probe reached its first checkpoint after 820.4 seconds: 1,000 uploads indexed; the last 200 had p50 373.92 ms, p95 1,102.23 ms and 2.05 uploads/s. The server used 56.2 s in Tokio runtime workers and 36.0 s in SQLite workers during that window. Host load was about 38 / 38 / 39. The checkpoint is committed in 3737cd4f; the probe is continuing toward 5k and 10k, then the 50k-file import.

Decision: retain this sequential API upload method because it measures the actual per-file Files endpoint and is using about 10% of one core on the shared host. Its full requested run is expected to take hours at the observed rate; I will preserve its checkpoint JSON and report completed counts if the host or job time limit stops it.

The 10k-Note upload probe reached its first checkpoint after 820.4 seconds: 1,000 uploads indexed; the last 200 had p50 373.92 ms, p95 1,102.23 ms and 2.05 uploads/s. The server used 56.2 s in Tokio runtime workers and 36.0 s in SQLite workers during that window. Host load was about 38 / 38 / 39. The checkpoint is committed in `3737cd4f`; the probe is continuing toward 5k and 10k, then the 50k-file import. Decision: retain this sequential API upload method because it measures the actual per-file Files endpoint and is using about 10% of one core on the shared host. Its full requested run is expected to take hours at the observed rate; I will preserve its checkpoint JSON and report completed counts if the host or job time limit stops it.

kayg commented

2026-09-27 14:20:28 +00:00

Author

Owner

Update: the 10k-Note API import has indexed 3,402 rows after 41m07s. The server is using about 15% of one core and 235 MB RSS; host load averages are 42.02 / 40.08 / 37.58. The import remains healthy. The latest interval was about 0.7 uploads/s under this load; the 5k and 10k checkpoints remain pending, followed by the 50k-file profile. I am continuing the requested run and will preserve its next harness checkpoint.

kayg commented

2026-09-27 14:29:46 +00:00

Author

Owner

The 10k Note upload profile reached 4,019 uploads after 50m28s, with 4,020 Files Index rows including the Notes folder. The server is at 242 MB RSS and about 15% average CPU; host load is 34.31 / 37.66 / 37.79. The current progress snapshot is committed as b1ceda41; the first latency window remains in 3737cd4f. The 5k checkpoint is close, after which the profile continues to 10k.

The 10k Note upload profile reached 4,019 uploads after 50m28s, with 4,020 Files Index rows including the Notes folder. The server is at 242 MB RSS and about 15% average CPU; host load is 34.31 / 37.66 / 37.79. The current progress snapshot is committed as `b1ceda41`; the first latency window remains in `3737cd4f`. The 5k checkpoint is close, after which the profile continues to 10k.

kayg commented

2026-09-27 14:41:25 +00:00

Author

Owner

The 10k Note API import reached 5,000 uploads at 3,722.5 seconds. The 200-upload window at 5k had p50 496.31 ms, p95 2,184.93 ms, and a rate of 1.36/s, compared with the 1k window p50 373.92 ms, p95 1,102.23 ms, 2.05/s. The p95 grew about 2x as the folder grew. The checkpoint is committed as 99f661bc; the run continues toward 10k.

The 10k Note API import reached 5,000 uploads at 3,722.5 seconds. The 200-upload window at 5k had p50 496.31 ms, p95 2,184.93 ms, and a rate of 1.36/s, compared with the 1k window p50 373.92 ms, p95 1,102.23 ms, 2.05/s. The p95 grew about 2x as the folder grew. The checkpoint is committed as `99f661bc`; the run continues toward 10k.

kayg commented

2026-09-27 15:22:13 +00:00

Author

Owner

The 10,000-Note API upload loop completed in 6,082.4 seconds. Its measured windows were: 1k p50 373.92 ms / p95 1,102.23 ms / 2.05 uploads/s; 5k p50 496.31 ms / p95 2,184.93 ms / 1.36 uploads/s; 10k p50 534.29 ms / p95 1,546.38 ms / 1.54 uploads/s. The three checkpoint rows are committed in 72d38962.

The harness is still waiting for post-upload server idle before it checks that the last upload is searchable. This JSON therefore has partial: true and does not yet include final resource and search results. The runner will proceed to the 50k-file import when this probe exits.

The 10,000-Note API upload loop completed in 6,082.4 seconds. Its measured windows were: 1k p50 373.92 ms / p95 1,102.23 ms / 2.05 uploads/s; 5k p50 496.31 ms / p95 2,184.93 ms / 1.36 uploads/s; 10k p50 534.29 ms / p95 1,546.38 ms / 1.54 uploads/s. The three checkpoint rows are committed in `72d38962`. The harness is still waiting for post-upload server idle before it checks that the last upload is searchable. This JSON therefore has `partial: true` and does not yet include final resource and search results. The runner will proceed to the 50k-file import when this probe exits.

kayg commented

2026-09-27 15:23:50 +00:00

Author

Owner

Update: the 10k Note probe's idle wait and search verification completed. It reported idle_reached: true, search probe HTTP 200, and search_probe_contains_last: true; complete measurements are committed in 73d025b4. Peak RSS was 341,082,112 bytes and peak sampled CPU was 195.66% of one core. The runner has started the 50k-file API import.

Update: the 10k Note probe's idle wait and search verification completed. It reported `idle_reached: true`, search probe HTTP 200, and `search_probe_contains_last: true`; complete measurements are committed in `73d025b4`. Peak RSS was 341,082,112 bytes and peak sampled CPU was 195.66% of one core. The runner has started the 50k-file API import.

kayg commented

2026-09-27 15:29:47 +00:00

Author

Owner

The 50k-file API import reached 1,000 uploads in 429.3 seconds. The last 200 had p50 272.71 ms, p95 562.92 ms and 3.09 uploads/s. This is faster than the Markdown Note API workload (1k window p50 373.92 ms, p95 1,102.23 ms, 2.05/s). The partial checkpoint is committed as 4f80cfee; the runner continues toward 10k and 50k.

The 50k-file API import reached 1,000 uploads in 429.3 seconds. The last 200 had p50 272.71 ms, p95 562.92 ms and 3.09 uploads/s. This is faster than the Markdown Note API workload (1k window p50 373.92 ms, p95 1,102.23 ms, 2.05/s). The partial checkpoint is committed as `4f80cfee`; the runner continues toward 10k and 50k.

kayg referenced this issue

2026-09-27 15:40:20 +00:00

Security: enforce the SPA shell Content-Security-Policy (now Report-Only) #118

kayg commented

2026-09-27 15:58:45 +00:00

Author

Owner

50k file import is at 6,089 / 50,000 files after about 35 minutes. The release server is healthy; RSS is 263 MB. Host load was 28.26 / 32.00 / 28.50 at the latest sample, so this throughput is under shared-host contention. The first 1k checkpoint remains 429.3 s, p50 272.71 ms, p95 562.92 ms, 3.09 uploads/s. No errors observed so far.

kayg commented

2026-09-27 16:13:30 +00:00

Author

Owner

The 50k file import is at 8,231 / 50,000 files after 51m24s. The release server is healthy with 289 MB RSS; host load was 32.28 / 32.13 / 30.91 at the latest sample. The 10k latency checkpoint is still pending. No upload errors or process restarts observed.

kayg commented

2026-09-27 16:27:45 +00:00

Author

Owner

The 50k file import reached its 10k checkpoint: 10000 uploads in 3876.7 s (64m37s). The last 200 uploads were p50 276.16 ms, p95 744.05 ms, 2.75/s; this is a 1.12x rate drop from the 1k checkpoint. The uploader has continued to 10081 files. Server RSS is 342 MB; host load was 28.75 / 31.18 / 30.21. No upload errors observed. I committed the machine-readable checkpoint and summary as fd7d3241.

The 50k file import reached its 10k checkpoint: 10000 uploads in 3876.7 s (64m37s). The last 200 uploads were p50 276.16 ms, p95 744.05 ms, 2.75/s; this is a 1.12x rate drop from the 1k checkpoint. The uploader has continued to 10081 files. Server RSS is 342 MB; host load was 28.75 / 31.18 / 30.21. No upload errors observed. I committed the machine-readable checkpoint and summary as fd7d3241.

kayg commented

2026-09-27 16:41:36 +00:00

Author

Owner

Continuing #164 under the 2026-09-27 finish directive. Finalization starts on branch job/perf at checkpoint fd7d32410e6607bb5e5c1d45b30ce14b8aed6691 (original base: dev at 82d4c258bbcc6f34e8805d0018004fa8e621b9a5). I will merge local dev once, run the requested final gates and one bounded adversarial round, capture production screenshots, then commit and push this branch. The 50k-file upload profile has a committed 10k checkpoint; I will report remaining uncompleted work.

Continuing #164 under the 2026-09-27 finish directive. Finalization starts on branch `job/perf` at checkpoint `fd7d32410e6607bb5e5c1d45b30ce14b8aed6691` (original base: `dev` at `82d4c258bbcc6f34e8805d0018004fa8e621b9a5`). I will merge local `dev` once, run the requested final gates and one bounded adversarial round, capture production screenshots, then commit and push this branch. The 50k-file upload profile has a committed 10k checkpoint; I will report remaining uncompleted work.

kayg referenced this issue

2026-09-27 17:08:53 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 17:46:39 +00:00

Author

Owner

Finished — #164

Branch: job/perf
Head: 2e97faf7dcb505307967718ded88117f0f439593 (pushed to origin/job/perf)
Merged dev once at 74d28480.

Built

Added the reusable production performance runner and comparer, route and Calendar profiles, and machine-readable baseline/report artifacts.
Batched startup Files Index reconciliation: one prior-row scan per folder and multi-row UPSERTs of 64 rows inside one folder transaction. This keeps each statement at 896 bind values and retains row triggers.
Preserved dev's five-second HTTP writer checkout deadline and SQLite busy retry behavior. The performance harness now uses separate instance-state and Home directories.
Fixed two Clippy findings in the conflict-resolved Files code in 2e97faf7 by boxing the large enum variant and grouping preparation options.

Key files: bench/run.sh, bench/compare.py, apps/web/e2e/route-perf.mjs, apps/web/e2e/calendar-perf.mjs, apps/web/e2e/photos-perf.mjs, apps/web/e2e/harness.mjs, docs/perf/, and crates/plugins/files/src/index.rs.

Gates

Verbatim gate summary:

>>> gate summary
cargo fmt --check 0
cargo clippy --all-targets -- -D warnings 124
cargo test 124
bun run check 0
bun run test 0

cargo fmt --check emitted no output. Clippy reached calternal-plugin-files, reported a large enum variant and a helper with too many arguments, then hit its 10-minute cap. Both findings are fixed in the pushed head. A targeted Files Clippy check started after the fix but hit its 90-second cap while compiling dependencies, before it reached the Files crate. The full cargo test run hit its 24-minute cap while later tests were still running; no failure summary was emitted.

bun run check output:

svelte-check found 0 errors and 0 warnings
EXIT_CODE=0

bun run test output:

Test Files  80 passed (80)
Tests  585 passed (585)
Duration  125.25s (transform 69%, import 12%, environment 11%, tests 6%, setup 2%)
EXIT_CODE=0

The production SPA build completed in 2m 5s. The bounded adversarial runner expired during server compilation before it reached setup or API probes, so this round is incomplete and produced no API findings.

Screenshots

Eight screenshots from the current production SPA and matching local server are attached:

1440px: Today, Files, Photos, Notes
390px: Today, Files, Photos, Notes

The Appearance screenshot capture did not complete because the current route did not expose the harness selector .sec-head.

Remaining work

Complete the full workspace Clippy and test gates on a host that can finish them. The post-fix targeted Files Clippy check is also incomplete.
Run the adversarial probes against the merged local server; the timebox expired during compilation.
Finish the 50k-file upload profile. The committed run reached its 10k checkpoint (10,081 uploads at the latest checkpoint), not 50k.
Remeasure route timings after the dev merge. Existing route and Calendar numbers are in docs/perf/2026-09-27.md; they predate this merge.
There is no Photos album API or active client sync endpoint in this build. The profile uses the real Photos timeline and restart reindex/import paths.

Decisions not set by DESIGN.md

Keep a 64-row SQL batch and one transaction per changed folder; 64 rows means 896 bind values, below SQLite's historical 999-variable limit.
Keep HTTP Index writes as single-row writes with the existing five-second writer checkout deadline; use the batch path for startup folder reconciliation.
Keep the benchmark harness on split instance-state and Home directories to match the current server configuration.

## Finished — #164 - Branch: `job/perf` - Head: `2e97faf7dcb505307967718ded88117f0f439593` (pushed to `origin/job/perf`) - Merged `dev` once at `74d28480`. ### Built - Added the reusable production performance runner and comparer, route and Calendar profiles, and machine-readable baseline/report artifacts. - Batched startup Files Index reconciliation: one prior-row scan per folder and multi-row UPSERTs of 64 rows inside one folder transaction. This keeps each statement at 896 bind values and retains row triggers. - Preserved `dev`'s five-second HTTP writer checkout deadline and SQLite busy retry behavior. The performance harness now uses separate instance-state and Home directories. - Fixed two Clippy findings in the conflict-resolved Files code in `2e97faf7` by boxing the large enum variant and grouping preparation options. Key files: `bench/run.sh`, `bench/compare.py`, `apps/web/e2e/route-perf.mjs`, `apps/web/e2e/calendar-perf.mjs`, `apps/web/e2e/photos-perf.mjs`, `apps/web/e2e/harness.mjs`, `docs/perf/`, and `crates/plugins/files/src/index.rs`. ### Gates Verbatim gate summary: ```text >>> gate summary cargo fmt --check 0 cargo clippy --all-targets -- -D warnings 124 cargo test 124 bun run check 0 bun run test 0 ``` `cargo fmt --check` emitted no output. Clippy reached `calternal-plugin-files`, reported a large enum variant and a helper with too many arguments, then hit its 10-minute cap. Both findings are fixed in the pushed head. A targeted Files Clippy check started after the fix but hit its 90-second cap while compiling dependencies, before it reached the Files crate. The full `cargo test` run hit its 24-minute cap while later tests were still running; no failure summary was emitted. `bun run check` output: ```text svelte-check found 0 errors and 0 warnings EXIT_CODE=0 ``` `bun run test` output: ```text Test Files 80 passed (80) Tests 585 passed (585) Duration 125.25s (transform 69%, import 12%, environment 11%, tests 6%, setup 2%) EXIT_CODE=0 ``` The production SPA build completed in 2m 5s. The bounded adversarial runner expired during server compilation before it reached setup or API probes, so this round is incomplete and produced no API findings. ### Screenshots Eight screenshots from the current production SPA and matching local server are attached: - 1440px: [Today](https://git.kayg.org/attachments/0385d3a9-ad8a-464f-833c-7881d5e8ae0f), [Files](https://git.kayg.org/attachments/5d84e481-74ee-4a65-87e1-1b15f9ba13d3), [Photos](https://git.kayg.org/attachments/34d8aa83-f0c9-4290-b71a-14bb4249776e), [Notes](https://git.kayg.org/attachments/e8a8d25f-725d-40e1-9b6a-f9fdaeee2aa9) - 390px: [Today](https://git.kayg.org/attachments/7bec7ca9-15d2-4d8b-8226-7769057aea7a), [Files](https://git.kayg.org/attachments/a8f35f71-c50b-4ba9-94ee-b07a08b4a0df), [Photos](https://git.kayg.org/attachments/7b85a994-d4e9-49b5-8f8f-b07fc82bb9e5), [Notes](https://git.kayg.org/attachments/a18b69b8-f906-442b-8108-e13a36e505a5) The Appearance screenshot capture did not complete because the current route did not expose the harness selector `.sec-head`. ### Remaining work - Complete the full workspace Clippy and test gates on a host that can finish them. The post-fix targeted Files Clippy check is also incomplete. - Run the adversarial probes against the merged local server; the timebox expired during compilation. - Finish the 50k-file upload profile. The committed run reached its 10k checkpoint (10,081 uploads at the latest checkpoint), not 50k. - Remeasure route timings after the `dev` merge. Existing route and Calendar numbers are in `docs/perf/2026-09-27.md`; they predate this merge. - There is no Photos album API or active client sync endpoint in this build. The profile uses the real Photos timeline and restart reindex/import paths. ### Decisions not set by DESIGN.md - Keep a 64-row SQL batch and one transaction per changed folder; 64 rows means 896 bind values, below SQLite's historical 999-variable limit. - Keep HTTP Index writes as single-row writes with the existing five-second writer checkout deadline; use the batch path for startup folder reconciliation. - Keep the benchmark harness on split instance-state and Home directories to match the current server configuration.

kayg commented

2026-09-27 17:47:56 +00:00

Author

Owner

Correction to the final report: the pushed branch HEAD is 2e97faf7b8c8ec1f39117574f1107b41b0b231b6. The earlier full SHA in my report was mistyped. The branch push succeeded and the worktree is clean.

Correction to the final report: the pushed branch HEAD is `2e97faf7b8c8ec1f39117574f1107b41b0b231b6`. The earlier full SHA in my report was mistyped. The branch push succeeded and the worktree is clean.

kayg commented

2026-09-27 17:51:19 +00:00

Author

Owner

Merged in 7b169436. Orchestrator re-ran the timed-out gates on the branch: clippy clean, calternal-plugin-files tests 106/106. Weekly timer installed next.

Merged in 7b169436. Orchestrator re-ran the timed-out gates on the branch: clippy clean, calternal-plugin-files tests 106/106. Weekly timer installed next.

kayg referenced this issue from a commit

2026-09-27 17:51:20 +00:00

Merge job/perf (#164)

kayg closed this issue

2026-09-27 17:51:20 +00:00

kayg referenced this issue

2026-09-27 18:16:12 +00:00

PERF: search & indexing — stupid fast and reliable (targets, benchmark, fixes, chaos) #258

kayg referenced this issue

2026-09-27 18:16:22 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg referenced this issue