kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 802 Pull requests Projects Releases Packages Wiki Activity Actions 462

PERF ROUND: benchmark + optimise average and worst-case CPU/memory (after current merges) #367

New issue
Open
opened 2026-09-28 17:35:11 +00:00 by kayg · 74 comments
kayg commented 2026-09-28 17:35:11 +00:00
Owner
Copy link

Request (owner, 2026-09-28)

"once all of those have been merged, i would like a performance review/benchmark/optimisation round so we can keep both median/average cpu/memory usage and worst case cpu/memory usage as low as possible."

When

Start after these merge: #355, #357, #358, #354, #321, #313, #326, #340/#366, #361, and the jobs running now. Claude launches this when the queue is drained. Measure on a quiet host: no Codex jobs running, load average below 2, with the load averages recorded for every run.

Phase 1: measure (baseline)

Use bench/run.sh --full against a production build of the dev head, and compare with docs/perf/2026-09-27.md. Also measure the following, raw numbers committed under docs/perf/:

  • Average: idle RSS and CPU% of the server over 10 minutes (one User, then 10 Users with real Homes); p50 and p95 latency of the top 20 API routes; web bundle size; LCP and INP for Calendar, Files, Notes, Photos, Search and Settings (production build, CPU throttled 4x).
  • Worst case: peak RSS and CPU during a request storm, a large Home (1M files / 100k notes), a big import (paperless-size PDF batch), a full search rebuild, the semantic embedding backfill, a large sync (10k changes), a thumbnail storm on 5k photos, 50 concurrent SSE clients, and a Mail full-history sync on a large mailbox. For each peak, say why: which allocation or task dominates (heaptrack or dhat for Rust, tokio-console or a sampling profile, and a Chrome performance trace for the web).
  • Browser: JS heap after 30 minutes of navigating every mode; long tasks over 50 ms; memory growth (leaks) across 100 open/close cycles of overlays, previews and the composer.
  • The arm64 target: repeat the idle and worst-case server set on o2.

Decision rule (fixed before the results)

  • A candidate optimisation is kept only if it improves its target metric by at least 10% (median of 5 interleaved A/B runs, CPU-pinned), with no other tracked metric regressing more than 3%, and with no loss of correctness (all gates green, adversarial round clean).
  • Priority: the worst-case peaks first (memory ceilings, unbounded queues, caches without bounds, whole-file reads), then the idle footprint, then the medians.
  • Every cap or bound is a named config value with a documented default, not a literal.

Phase 2: optimise

Split the findings: one fj issue per root cause, each carrying its measurement. Claude launches one job per independent root cause (one owner per shared primitive). Each job re-measures with the same protocol and commits before/after numbers.

Phase 3: report

Write docs/perf/-review.md with a before/after table for every metric, what changed, and what was tried and rejected (with numbers). Update the baseline the weekly timer compares against.

Rules: performance is never a merge gate (CLAUDE.md), but this round is an explicit owner request. Keep build output small (CARGO_PROFILE_DEV_DEBUG=line-tables-only). No fake data in the UI; bench fixtures live in bench/ only.

## Request (owner, 2026-09-28) "once all of those have been merged, i would like a performance review/benchmark/optimisation round so we can keep both median/average cpu/memory usage and worst case cpu/memory usage as low as possible." ## When Start after these merge: #355, #357, #358, #354, #321, #313, #326, #340/#366, #361, and the jobs running now. Claude launches this when the queue is drained. Measure on a **quiet host**: no Codex jobs running, load average below 2, with the load averages recorded for every run. ## Phase 1: measure (baseline) Use `bench/run.sh --full` against a production build of the dev head, and compare with docs/perf/2026-09-27.md. Also measure the following, raw numbers committed under docs/perf/: - **Average:** idle RSS and CPU% of the server over 10 minutes (one User, then 10 Users with real Homes); p50 and p95 latency of the top 20 API routes; web bundle size; LCP and INP for Calendar, Files, Notes, Photos, Search and Settings (production build, CPU throttled 4x). - **Worst case:** peak RSS and CPU during a request storm, a large Home (1M files / 100k notes), a big import (paperless-size PDF batch), a full search rebuild, the semantic embedding backfill, a large sync (10k changes), a thumbnail storm on 5k photos, 50 concurrent SSE clients, and a Mail full-history sync on a large mailbox. For each peak, say **why**: which allocation or task dominates (heaptrack or dhat for Rust, `tokio-console` or a sampling profile, and a Chrome performance trace for the web). - Browser: JS heap after 30 minutes of navigating every mode; long tasks over 50 ms; memory growth (leaks) across 100 open/close cycles of overlays, previews and the composer. - The arm64 target: repeat the idle and worst-case server set on o2. ## Decision rule (fixed before the results) - A candidate optimisation is kept only if it improves its target metric by at least 10% (median of 5 interleaved A/B runs, CPU-pinned), with no other tracked metric regressing more than 3%, and with no loss of correctness (all gates green, adversarial round clean). - Priority: the worst-case peaks first (memory ceilings, unbounded queues, caches without bounds, whole-file reads), then the idle footprint, then the medians. - Every cap or bound is a named config value with a documented default, not a literal. ## Phase 2: optimise Split the findings: one fj issue per root cause, each carrying its measurement. Claude launches one job per independent root cause (one owner per shared primitive). Each job re-measures with the same protocol and commits before/after numbers. ## Phase 3: report Write docs/perf/<date>-review.md with a before/after table for every metric, what changed, and what was tried and rejected (with numbers). Update the baseline the weekly timer compares against. Rules: performance is never a merge gate (CLAUDE.md), but this round is an explicit owner request. Keep build output small (CARGO_PROFILE_DEV_DEBUG=line-tables-only). No fake data in the UI; bench fixtures live in bench/ only.
route-review-screenshots.zip
11 MiB
kayg commented 2026-09-29 03:01:11 +00:00
Author
Owner
Copy link

Scope addition (Claude): WebDAV small-file throughput. job/webdav (#339) measured ~2.6 files/s for 30 KB files with parallel rclone on the loaded host; the comparison was inconclusive. On the quiet host: per-stage profile of one PUT (auth verify, quota, atomic write+fsync, Index row, feed row, SSE, Search/thumbnail enqueue) and rclone --transfers 16 on 10k x 30 KB and 1k x 3 MB. Target: >= 200 small files/s, or a measured explanation.

Scope addition (Claude): WebDAV small-file throughput. job/webdav (#339) measured ~2.6 files/s for 30 KB files with parallel rclone on the loaded host; the comparison was inconclusive. On the quiet host: per-stage profile of one PUT (auth verify, quota, atomic write+fsync, Index row, feed row, SSE, Search/thumbnail enqueue) and rclone --transfers 16 on 10k x 30 KB and 1k x 3 MB. Target: >= 200 small files/s, or a measured explanation.
kayg commented 2026-09-29 07:31:13 +00:00
Author
Owner
Copy link

Finding from the 2026-09-29 API-only adversarial run: the probe now targets the backend directly (127.0.0.1:16223 in this run), with no frontend proxy between the client and the API. The Task create baseline reported p50 21.472s with 4/5 successful requests. During the Task storm, requests 6, 7, 8, 10, 11, 12, 13 and 14 returned NO RESPONSE (b'timed out') at the probe's 30-second socket timeout. These are backend timeouts, not synthetic proxy 502s. The shared host was busy at the same time: ps showed other server instances, two other adversarial attack.py processes, and active cargo builds. This is actionable mixed-load evidence, but it does not isolate server cost from host contention; please include a quiet-host profile in the #367 review before attributing a root cause.

Finding from the 2026-09-29 API-only adversarial run: the probe now targets the backend directly (`127.0.0.1:16223` in this run), with no frontend proxy between the client and the API. The Task create baseline reported p50 21.472s with 4/5 successful requests. During the Task storm, requests 6, 7, 8, 10, 11, 12, 13 and 14 returned `NO RESPONSE (b'timed out')` at the probe's 30-second socket timeout. These are backend timeouts, not synthetic proxy 502s. The shared host was busy at the same time: `ps` showed other server instances, two other adversarial `attack.py` processes, and active cargo builds. This is actionable mixed-load evidence, but it does not isolate server cost from host contention; please include a quiet-host profile in the #367 review before attributing a root cause.
kayg commented 2026-09-29 07:32:39 +00:00
Author
Owner
Copy link

Quiet host available (owner, 2026-09-29): the perf-test VM (netbird ssh --no-browser --user root 10.69.69.63; Debian 13, 4 vCPU, 7 GB RAM, intentionally low specs) is dedicated to performance work. Run the round there instead of waiting for this build host to be idle. #409 installs a release server on it first.

Quiet host available (owner, 2026-09-29): the **perf-test VM** (netbird ssh --no-browser --user root 10.69.69.63; Debian 13, 4 vCPU, 7 GB RAM, intentionally low specs) is dedicated to performance work. Run the round there instead of waiting for this build host to be idle. #409 installs a release server on it first.
kayg commented 2026-09-29 07:40:00 +00:00
Author
Owner
Copy link

Follow-up from the same direct-backend API round: after the 120-photo upload burst, POST /api/v1/calendar/events/from-log returned no response within the probe's 30-second timeout (calendar Event from Log: NO RESPONSE (b'timed out')). This request bypassed the Node frontend proxy. The host contention noted in my previous comment was still present, so this is a real backend timeout under mixed load but not an isolated quiet-host measurement.

Follow-up from the same direct-backend API round: after the 120-photo upload burst, `POST /api/v1/calendar/events/from-log` returned no response within the probe's 30-second timeout (`calendar Event from Log: NO RESPONSE (b'timed out')`). This request bypassed the Node frontend proxy. The host contention noted in my previous comment was still present, so this is a real backend timeout under mixed load but not an isolated quiet-host measurement.
kayg commented 2026-09-29 09:22:18 +00:00
Author
Owner
Copy link

Started now (job perf-367), without waiting for the job queue to drain. Measurements run on the quiet perf-test VM (4 vCPU, 7 GB), not on the busy build host. Phase 1 and the per-root-cause split come first; optimisation jobs follow from those issues.

Started now (job perf-367), without waiting for the job queue to drain. Measurements run on the quiet perf-test VM (4 vCPU, 7 GB), not on the busy build host. Phase 1 and the per-root-cause split come first; optimisation jobs follow from those issues.
kayg commented 2026-09-29 09:22:50 +00:00
Author
Owner
Copy link

Starting Phase 1 measurements and Phase 2 root-cause split. Branch: job/perf-367. Base/dev SHA: 191b179baa. I have read the #367 protocol and will build on the host, measure on the perf-test VM, and report load averages with each run.

Starting Phase 1 measurements and Phase 2 root-cause split. Branch: job/perf-367. Base/dev SHA: 191b179baac3ef4f5bebfe07ce91c4b7a887ace2. I have read the #367 protocol and will build on the host, measure on the perf-test VM, and report load averages with each run.
kayg commented 2026-09-29 09:27:57 +00:00
Author
Owner
Copy link

Environment finding: at 11:23 UTC the perf-test VM reported load averages 0.35, 0.41, 0.72, Debian 13 x86_64, 7.1 GiB available RAM, and 83 GiB free disk. Bun, Cargo, Python and perf are installed; Chromium is not. The existing runner builds and measures on one host, so commit d6855f97 adds a measure-only mode for copied production artifacts and records uptime alongside each probe. The gate applies only to the VM mode and checks its one-minute load average is below 1.

Environment finding: at 11:23 UTC the perf-test VM reported load averages 0.35, 0.41, 0.72, Debian 13 x86_64, 7.1 GiB available RAM, and 83 GiB free disk. Bun, Cargo, Python and perf are installed; Chromium is not. The existing runner builds and measures on one host, so commit d6855f97 adds a measure-only mode for copied production artifacts and records `uptime` alongside each probe. The gate applies only to the VM mode and checks its one-minute load average is below 1.
kayg commented 2026-09-29 09:31:54 +00:00
Author
Owner
Copy link

Load-gate finding: the perf VM reported 3.04, 1.68, 1.16 at 11:31 UTC. Its process list showed a debug calternal server with a Bun browser run from another checkout, plus a Python process at 249% CPU. I did not stop processes outside this job. No performance probe has run. bench/run.sh --measure-only waits for the one-minute load average to fall below 1 before each probe and records uptime before and after it.

Load-gate finding: the perf VM reported 3.04, 1.68, 1.16 at 11:31 UTC. Its process list showed a debug calternal server with a Bun browser run from another checkout, plus a Python process at 249% CPU. I did not stop processes outside this job. No performance probe has run. `bench/run.sh --measure-only` waits for the one-minute load average to fall below 1 before each probe and records `uptime` before and after it.
kayg commented 2026-09-29 09:51:28 +00:00
Author
Owner
Copy link

Update for #367 (2026-09-29 11:50 UTC): no benchmark probe has started because the perf VM is not below the required one-minute load average of 1. uptime reported 1.88, 2.59, 2.63. ps showed an unrelated calternal-server and Chromium benchmark process active. I have not stopped processes from another checkout. The measurement runner remains gated and will capture uptime before and after every run.

Update for #367 (2026-09-29 11:50 UTC): no benchmark probe has started because the perf VM is not below the required one-minute load average of 1. `uptime` reported 1.88, 2.59, 2.63. `ps` showed an unrelated calternal-server and Chromium benchmark process active. I have not stopped processes from another checkout. The measurement runner remains gated and will capture uptime before and after every run.
kayg commented 2026-09-29 09:52:05 +00:00
Author
Owner
Copy link

Timestamp correction: the load snapshot in my previous update was taken at 11:50 Europe/Berlin (09:50 UTC), not 11:50 UTC. The reported load averages and process observation are unchanged.

Timestamp correction: the load snapshot in my previous update was taken at 11:50 Europe/Berlin (09:50 UTC), not 11:50 UTC. The reported load averages and process observation are unchanged.
kayg commented 2026-09-29 09:54:34 +00:00
Author
Owner
Copy link

Finding for #367: Mail Search over 100k messages cannot be measured on the current head. crates/plugins/mail/src/routes.rs registers account, folder, message, and sync routes, but no Search route; the Mail module has no search provider. The shared Search request deadline is 200 ms in crates/calternal-server/src/main.rs. Issue #396 already tracks Mail M2 (reader and Search), so I will report this as a protocol gap and avoid creating a duplicate issue or seeding a fake Mail Search table.

Finding for #367: Mail Search over 100k messages cannot be measured on the current head. `crates/plugins/mail/src/routes.rs` registers account, folder, message, and sync routes, but no Search route; the Mail module has no search provider. The shared Search request deadline is 200 ms in `crates/calternal-server/src/main.rs`. Issue #396 already tracks Mail M2 (reader and Search), so I will report this as a protocol gap and avoid creating a duplicate issue or seeding a fake Mail Search table.
kayg commented 2026-09-29 10:00:47 +00:00
Author
Owner
Copy link

Measurement tooling update: --full now holds 50 authenticated SSE connections for 60 seconds (25 Files and 25 Notes) and records setup latency, server RSS/CPU, and host load. This uses real event routes and disposable credentials, with no seeded content. The existing reporter table expectation remains unchanged; LCP is in a separate section. python3 -m unittest bench.test_record passes (3 tests).

Measurement tooling update: `--full` now holds 50 authenticated SSE connections for 60 seconds (25 Files and 25 Notes) and records setup latency, server RSS/CPU, and host load. This uses real event routes and disposable credentials, with no seeded content. The existing reporter table expectation remains unchanged; LCP is in a separate section. `python3 -m unittest bench.test_record` passes (3 tests).
kayg commented 2026-09-29 10:01:18 +00:00
Author
Owner
Copy link

Perf VM status: the load gate remains closed. The latest uptime sample was 12:00 Europe/Berlin with load averages 6.02, 4.18, 3.25. ps showed server and Chromium processes from other worktrees. No #367 probe has run, and I have left those processes untouched.

Perf VM status: the load gate remains closed. The latest `uptime` sample was 12:00 Europe/Berlin with load averages 6.02, 4.18, 3.25. `ps` showed server and Chromium processes from other worktrees. No #367 probe has run, and I have left those processes untouched.
kayg commented 2026-09-29 10:08:35 +00:00
Author
Owner
Copy link

Benchmark decision for the unspecified “paperless-size PDF batch”: use 10,000 generated, valid, searchable 64 KiB PDFs (about 640 MiB total) uploaded through the real Tus API. The probe caps configurable batches at 1 GiB and records this fixture size with its results. This is test-only generated content; no source PDFs or private data are used. python3 -m unittest bench.test_record bench.test_pdf_batch passes (5 tests).

Benchmark decision for the unspecified “paperless-size PDF batch”: use 10,000 generated, valid, searchable 64 KiB PDFs (about 640 MiB total) uploaded through the real Tus API. The probe caps configurable batches at 1 GiB and records this fixture size with its results. This is test-only generated content; no source PDFs or private data are used. `python3 -m unittest bench.test_record bench.test_pdf_batch` passes (5 tests).
kayg commented 2026-09-29 10:22:13 +00:00
Author
Owner
Copy link

Phase 1 remains blocked by the perf-test VM load gate. At 2026-09-29T10:21:49Z, uptime reported load average: 4.69, 5.07, 4.77 (required one-minute load below 1). The VM has unrelated /opt/calternal-webdav-perf and /root/jank414 server/Chromium processes. I have not started any #367 measurement or stopped another job. Continuing local harness and build work while waiting for a quiet sample.

Phase 1 remains blocked by the perf-test VM load gate. At 2026-09-29T10:21:49Z, `uptime` reported `load average: 4.69, 5.07, 4.77` (required one-minute load below 1). The VM has unrelated `/opt/calternal-webdav-perf` and `/root/jank414` server/Chromium processes. I have not started any #367 measurement or stopped another job. Continuing local harness and build work while waiting for a quiet sample.
kayg commented 2026-09-29 10:26:07 +00:00
Author
Owner
Copy link

The current docs/perf/baseline.json is from calternal-dev (37788888) and records load averages of 25.38 / 28.87 / 30.08 before the run and 26.34 / 27.47 / 29.24 after it. That does not match the isolated perf VM. I added PERF_PROMOTE_BASELINE=1 so the first quiet VM result becomes the new baseline without reporting a cross-host comparison as a regression. The VM run will use this mode and the dated report will identify the baseline host.

The current `docs/perf/baseline.json` is from `calternal-dev` (`37788888`) and records load averages of 25.38 / 28.87 / 30.08 before the run and 26.34 / 27.47 / 29.24 after it. That does not match the isolated perf VM. I added `PERF_PROMOTE_BASELINE=1` so the first quiet VM result becomes the new baseline without reporting a cross-host comparison as a regression. The VM run will use this mode and the dated report will identify the baseline host.
kayg commented 2026-09-29 10:36:50 +00:00
Author
Owner
Copy link

Decision not specified by DESIGN: the full run uses a 30-minute Browser soak at a fixed 1440×900 viewport and 4× CPU throttle, then performs exactly 100 cycles each for Search, Quick Look and Composer. A fixed viewport keeps heap samples comparable; route-perf separately covers 390, 820 and 1440 px in light and dark. The duration can be shortened or extended with PERF_BROWSER_SOAK_MINUTES; cycle count stays 100 for this profile.

Decision not specified by DESIGN: the full run uses a 30-minute Browser soak at a fixed 1440×900 viewport and 4× CPU throttle, then performs exactly 100 cycles each for Search, Quick Look and Composer. A fixed viewport keeps heap samples comparable; route-perf separately covers 390, 820 and 1440 px in light and dark. The duration can be shortened or extended with `PERF_BROWSER_SOAK_MINUTES`; cycle count stays 100 for this profile.
kayg commented 2026-09-29 10:46:20 +00:00
Author
Owner
Copy link

At 2026-09-29T10:45:43Z the perf-test VM reported load average: 4.75, 3.65, 3.77, still above the required one-minute load below 1. The sample previously fell to 1.29 at 10:38:10Z, then rose again. The runner files and release build continue preparing, but Phase 1 probes remain unstarted.

At 2026-09-29T10:45:43Z the perf-test VM reported `load average: 4.75, 3.65, 3.77`, still above the required one-minute load below 1. The sample previously fell to 1.29 at 10:38:10Z, then rose again. The runner files and release build continue preparing, but Phase 1 probes remain unstarted.
kayg commented 2026-09-29 11:00:42 +00:00
Author
Owner
Copy link

At 2026-09-29T11:00:37Z the perf-test VM reported load average: 4.62, 4.52, 4.14 (one-minute gate is below 1). /opt/calternal-webdav-perf/bin/calternal-server, /root/jank414/target/debug/calternal-server, and Chromium are still active. I have not stopped them or started #367 probes. The newest release server binary is still building on the build host.

At 2026-09-29T11:00:37Z the perf-test VM reported `load average: 4.62, 4.52, 4.14` (one-minute gate is below 1). `/opt/calternal-webdav-perf/bin/calternal-server`, `/root/jank414/target/debug/calternal-server`, and Chromium are still active. I have not stopped them or started #367 probes. The newest release server binary is still building on the build host.
kayg commented 2026-09-29 11:16:09 +00:00
Author
Owner
Copy link

At 2026-09-29T11:14:56Z the perf VM was at 1.46 one-minute load, then at 2026-09-29T11:16:02Z it was at 1.86. Both are above the required below-1 gate, so the probe has not started. The release build remains active on the build host.

At 2026-09-29T11:14:56Z the perf VM was at 1.46 one-minute load, then at 2026-09-29T11:16:02Z it was at 1.86. Both are above the required below-1 gate, so the probe has not started. The release build remains active on the build host.
kayg commented 2026-09-29 11:22:49 +00:00
Author
Owner
Copy link

The perf-test VM has now stayed below the gate in successive samples: one-minute load 0.34 at 11:20:17Z, 0.09 at 11:21:35Z, and 0.04 at 11:22:14Z. The production web build and runner are ready on the VM. The release server binary is still compiling on the build host, so no measurements have started yet.

The perf-test VM has now stayed below the gate in successive samples: one-minute load 0.34 at 11:20:17Z, 0.09 at 11:21:35Z, and 0.04 at 11:22:14Z. The production web build and runner are ready on the VM. The release server binary is still compiling on the build host, so no measurements have started yet.
kayg commented 2026-09-29 11:30:07 +00:00
Author
Owner
Copy link

At 2026-09-29T11:30:02Z, the perf-test VM again reports load average: 1.99, 1.94, 2.48. The brief quiet interval ended before the release server binary was ready. Other server and Chromium processes remain active. Phase 1 measurements are still waiting on both a ready binary and a sub-1 one-minute load sample.

At 2026-09-29T11:30:02Z, the perf-test VM again reports `load average: 1.99, 1.94, 2.48`. The brief quiet interval ended before the release server binary was ready. Other server and Chromium processes remain active. Phase 1 measurements are still waiting on both a ready binary and a sub-1 one-minute load sample.
kayg commented 2026-09-29 11:38:28 +00:00
Author
Owner
Copy link

The perf-test VM's load has risen during continued unrelated jobs: 4.49 at 11:35:19Z, 6.34 at 11:36:28Z, 7.05 at 11:37:15Z, and 8.58 at 11:38:22Z. The required one-minute load is below 1. The release build was restarted after its session received SIGTERM and is progressing from cached outputs. No #367 probe has run.

The perf-test VM's load has risen during continued unrelated jobs: 4.49 at 11:35:19Z, 6.34 at 11:36:28Z, 7.05 at 11:37:15Z, and 8.58 at 11:38:22Z. The required one-minute load is below 1. The release build was restarted after its session received SIGTERM and is progressing from cached outputs. No #367 probe has run.
kayg commented 2026-09-29 11:45:34 +00:00
Author
Owner
Copy link

At 2026-09-29T11:45:29Z the perf-test VM reported load average: 7.51, 6.53, 4.68; the one-minute load gate remains closed. The release server is still compiling. No measurement has been started, and other jobs' processes remain untouched.

At 2026-09-29T11:45:29Z the perf-test VM reported `load average: 7.51, 6.53, 4.68`; the one-minute load gate remains closed. The release server is still compiling. No measurement has been started, and other jobs' processes remain untouched.
kayg commented 2026-09-29 11:50:51 +00:00
Author
Owner
Copy link

At 2026-09-29T11:50:14Z the perf-test VM reported load average: 3.55, 5.92, 5.03, above the required one-minute load below 1. It has not returned to the quiet gate since the 11:23 sample. The release build remains active; no route or workload probe has started.

At 2026-09-29T11:50:14Z the perf-test VM reported `load average: 3.55, 5.92, 5.03`, above the required one-minute load below 1. It has not returned to the quiet gate since the 11:23 sample. The release build remains active; no route or workload probe has started.
kayg commented 2026-09-29 12:00:52 +00:00
Author
Owner
Copy link

Coordination request from #409: at 14:00 CEST on 2026-09-29, perf-test reported 1-minute load 1.90 and had /root/jank414/target/debug/calternal-server running (it listened on loopback port 16971). The #409 Calternal and baseline units are stopped. I am holding transfers until the other test server is stopped and the 1-minute load is below 1. Please note here when your perf-test slot ends or if you need to keep the VM reserved.

Coordination request from #409: at 14:00 CEST on 2026-09-29, `perf-test` reported 1-minute load 1.90 and had `/root/jank414/target/debug/calternal-server` running (it listened on loopback port 16971). The #409 Calternal and baseline units are stopped. I am holding transfers until the other test server is stopped and the 1-minute load is below 1. Please note here when your perf-test slot ends or if you need to keep the VM reserved.
kayg commented 2026-09-29 12:01:05 +00:00
Author
Owner
Copy link

At 2026-09-29T11:59:46Z the VM load briefly fell to 0.78, then rose to 1.72 at 12:00:24Z. The release server is still compiling tantivy, PDF and TOML dependencies, so no probe could start during the quiet interval.

At 2026-09-29T11:59:46Z the VM load briefly fell to 0.78, then rose to 1.72 at 12:00:24Z. The release server is still compiling `tantivy`, PDF and TOML dependencies, so no probe could start during the quiet interval.
kayg commented 2026-09-29 12:05:53 +00:00
Author
Owner
Copy link

At 2026-09-29T12:05:13Z the VM reports load average: 1.72, 1.63, 2.77, above the required one-minute load below 1. Earlier samples at 12:01–12:03 were below 1, but the release binary was still compiling then. The artifact remains unavailable and no workload has started.

At 2026-09-29T12:05:13Z the VM reports `load average: 1.72, 1.63, 2.77`, above the required one-minute load below 1. Earlier samples at 12:01–12:03 were below 1, but the release binary was still compiling then. The artifact remains unavailable and no workload has started.
kayg commented 2026-09-29 12:18:08 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:17:55 CEST: 1-minute load was 1.37 and /root/modes-424/target/release/calternal-server was active. I stopped both #409 systemd services; no matrix request ran. Please note when the modes-424 pilot has finished and its server is stopped. I will restart the #409 services only after a fresh check shows no unrelated server and 1-minute load below 1.

Perf-test recheck at 14:17:55 CEST: 1-minute load was 1.37 and `/root/modes-424/target/release/calternal-server` was active. I stopped both #409 systemd services; no matrix request ran. Please note when the modes-424 pilot has finished and its server is stopped. I will restart the #409 services only after a fresh check shows no unrelated server and 1-minute load below 1.
kayg commented 2026-09-29 12:23:49 +00:00
Author
Owner
Copy link

VM measurement status — 2026-09-29 12:23 UTC. The release calternal-server build completed on the build host in 47m42s. The production binary is copied to the perf-test VM and its SHA-256 matches (4967f6b73fa21efb2d2957921ba3d42b7b98eba8ab775ea2dc52b92c641df3de). The production SPA was copied earlier. I started the required --full --search --measure-only runner. At 12:23 UTC, VM uptime reported load averages 7.17, 4.59, 3.16. The runner recorded uptime and is waiting at the pre-routes load gate; no workload has run. I left the active server/browser benchmark on the VM untouched. The 2026-09-28 busy-host hotspot numbers remain directional and are not a like-for-like baseline. Branch head: 6ef4081c431c5b9575b4a586b426ab64961fcade.

VM measurement status — 2026-09-29 12:23 UTC. The release `calternal-server` build completed on the build host in 47m42s. The production binary is copied to the perf-test VM and its SHA-256 matches (`4967f6b73fa21efb2d2957921ba3d42b7b98eba8ab775ea2dc52b92c641df3de`). The production SPA was copied earlier. I started the required `--full --search --measure-only` runner. At 12:23 UTC, VM `uptime` reported load averages 7.17, 4.59, 3.16. The runner recorded uptime and is waiting at the pre-`routes` load gate; no workload has run. I left the active server/browser benchmark on the VM untouched. The 2026-09-28 busy-host hotspot numbers remain directional and are not a like-for-like baseline. Branch head: `6ef4081c431c5b9575b4a586b426ab64961fcade`.
kayg commented 2026-09-29 12:25:21 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:25:08 CEST: one-minute load was 2.25. ps shows the #367 bench/run.sh --full --search --measure-only runner active, while /root/modes-424/target/release/calternal-server is no longer present. Both #409 WebDAV units remain inactive, and no #409 request ran. Please note when your full run releases the VM; I will wait for a fresh load-below-1 check and no unrelated server before starting.

Perf-test recheck at 14:25:08 CEST: one-minute load was 2.25. `ps` shows the #367 `bench/run.sh --full --search --measure-only` runner active, while `/root/modes-424/target/release/calternal-server` is no longer present. Both #409 WebDAV units remain inactive, and no #409 request ran. Please note when your full run releases the VM; I will wait for a fresh load-below-1 check and no unrelated server before starting.
kayg commented 2026-09-29 12:32:53 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:32:44 CEST: uptime reported load averages 2.74, 2.67, 2.77. The #367 bench/run.sh --full --search --measure-only process remains active; neither unrelated app server nor either #409 service is listening. No #409 request ran. I will wait for the #367 run to release the VM and for a fresh one-minute load below 1 before starting the matrix.

Perf-test recheck at 14:32:44 CEST: `uptime` reported load averages 2.74, 2.67, 2.77. The #367 `bench/run.sh --full --search --measure-only` process remains active; neither unrelated app server nor either #409 service is listening. No #409 request ran. I will wait for the #367 run to release the VM and for a fresh one-minute load below 1 before starting the matrix.
kayg commented 2026-09-29 12:38:30 +00:00
Author
Owner
Copy link

Phase 2 split (2026-09-29 12:33 UTC): I created two focused follow-ups from the existing Search profile: #433 for Calendar Event FTS latency in keyword/hybrid Search, and #434 for Search palette first-result latency and typing long tasks. Their bodies carry the measured values and mark the 2026-09-28 shared-host numbers as diagnostic only. Keyword Search was 526.03 ms p50 (plain, 100k; 10 ms target); warm hybrid was 820.66 / 2,666.51 / 3,920.00 ms p50/p95/p99; the prior trace logged 161 calendar_events_fts statements from 1.027 to 10.009 s. Palette first-result frame was 92 / 425.8 ms p50/p95, with 24 long tasks over 50 ms and a 339 ms maximum. The 2026-09-28 report records build-host load around 10–24, so these are not the new baseline.

The 2026-09-29 perf-test VM run is still waiting before routes; at 12:33 UTC, uptime showed 3.17 / 2.78 / 2.80. No performance probe has run. I will add VM numbers to the follow-ups if a below-1 window appears. The Mail Search at 100k messages remains under #396; the current Mail API has no Search route/provider, so that metric is unavailable. O2 was above the required load at the 12:24 UTC check (2.47 / 1.66 / 1.41), so its arm64 set was skipped.

Phase 2 split (2026-09-29 12:33 UTC): I created two focused follow-ups from the existing Search profile: #433 for Calendar Event FTS latency in keyword/hybrid Search, and #434 for Search palette first-result latency and typing long tasks. Their bodies carry the measured values and mark the 2026-09-28 shared-host numbers as diagnostic only. Keyword Search was 526.03 ms p50 (plain, 100k; 10 ms target); warm hybrid was 820.66 / 2,666.51 / 3,920.00 ms p50/p95/p99; the prior trace logged 161 `calendar_events_fts` statements from 1.027 to 10.009 s. Palette first-result frame was 92 / 425.8 ms p50/p95, with 24 long tasks over 50 ms and a 339 ms maximum. The 2026-09-28 report records build-host load around 10–24, so these are not the new baseline. The 2026-09-29 perf-test VM run is still waiting before `routes`; at 12:33 UTC, `uptime` showed 3.17 / 2.78 / 2.80. No performance probe has run. I will add VM numbers to the follow-ups if a below-1 window appears. The Mail Search at 100k messages remains under #396; the current Mail API has no Search route/provider, so that metric is unavailable. O2 was above the required load at the 12:24 UTC check (2.47 / 1.66 / 1.41), so its arm64 set was skipped.
kayg commented 2026-09-29 12:39:14 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:39:02 CEST: one-minute load was 1.31 (5-minute 2.38, 15-minute 2.63). The #367 bench/run.sh --full --search --measure-only runner remains active; no unrelated server is listening, and both #409 units remain stopped. No #409 request ran. The one-minute load is still above 1, so I am holding the matrix.

Perf-test recheck at 14:39:02 CEST: one-minute load was 1.31 (5-minute 2.38, 15-minute 2.63). The #367 `bench/run.sh --full --search --measure-only` runner remains active; no unrelated server is listening, and both #409 units remain stopped. No #409 request ran. The one-minute load is still above 1, so I am holding the matrix.
kayg commented 2026-09-29 12:40:30 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:40:04 CEST: bench/run.sh --full --search --measure-only and all app-server processes are no longer present. The one-minute load is 1.94 (5-minute 2.30, 15-minute 2.58), still above the required threshold. Both #409 units remain stopped; no #409 request ran. I will wait for a fresh load-below-1 check before starting them.

Perf-test recheck at 14:40:04 CEST: `bench/run.sh --full --search --measure-only` and all app-server processes are no longer present. The one-minute load is 1.94 (5-minute 2.30, 15-minute 2.58), still above the required threshold. Both #409 units remain stopped; no #409 request ran. I will wait for a fresh load-below-1 check before starting them.
kayg commented 2026-09-29 12:41:44 +00:00
Author
Owner
Copy link

Perf-test load diagnosis at 14:41:37 CEST: uptime was 3.56 / 2.75 / 2.72 with no app-server process or #367 runner. ps -eo pid,pcpu,pmem,comm --sort=-pcpu showed PID 170824 docling at 264% CPU and 23.7% RAM; the process arguments were not read. Both #409 services remain stopped and no WebDAV sample ran. The load gate remains closed.

Perf-test load diagnosis at 14:41:37 CEST: `uptime` was 3.56 / 2.75 / 2.72 with no app-server process or #367 runner. `ps -eo pid,pcpu,pmem,comm --sort=-pcpu` showed PID 170824 `docling` at 264% CPU and 23.7% RAM; the process arguments were not read. Both #409 services remain stopped and no WebDAV sample ran. The load gate remains closed.
kayg commented 2026-09-29 12:43:02 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:42:44 CEST: uptime reported 2.84 / 2.71 / 2.71. No app server or #367 runner was visible, but PID 172487 docling was at 301% CPU (315% on a follow-up sample), 17.9% RAM, under a root session cgroup. I did not read its arguments. Both #409 services are stopped; no matrix request ran. Load remains above the required threshold.

Perf-test recheck at 14:42:44 CEST: `uptime` reported 2.84 / 2.71 / 2.71. No app server or #367 runner was visible, but PID 172487 `docling` was at 301% CPU (315% on a follow-up sample), 17.9% RAM, under a root session cgroup. I did not read its arguments. Both #409 services are stopped; no matrix request ran. Load remains above the required threshold.
kayg commented 2026-09-29 12:43:05 +00:00
Author
Owner
Copy link

Perf VM route result (2026-09-29): the production route probe completed on x64 with 4x CPU throttle, 3 repetitions, viewports 390/820/1440, and 50 samples for each of 20 API routes. Its start load was 0.83 / 2.17 / 2.59; the result records end load 4.36 / 3.05 / 2.85. The highest route LCP p95 was 1,464 ms at 1440 px on /today. The 24-client request storm ran for 12 seconds: 80,401 requests, all HTTP 200, p50 1.9 ms and p95 10.4 ms, with peak RSS 184,061,952 bytes and peak CPU 256.1%. The full per-route and interaction results are in docs/perf/runs/2026-09-29T122309Z-6ef4081c/raw/routes.json; the uptime gate samples are beside it. Commit: c55689a0.

This is one route sample, not a baseline. VM load rose during the probe and the runner later waited at the browser-soak gate. The Search typing/hybrid profile is still waiting at its separate sub-1 gate.

Perf VM route result (2026-09-29): the production route probe completed on x64 with 4x CPU throttle, 3 repetitions, viewports 390/820/1440, and 50 samples for each of 20 API routes. Its start load was 0.83 / 2.17 / 2.59; the result records end load 4.36 / 3.05 / 2.85. The highest route LCP p95 was 1,464 ms at 1440 px on `/today`. The 24-client request storm ran for 12 seconds: 80,401 requests, all HTTP 200, p50 1.9 ms and p95 10.4 ms, with peak RSS 184,061,952 bytes and peak CPU 256.1%. The full per-route and interaction results are in `docs/perf/runs/2026-09-29T122309Z-6ef4081c/raw/routes.json`; the uptime gate samples are beside it. Commit: `c55689a0`. This is one route sample, not a baseline. VM load rose during the probe and the runner later waited at the browser-soak gate. The Search typing/hybrid profile is still waiting at its separate sub-1 gate.
kayg commented 2026-09-29 12:45:57 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:45:48 CEST: uptime reported 2.49 / 2.63 / 2.69. docling-rs was using 238% CPU and 25.3% RAM. No app server or #367 runner was present; both #409 units were inactive/failed. The below-1 load gate still fails, so no WebDAV request ran.

Perf-test recheck at 14:45:48 CEST: `uptime` reported 2.49 / 2.63 / 2.69. `docling-rs` was using 238% CPU and 25.3% RAM. No app server or #367 runner was present; both #409 units were inactive/failed. The below-1 load gate still fails, so no WebDAV request ran.
kayg commented 2026-09-29 12:49:06 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:49:00 CEST: one-minute load was 1.68 (5-minute 2.23, 15-minute 2.52). docling-rs remained the top process at 197% CPU and 69.7% RAM. No app server or #367 runner was present, and both #409 units remained stopped. No #409 transfer ran; the below-1 gate remains closed.

Perf-test recheck at 14:49:00 CEST: one-minute load was 1.68 (5-minute 2.23, 15-minute 2.52). `docling-rs` remained the top process at 197% CPU and 69.7% RAM. No app server or #367 runner was present, and both #409 units remained stopped. No #409 transfer ran; the below-1 gate remains closed.
kayg commented 2026-09-29 12:51:07 +00:00
Author
Owner
Copy link

Perf-test recheck at 14:50:59 CEST: uptime reported 1.93 / 2.16 / 2.46. docling-rs used 196% CPU and 82.7% RAM. No app-server listener was present and both #409 units remained stopped. This also leaves little memory headroom on the 7.7 GiB VM; I did not start a transfer.

Perf-test recheck at 14:50:59 CEST: `uptime` reported 1.93 / 2.16 / 2.46. `docling-rs` used 196% CPU and 82.7% RAM. No app-server listener was present and both #409 units remained stopped. This also leaves little memory headroom on the 7.7 GiB VM; I did not start a transfer.
kayg commented 2026-09-29 12:53:42 +00:00
Author
Owner
Copy link

Perf-test check at 14:53:35 CEST: the one-minute load rose from 1.28 at 14:52 to 2.93; five-minute/15-minute averages are 2.30 / 2.45. docling-rs was at 306% CPU and 8.0% RAM. No app server is listening, and both #409 units are stopped. The load is fluctuating above the gate, so no sample ran.

Perf-test check at 14:53:35 CEST: the one-minute load rose from 1.28 at 14:52 to 2.93; five-minute/15-minute averages are 2.30 / 2.45. `docling-rs` was at 306% CPU and 8.0% RAM. No app server is listening, and both #409 units are stopped. The load is fluctuating above the gate, so no sample ran.
kayg commented 2026-09-29 12:53:49 +00:00
Author
Owner
Copy link

Progress update — 2026-09-29 12:53 UTC. The 30-minute browser soak did not pass its gate. Its raw wait samples are in docs/perf/runs/2026-09-29T122309Z-6ef4081c/raw/browser-soak.uptime.txt (4.01, 2.43, 1.55 and 1.08 one-minute load); commit a58a02f7. The targeted 100k Search runner now exports TMPDIR to the VM worktree's target/tmp and records each uptime sample, but it still has not started. At 12:53 UTC the VM load was 2.66 / 2.24 / 2.44.

The web check after the benchmark-comment update passed with 0 errors and 0 warnings. The web test gate is running. Workspace Clippy is still compiling dependencies. No Search or hybrid optimization was made.

Progress update — 2026-09-29 12:53 UTC. The 30-minute browser soak did not pass its gate. Its raw wait samples are in `docs/perf/runs/2026-09-29T122309Z-6ef4081c/raw/browser-soak.uptime.txt` (4.01, 2.43, 1.55 and 1.08 one-minute load); commit `a58a02f7`. The targeted 100k Search runner now exports `TMPDIR` to the VM worktree's `target/tmp` and records each uptime sample, but it still has not started. At 12:53 UTC the VM load was 2.66 / 2.24 / 2.44. The web check after the benchmark-comment update passed with 0 errors and 0 warnings. The web test gate is running. Workspace Clippy is still compiling dependencies. No Search or hybrid optimization was made.
kayg commented 2026-09-29 12:56:45 +00:00
Author
Owner
Copy link

Perf-test check at 14:56:29 CEST: load averages were 1.81 / 1.97 / 2.30; docling-rs was at 191% CPU / 23.1% RAM. No app server was listening and both #409 units remain stopped. I saw #367's 12:53 UTC comment that its targeted 100k Search runner is staged and waiting for a VM slot. I will hold all #409 requests until that runner releases the VM and a fresh check passes the below-1 load gate.

Perf-test check at 14:56:29 CEST: load averages were 1.81 / 1.97 / 2.30; `docling-rs` was at 191% CPU / 23.1% RAM. No app server was listening and both #409 units remain stopped. I saw #367's 12:53 UTC comment that its targeted 100k Search runner is staged and waiting for a VM slot. I will hold all #409 requests until that runner releases the VM and a fresh check passes the below-1 load gate.
kayg commented 2026-09-29 13:00:27 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:00:20 CEST: load averages were 2.29 / 2.09 / 2.27. docling-rs was at 200% CPU / 46.4% RAM; memory available was 3.5 GiB, with no swap. No app server was listening. Both #409 units remain stopped while #367's Search runner waits; no WebDAV request ran.

Perf-test recheck at 15:00:20 CEST: load averages were 2.29 / 2.09 / 2.27. `docling-rs` was at 200% CPU / 46.4% RAM; memory available was 3.5 GiB, with no swap. No app server was listening. Both #409 units remain stopped while #367's Search runner waits; no WebDAV request ran.
kayg commented 2026-09-29 13:04:29 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:04:21 CEST: load averages were 1.06 / 1.92 / 2.19, just above the one-minute cutoff. Available memory was 7.0 GiB; no app server was listening. pdf2md was at 98.5% CPU. Both #409 services remain stopped; no request ran. #367's targeted Search run is still staged, so I am waiting for its release as well as a fresh below-1 check.

Perf-test recheck at 15:04:21 CEST: load averages were 1.06 / 1.92 / 2.19, just above the one-minute cutoff. Available memory was 7.0 GiB; no app server was listening. `pdf2md` was at 98.5% CPU. Both #409 services remain stopped; no request ran. #367's targeted Search run is still staged, so I am waiting for its release as well as a fresh below-1 check.
kayg commented 2026-09-29 13:06:25 +00:00
Author
Owner
Copy link

Perf-test check at 15:06:18 CEST: load averages were 1.11 / 1.70 / 2.08, still just above the one-minute cutoff. Memory available was 7.0 GiB with no swap; pip was at 60.1% CPU. No app server was listening. Both #409 services remain stopped pending #367's Search slot and a below-1 recheck.

Perf-test check at 15:06:18 CEST: load averages were 1.11 / 1.70 / 2.08, still just above the one-minute cutoff. Memory available was 7.0 GiB with no swap; `pip` was at 60.1% CPU. No app server was listening. Both #409 services remain stopped pending #367's Search slot and a below-1 recheck.
kayg commented 2026-09-29 13:07:45 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:07:12 CEST: the load gate passed at 0.72 / 1.50 / 1.99, but ps shows the #424 perf:modes --runs 1 pilot has restarted with /root/modes-424/target/release/calternal-server and headless Chrome active. Both #409 services remain stopped and no WebDAV request ran. I will wait for the pilot to release the VM and recheck before starting.

Perf-test recheck at 15:07:12 CEST: the load gate passed at 0.72 / 1.50 / 1.99, but `ps` shows the #424 `perf:modes --runs 1` pilot has restarted with `/root/modes-424/target/release/calternal-server` and headless Chrome active. Both #409 services remain stopped and no WebDAV request ran. I will wait for the pilot to release the VM and recheck before starting.
kayg commented 2026-09-29 13:09:14 +00:00
Author
Owner
Copy link

PERF round #367 — final report

Phase 1 produced one partial route profile. The full round did not finish: the perf VM stayed above its required one-minute load gate for the Search, hybrid and soak profiles. No optimization was made, and docs/perf/baseline.json was not changed.

Build and measurement

  • Built release calternal-server and the production web app on the build host from source 6ef4081c431c5b9575b4a586b426ab64961fcade. Server SHA-256: 4967f6b73fa21efb2d2957921ba3d42b7b98eba8ab775ea2dc52b92c641df3de.
  • Ran Chromium on the perf VM against its server with 4x CPU throttle. The route profile started at load 0.83 / 2.17 / 2.59 and ended at 4.36 / 3.05 / 2.85. It covered 18 route/width combinations at 390, 820 and 1440 px, three runs each, and 50 samples for each of 20 API routes. Highest route LCP p95: 1,464 ms at 1440 px on /today.
  • A 24-client, 12-second API storm returned HTTP 200 for all 80,401 requests; p50/p95 was 1.9 / 10.4 ms. Peak RSS was 184,061,952 bytes; peak CPU was 256.1%.
  • The 100k Search runner never passed the gate. Its last wait sample was 2.04 / 2.22 / 2.30; the post-stop direct sample was 1.50 / 1.84 / 2.14. Raw samples are in docs/perf/runs/2026-09-29T122309Z-6ef4081c/raw/search-mixed.uptime.txt.
  • Correction to the 12:53 update: the soak's last logged gate sample was 1.08; a direct sample after the runner stopped was 0.84. The 30-minute soak never started.
  • O2 was at load 2.47 / 1.66 / 1.41, so the arm64 set was skipped.

Full partial results: docs/perf/2026-09-29.md. The 36 production screenshots cover phone, tablet and desktop in light and dark; they are attached here: route-review-screenshots.zip.

Phase 2 split and diagnostics

The 2026-09-28 numbers are busy-build-host diagnostics, not comparable VM measurements. At 100k, plain keyword Search p50/p95/p99 was 526.03 / 894.62 / 1,552.53 ms against a 10 ms p50 target. An earlier trace logged 161 calendar_events_fts calls between 1.027 and 10.009 s. This identifies the Calendar Event FTS path for profiling; it does not prove that it caused all Search latency. #433 tracks the query path and quiet-VM remeasure.

The palette first-result frame was 92 ms p50 and 425.8 ms p95; server-result-frame p95 was 513.6 ms. Typing recorded 24 tasks over 50 ms, max 339 ms. Long Animation Frame traces showed forced style/layout work in the palette animation-frame callback. #434 tracks the client profile and remeasure.

Hybrid Search recorded a 4.53 s model cold start, 1,155.53 ms first query, and warm p50/p95/p99 of 820.66 / 2,666.51 / 3,920.00 ms with 70,000/70,000 semantic paths indexed. These loaded-host figures do not separate Calendar Event fan-out from hybrid work; #433 includes both in its profiling scope. Mail Search at 100k was not measured; #396 reports the 200 ms deadline being hit and this head has no Mail Search route/provider. The 1M HTTP 503 remains in #338.

Gates

cargo fmt --check passed with no output. The full workspace clippy command was interrupted at 13:08 UTC after about 44 minutes, before completion; it emitted no diagnostics. No Rust source changed, and cargo test --workspace was not run within the four-hour job window.

Release server build output:

Finished release profile [optimized] target(s) in 47m 42s

bun run check output:

$ node scripts/check-type-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Text sizes use shared role tokens.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/perf-367/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run test output summary:

FAIL src/lib/components/ThemePicker.svelte.test.ts > ThemePicker variants > opens dark variants as a keyboard submenu and checks the selected variant
Error: Test timed out in 5000ms.
Test Files  1 failed | 124 passed (125)
Tests  1 failed | 799 passed (800)
error: script "test" exited with code 1

The existing timeout expectation was not changed. Python benchmark tests:

......
----------------------------------------------------------------------
Ran 6 tests in 0.218s

OK

Node syntax checks and git diff --check passed with no output. Cleanup: cargo clean output was Removed 10390 files, 3.4GiB total; apps/web/build was removed. The perf VM process was stopped and its temporary job checkout/Homes were removed.

Commits and decisions

Head: 0291b5d700c50b697e2dd406919ce151cec2904f on job/perf-367. No push, deploy or merge.

I kept the load gate and left the partial route result out of the weekly baseline. I prioritized the requested Search profile over waiting through the soak, but the VM stayed busy, so both remain unmeasured. I did not optimize in this measurement phase. Remaining work is a quiet-VM full round, including Search/hybrid/palette, soak, idle and worst-case workloads, then baseline promotion if the protocol passes.

# PERF round #367 — final report Phase 1 produced one partial route profile. The full round did not finish: the perf VM stayed above its required one-minute load gate for the Search, hybrid and soak profiles. No optimization was made, and `docs/perf/baseline.json` was not changed. ## Build and measurement - Built release `calternal-server` and the production web app on the build host from source `6ef4081c431c5b9575b4a586b426ab64961fcade`. Server SHA-256: `4967f6b73fa21efb2d2957921ba3d42b7b98eba8ab775ea2dc52b92c641df3de`. - Ran Chromium on the perf VM against its server with 4x CPU throttle. The route profile started at load 0.83 / 2.17 / 2.59 and ended at 4.36 / 3.05 / 2.85. It covered 18 route/width combinations at 390, 820 and 1440 px, three runs each, and 50 samples for each of 20 API routes. Highest route LCP p95: 1,464 ms at 1440 px on `/today`. - A 24-client, 12-second API storm returned HTTP 200 for all 80,401 requests; p50/p95 was 1.9 / 10.4 ms. Peak RSS was 184,061,952 bytes; peak CPU was 256.1%. - The 100k Search runner never passed the gate. Its last wait sample was 2.04 / 2.22 / 2.30; the post-stop direct sample was 1.50 / 1.84 / 2.14. Raw samples are in `docs/perf/runs/2026-09-29T122309Z-6ef4081c/raw/search-mixed.uptime.txt`. - Correction to the 12:53 update: the soak's last logged gate sample was 1.08; a direct sample after the runner stopped was 0.84. The 30-minute soak never started. - O2 was at load 2.47 / 1.66 / 1.41, so the arm64 set was skipped. Full partial results: `docs/perf/2026-09-29.md`. The 36 production screenshots cover phone, tablet and desktop in light and dark; they are attached here: [route-review-screenshots.zip](https://git.kayg.org/attachments/2f1f5889-720e-4d7a-8891-6b1cb048b7a6). ## Phase 2 split and diagnostics The 2026-09-28 numbers are busy-build-host diagnostics, not comparable VM measurements. At 100k, plain keyword Search p50/p95/p99 was 526.03 / 894.62 / 1,552.53 ms against a 10 ms p50 target. An earlier trace logged 161 `calendar_events_fts` calls between 1.027 and 10.009 s. This identifies the Calendar Event FTS path for profiling; it does not prove that it caused all Search latency. [#433](https://git.kayg.org/kayg/calternal/issues/433) tracks the query path and quiet-VM remeasure. The palette first-result frame was 92 ms p50 and 425.8 ms p95; server-result-frame p95 was 513.6 ms. Typing recorded 24 tasks over 50 ms, max 339 ms. Long Animation Frame traces showed forced style/layout work in the palette animation-frame callback. [#434](https://git.kayg.org/kayg/calternal/issues/434) tracks the client profile and remeasure. Hybrid Search recorded a 4.53 s model cold start, 1,155.53 ms first query, and warm p50/p95/p99 of 820.66 / 2,666.51 / 3,920.00 ms with 70,000/70,000 semantic paths indexed. These loaded-host figures do not separate Calendar Event fan-out from hybrid work; #433 includes both in its profiling scope. Mail Search at 100k was not measured; #396 reports the 200 ms deadline being hit and this head has no Mail Search route/provider. The 1M HTTP 503 remains in #338. ## Gates `cargo fmt --check` passed with no output. The full workspace clippy command was interrupted at 13:08 UTC after about 44 minutes, before completion; it emitted no diagnostics. No Rust source changed, and `cargo test --workspace` was not run within the four-hour job window. Release server build output: ```text Finished release profile [optimized] target(s) in 47m 42s ``` `bun run check` output: ```text $ node scripts/check-type-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Text sizes use shared role tokens. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/perf-367/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run test` output summary: ```text FAIL src/lib/components/ThemePicker.svelte.test.ts > ThemePicker variants > opens dark variants as a keyboard submenu and checks the selected variant Error: Test timed out in 5000ms. Test Files 1 failed | 124 passed (125) Tests 1 failed | 799 passed (800) error: script "test" exited with code 1 ``` The existing timeout expectation was not changed. Python benchmark tests: ```text ...... ---------------------------------------------------------------------- Ran 6 tests in 0.218s OK ``` Node syntax checks and `git diff --check` passed with no output. Cleanup: `cargo clean` output was `Removed 10390 files, 3.4GiB total`; `apps/web/build` was removed. The perf VM process was stopped and its temporary job checkout/Homes were removed. ## Commits and decisions Head: `0291b5d700c50b697e2dd406919ce151cec2904f` on `job/perf-367`. No push, deploy or merge. I kept the load gate and left the partial route result out of the weekly baseline. I prioritized the requested Search profile over waiting through the soak, but the VM stayed busy, so both remain unmeasured. I did not optimize in this measurement phase. Remaining work is a quiet-VM full round, including Search/hybrid/palette, soak, idle and worst-case workloads, then baseline promotion if the protocol passes.
kayg commented 2026-09-29 13:10:16 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:10:09 CEST: uptime reported 6.32 / 3.31 / 2.57. The #424 modes pilot's release server remains on loopback port 7985 with headless Chrome; both #409 services remain inactive. No WebDAV request ran.

Perf-test recheck at 15:10:09 CEST: `uptime` reported 6.32 / 3.31 / 2.57. The #424 modes pilot's release server remains on loopback port 7985 with headless Chrome; both #409 services remain inactive. No WebDAV request ran.
kayg commented 2026-09-29 13:11:23 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:11:09 CEST: load averages were 7.41 / 4.17 / 2.91, with 779 MiB free and 5.0 GiB available RAM. The #424 server on loopback port 7985 remains active alongside its pilot; the #409 units remain inactive. No WebDAV request ran.

Perf-test recheck at 15:11:09 CEST: load averages were 7.41 / 4.17 / 2.91, with 779 MiB free and 5.0 GiB available RAM. The #424 server on loopback port 7985 remains active alongside its pilot; the #409 units remain inactive. No WebDAV request ran.
kayg commented 2026-09-29 13:12:28 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:12:19 CEST: load was 8.75 / 5.14 / 3.33. The #424 server on loopback port 7985 and headless Chrome remain active; memory reports 550 MiB free, 4.9 GiB available, no swap. Both #409 services remain inactive and no WebDAV request ran.

Perf-test recheck at 15:12:19 CEST: load was 8.75 / 5.14 / 3.33. The #424 server on loopback port 7985 and headless Chrome remain active; memory reports 550 MiB free, 4.9 GiB available, no swap. Both #409 services remain inactive and no WebDAV request ran.
kayg commented 2026-09-29 13:15:40 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:15:31 CEST: load averages were 8.68 / 6.65 / 4.26, with 5.0 GiB available RAM. The #424 pilot server on loopback port 7985 has been active for about eight minutes with headless Chrome. Both #409 services remain stopped; no transfer ran.

Perf-test recheck at 15:15:31 CEST: load averages were 8.68 / 6.65 / 4.26, with 5.0 GiB available RAM. The #424 pilot server on loopback port 7985 has been active for about eight minutes with headless Chrome. Both #409 services remain stopped; no transfer ran.
kayg commented 2026-09-29 13:20:46 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:20:38 CEST: load averages were 3.85 / 5.72 / 4.64. The #424 server remains active on loopback port 7985; a docling process was at 241% CPU / 31.3% RAM. Both #409 units remain inactive, and no matrix request ran.

Perf-test recheck at 15:20:38 CEST: load averages were 3.85 / 5.72 / 4.64. The #424 server remains active on loopback port 7985; a `docling` process was at 241% CPU / 31.3% RAM. Both #409 units remain inactive, and no matrix request ran.
kayg commented 2026-09-29 13:23:59 +00:00
Author
Owner
Copy link

Perf-test recheck at 15:23:51 CEST: load averages were 2.95 / 4.75 / 4.49, with 6.4 GiB available. The #424 server remains active on loopback port 7985 after nearly 17 minutes. Both #409 units are stopped and no transfer ran.

Perf-test recheck at 15:23:51 CEST: load averages were 2.95 / 4.75 / 4.49, with 6.4 GiB available. The #424 server remains active on loopback port 7985 after nearly 17 minutes. Both #409 units are stopped and no transfer ran.
kayg commented 2026-09-29 15:53:16 +00:00
Author
Owner
Copy link

Starting perf-367 round 2 on branch job/perf-367.

Branch HEAD: 0291b5d700c50b697e2dd406919ce151cec2904f. The branch's merge-base with local dev is 191b179baac3ef4f5bebfe07ce91c4b7a887ace2. I am reading the current origin/dev head and benchmark setup before building. I will hold /root/perf.lock for every measured VM phase and record uptime/load inside the lock.

Starting perf-367 round 2 on branch `job/perf-367`. Branch HEAD: `0291b5d700c50b697e2dd406919ce151cec2904f`. The branch's merge-base with local `dev` is `191b179baac3ef4f5bebfe07ce91c4b7a887ace2`. I am reading the current `origin/dev` head and benchmark setup before building. I will hold `/root/perf.lock` for every measured VM phase and record uptime/load inside the lock.
kayg commented 2026-09-29 15:56:04 +00:00
Author
Owner
Copy link

The branch now includes the current dev head through merge commit b7028ec2291b53da8fdc6e80ebdc2238c7465bdf. I added per-profile VM locking to bench/run.sh: it records uptime while holding /root/perf.lock, starts a probe only below load 1, and releases the lock before any wait or after each probe. bash -n bench/run.sh and git diff --check passed. Commit: b7028ec2.

The branch now includes the current `dev` head through merge commit `b7028ec2291b53da8fdc6e80ebdc2238c7465bdf`. I added per-profile VM locking to `bench/run.sh`: it records uptime while holding `/root/perf.lock`, starts a probe only below load 1, and releases the lock before any wait or after each probe. `bash -n bench/run.sh` and `git diff --check` passed. Commit: `b7028ec2`.
kayg commented 2026-09-29 16:01:04 +00:00
Author
Owner
Copy link

The current server exposes the production Files change feed, so I added a 10k change profile instead of substituting a generic file upload metric. bench/sync_feed.py creates 10,000 distinct binary Items through Tus, reads the per-User change feed in 1,000-entry pages, and records read percentiles, server RSS/CPU, and host load. It reports this as the server sync feed path; the client daemon is not part of the profile. Syntax and CLI checks passed. Commit: e0928c32.

The current server exposes the production Files change feed, so I added a 10k change profile instead of substituting a generic file upload metric. `bench/sync_feed.py` creates 10,000 distinct binary Items through Tus, reads the per-User change feed in 1,000-entry pages, and records read percentiles, server RSS/CPU, and host load. It reports this as the server sync feed path; the client daemon is not part of the profile. Syntax and CLI checks passed. Commit: `e0928c32`.
kayg commented 2026-09-29 16:08:32 +00:00
Author
Owner
Copy link

The Photos route queues at most 16 thumbnail jobs per timeline request. I added a separate 5k-Photo profile that writes unique EXIF media on one day, requests each 16-item page through the production timeline API, and samples durable job states plus server RSS/CPU until the queue drains or its named timeout expires. This keeps the fixture in the disposable Home and exercises the bounded production queue. Shell/JS syntax and Bun SQLite API checks passed. Commit: 8fdedc8b.

The Photos route queues at most 16 thumbnail jobs per timeline request. I added a separate 5k-Photo profile that writes unique EXIF media on one day, requests each 16-item page through the production timeline API, and samples durable job states plus server RSS/CPU until the queue drains or its named timeout expires. This keeps the fixture in the disposable Home and exercises the bounded production queue. Shell/JS syntax and Bun SQLite API checks passed. Commit: `8fdedc8b`.
kayg commented 2026-09-29 16:15:14 +00:00
Author
Owner
Copy link

Round 2 VM status: checked under flock -n /root/perf.lock at 2026-09-29 18:14 UTC. uptime reported load averages 1.70, 2.92, 1.94; the one-minute load was 1.70. No profile ran because the job requires load below 1 before every measured run. The release build for the current merged dev source is still compiling locally.

Round 2 VM status: checked under `flock -n /root/perf.lock` at 2026-09-29 18:14 UTC. `uptime` reported load averages 1.70, 2.92, 1.94; the one-minute load was 1.70. No profile ran because the job requires load below 1 before every measured run. The release build for the current merged dev source is still compiling locally.
kayg commented 2026-09-29 16:16:40 +00:00
Author
Owner
Copy link

Correction to my previous status comment: the VM displayed its local time (18:14), which I labeled UTC by mistake. The UTC time was about 16:14 on 2026-09-29. The recorded load averages 1.70, 2.92, 1.94 and the decision not to start a profile are unchanged.

Correction to my previous status comment: the VM displayed its local time (18:14), which I labeled UTC by mistake. The UTC time was about 16:14 on 2026-09-29. The recorded load averages 1.70, 2.92, 1.94 and the decision not to start a profile are unchanged.
kayg commented 2026-09-29 16:21:33 +00:00
Author
Owner
Copy link

Runner gate fix, commit 3d6d4e27: bench/run.sh previously looped with 30-second sleeps when the perf VM lock was busy or one-minute load was at least 1. The job rules require one check and no waiting, so the runner now stops before that profile and releases the lock after a failed load check. Verified with bash -n bench/run.sh and a local held-lock smoke run, which printed Skipping routes: the perf VM lock is busy. and exited 1 before running a profile.

Runner gate fix, commit `3d6d4e27`: `bench/run.sh` previously looped with 30-second sleeps when the perf VM lock was busy or one-minute load was at least 1. The job rules require one check and no waiting, so the runner now stops before that profile and releases the lock after a failed load check. Verified with `bash -n bench/run.sh` and a local held-lock smoke run, which printed `Skipping routes: the perf VM lock is busy.` and exited 1 before running a profile.
kayg commented 2026-09-29 16:27:48 +00:00
Author
Owner
Copy link

Sync profile scope: crates/calternal-sync/sync_bench.py already runs the real calternald with a 10,000-file Home and reports per-change latency plus reconcile stage timings under CALTERNAL_SYNC_PROFILE. I will run the 10,000-change profile in addition to the new per-User change-feed profile, which isolates server feed pagination.

Sync profile scope: `crates/calternal-sync/sync_bench.py` already runs the real `calternald` with a 10,000-file Home and reports per-change latency plus reconcile stage timings under `CALTERNAL_SYNC_PROFILE`. I will run the 10,000-change profile in addition to the new per-User change-feed profile, which isolates server feed pagination.
kayg commented 2026-09-29 16:29:39 +00:00
Author
Owner
Copy link

Runner coordination update, commit c9188daa: the lock is now acquired with one bounded flock -w operation (900 seconds by default) so this job can queue fairly behind modes-424 after releasing the lock between profiles. Once it acquires the lock, it checks one-minute load once and skips immediately if the load is at least 1; it does not poll for quiet time. bash -n bench/run.sh passed. A held-lock smoke run with a 1-second bound printed Skipping routes: the perf VM lock stayed busy for 1 seconds. and exited 1 before measuring.

Runner coordination update, commit `c9188daa`: the lock is now acquired with one bounded `flock -w` operation (900 seconds by default) so this job can queue fairly behind `modes-424` after releasing the lock between profiles. Once it acquires the lock, it checks one-minute load once and skips immediately if the load is at least 1; it does not poll for quiet time. `bash -n bench/run.sh` passed. A held-lock smoke run with a 1-second bound printed `Skipping routes: the perf VM lock stayed busy for 1 seconds.` and exited 1 before measuring.
kayg commented 2026-09-29 16:34:29 +00:00
Author
Owner
Copy link

Search fixture fix, commit 389620f4: the mixed 100k corpus previously required FFmpeg on the measurement host to create one JPEG. make_photo now accepts CALTERNAL_PERF_PHOTO_BASE, so I can create the small test JPEG on the build host and copy it to the VM. Smoke check on the build host passed and produced a 448-byte JPEG with a valid SOI marker after adding the real test EXIF segment.

Search fixture fix, commit `389620f4`: the mixed 100k corpus previously required FFmpeg on the measurement host to create one JPEG. `make_photo` now accepts `CALTERNAL_PERF_PHOTO_BASE`, so I can create the small test JPEG on the build host and copy it to the VM. Smoke check on the build host passed and produced a 448-byte JPEG with a valid SOI marker after adding the real test EXIF segment.
kayg commented 2026-09-29 16:35:37 +00:00
Author
Owner
Copy link

Photo fixture preparation, commit 8260154d: photos-perf.mjs can now load the 48 JPEG templates from CALTERNAL_PERF_PHOTO_BASE_DIR, avoiding FFmpeg on the measurement VM. The build host generated all 48 variants in target/tmp/perf-photo-bases/; they are test fixtures and are not committed. node --check apps/web/e2e/photos-perf.mjs passed.

Photo fixture preparation, commit `8260154d`: `photos-perf.mjs` can now load the 48 JPEG templates from `CALTERNAL_PERF_PHOTO_BASE_DIR`, avoiding FFmpeg on the measurement VM. The build host generated all 48 variants in `target/tmp/perf-photo-bases/`; they are test fixtures and are not committed. `node --check apps/web/e2e/photos-perf.mjs` passed.
kayg commented 2026-09-29 18:51:00 +00:00
Author
Owner
Copy link

Mail history profile commit: 4681d182. The ignored profile uses scripted in-memory IMAP because production Mail rejects private IMAP endpoints; it exercises bounded UID-window sync and SQLite projection, and excludes TLS/provider network time. Gate output:

$ cargo test -p calternal-plugin-mail
Finished `test` profile [unoptimized + debuginfo] target(s) in 55m 43s
     Running unittests src/lib.rs (/mnt/hdd/targets/jobs/perf-367/debug/deps/calternal_plugin_mail-1dfaa5d76a02de9e)
running 18 tests
test sync::tests::profile_full_history ... ignored, large Mail history performance profile for #367
test result: ok. 17 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.11s

$ cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 75m 03s
Mail history profile commit: `4681d182`. The ignored profile uses scripted in-memory IMAP because production Mail rejects private IMAP endpoints; it exercises bounded UID-window sync and SQLite projection, and excludes TLS/provider network time. Gate output: ```text $ cargo test -p calternal-plugin-mail Finished `test` profile [unoptimized + debuginfo] target(s) in 55m 43s Running unittests src/lib.rs (/mnt/hdd/targets/jobs/perf-367/debug/deps/calternal_plugin_mail-1dfaa5d76a02de9e) running 18 tests test sync::tests::profile_full_history ... ignored, large Mail history performance profile for #367 test result: ok. 17 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.11s $ cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 75m 03s ```
kayg commented 2026-09-29 19:44:27 +00:00
Author
Owner
Copy link

Round 2 report (2026-09-29)

Branch: job/perf-367. Head: 4681d182fc801591198ae29720ed9c1c2c254ce4.

Built and committed the bounded-lock / single-load-check runner behavior; a 10k Files sync-feed profile; a 5k Photo thumbnail-storm profile; reusable Search Photo fixtures; and an ignored Mail full-history profile using bounded UID windows and SQLite. Added scope and run documentation in docs/perf/README.md. No baseline values were promoted.

Files changed across the job: bench/run.sh, bench/sync_feed.py, apps/web/e2e/photos-perf.mjs, tests/perf/search_scale.py, crates/plugins/mail/src/sync.rs, and docs/perf/README.md.

Gate output:

cargo fmt --all --check

Passed with no output.

cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 75m 03s

cargo test -p calternal-plugin-mail
Finished `test` profile [unoptimized + debuginfo] target(s) in 55m 43s
running 18 tests
17 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.11s

The ignored test is sync::tests::profile_full_history. bun run build, node --check apps/web/e2e/photos-perf.mjs, and bash -n bench/run.sh passed. Files clippy was interrupted with exit 130 while compiling dependencies; Files tests were not run. The production release server build was also interrupted with exit 130. The current production web build completed, but no server/web deployment or VM profile ran.

Known gaps: the Search keyword/tag/type/folder p50/p95/p99 profiles, hybrid and semantic query, palette frame profile, 30-minute browser soak, idle 1- and 10-user RSS/CPU, 1M-file Home, large import, full Search rebuild, embedding backfill, 10k client sync, 5k Photo storm, 50 SSE clients, and large Mail history profile were not measured on the quiet VM. VM load checks prevented earlier runs, and this round reached the four-hour job limit before the refreshed server build completed. No quiet-VM numbers were added to #433, #434 or #429; each issue now has a status comment. No arm64 run was made because o2 was unavailable and its last known load was above 1. docs/perf/ baseline is unchanged. Full workspace Rust gates and web check/test remain unrun.

Decisions not specified by the design docs: the Files feed profile measures the real per-user change feed through its HTTP pages rather than exercising the full daemon; the Mail history profile uses a scripted in-memory IMAP peer because production blocks private provider endpoints; the 5k Photo storm uses a set of local fixture JPEGs so the VM does not need FFmpeg.

Round 2 report (2026-09-29) Branch: `job/perf-367`. Head: `4681d182fc801591198ae29720ed9c1c2c254ce4`. Built and committed the bounded-lock / single-load-check runner behavior; a 10k Files sync-feed profile; a 5k Photo thumbnail-storm profile; reusable Search Photo fixtures; and an ignored Mail full-history profile using bounded UID windows and SQLite. Added scope and run documentation in `docs/perf/README.md`. No baseline values were promoted. Files changed across the job: `bench/run.sh`, `bench/sync_feed.py`, `apps/web/e2e/photos-perf.mjs`, `tests/perf/search_scale.py`, `crates/plugins/mail/src/sync.rs`, and `docs/perf/README.md`. Gate output: ```text cargo fmt --all --check ``` Passed with no output. ```text cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 75m 03s ``` ```text cargo test -p calternal-plugin-mail Finished `test` profile [unoptimized + debuginfo] target(s) in 55m 43s running 18 tests 17 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.11s ``` The ignored test is `sync::tests::profile_full_history`. `bun run build`, `node --check apps/web/e2e/photos-perf.mjs`, and `bash -n bench/run.sh` passed. Files clippy was interrupted with exit 130 while compiling dependencies; Files tests were not run. The production release server build was also interrupted with exit 130. The current production web build completed, but no server/web deployment or VM profile ran. Known gaps: the Search keyword/tag/type/folder p50/p95/p99 profiles, hybrid and semantic query, palette frame profile, 30-minute browser soak, idle 1- and 10-user RSS/CPU, 1M-file Home, large import, full Search rebuild, embedding backfill, 10k client sync, 5k Photo storm, 50 SSE clients, and large Mail history profile were not measured on the quiet VM. VM load checks prevented earlier runs, and this round reached the four-hour job limit before the refreshed server build completed. No quiet-VM numbers were added to #433, #434 or #429; each issue now has a status comment. No arm64 run was made because o2 was unavailable and its last known load was above 1. `docs/perf/` baseline is unchanged. Full workspace Rust gates and web check/test remain unrun. Decisions not specified by the design docs: the Files feed profile measures the real per-user change feed through its HTTP pages rather than exercising the full daemon; the Mail history profile uses a scripted in-memory IMAP peer because production blocks private provider endpoints; the 5k Photo storm uses a set of local fixture JPEGs so the VM does not need FFmpeg.
kayg commented 2026-09-30 08:46:05 +00:00
Author
Owner
Copy link

#367 round 3: quiet-VM baseline recorded and promoted

Head: 85e9df84acd8b75ba9fd7726304b7eccea195211 on job/perf-367 (merged dev at 369ab6a2f). No push, merge or deploy.

Setup. The release server (/mnt/hdd/targets/release-shared, built from 369ab6a2f, SHA-256 bb3dde66…) and the production SPA were built on the build host and copied to the perf-test VM (x86-64, 4 vCPU, 7.7 GiB). Nothing was compiled on the VM. bench/run.sh --full --search --measure-only --runs 5 ran every profile under /root/perf.lock, one lock hold per profile. Each profile started only at a one-minute load below 1. The start loads were 0.00 to 0.99; the samples are in raw/*.uptime.txt. The lock was released between profiles, and caldav-stress-457 took its turns.

Decision rule (fixed before the full run): a result counts only if it passed the lock and load gate and reported success. Average metrics come from 5 interleaved route runs or one 600 s idle sample; worst-case profiles run once (single samples). A finding is filed when it misses a DESIGN §18 budget by > 10%, when a peak exceeds 1 GiB or grows without a bound, when throughput is ≥ 3× below a reference, or when compare.py reports a regression. Findings are ranked by user impact. Optimisations follow the #367 keep rule (≥ 10%, median of 5 A/B, no other metric > 3% worse).

Baseline (docs/perf/baseline.json = runs/2026-09-30T042833Z-369ab6a2.json)

Metric Value
Route ready p50 / p95, 390 px 821–1,037 / 923–1,268 ms
Route LCP p50 / p95, 1440 px (4× throttle) 1,048–1,380 / 1,128–1,636 ms (/today worst)
JS gzip per route 326,043–400,869 B (budget 200 kB); root-layout closure 321,748 B
20 core API reads p50 1.1–2.7 ms, p95 2.1–4.8 ms
Interactions visible p50 / p95; INP palette 191 / 405 ms, INP 328; mode switch 316 / 594 ms, INP 216; composer 185 / 349 ms, INP 112
Idle server, 1 / 10 Users (600 s) 144.0 / 154.7 MiB mean RSS; 0.128 / 0.133% CPU
Request storm, 24 clients, 12 s 71,751 × 200; p50 2.1, p95 11.9 ms; peak 177 MiB, 244% CPU
50 SSE clients, 60 s setup p95 31.4 ms; peak 129.9 MiB, 51.9% CPU
Keyword Search 100k (plain) / 1M (plain) 6.22 / 18.08 / 23.50 ms · 16.73 / 27.44 / 32.13 ms (p50/p95/p99)
Hybrid query 100k 203.1 / 213.2 / 214.7 ms, 0 hits (#433)
Palette keystroke → first frame, 100k 100.5 / 218.4 ms (#434)
Search: 100k first index / full rebuild / 1M first index 22.7 s, 479 MiB / 39.8 s, 1,133 MiB / 283.6 s, 1,895 MiB (#496)
Semantic backfill 70k paths peak 874 MiB; model cold start 7.4 s
10k Notes import 1,069.7 s, p50 104 ms; 1,547.6 server CPU-s (#494)
50k files import (one folder) 5,529.6 s, 8.26/s, p50 111 ms (#429)
10k PDF import, 64 KiB 4,128.2 s, 2.42/s, p50 294 ms, 33 busy retries (#429)
10k change feed 10 pages, p95 7.7 ms
Client sync, 10k files, 200 remote changes p50 248, p95 408 ms edit-to-visible
5k Photo thumbnail storm drained in 116.3 s, 0 dead, peak 473 MiB
Mail history 100k (in-memory IMAP) 33.4 s, 2,997 msg/s, 13 worker runs, 16 MiB
Browser soak 30 min / 100 overlay cycles 1,915 navigations, 4,068 tasks > 50 ms (max 2,175 ms); heap 7.0 MB after GC, flat across cycles
Calendar week, desktop pinch 10.1 fps, fling 10.3 fps (#498)
Large Home 1M files / 100k Notes / 20k Photos Files ~1,330 s, Notes ~1,520 s, Photos never; peak 6.65 GB RSS (#503, #495)

compare.py against the old 09-27 baseline: bundle +19.8 to +25.0% on every route, idle RSS +24.8% (148.4 → 185.2 MB).

Issues filed

  • #494 — Notes: every .md change batch re-indexes every Note (277,476 store::index calls for 3,000 Notes; 155 ms CPU per Note at 10k).
  • #503 — Large Home at 6.65 GB RSS; heaptrack attributes the peak to Search whole-Home maps (0.6 GB at 30% scale), two ONNX models (0.45 GB) and SQLite (0.5 GB).
  • #496 — Search indexing memory above its caps (1.9 GB at 1M, 1.1 GB rebuild at 100k).
  • #495 — Photos full rebuild never finishes in a 1M-file Home (whole-Home row scan before any write). #470 — Photos misses offline-written media at startup.
  • #497 — Initial JS 322 kB gzip before route code, +20–25%.
  • #498 — Desktop Calendar zoom/fling at 10 fps.
  • #499 — Search palette shifts 8 px (UI; blocks the search-ui profile).
  • Updated #429 (durable write latency, with the upload numbers above), #433 (keyword now meets the budget; hybrid waits 200 ms and returns 0 hits), #434 (palette frame numbers).

Harness changes (this round)

bench/run.sh: one failed profile no longer aborts the run; PERF_ONLY / PERF_RAW_DIR rerun single profiles; bounded cooldown while holding the lock; client-sync (calternald) and Mail-history profiles. The SSE, PDF and idle probes create the data directory (they exited with NotFound). calendar-perf finds the Daily note anywhere. photos-perf forces the Photos rebuild (#470 workaround). search_scale waits for the queued rebuild job (202). Upload probes retry 503 busy. The Mail profile rediscovers folders each worker run (it failed above 8,000 messages).

Gates

$ cargo fmt --all --check
(no output)
$ cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15m 07s
$ cargo test -p calternal-plugin-mail
    Finished `test` profile [unoptimized + debuginfo] target(s) in 17m 04s
test result: ok. 17 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.10s
$ python3 -m unittest discover -s bench -p 'test_*.py'
Ran 12 tests in 0.704s
OK
$ python3 -m unittest discover -s tests/perf -p 'test_*.py'
Ran 3 tests in 0.136s
OK

node --check on the changed e2e probes, bash -n bench/run.sh and git diff --check all passed. No server or web source changed, so the full workspace and web gates were not run.

Not done

  • There were no optimisations this round, because every finding is a design-level change.
  • The large-Home render numbers are missing because Photos never indexed (#495).
  • The browser soak uses full page loads, so it does not measure SPA leaks yet.
  • The arm64 set on o2 was not run.
# #367 round 3: quiet-VM baseline recorded and promoted **Head:** `85e9df84acd8b75ba9fd7726304b7eccea195211` on `job/perf-367` (merged `dev` at `369ab6a2f`). No push, merge or deploy. **Setup.** The release server (`/mnt/hdd/targets/release-shared`, built from `369ab6a2f`, SHA-256 `bb3dde66…`) and the production SPA were built on the build host and copied to the perf-test VM (x86-64, 4 vCPU, 7.7 GiB). Nothing was compiled on the VM. `bench/run.sh --full --search --measure-only --runs 5` ran every profile under `/root/perf.lock`, one lock hold per profile. Each profile started only at a one-minute load below 1. The start loads were 0.00 to 0.99; the samples are in `raw/*.uptime.txt`. The lock was released between profiles, and `caldav-stress-457` took its turns. **Decision rule (fixed before the full run):** a result counts only if it passed the lock and load gate and reported success. Average metrics come from 5 interleaved route runs or one 600 s idle sample; worst-case profiles run once (single samples). A finding is filed when it misses a DESIGN §18 budget by > 10%, when a peak exceeds 1 GiB or grows without a bound, when throughput is ≥ 3× below a reference, or when `compare.py` reports a regression. Findings are ranked by user impact. Optimisations follow the #367 keep rule (≥ 10%, median of 5 A/B, no other metric > 3% worse). ## Baseline (`docs/perf/baseline.json` = `runs/2026-09-30T042833Z-369ab6a2.json`) | Metric | Value | |---|---| | Route ready p50 / p95, 390 px | 821–1,037 / 923–1,268 ms | | Route LCP p50 / p95, 1440 px (4× throttle) | 1,048–1,380 / 1,128–1,636 ms (`/today` worst) | | JS gzip per route | 326,043–400,869 B (budget 200 kB); root-layout closure 321,748 B | | 20 core API reads | p50 1.1–2.7 ms, p95 2.1–4.8 ms | | Interactions visible p50 / p95; INP | palette 191 / 405 ms, INP 328; mode switch 316 / 594 ms, INP 216; composer 185 / 349 ms, INP 112 | | Idle server, 1 / 10 Users (600 s) | 144.0 / 154.7 MiB mean RSS; 0.128 / 0.133% CPU | | Request storm, 24 clients, 12 s | 71,751 × 200; p50 2.1, p95 11.9 ms; peak 177 MiB, 244% CPU | | 50 SSE clients, 60 s | setup p95 31.4 ms; peak 129.9 MiB, 51.9% CPU | | Keyword Search 100k (plain) / 1M (plain) | 6.22 / 18.08 / 23.50 ms · 16.73 / 27.44 / 32.13 ms (p50/p95/p99) | | Hybrid query 100k | 203.1 / 213.2 / 214.7 ms, 0 hits (#433) | | Palette keystroke → first frame, 100k | 100.5 / 218.4 ms (#434) | | Search: 100k first index / full rebuild / 1M first index | 22.7 s, 479 MiB / 39.8 s, 1,133 MiB / 283.6 s, 1,895 MiB (#496) | | Semantic backfill 70k paths | peak 874 MiB; model cold start 7.4 s | | 10k Notes import | 1,069.7 s, p50 104 ms; **1,547.6 server CPU-s** (#494) | | 50k files import (one folder) | 5,529.6 s, 8.26/s, p50 111 ms (#429) | | 10k PDF import, 64 KiB | 4,128.2 s, 2.42/s, p50 294 ms, 33 busy retries (#429) | | 10k change feed | 10 pages, p95 7.7 ms | | Client sync, 10k files, 200 remote changes | p50 248, p95 408 ms edit-to-visible | | 5k Photo thumbnail storm | drained in 116.3 s, 0 dead, peak 473 MiB | | Mail history 100k (in-memory IMAP) | 33.4 s, 2,997 msg/s, 13 worker runs, 16 MiB | | Browser soak 30 min / 100 overlay cycles | 1,915 navigations, 4,068 tasks > 50 ms (max 2,175 ms); heap 7.0 MB after GC, flat across cycles | | Calendar week, desktop | pinch 10.1 fps, fling 10.3 fps (#498) | | Large Home 1M files / 100k Notes / 20k Photos | Files ~1,330 s, Notes ~1,520 s, Photos never; **peak 6.65 GB RSS** (#503, #495) | `compare.py` against the old 09-27 baseline: bundle +19.8 to +25.0% on every route, idle RSS +24.8% (148.4 → 185.2 MB). ## Issues filed - #494 — Notes: every `.md` change batch re-indexes every Note (277,476 `store::index` calls for 3,000 Notes; 155 ms CPU per Note at 10k). - #503 — Large Home at 6.65 GB RSS; heaptrack attributes the peak to Search whole-Home maps (0.6 GB at 30% scale), two ONNX models (0.45 GB) and SQLite (0.5 GB). - #496 — Search indexing memory above its caps (1.9 GB at 1M, 1.1 GB rebuild at 100k). - #495 — Photos full rebuild never finishes in a 1M-file Home (whole-Home row scan before any write). #470 — Photos misses offline-written media at startup. - #497 — Initial JS 322 kB gzip before route code, +20–25%. - #498 — Desktop Calendar zoom/fling at 10 fps. - #499 — Search palette shifts 8 px (UI; blocks the `search-ui` profile). - Updated #429 (durable write latency, with the upload numbers above), #433 (keyword now meets the budget; hybrid waits 200 ms and returns 0 hits), #434 (palette frame numbers). ## Harness changes (this round) `bench/run.sh`: one failed profile no longer aborts the run; `PERF_ONLY` / `PERF_RAW_DIR` rerun single profiles; bounded cooldown while holding the lock; client-sync (`calternald`) and Mail-history profiles. The SSE, PDF and idle probes create the data directory (they exited with `NotFound`). `calendar-perf` finds the Daily note anywhere. `photos-perf` forces the Photos rebuild (#470 workaround). `search_scale` waits for the queued rebuild job (202). Upload probes retry 503 busy. The Mail profile rediscovers folders each worker run (it failed above 8,000 messages). ## Gates ```text $ cargo fmt --all --check (no output) $ cargo clippy -p calternal-plugin-mail --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 15m 07s $ cargo test -p calternal-plugin-mail Finished `test` profile [unoptimized + debuginfo] target(s) in 17m 04s test result: ok. 17 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.10s $ python3 -m unittest discover -s bench -p 'test_*.py' Ran 12 tests in 0.704s OK $ python3 -m unittest discover -s tests/perf -p 'test_*.py' Ran 3 tests in 0.136s OK ``` `node --check` on the changed e2e probes, `bash -n bench/run.sh` and `git diff --check` all passed. No server or web source changed, so the full workspace and web gates were not run. ## Not done - There were no optimisations this round, because every finding is a design-level change. - The large-Home render numbers are missing because Photos never indexed (#495). - The browser soak uses full page loads, so it does not measure SPA leaks yet. - The arm64 set on o2 was not run.
kayg commented 2026-09-30 13:09:41 +00:00
Author
Owner
Copy link

All five branches are committed on job/merge-round-2. Validation remains incomplete at the four-hour limit.
Head: 84d4258f130ebebe748a9b3a08f03c8741d01c45. Base: 6c87f5ff9442cd658572139bc536d018fd5222a4.
No push, deploy, or change to dev. No branch was dropped: no included branch produced a Rust code/test failure; unfinished validation is listed below.

Included: job/location (#391), job/webcal-431 (#431), job/small-bugs-4 (#459/#463/#464), job/photos-470 (#470), job/perf-367 (#367). All five heads are ancestors. The required final fetch and origin/dev merge returned Already up to date.

Integration fixes and files:

  • Settings canonical links preserve Maintenance query state and stable Saved place UUID fragments (apps/web/src/routes/settings/[...path]/+page.svelte, registry tests).
  • Authenticated Location routing coexists with public Calendar feeds (crates/calternal-server/src/wire.rs).
  • Mail fixtures retain MIME reader regressions and the full-history profile (crates/plugins/mail/src/sync.rs). Calendar benchmarks use the canonical Daily note first, then search only that User's Home (apps/web/e2e/calendar-perf.mjs).
  • Location uses shared leading and shape tokens (apps/web/src/routes/settings/account/LocationGroup.svelte).
  • Location operations and Saved place identities have explicit fail-closed classifications with regression tests (tests/adversarial/authz_matrix.py, xuser_matrix.py, test_xuser_classification.py).
  • Updated the shared capture harness and its regression tests (apps/web/e2e/harness.mjs, harness.test.mjs, appearance-review.mjs). Earlier failing capture output is preserved; no existing numeric assertion was weakened.
  • Regenerated contracts/openapi.json, packages/api-client/src/generated.ts, and docs/parity-*. No generated files were hand-merged.
  • Reviewed module/function comments were updated with the merge fixes. Imported feature files remain in the five branch histories. The complete changed-file list is artifacts/merge-round-2/files.txt.

Migrations: no duplicate numeric prefixes in any migrations folder. Notes adds 0020 after origin/dev's 0019; Calendar adds 0004 after origin/dev's 0003. Notes bridge is excluded.

Decisions:

  • Keep the existing API adapters' scope. Record the new Location/Calendar adapter gaps in the generated parity inventory rather than invent new CLI/MCP/WebMCP behavior in this merge.
  • Keep both Mail fixture behaviors in one helper. Keep the Calendar fallback scoped to the current User's Home.
  • Preserve dev's existing test expectations. The two web failures were timeouts and were rerun alone. Restore the exact 16-family theme expectation from dev; count dark variant submenu parents as families and use the existing keyboard submenu interaction.
  • In the capture harness, check the saved Auto/System preference separately from its rendered Light/Dark CSS phase. Eight regressions verify both phases and reject wrong persistence or rendering.
  • Seed the review's initial preference through the real API before SPA navigation. Wait for hydration before choosing a mode, and await persistence with bounded API reads on the host. Playwright 1.63 treats the former async predicate as truthy before its Promise resolves.

Known gaps:

  • calternal-server tests were stopped during compilation at the approximately four-hour limit (exit -15). Server clippy passed. No server test assertion result was produced. Run cargo test -p calternal-server before merge.
  • Standalone vendored async-imap clippy/test were initially cancelled while waiting for the build lock and were not completed before the limit. Mail clippy and tests checked/exercised its Tokio dependency path.
  • The ignored large Mail history and worst-case performance profiles were not rerun. The requested local one-item Photos smoke completed; the imported baseline remains unchanged.
  • The live hostile-input and race adversarial round was not run under this session's safety limits. Offline classification and probe-helper tests do not replace it. This branch is not certified ready to merge until that round is completed.
  • Static parity records 188 actions with adapter gaps. New Location operations lack CLI/MCP/WebMCP adapters; Calendar feeds/subscriptions lack MCP adapters. These imported scope gaps remain documented.
  • The official generated-check wrapper returned 143 after its build/generation output. The equivalent OpenAPI generation, client generation, unique-operation check, and clean generated diff were rerun explicitly and all returned 0.
  • The local one-photo debug smoke is not comparable with the 5,000-photo baseline; no regression verdict was made. Its JSON commit field is unknown.

Browser evidence: Calendar feeds passed with 36 screenshots. Location passed with six screenshots, including stable Saved place link restoration and cap-height assertions. The full Appearance review passed with 71 screenshots. Notes, Photos and the Log with its Saved place passed with 18 more screenshots. All required surface matrices cover 390/820/1440 px and light/dark. The additional Unsplash previews use third-party test fixtures; ordinary Notes/Photos/Location/Calendar data comes from the real local API. The production app supplied the screenshots; visual quality review remains with the orchestrator.
Appearance, Notes, Photos and Log matrix. Location full matrix. Calendar full matrix.

Local performance smoke (photos-perf.mjs --items 1 --home-only): server start 1101 ms; all indexed 4060 ms; rebuild requested 2582 ms; mean/peak RSS 350938740/432443392 bytes; mean/peak CPU 25.24/132.98%; CPU 23.74 s. Load after: 33.2/38.3/40.6. Baseline docs/perf/baseline.json, commit 369ab6a2f9fc673e3564b94857fbecfeb04df404, 5000 photos: server start 290 ms; indexed 63230 ms; rebuild 719 ms; mean/peak RSS 360533602/495759360 bytes; mean/peak CPU 63.05/235.82%; CPU 113.24 s. Different dataset and environment: these numbers show the smoke completed, not a regression comparison.

Gate outputs follow. Full logs are under artifacts/merge-round-2/ in the worktree; screenshots and logs are not committed.

Required environment: CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 TMPDIR=$PWD/target/tmp; the preset CARGO_TARGET_DIR was retained. Rust gates ran per crate.

cargo fmt --check: exit 0, no output. git diff --check: exit 0, no output.

Per-crate exit statuses, verbatim:

calternal-cli	clippy	0
calternal-cli	test	0
calternal-dav	clippy	0
calternal-dav	test	0
calternal-location	clippy	0
calternal-location	test	0
calternal-notes-core	clippy	0
calternal-notes-core	test	0
calternal-plugin	clippy	0
calternal-plugin	test	0
calternal-plugin-calendar	clippy	0
calternal-plugin-calendar	test	0
calternal-plugin-files	clippy	0
calternal-plugin-files	test	0
calternal-plugin-mail	clippy	0
calternal-plugin-mail	test	0
calternal-plugin-notes	clippy	0
calternal-plugin-notes	test	0
calternal-plugin-photos	clippy	0
calternal-plugin-photos	test	0
calternal-server	clippy	0
calternal-server	test	-15

calternal-cli clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 16m 25s

calternal-cli test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 57m 13s
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.09s

calternal-dav clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 25s

calternal-dav test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 06s
test result: ok. 41 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.46s
test result: ok. 33 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-location clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 05s

calternal-location test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 57.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.47s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-notes-core clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 29s

calternal-notes-core test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 31s
test result: ok. 506 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.74s
test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.47s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.53s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 9m 55s

calternal-plugin test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 21s
test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.05s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-calendar clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 29m 42s

calternal-plugin-calendar test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 46s
test result: ok. 79 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.33s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-files clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 08s

calternal-plugin-files test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 35s
test result: ok. 129 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 220.28s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-mail clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 36s

calternal-plugin-mail test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
test result: ok. 35 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.17s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-notes clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 36s

calternal-plugin-notes test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 40s
test result: ok. 130 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 223.97s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.83s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-plugin-photos clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 48s

calternal-plugin-photos test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 06s
test result: ok. 44 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 25.16s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server clippy:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 02s

Server test run: cancelled during compilation; no test result. Last compiler output, verbatim:

   Compiling calternal-plugin-money v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/money)
   Compiling rmcp v3.5.0
   Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/files)
   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/calternal-collab)
   Compiling calternal-plugin-calendar v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/calendar)
   Compiling calternal-plugin-ai v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/ai)
   Compiling calternal-plugin-photos v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/photos)
   Compiling calternal-plugin-video v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/video)
   Compiling calternal-plugin-notifications v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/notifications)
   Compiling calternal-plugin-analytics v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/analytics)

Web, contract, classification, parity, offline helper and browser output excerpts follow verbatim. Full output, including earlier failures and isolated reruns, is in the attached log archive.

bun-install.log

Checked 631 installs across 749 packages (no changes) [25.76s]

web-check-final.log

Text sizes and UI shape values use shared role tokens.
svelte-check found 0 errors and 0 warnings

web-test.log

 ❯ |component| src/lib/components/KeyboardShortcutsCard.svelte.test.ts (1 test | 1 failed) 5696ms
 ❯ |unit| src/lib/calendar/zones.test.ts (21 tests | 1 failed) 6881ms
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯
Error: Test timed out in 5000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
Error: Test timed out in 5000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
 Test Files  2 failed | 137 passed (139)
      Tests  2 failed | 897 passed (899)
   Duration  416.06s (transform 66%, import 12%, environment 12%, tests 7%, setup 3%)

web-zones-alone.log

 Test Files  1 passed (1)
      Tests  21 passed (21)
   Duration  88.80s (transform 94%, import 6%)

web-shortcuts-alone.log

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Duration  126.01s (transform 79%, environment 11%, import 5%, setup 3%, tests 2%)

settings-alone.log

 Test Files  1 passed (1)
      Tests  8 passed (8)
   Duration  6.85s (transform 85%, import 10%, tests 4%)

api-client-tests.log

(pass) apiFetch > keeps a typed failure body that is not an error envelope [0.23ms]
 9 pass
 0 fail
 31 expect() calls

bench-tests.log

............
----------------------------------------------------------------------
Ran 12 tests in 1.213s

OK

dav-probe-contract-tests.log

.......
----------------------------------------------------------------------
Ran 7 tests in 0.029s

OK

appearance-probe-contract-tests.log

....
----------------------------------------------------------------------
Ran 4 tests in 0.008s

OK

classification-final-tests.log

....
----------------------------------------------------------------------
Ran 4 tests in 0.284s

OK

classify-final.log

Cross-User classification gate: 328 operations classified

parity-final.log

Parity matrix: 206 web API actions, 113 shortcuts, 2 static commands, 131 menu actions, 33 settings groups, 188 actions with adapter gaps

generated-check-confirmed.log

$ /mnt/hdd/targets/jobs/merge-round-2/debug/calternal-server openapi
exit=0
$ bun run --cwd packages/api-client generate
$ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts
✨ openapi-typescript 7.13.0
🚀 ../../contracts/openapi.json → src/generated.ts [3s]
exit=0
$ git diff --exit-code -- contracts/openapi.json packages/api-client/src/generated.ts
exit=0
OpenAPI operation IDs: 328 unique

calendar-feeds.log

Calendar feeds proof passed; screenshots: /home/kayg/Developer/calternal-wt/merge-web/artifacts/merge-round-2/calendar-feeds

location-review.log

PASS appearance review captures: 6 screenshots in /home/kayg/Developer/calternal-wt/merge-web/artifacts/location

photos-home-local.log

  indexed: {
{"items":1,"commit":"unknown","recordedAt":"2026-09-30T10:14:15.246Z","homeOnly":true,"environment":{"host":"calternal-dev","platform":"linux","architecture":"x64"},"homeFixtures":{"photos":1,"photoDays":1,"files":0,"folderItems":0,"notes":0,"logEntries":0,"dailyNotes":0,"largeNoteBytes":0},"homeFixtureWriteMs":10277,"indexing":{"serverUpMs":1101,"allIndexedMs":4060,"photoRebuildRequestedMs":2582,"indexed":{"photos":1,"folderItems":0,"restItems":0,"notes":0,"logEntries":0}},"homeRenders":{"cpuThrottle":4,"runs":3,"order":[["files5k","analyticsYear","note1MiB"],["analyticsYear","note1MiB","files5k"],["note1MiB","files5k","analyticsYear"]],"files5k":{"skipped":true},"analyticsYear":{"skipped":true},"note1MiB":{"skipped":true}},"loadAverageAfter":[33.2,38.3,40.6],"homeResources":{"meanRssBytes":350938740,"peakRssBytes":432443392,"meanCpuPercent":25.24,"peakCpuPercent":132.98,"cpuSeconds":23.74,"samples":186}}

harness-test-final.log:


 8 pass
 0 fail
Ran 8 tests across 1 file. [474.00ms]

appearance-review-awaited.log:

PASS appearance review captures: 71 screenshots in /home/kayg/Developer/calternal-wt/merge-web/artifacts/location

reconcile-review-normal-note.log:

PASS Notes and Photos review: 18 production screenshots, 390/820/1440 px, light/dark

cargo-clean.log:

     Removed 23900 files, 14.0GiB total

migrations.log:

crates/calternal-auth/migrations: 10 numbered migrations; no duplicates
crates/calternal-db/src/migrations: 6 numbered migrations; no duplicates
crates/calternal-plugin/migrations: 1 numbered migrations; no duplicates
crates/calternal-search/migrations: 3 numbered migrations; no duplicates
crates/calternal-tags/migrations: 2 numbered migrations; no duplicates
crates/plugins/ai/migrations: 4 numbered migrations; no duplicates
crates/plugins/analytics/migrations: 2 numbered migrations; no duplicates
crates/plugins/calendar/migrations: 4 numbered migrations; no duplicates
crates/plugins/files/migrations: 15 numbered migrations; no duplicates
crates/plugins/mail/migrations: 8 numbered migrations; no duplicates
crates/plugins/notes/migrations: 20 numbered migrations; no duplicates
crates/plugins/notifications/migrations: 4 numbered migrations; no duplicates
crates/plugins/photos/migrations: 6 numbered migrations; no duplicates
crates/plugins/video/migrations: 1 numbered migrations; no duplicates
PASS: 14 migration folders have unique numeric prefixes

bun run build: exit 0. Output, verbatim:

✓ built in 2m 29s
✓ built in 527ms
✓ built in 4m 52s
> Using @sveltejs/adapter-static

Official generated-check wrapper exit: 143. Confirmed equivalent generation/uniqueness/diff steps: all exit 0, quoted above.
Standalone vendor attempt statuses, verbatim:

async-imap	clippy	-15
async-imap	test	-15

Cleanup: cargo clean exit 0; web build and Svelte build output deleted. Screenshots and logs remain uncommitted. Final git status is clean.

Full gate output, earlier failures, isolated reruns, performance smoke and changed-file manifest.

All five branches are committed on `job/merge-round-2`. Validation remains incomplete at the four-hour limit. Head: `84d4258f130ebebe748a9b3a08f03c8741d01c45`. Base: `6c87f5ff9442cd658572139bc536d018fd5222a4`. No push, deploy, or change to dev. No branch was dropped: no included branch produced a Rust code/test failure; unfinished validation is listed below. Included: `job/location` (#391), `job/webcal-431` (#431), `job/small-bugs-4` (#459/#463/#464), `job/photos-470` (#470), `job/perf-367` (#367). All five heads are ancestors. The required final fetch and origin/dev merge returned `Already up to date.` Integration fixes and files: - Settings canonical links preserve Maintenance query state and stable Saved place UUID fragments (`apps/web/src/routes/settings/[...path]/+page.svelte`, registry tests). - Authenticated Location routing coexists with public Calendar feeds (`crates/calternal-server/src/wire.rs`). - Mail fixtures retain MIME reader regressions and the full-history profile (`crates/plugins/mail/src/sync.rs`). Calendar benchmarks use the canonical Daily note first, then search only that User's Home (`apps/web/e2e/calendar-perf.mjs`). - Location uses shared leading and shape tokens (`apps/web/src/routes/settings/account/LocationGroup.svelte`). - Location operations and Saved place identities have explicit fail-closed classifications with regression tests (`tests/adversarial/authz_matrix.py`, `xuser_matrix.py`, `test_xuser_classification.py`). - Updated the shared capture harness and its regression tests (`apps/web/e2e/harness.mjs`, `harness.test.mjs`, `appearance-review.mjs`). Earlier failing capture output is preserved; no existing numeric assertion was weakened. - Regenerated `contracts/openapi.json`, `packages/api-client/src/generated.ts`, and `docs/parity-*`. No generated files were hand-merged. - Reviewed module/function comments were updated with the merge fixes. Imported feature files remain in the five branch histories. The complete changed-file list is `artifacts/merge-round-2/files.txt`. Migrations: no duplicate numeric prefixes in any migrations folder. Notes adds 0020 after origin/dev's 0019; Calendar adds 0004 after origin/dev's 0003. Notes bridge is excluded. Decisions: - Keep the existing API adapters' scope. Record the new Location/Calendar adapter gaps in the generated parity inventory rather than invent new CLI/MCP/WebMCP behavior in this merge. - Keep both Mail fixture behaviors in one helper. Keep the Calendar fallback scoped to the current User's Home. - Preserve dev's existing test expectations. The two web failures were timeouts and were rerun alone. Restore the exact 16-family theme expectation from dev; count dark variant submenu parents as families and use the existing keyboard submenu interaction. - In the capture harness, check the saved Auto/System preference separately from its rendered Light/Dark CSS phase. Eight regressions verify both phases and reject wrong persistence or rendering. - Seed the review's initial preference through the real API before SPA navigation. Wait for hydration before choosing a mode, and await persistence with bounded API reads on the host. Playwright 1.63 treats the former async predicate as truthy before its Promise resolves. Known gaps: - `calternal-server` tests were stopped during compilation at the approximately four-hour limit (exit -15). Server clippy passed. No server test assertion result was produced. Run `cargo test -p calternal-server` before merge. - Standalone vendored `async-imap` clippy/test were initially cancelled while waiting for the build lock and were not completed before the limit. Mail clippy and tests checked/exercised its Tokio dependency path. - The ignored large Mail history and worst-case performance profiles were not rerun. The requested local one-item Photos smoke completed; the imported baseline remains unchanged. - The live hostile-input and race adversarial round was not run under this session's safety limits. Offline classification and probe-helper tests do not replace it. This branch is not certified ready to merge until that round is completed. - Static parity records 188 actions with adapter gaps. New Location operations lack CLI/MCP/WebMCP adapters; Calendar feeds/subscriptions lack MCP adapters. These imported scope gaps remain documented. - The official generated-check wrapper returned 143 after its build/generation output. The equivalent OpenAPI generation, client generation, unique-operation check, and clean generated diff were rerun explicitly and all returned 0. - The local one-photo debug smoke is not comparable with the 5,000-photo baseline; no regression verdict was made. Its JSON commit field is `unknown`. Browser evidence: Calendar feeds passed with 36 screenshots. Location passed with six screenshots, including stable Saved place link restoration and cap-height assertions. The full Appearance review passed with 71 screenshots. Notes, Photos and the Log with its Saved place passed with 18 more screenshots. All required surface matrices cover 390/820/1440 px and light/dark. The additional Unsplash previews use third-party test fixtures; ordinary Notes/Photos/Location/Calendar data comes from the real local API. The production app supplied the screenshots; visual quality review remains with the orchestrator. [Appearance, Notes, Photos and Log matrix](https://git.kayg.org/attachments/2a0c2168-6199-4d45-9ff2-ad676518e88e). [Location full matrix](https://git.kayg.org/attachments/6078bc47-ceb1-4566-9b5e-63ef5c52b9a1). [Calendar full matrix](https://git.kayg.org/attachments/03f5976a-28a3-4384-9aae-a995bac45409). Local performance smoke (`photos-perf.mjs --items 1 --home-only`): server start 1101 ms; all indexed 4060 ms; rebuild requested 2582 ms; mean/peak RSS 350938740/432443392 bytes; mean/peak CPU 25.24/132.98%; CPU 23.74 s. Load after: 33.2/38.3/40.6. Baseline `docs/perf/baseline.json`, commit `369ab6a2f9fc673e3564b94857fbecfeb04df404`, 5000 photos: server start 290 ms; indexed 63230 ms; rebuild 719 ms; mean/peak RSS 360533602/495759360 bytes; mean/peak CPU 63.05/235.82%; CPU 113.24 s. Different dataset and environment: these numbers show the smoke completed, not a regression comparison. Gate outputs follow. Full logs are under `artifacts/merge-round-2/` in the worktree; screenshots and logs are not committed. Required environment: `CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 TMPDIR=$PWD/target/tmp`; the preset CARGO_TARGET_DIR was retained. Rust gates ran per crate. `cargo fmt --check`: exit 0, no output. `git diff --check`: exit 0, no output. Per-crate exit statuses, verbatim: ```text calternal-cli clippy 0 calternal-cli test 0 calternal-dav clippy 0 calternal-dav test 0 calternal-location clippy 0 calternal-location test 0 calternal-notes-core clippy 0 calternal-notes-core test 0 calternal-plugin clippy 0 calternal-plugin test 0 calternal-plugin-calendar clippy 0 calternal-plugin-calendar test 0 calternal-plugin-files clippy 0 calternal-plugin-files test 0 calternal-plugin-mail clippy 0 calternal-plugin-mail test 0 calternal-plugin-notes clippy 0 calternal-plugin-notes test 0 calternal-plugin-photos clippy 0 calternal-plugin-photos test 0 calternal-server clippy 0 calternal-server test -15 ``` calternal-cli clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 16m 25s ``` calternal-cli test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 57m 13s test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.09s ``` calternal-dav clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 25s ``` calternal-dav test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 06s test result: ok. 41 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.46s test result: ok. 33 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-location clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 05s ``` calternal-location test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 57.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.47s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-notes-core clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 29s ``` calternal-notes-core test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 31s test result: ok. 506 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.74s test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.47s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.53s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-plugin clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 9m 55s ``` calternal-plugin test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 21s test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.05s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-plugin-calendar clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 29m 42s ``` calternal-plugin-calendar test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 46s test result: ok. 79 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.33s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-plugin-files clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 08s ``` calternal-plugin-files test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 35s test result: ok. 129 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 220.28s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-plugin-mail clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 36s ``` calternal-plugin-mail test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s test result: ok. 35 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.17s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-plugin-notes clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 36s ``` calternal-plugin-notes test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 40s test result: ok. 130 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 223.97s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.83s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-plugin-photos clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 48s ``` calternal-plugin-photos test: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 06s test result: ok. 44 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 25.16s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` calternal-server clippy: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 02s ``` Server test run: cancelled during compilation; no test result. Last compiler output, verbatim: ```text Compiling calternal-plugin-money v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/money) Compiling rmcp v3.5.0 Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/files) Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/calternal-collab) Compiling calternal-plugin-calendar v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/calendar) Compiling calternal-plugin-ai v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/ai) Compiling calternal-plugin-photos v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/photos) Compiling calternal-plugin-video v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/video) Compiling calternal-plugin-notifications v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/notifications) Compiling calternal-plugin-analytics v0.0.1 (/home/kayg/Developer/calternal-wt/merge-web/crates/plugins/analytics) ``` Web, contract, classification, parity, offline helper and browser output excerpts follow verbatim. Full output, including earlier failures and isolated reruns, is in the attached log archive. ### bun-install.log ```text Checked 631 installs across 749 packages (no changes) [25.76s] ``` ### web-check-final.log ```text Text sizes and UI shape values use shared role tokens. svelte-check found 0 errors and 0 warnings ``` ### web-test.log ```text ❯ |component| src/lib/components/KeyboardShortcutsCard.svelte.test.ts (1 test | 1 failed) 5696ms ❯ |unit| src/lib/calendar/zones.test.ts (21 tests | 1 failed) 6881ms ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯ Error: Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout". Error: Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout". Test Files 2 failed | 137 passed (139) Tests 2 failed | 897 passed (899) Duration 416.06s (transform 66%, import 12%, environment 12%, tests 7%, setup 3%) ``` ### web-zones-alone.log ```text Test Files 1 passed (1) Tests 21 passed (21) Duration 88.80s (transform 94%, import 6%) ``` ### web-shortcuts-alone.log ```text Test Files 1 passed (1) Tests 1 passed (1) Duration 126.01s (transform 79%, environment 11%, import 5%, setup 3%, tests 2%) ``` ### settings-alone.log ```text Test Files 1 passed (1) Tests 8 passed (8) Duration 6.85s (transform 85%, import 10%, tests 4%) ``` ### api-client-tests.log ```text (pass) apiFetch > keeps a typed failure body that is not an error envelope [0.23ms] 9 pass 0 fail 31 expect() calls ``` ### bench-tests.log ```text ............ ---------------------------------------------------------------------- Ran 12 tests in 1.213s OK ``` ### dav-probe-contract-tests.log ```text ....... ---------------------------------------------------------------------- Ran 7 tests in 0.029s OK ``` ### appearance-probe-contract-tests.log ```text .... ---------------------------------------------------------------------- Ran 4 tests in 0.008s OK ``` ### classification-final-tests.log ```text .... ---------------------------------------------------------------------- Ran 4 tests in 0.284s OK ``` ### classify-final.log ```text Cross-User classification gate: 328 operations classified ``` ### parity-final.log ```text Parity matrix: 206 web API actions, 113 shortcuts, 2 static commands, 131 menu actions, 33 settings groups, 188 actions with adapter gaps ``` ### generated-check-confirmed.log ```text $ /mnt/hdd/targets/jobs/merge-round-2/debug/calternal-server openapi exit=0 $ bun run --cwd packages/api-client generate $ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts ✨ openapi-typescript 7.13.0 🚀 ../../contracts/openapi.json → src/generated.ts [3s] exit=0 $ git diff --exit-code -- contracts/openapi.json packages/api-client/src/generated.ts exit=0 OpenAPI operation IDs: 328 unique ``` ### calendar-feeds.log ```text Calendar feeds proof passed; screenshots: /home/kayg/Developer/calternal-wt/merge-web/artifacts/merge-round-2/calendar-feeds ``` ### location-review.log ```text PASS appearance review captures: 6 screenshots in /home/kayg/Developer/calternal-wt/merge-web/artifacts/location ``` ### photos-home-local.log ```text indexed: { {"items":1,"commit":"unknown","recordedAt":"2026-09-30T10:14:15.246Z","homeOnly":true,"environment":{"host":"calternal-dev","platform":"linux","architecture":"x64"},"homeFixtures":{"photos":1,"photoDays":1,"files":0,"folderItems":0,"notes":0,"logEntries":0,"dailyNotes":0,"largeNoteBytes":0},"homeFixtureWriteMs":10277,"indexing":{"serverUpMs":1101,"allIndexedMs":4060,"photoRebuildRequestedMs":2582,"indexed":{"photos":1,"folderItems":0,"restItems":0,"notes":0,"logEntries":0}},"homeRenders":{"cpuThrottle":4,"runs":3,"order":[["files5k","analyticsYear","note1MiB"],["analyticsYear","note1MiB","files5k"],["note1MiB","files5k","analyticsYear"]],"files5k":{"skipped":true},"analyticsYear":{"skipped":true},"note1MiB":{"skipped":true}},"loadAverageAfter":[33.2,38.3,40.6],"homeResources":{"meanRssBytes":350938740,"peakRssBytes":432443392,"meanCpuPercent":25.24,"peakCpuPercent":132.98,"cpuSeconds":23.74,"samples":186}} ``` harness-test-final.log: ```text 8 pass 0 fail Ran 8 tests across 1 file. [474.00ms] ``` appearance-review-awaited.log: ```text PASS appearance review captures: 71 screenshots in /home/kayg/Developer/calternal-wt/merge-web/artifacts/location ``` reconcile-review-normal-note.log: ```text PASS Notes and Photos review: 18 production screenshots, 390/820/1440 px, light/dark ``` cargo-clean.log: ```text Removed 23900 files, 14.0GiB total ``` migrations.log: ```text crates/calternal-auth/migrations: 10 numbered migrations; no duplicates crates/calternal-db/src/migrations: 6 numbered migrations; no duplicates crates/calternal-plugin/migrations: 1 numbered migrations; no duplicates crates/calternal-search/migrations: 3 numbered migrations; no duplicates crates/calternal-tags/migrations: 2 numbered migrations; no duplicates crates/plugins/ai/migrations: 4 numbered migrations; no duplicates crates/plugins/analytics/migrations: 2 numbered migrations; no duplicates crates/plugins/calendar/migrations: 4 numbered migrations; no duplicates crates/plugins/files/migrations: 15 numbered migrations; no duplicates crates/plugins/mail/migrations: 8 numbered migrations; no duplicates crates/plugins/notes/migrations: 20 numbered migrations; no duplicates crates/plugins/notifications/migrations: 4 numbered migrations; no duplicates crates/plugins/photos/migrations: 6 numbered migrations; no duplicates crates/plugins/video/migrations: 1 numbered migrations; no duplicates PASS: 14 migration folders have unique numeric prefixes ``` `bun run build`: exit 0. Output, verbatim: ```text ✓ built in 2m 29s ✓ built in 527ms ✓ built in 4m 52s > Using @sveltejs/adapter-static ``` Official generated-check wrapper exit: 143. Confirmed equivalent generation/uniqueness/diff steps: all exit 0, quoted above. Standalone vendor attempt statuses, verbatim: ```text async-imap clippy -15 async-imap test -15 ``` Cleanup: cargo clean exit 0; web build and Svelte build output deleted. Screenshots and logs remain uncommitted. Final git status is clean. [Full gate output, earlier failures, isolated reruns, performance smoke and changed-file manifest](https://git.kayg.org/attachments/bd32de61-0a7d-4999-b66c-134546087c66).
kayg commented 2026-09-30 14:07:22 +00:00
Author
Owner
Copy link

Final-tree local Photos smoke passed: photos-perf.mjs --items 1 --home-only, commit 42c63668fdfb5cc43bf6a790cf002023eddd7f89.

Metric Local smoke (1 Photo, debug) Baseline (5000 Photos, perf-test)
Server start, ms 2805 290
Indexed, ms 4438 63230
Mean RSS, bytes 348473058 360533602
Peak RSS, bytes 445788160 495759360
Mean CPU, % 24.1 63.05
Peak CPU, % 116.46 235.82
CPU, seconds 22.76 113.24

Local load before/after: [21.6, 26.8, 28.3] / [25.2, 26.6, 28.1]. The one Photo indexed without an explicit rebuild (rebuild request time is null). Dataset and environment differ; these numbers prove the smoke ran and do not support a regression verdict. Settings effects is rerunning with the Nix C++ runtime paths after the native Sharp prerequisite failed. Standalone IMAP retains one failing existing error-text expectation; details on #391/#459.

Final-tree local Photos smoke passed: `photos-perf.mjs --items 1 --home-only`, commit `42c63668fdfb5cc43bf6a790cf002023eddd7f89`. | Metric | Local smoke (1 Photo, debug) | Baseline (5000 Photos, perf-test) | | --- | ---: | ---: | | Server start, ms | 2805 | 290 | | Indexed, ms | 4438 | 63230 | | Mean RSS, bytes | 348473058 | 360533602 | | Peak RSS, bytes | 445788160 | 495759360 | | Mean CPU, % | 24.1 | 63.05 | | Peak CPU, % | 116.46 | 235.82 | | CPU, seconds | 22.76 | 113.24 | Local load before/after: [21.6, 26.8, 28.3] / [25.2, 26.6, 28.1]. The one Photo indexed without an explicit rebuild (rebuild request time is null). Dataset and environment differ; these numbers prove the smoke ran and do not support a regression verdict. Settings effects is rerunning with the Nix C++ runtime paths after the native Sharp prerequisite failed. Standalone IMAP retains one failing existing error-text expectation; details on #391/#459.
kayg commented 2026-09-30 16:49:36 +00:00
Author
Owner
Copy link

Merge round 2 final report

Head: 192ea5457f1fc17735d876e70b41cb506b2a8274 on job/merge-round-2. origin/dev was fetched and merged at 1389119e33831275e6b56b9181ef42838003a4b3; job/bg-422 was already an ancestor (git merge --no-edit job/bg-422 → Already up to date.). No push, deploy, or merge to dev.

Built

Integrated the #391 Location/Appearance, #431 calendar feeds, #459 bug fixes, #470 Photos and #367 performance changes. Follow-up 76d1e71e1 makes Location probes await settled server state. Follow-up 192ea5457 fixes the Photos/Backgrounds SSE ordering race: preserve one event invalidation while the stable folder ID resolves, then refetch once. Updated the production Settings sweep to cover Account → Location and real Photos folder actions.

Files changed by the follow-up:

  • apps/web/src/lib/photos/BackgroundsFolderView.svelte
  • apps/web/e2e/settings-effects.mjs
  • tests/adversarial/appearance_auto_scheme.mjs

Full integration manifest: artifacts/merge-round-2/files.txt.

Gates (output verbatim)

cargo fmt --check: exit 0, no output.

cargo clippy -p calternal-server --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 06s

cargo test -p calternal-server
    Finished `test` profile [unoptimized + debuginfo] target(s) in 9m 02s
test result: ok. 93 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 22.86s

async-imap clippy
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.21s

async-imap test
---- client::tests::test_parsing_error stdout ----
thread 'client::tests::test_parsing_error' panicked at crates/plugins/mail/vendor/async-imap/src/client.rs:2724:9:
assertion failed: session.noop().await.unwrap_err().to_string().contains("220 mail.example.org ESMTP Postcow")
test client::tests::test_parsing_error ... FAILED
test result: FAILED. 69 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.40s

bun run check
Text sizes and UI shape values use shared role tokens.
svelte-check found 0 errors and 0 warnings

bun run build
✓ built in 1m 37s
> Using @sveltejs/adapter-static
  Wrote site to "build"
  ✔ done

bun run test (final run)
 ❯ |component| src/lib/themePicker.svelte.test.ts (7 tests | 1 failed) 11924ms
 FAIL |component| src/lib/themePicker.svelte.test.ts > theme menu > reveals the trigger in the sheet scrollport before opening
Error: Test timed out in 5000ms.
 Test Files  1 failed | 138 passed (139)
      Tests  1 failed | 909 passed (910)
   Start at  18:24:29
   Duration  271.28s (transform 61%, environment 16%, import 10%, tests 10%, setup 3%)

generated contract
Finished `dev` profile [unoptimized + debuginfo] target(s) in 13m 26s
Running `/mnt/hdd/targets/jobs/merge-round-2/debug/calternal-server openapi`
$ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts
✨ openapi-typescript 7.13.0
🚀 ../../contracts/openapi.json → src/generated.ts [1.9s]

parity check
Parity matrix: 206 web API actions, 113 shortcuts, 2 static commands, 131 menu actions, 33 settings groups, 188 actions with adapter gaps

Live probes

  • Location: Location API probe: malformed and oversized payloads and file, exact coordinates, Unicode place, concurrent writes, fixture restore, and cross-User isolation passed; Appearance Auto/Fonts/Background burst: 48 concurrent writes, all 200.
  • Webcal: Calendar feeds proof passed; screenshots: /home/kayg/Developer/calternal-wt/merge-web/artifacts/webcal-431.
  • WebDAV race: WebDAV scripted probes passed.
  • Settings and Photos/Backgrounds: PASS settings effects: Appearance, Photos/Backgrounds, Calendar time preview; screenshots in /home/kayg/Developer/calternal-wt/merge-web/artifacts/bg-422.
  • Two-User matrix: 328 operations classified; 157 operations replayed; 565 A-ID vs missing-ID comparisons across B, C, D and anonymous; median absolute timing delta 6.1 ms; ownership: 77 comparisons; 0 denial failures.
  • Media had one SLOW-only finding: the PDF worker list took 5.39 s with HTTP 200 against the probe's 5.0 s threshold. No hostile-input or 5xx finding remained.
  • Local photos-perf.mjs --items 1 --home-only: at load average 38.2/42.8/43, server start 2850 ms, indexed 4790 ms, mean/peak RSS 335424222/425738240 bytes, mean/peak CPU 23.37/95.95%, CPU 22.16 s. This one-photo local smoke is not comparable to the 5,000-photo perf VM baseline.

Known gaps and decisions

  • Standalone IMAP tests retain the existing display expectation and fail client::tests::test_parsing_error; clippy passes. No test expectation was changed.
  • The final web test gate has the single five-second ThemePicker timeout above. The component and test are unchanged from origin/dev; this is reported as a gate failure.
  • Media's 5.39-second response is SLOW-only shared-host load.
  • DESIGN §35 does not specify an SSE event arriving before stable folder identity resolves. Decision: coalesce early events into one list refresh after resolution. DESIGN §47 L1 defines Account as Location consent owner; Auto uses the exact saved location.
  • Settings effects used Node 22 because Bun could not load host Sharp (libstdc++.so.6 missing). No app code changed for the host workaround.

Production screenshots

The new Settings/Photos/Backgrounds matrix covers 390/820/1440 px in light and dark: download review ZIP. Existing full matrices: Location, Appearance/Notes/Photos/Log, Calendar feeds. Screenshots and logs are not committed.

## Merge round 2 final report **Head:** `192ea5457f1fc17735d876e70b41cb506b2a8274` on `job/merge-round-2`. `origin/dev` was fetched and merged at `1389119e33831275e6b56b9181ef42838003a4b3`; `job/bg-422` was already an ancestor (`git merge --no-edit job/bg-422` → `Already up to date.`). No push, deploy, or merge to dev. ### Built Integrated the #391 Location/Appearance, #431 calendar feeds, #459 bug fixes, #470 Photos and #367 performance changes. Follow-up `76d1e71e1` makes Location probes await settled server state. Follow-up `192ea5457` fixes the Photos/Backgrounds SSE ordering race: preserve one event invalidation while the stable folder ID resolves, then refetch once. Updated the production Settings sweep to cover Account → Location and real Photos folder actions. Files changed by the follow-up: - `apps/web/src/lib/photos/BackgroundsFolderView.svelte` - `apps/web/e2e/settings-effects.mjs` - `tests/adversarial/appearance_auto_scheme.mjs` Full integration manifest: `artifacts/merge-round-2/files.txt`. ### Gates (output verbatim) `cargo fmt --check`: exit 0, no output. ```text cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 06s cargo test -p calternal-server Finished `test` profile [unoptimized + debuginfo] target(s) in 9m 02s test result: ok. 93 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 22.86s async-imap clippy Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.21s async-imap test ---- client::tests::test_parsing_error stdout ---- thread 'client::tests::test_parsing_error' panicked at crates/plugins/mail/vendor/async-imap/src/client.rs:2724:9: assertion failed: session.noop().await.unwrap_err().to_string().contains("220 mail.example.org ESMTP Postcow") test client::tests::test_parsing_error ... FAILED test result: FAILED. 69 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.40s bun run check Text sizes and UI shape values use shared role tokens. svelte-check found 0 errors and 0 warnings bun run build ✓ built in 1m 37s > Using @sveltejs/adapter-static Wrote site to "build" ✔ done bun run test (final run) ❯ |component| src/lib/themePicker.svelte.test.ts (7 tests | 1 failed) 11924ms FAIL |component| src/lib/themePicker.svelte.test.ts > theme menu > reveals the trigger in the sheet scrollport before opening Error: Test timed out in 5000ms. Test Files 1 failed | 138 passed (139) Tests 1 failed | 909 passed (910) Start at 18:24:29 Duration 271.28s (transform 61%, environment 16%, import 10%, tests 10%, setup 3%) generated contract Finished `dev` profile [unoptimized + debuginfo] target(s) in 13m 26s Running `/mnt/hdd/targets/jobs/merge-round-2/debug/calternal-server openapi` $ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts ✨ openapi-typescript 7.13.0 🚀 ../../contracts/openapi.json → src/generated.ts [1.9s] parity check Parity matrix: 206 web API actions, 113 shortcuts, 2 static commands, 131 menu actions, 33 settings groups, 188 actions with adapter gaps ``` ### Live probes - Location: `Location API probe: malformed and oversized payloads and file, exact coordinates, Unicode place, concurrent writes, fixture restore, and cross-User isolation passed`; `Appearance Auto/Fonts/Background burst: 48 concurrent writes, all 200`. - Webcal: `Calendar feeds proof passed; screenshots: /home/kayg/Developer/calternal-wt/merge-web/artifacts/webcal-431`. - WebDAV race: `WebDAV scripted probes passed`. - Settings and Photos/Backgrounds: `PASS settings effects: Appearance, Photos/Backgrounds, Calendar time preview; screenshots in /home/kayg/Developer/calternal-wt/merge-web/artifacts/bg-422`. - Two-User matrix: `328 operations classified; 157 operations replayed; 565 A-ID vs missing-ID comparisons across B, C, D and anonymous; median absolute timing delta 6.1 ms`; ownership: `77 comparisons; 0 denial failures`. - Media had one SLOW-only finding: the PDF worker list took 5.39 s with HTTP 200 against the probe's 5.0 s threshold. No hostile-input or 5xx finding remained. - Local `photos-perf.mjs --items 1 --home-only`: at load average 38.2/42.8/43, server start 2850 ms, indexed 4790 ms, mean/peak RSS 335424222/425738240 bytes, mean/peak CPU 23.37/95.95%, CPU 22.16 s. This one-photo local smoke is not comparable to the 5,000-photo perf VM baseline. ### Known gaps and decisions - Standalone IMAP tests retain the existing display expectation and fail `client::tests::test_parsing_error`; clippy passes. No test expectation was changed. - The final web test gate has the single five-second ThemePicker timeout above. The component and test are unchanged from origin/dev; this is reported as a gate failure. - Media's 5.39-second response is SLOW-only shared-host load. - DESIGN §35 does not specify an SSE event arriving before stable folder identity resolves. Decision: coalesce early events into one list refresh after resolution. DESIGN §47 L1 defines Account as Location consent owner; Auto uses the exact saved location. - Settings effects used Node 22 because Bun could not load host Sharp (`libstdc++.so.6` missing). No app code changed for the host workaround. ### Production screenshots The new Settings/Photos/Backgrounds matrix covers 390/820/1440 px in light and dark: [download review ZIP](https://git.kayg.org/attachments/ef08eb55-75be-4234-a946-b5557db27745). Existing full matrices: [Location](https://git.kayg.org/attachments/6078bc47-ceb1-4566-9b5e-63ef5c52b9a1), [Appearance/Notes/Photos/Log](https://git.kayg.org/attachments/2a0c2168-6199-4d45-9ff2-ad676518e88e), [Calendar feeds](https://git.kayg.org/attachments/03f5976a-28a3-4384-9aae-a995bac45409). Screenshots and logs are not committed.
kayg commented 2026-09-30 17:20:51 +00:00
Author
Owner
Copy link

Merged into dev and deployed to calternal.cloud at aa372eef6 via merge round 2 (server tests 93 passed, web 914/914, live location/webcal/WebDAV/Settings-Photos/two-User probes passed).

Merged into dev and deployed to calternal.cloud at aa372eef6 via merge round 2 (server tests 93 passed, web 914/914, live location/webcal/WebDAV/Settings-Photos/two-User probes passed).
kayg commented 2026-10-03 12:47:46 +00:00
Author
Owner
Copy link

Audit against origin/dev: docs/perf/BASELINE.md still marks the #367 baseline incomplete. It records an incomplete Photos rebuild, a large Home run that did not finish, and the o2 arm64 workload not run. The benchmark round is therefore still open.

Audit against origin/dev: `docs/perf/BASELINE.md` still marks the #367 baseline incomplete. It records an incomplete Photos rebuild, a large Home run that did not finish, and the o2 arm64 workload not run. The benchmark round is therefore still open.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
wip/hiderev-1153
wip/hide4-1153
wip/hide3-1153
wip/hide2-1153
wip/hide-1153
wip/editreg-1132
wip/editorrail3-1113
wip/editorrail2-1113
wip/editorrail-1113
wip/e2e-b2-1071
wip/e2e-b-1071
wip/draw4-1101
wip/draw3-1101
wip/draw2-1101
wip/draw-1101
wip/directory-1199
wip/delete-1119
wip/collabloss-1197
wip/cards2-1083
wip/cards-1083
wip/canvas-visual
wip/canvasvis2-976
wip/calhdr-1112
wip/calcards-1115
wip/browserfix
wip/blocks-1125
wip/allday-1107
wip/agenda-decks
wip/agenda-1086
wip/adv7c-1105
wip/txentry-1198
wip/trayicons2-1095
wip/trayicons-1095
wip/tagperf-1186
wip/sidebar3-1094
wip/segmented-1200
wip/rev2-webperf
wip/rev2-money-ident
wip/previewcard-1098
job/collabloss-1197
wip/palette2-1123
wip/palette-1093
wip/onboard2-1141
job/restyle-settings
wip/onboard-1141.aborted-early
wip/onboard-1141
wip/notifloop-1194
wip/nlpchip-1127
wip/morph-1104
wip/merge-round-7c5
wip/merge-round-7c4
wip/merge-round-7c3
wip/merge-round-7c2
wip/merge-round-7c
wip/mchrome-1084
wip/mailghost2-1094
wip/mailghost-1094
wip/kbpreview2-1118
job/restyle-files
wip/kbpreview-1118
wip/kanban-1092
wip/importhang-1121
job/tagdnd-1187
job/merge30
job/perf-1124
job/tocrail-1191
job/cards-1179
wip/cards2-1179
wip/cards-1179
job/segmented-1200
wip/tocrail-1191
job/hide-1153
wip/tagdnd-1187
wip/restyle-files
wip/perf-1124
wip/merge30j
job/onboard-1141
job/restyle-notes
wip/restyle-notes
job/wizchoices-1140
job/adv-1202
job/notifloop-1194
wip/wizchoices-1140
wip/restyle-1190
job/moneyfmt-1180
job/txentry-1198
wip/moneyfmt2-1180
wip/moneyfmt-1180-r
wip/moneyfmt-1180
job/tagperf-1186
job/pillglass-1189
job/flags-1181
wip/flags-1181
job/restyle-1190
job/restyle-mailmoney
job/restyle-search
job/settingsreg-1195
job/wizard-1140
site/website
wip/wizardrev2-1140
wip/wizardrev-1140
wip/wizard5-1140
wip/wizard4-1140
wip/wizard3-1140
wip/wizard2-1140
wip/wizard-1140
wip/pillglass-1189
wip/settingsreg-1195
job/merge29
job/fu-1171
wip/merge29j
wip/fu-1171
job/fu-1166
job/directory-1199
job/txresearch-1188
wip/fu-1166
job/merge28
job/search-1066
wip/search-1066
wip/merge28j
job/gateslot-1182
job/bulkimport-1157
job/mailnet-1160
wip/mailnetrev-1160
wip/mailnet-1160
wip/bulkrev-1157
wip/bulkimport-1157
job/startup-1161
wip/startup-1161
job/merge27
job/linkcards-1151
wip/linkcards3-1151
wip/linkcards2-1151
wip/linkcards-1151
job/traydate-1144
wip/traydate3-1144
wip/traydate2-1144
wip/traydate-1144
job/draw-1101
wip/merge27j
job/blockpill-1152
wip/blockpill3-1152
wip/blockpill2-1152
wip/blockpill-1152
job/minihover-1149
wip/minihover2-1149
wip/minihover-1149
job/merge25
wip/merge25-r
wip/merge25b
wip/merge25
job/inspector-1129
job/tags-1110
wip/inspector3-1129
wip/inspector2-1129
wip/inspector-1129
wip/tagsrev-1110
wip/tags2-1110
wip/tags-1110
job/dates-1148
wip/datesrev-1148
wip/dates2-1148
wip/dates-1148
job/licence-1145
wip/licence2-1145
wip/licence-1145
job/selfhost-1156
job/merge23
wip/merge23
job/tagfilter-1109
wip/tagfilter2-1109
wip/tagfilter-1109
job/kbd-1134
wip/kbd2-1134
wip/kbd-1134
job/palfoot-1137
wip/selfhost-1156
wip/palfoot2-1137
wip/palfoot-1137
job/toggle-1158
wip/toggle-1158
job/kbpreview-1118
job/docratchet-1155
job/perflint-1133
job/devtests-1159
wip/docratchet-1155
wip/devtests-1159
job/segv-1136
wip/toast-1142
wip/segv-1136
job/toast-1142
job/blockreload-1147
wip/blockreload-1147
job/font-1150
wip/font-1150
job/importui-1120
job/minimonth-1149
wip/importui-1120
wip/minimonth-1149
job/depcheck-1146
wip/perflint-1133
wip/depcheck-1146
job/calcards-1115
job/blocks-1125
job/plus-1128
job/shift-1138
wip/plus2-1128
wip/plus-1128
wip/shift-1138
job/moneyfid-1130
job/editorrail-1113
wip/moneyrev-1130
wip/moneyfid-1130
job/noext-851
wip/noext-851
wip/noext3-851
wip/noext2-851
job/week-1135
wip/week-1135
job/editreg-1132
job/smoke-1122
wip/smoke-1122
job/docs-1143
job/palette2-1123
job/calhdr-1112
job/nlpchip-1127
job/mailghost-1094
job/reconnect-1131
wip/reconnect-1131
job/trayicons-1095
job/delete-1119
job/importhang-1121
job/cards-1083
job/palette-1093
job/mchrome-1084
job/e2e-a-1071
job/canvas-visual
job/previewcard-1098
job/allday-1107
wip/e2e-a2-1071
wip/e2e-a-1071
job/e2e-b-1071
job/adv7c-1105
job/kanban-1092
job/agenda-1086
job/merge-round-7c
job/morph-1104
wip/surfaces-p2
job/merge-round-9
wip/merge-round-9
job/7cfix-small
wip/7cfix-small
job/mailui-1078
job/merge-round-8
wip/merge-round-8
wip/mailui-1078
job/mailround-1038
job/applemail-accept
wip/settitle-1068
wip/mailround2-1038
wip/mailround-1038
wip/e2e-7b
job/crash-1069
wip/crash-1069
job/searchlost-1066
wip/searchlost-1066
job/7b-reconcile
job/flake-1065
wip/flake-1065
wip/merge-round-7b7
wip/merge-round-7b6
wip/merge-round-7b5
wip/merge-round-7b4
wip/7b-reconcile
job/appupdate-1059
job/nfd-1044
wip/appupdate-1059
job/e2e-7b
job/loop-1062
wip/loop-1062
job/pdfprev-1045
job/invtoggle-1053
wip/pdfprev-1045
wip/nfd-1044
wip/invtoggle-1053
job/7bfix-e2e
job/mailstress-b
wip/7bfix-e2e
wip/mailstress-b
job/7bfix-adv
wip/7bfix-adv
job/mailstress-a
job/stack-1054
wip/stack-1054
wip/mailstress-a
job/mailstress-1038
wip/mailstress-1038
job/upload500-1051
wip/upload500-1051
job/share-1034
wip/share-1034
job/syncerr-1037
job/7bfix-photos
wip/7bfix-photos
job/paste-1036
job/setside-1039
wip/setside-1039
wip/paste-1036
job/lease-1042
wip/syncerr-1037
wip/lease-1042
job/7bfix-data
job/passkeybind-1043
wip/apprevoke-1041
job/invite-1035
wip/invite-1035
job/merge-round-7b2
wip/merge-round-7b2
job/mailproxy-486
job/apprevoke-1041
job/rebuild-1033
job/pillborder-1029
wip/pillborder-1029
wip/mailproxy-486
wip/applemail-486
job/headless-998
wip/headless-998
job/groups-1028
wip/groups-1028
job/rebuildwarn-1016
wip/rebuildwarn-1016
job/startup-1011
wip/startup-1011
job/monthpill-1009
job/bgthumb-1025
job/sharetitle-1012
wip/monthpill-1009
wip/bgthumb-1025
wip/sharetitle-1012
job/canvas-cards-977
wip/canvas-cards-977
job/canvas-pencil-978
job/canvas-sketch-990
wip/canvas-sketch-990
wip/canvas-pencil-978
job/canvas-files-989
wip/canvas-files-989
job/canvas-collab-991
wip/canvas-collab-991
job/weekscroll-1018
wip/weekscroll-1018
wip/canvas-core-976
job/canvas-core-976
job/round-drag
wip/round-drag
job/round-settings
job/browserfix
wip/oapi-974
job/oapi-974
job/hist2-integrate
job/mailhtml-726
wip/mailhtml-726
wip/hist2-integrate
job/moneyfu-984
job/drag-1015
wip/drag-1015
job/rename-1017
wip/rename-1017
job/hist2-api
wip/hist2-api
job/oneacct-1014
wip/oneacct-1014
wip/moneyfu-984
job/hist2-bench
job/hist2-restore
wip/hist2-bench
job/hist2-write
job/hotfix-724
wip/hotfix-724
wip/hist2-write
wip/hist2-restore
job/hist2-store
job/hist2-ui
wip/hist2-ui
wip/hist2-store
job/searchstarve-965
job/shutdown-963
wip/shutdown-963
wip/pubedit-981
job/pubedit-981
job/analytics-973
wip/searchstarve-965
job/authflash-850
job/weeklane-969
job/pvtitle-1004
job/hist-975
wip/authflash-850
job/voicepill-617
wip/pvtitle-1004
job/headring-1003
wip/weeklane-969
wip/voicepill-617
wip/headring-1003
wip/analytics-973
job/agentscope-980
wip/thumbsandbox-988
job/thumbsandbox-988
wip/hist-975
job/links-856
wip/links-856
job/davetag-966
wip/davetag-966
job/filesstorm-1000
job/hoverpad-725
wip/filesstorm-1000
job/ffmpegblas-993
job/merge-round-7a
wip/hoverpad-725
wip/ffmpegblas-993
job/nowdot-1002
wip/verify-7a
job/noteid-857
wip/nowdot-1002
wip/noteid-857
wip/merge-round-7a
wip/agentscope-980
job/imapedge
job/a11yfix2
wip/imapedge-941
wip/imapedge
wip/a11yfix2
job/notetask-986
job/logheading
wip/logheading-998
job/textthumb-652
job/photolive-987
wip/photolive-987
job/davactive-983
job/savefix-985
job/tabicons-607
wip/davactive-983
wip/tabicons-607
wip/notetask-986
wip/savefix-985
job/dirid-627
job/buildspeed-1007
wip/dirid-627
job/agenda-decks
job/perfguards-impl
job/undo-a11y
wip/undo-a11y
job/mailperf
job/wal-824
wip/settings-50
job/settings-50
job/notesfilter-606
wip/notesfilter-606
job/surfaces-p2
wip/wal-824
job/maillayouts
wip/mailperf
wip/maillayouts
job/taskmeta-659
job/money-ident
wip/money-ident
wip/taskmeta-659
job/errstates
wip/perfguards-impl
job/headings-881
wip/headings-881
wip/errstates
job/voice-619
job/gaps-827
job/notesperf
wip/notesperf
wip/voice-619
job/hddsql-549
job/perf-stream-668
wip/perf-stream-668
wip/deeplinks-fix
job/deeplinks-fix
job/authfix
job/docsfix-rust
wip/docsfix-rust
job/webperf
job/docsfix-web
job/datafix2
job/webdav-lock-476
job/copyfix
wip/copyfix
wip/webperf
job/focus-658
wip/protofix
job/mediafix
job/protofix
wip/mediafix
job/agentfix
job/hhmm-724
wip/agentfix
job/undo-722
job/reuse
wip/webdav-lock-476
wip/reuse
job/scopefix
job/datafix
wip/hhmm-724
wip/undo-722
job/surfaces-p1
wip/hddsql-549
job/voicememos-618
wip/datafix2
wip/surfaces-p1
job/fix-940
wip/fix-940
job/blaze-surfaces
wip/datafix
wip/blaze-surfaces
job/taskday-655
job/linknav-639
wip/linknav-639
wip/gaps-827
job/isolation-707
job/audiophotos-720
wip/audiophotos-720
job/advfind-664
wip/voicememos-618
wip/taskday-655
wip/isolation-707
wip/advfind-664
wip/scopefix
wip/focus-658
job/testgaps
wip/testgaps
job/overscroll-718
wip/authfix
job/deps
wip/overscroll-718
job/rev2-agentfix
job/rev2-money-ident
job/rev2-mailperf
wip/deps
job/hardening-728
wip/hardening-728
job/searchgen-832
wip/searchgen-832
job/photopw-849
job/mailsql-825
wip/photopw-849
job/sharefix
wip/sharefix
job/rev2-mailhtml-726
job/rev2-perfguards
job/copyval-723
job/lightglass-r2
wip/lightglass-r2
wip/docsfix-web
job/copy-audit
job/macinterop-staging-r2
job/design-sync
job/rev2-taskmeta-659
job/rev2-webperf
job/docs-audit
job/rev2-advfind-664
job/rev2-mailproxy-486
job/states-audit
job/rev2-datafix
job/design-drift
job/test-gaps
job/rev2-voicememos-618
job/rev2-mediafix
job/rev2-deps
job/rev2-datafix2
job/licence-audit
job/issue-hygiene
job/rev2-protofix
job/rev2-voice-619
job/rev2-isolation-707
job/rev2-surfaces-p1
job/deeplink-audit2
job/rev2-audiophotos-720
wip/test-gaps
job/rev2-overscroll-718
job/rev2-undo-722
wip/states-audit
job/rev2-dropmd-719
job/rev2-linknav-639
job/merge-7b-plan
wip/merge-7b-plan
job/rev2-taskday-655
wip/mailsql-825
job/rev2-webdav-lock-476
job/rev2-browserfix
wip/design-drift
job/rev2-hddsql-549
wip/deeplink-audit2
job/rev2-scopefix
job/rev2-authfix
job/rev2-hardening-728
job/rev2-wal-824
job/rev2-sharefix
job/calsidebar-638
job/chrome-audit
job/ioperf
wip/ioperf
wip/chrome-audit
wip/calsidebar-638
job/dropmd-719
wip/dropmd-719
job/ocr-build
wip/ocr-build
job/blaze-settings
wip/copyval-723
job/toastring-721
wip/toastring-721
job/deployfix-732
wip/deployfix-732
wip/blaze-settings
job/money-import-recheck
job/rev-a11y
job/perf-arch-db
job/rev-7b-data
wip/textthumb-652
wip/perf-arch-db
job/sec-protocols
job/sidehdr-660
job/rev-7b-security
job/research-surfaces
job/rev-design-gaps
job/rev-mcp-api
wip/sidehdr-660
job/perf-arch-memory
wip/sec-protocols
job/perf-arch-bundle
job/snapedge-714
wip/rev-mcp-api
job/sec-supplychain
wip/research-surfaces
job/perf-arch-sync
job/rev-consistency
job/perf-arch-server
wip/perf-arch-server
wip/perf-arch-memory
job/perf-arch-io
job/perf-arch-client
job/sec-fs
job/sec-mcp-scopes
job/sec-sharing
job/perf-guards
job/sec-browser
job/sec-admin-deploy
job/sec-auth
wip/snapedge-714
job/bgpicker-717
wip/perf-arch-bundle
wip/money-import-recheck
job/advsetup-654
wip/bgpicker-717
wip/advsetup-654
job/burst-709
job/kbdcaps-710
job/app-pw-chooser
wip/burst-709
wip/app-pw-chooser
job/imaptest-625
wip/kbdcaps-710
job/fix-499
wip/fix-499
job/perf-mut-667
job/calimg-589
job/perf-snap-666
wip/calimg-589
wip/perf-snap-666
wip/perf-mut-667
job/perf-cache-665
wip/perf-cache-665
job/voicefiles-620
wip/voicefiles-620
job/admin-burst-705
wip/admin-burst-705
job/voicememos-review
wip/voicememos-review
wip/ryw-653
job/ryw-653
job/writeonopen-661
job/instant-663
wip/writeonopen-661
job/money-import-review
wip/money-import-review
wip/importjs-610
review/integrations-407-round6
wip/integrations-review
job/dragghost-612
wip/dragghost-612
job/integrations
wip/integrations
job/decider-656
job/merge-round-6
job/perf-rerun
wip/merge-round-6
job/integrations-review-round5
job/selalign-576
wip/selalign-576
job/mcp-events-491
job/files-631
job/cal-e2e-569
wip/cal-e2e-569
job/reload-423
wip/reload-423
wip/mcp-events-491
wip/files-631
job/notesbridge-644
wip/notesbridge-644
job/editor-series
job/calcard-series
wip/calcard-series
job/mcp-events-review-491
wip/mcp-events-review
wip/editor-series
job/quirks-546
job/integrations-recheck
job/tocrail-636
wip/tocrail-636
wip/quirks-546
wip/reminders-643
job/reminders-643
wip/davscale-573
job/davscale-573
job/integrations-review
wip/ocr-eval-584
job/ocr-eval-584
job/esc-537
wip/esc-537
job/toastname-586
wip/toastname-586
job/submenu-579
wip/submenu-579
job/tasks-mode
wip/tasks-mode
job/agentdocs-630
job/dupwrite-634
wip/agentdocs-630
wip/dupwrite-634
job/lightglass-588
wip/lightglass-588
job/tabswitch-549
job/ghosttask-623
wip/ghosttask-623
job/toaststack-616
job/weekstate-609
job/mailsync-613
wip/mailsync-613
wip/weekstate-609
job/maildup-626
wip/tabswitch-549
wip/maildup-626
wip/toaststack-616
job/motion-611
wip/motion-611
job/tlstest-601
wip/tlstest-601
job/perf-495
job/floating-sheet
wip/floating-sheet
job/remdup-585
wip/remdup-585
job/fix-502
wip/fix-502
job/attachplay-622
job/perf-batch
wip/perf-batch-563
wip/perf-495
hotfix/mail-sync-diag
job/mail-m3
wip/mail-m3
job/attach-poof-603
job/calhover-608
job/editorbar-604
job/mentions-605
job/merge-round-4
job/allday-514
wip/merge-round-4
wip/allday-514
job/merge-round-4a
wip/merge-round-4a
job/sharestack-580
job/fix-501
wip/sharestack-580
wip/fix-501
job/perf-batch-563
job/apw-cache-review
wip/apw-cache-review
job/probe-520
wip/probe-520
job/mac-393
wip/mac-393
job/header-571
job/flake-513
wip/flake-513
job/docs-thumb-547
wip/header-571
job/webcal-572
wip/webcal-572
wip/shortcuts-542
job/shortcuts-542
wip/docs-thumb-547
job/caldav-stress
wip/caldav-stress
wip/sweep-478
job/apw-cache-512
wip/apw-cache-512
job/money-empty-540
wip/restart-505
wip/money-empty-540
wip/fix-510
job/restart-505
job/fix-503
job/perf-496
wip/perf-496
job/fix-498
wip/fix-498
job/info-inspector-465
wip/info-inspector-465
job/fix-510
job/fix-507
wip/fix-507
wip/fix-503
job/fix-493
job/money-kinds
wip/money-kinds
job/hygiene-548
job/merge-round-3
wip/fix-493
job/drag-snap-536
wip/merge-round-3
wip/merge-round-0930
wip/drag-snap-536
job/align-538
wip/align-538
job/bg-flash
wip/bg-flash
job/money-import
job/search-count-544
wip/search-count-544
wip/money-import
job/settings-key-541
wip/settings-key-541
job/toast-539
job/preview-421
wip/preview-421
wip/toast-539
job/tasks-500-531
job/title-plain-526
wip/title-plain-526
wip/tasks-500-531
job/notes-bridge
wip/parity-484
job/parity-484
job/files-slow
job/crash-525
wip/notes-bridge
wip/files-slow
wip/crash-525
job/kbd-motion-527
wip/bg-422
job/analytics-504
wip/analytics-504
wip/kbd-motion-527
job/upload-pill-523
wip/upload-pill-523
wip/tray-order
job/tray-order
wip/overflow-mid
wip/merge-round-2
job/perf-494
wip/perf-494
wip/mcp-fast-492
wip/motion-477
wip/asr-ab-489
wip/theme-variants-506
wip/overflow-511
wip/week-header-508
wip/attach-427
job/dav-delete-471
job/iso-435
wip/iso-435
wip/files-sel-keys
wip/dav-delete-471
job/align-253
job/siwc-490
wip/siwc-490
job/money-kinds-review
wip/align-253
wip/money-kinds-review
job/small-bugs-3
wip/overlay-title-487
wip/multiget-500
wip/hidden-420
wip/webcal-ui
wip/webcal-431
job/perf-367
job/location
wip/small-bugs-3
wip/location
wip/perf-367
wip/admin-deny-483
job/tag-unicode-473
wip/tag-unicode-473
job/blur-436
wip/photos-470
wip/blur-436
wip/small-bugs-4
wip/hunt-20260930
wip/settings-hdr-482
wip/chips-416
job/dedup-375
wip/dedup-375
job/doc-stack
wip/doc-stack
job/tokens-literals
wip/tokens-literals
job/jobs-leftovers
wip/send-fast
wip/paste-467
wip/money-numbers
job/money-plugin
wip/money-plugin
job/break-dav
wip/merge-batch
wip/crossday-469
wip/mac-verify
wip/mail-m2
wip/break-dav
wip/money-review2
job/money-md
job/modes-424
wip/money-md
wip/jobs-leftovers
job/agenda-413
wip/agenda-413
wip/modes-424
job/recog-417
wip/recog-417
wip/bounce-425
wip/ab-384-luna
job/webdav-perf
wip/webdav-perf
job/toast-ring
wip/toast-ring
job/money-review
wip/money-review
wip/micro-motion
wip/settings-card
wip/minical
job/notes-imap-428
job/least-priv
wip/ui-small-2
wip/flaky-426
wip/drag-end-418
job/jank
wip/jank
wip/least-priv
wip/docs-site
job/agenda
job/sec-batch
wip/sec-batch
wip/per-user-index
job/area-calendars
wip/area-calendars
job/parity
wip/parity
job/documents-research
wip/documents-research
job/test-infra
job/reminders-sync
wip/small-bugs-2
wip/reminders-sync
wip/gestures
job/google-oauth
wip/tags-merge
wip/tags
job/e2e-theme
wip/e2e-theme
job/icon-align
wip/test-infra
wip/select-align
wip/editor-385
job/voice
wip/webdav
job/webdav
job/app-pw-ui
job/editor-integrity
wip/editor-integrity
wip/voice
wip/quota
wip/cal-followups
wip/icon-align
job/composer-scale
wip/composer-scale
job/jobs-page
wip/jobs-page
job/hig-type
wip/hig-type
wip/app-pw-ui
job/motion-spring
job/mcp
wip/motion-spring
wip/mcp
job/small-bugs
wip/push-hosts
job/profile-sign
wip/touch-369
wip/profile-sign
job/mobile-focus
wip/mobile-focus
wip/ui-polish-354
wip/small-bugs
wip/dup-task
job/toast-polish
job/app-pw-scopes
wip/toast-polish
wip/app-pw-scopes
wip/cli-agent
wip/selection-pills
job/preview-attach
wip/preview-attach
job/dav-proppatch
wip/dav-proppatch
wip/cal-switcher
job/atomic-race
wip/atomic-race
job/photos-shared
wip/photos-shared
wip/cal-grid
wip/note-rewrite
wip/search-rebuild
job/mail-m1
job/paperless-import
wip/paperless-import
wip/mail-m1
wip/hidden-activity
wip/search-d
wip/pricing-research
wip/cursors
wip/auto-scheme
job/single-pills
wip/single-pills
wip/xuser-matrix
wip/money-format
wip/app-pw-setup
wip/purge-dos
wip/vault-health
wip/caldav-apple
wip/xuser-audit
wip/e2e-green
wip/tabbar
wip/adv-harness
wip/maple-mono
job/search-fix
wip/search-fix
wip/search-perf-c
job/adv-harness
wip/sidebar-headers
job/glass
wip/temp-index
job/polish
wip/polish
wip/file-protocols
wip/money-research
wip/glass
wip/voice-models
wip/collab-redo
job/voice-research
wip/hunt-20260928
wip/notes-actions-research
wip/search-pad
wip/search-perf
wip/search-sticky
wip/editor-undo
wip/chrome-rules
wip/motion
wip/appearance-research
wip/appearance
wip/audit-bugs
wip/cal-glass
wip/block-actions
wip/authz-order
wip/event-stripes
wip/chrome-sidebar
wip/auth-flaky
wip/robust-2
wip/gate-fix
wip/menu-blur
wip/import-calternaljs
wip/tray-fix
job/import-calternaljs
wip/index-order
wip/audit-fixes
wip/search-chevrons
research/mail
wip/phone-chrome
wip/dedup-break
wip/csp
wip/ui-audit
wip/select-toast
wip/perf
wip/flat-layout
wip/fonts
wip/event-tint
wip/sync-converge
wip/data-split
wip/glass-audit
wip/robustness
wip/sync-chaos
wip/search-thumbs
wip/fuzz
wip/menu-icons
wip/search-pill
wip/sync-changing
wip/heading-links
wip/date-formats
wip/a11y
wip/break-editor
wip/e2e-fix
wip/settings-sections
wip/sync-root-guard
wip/search-palette
wip/share-edit
job/toasts
wip/toasts
wip/cont-analytics
wip/authz-review
wip/popovers
wip/overlay-glass
wip/change-feed
wip/editor-modes
wip/composer-align
wip/cont-agenda
wip/agenda-merge
job/agent-conventions
wip/agent-conventions
wip/backend-misc
job/route-audit
wip/route-audit
wip/ui-batch
wip/heif-hardening
wip/grid-resize
wip/ask-page
wip/webmcp
job/deeplink-audit
wip/deeplinks
wip/shortcuts
wip/cont-tz-days
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#367
No description provided.