kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 803 Pull requests Projects Releases Packages Wiki Activity Actions 430

PERF: one per-User change stream and capped delta endpoint (#663) #668

New issue
Open
opened 2026-10-02 05:20:43 +00:00 by kayg · 24 comments
kayg commented 2026-10-02 05:20:43 +00:00
Owner
Copy link

Context: #663 instant-interaction audit. No product change is part of the audit job.

Own rule 5 in DESIGN §58. This is the sole owner of the app-wide change sequence, SSE wake-up and delta protocol. Tab issues only contribute projections and adapters.

Evidence at c4a61e8cf0:

  • Files already has a durable change feed and User cursors (DESIGN §24; crates/plugins/files/src/lib.rs:3904). Reuse it and preserve existing sync clients.
  • Files SSE (lib.rs:3834) sends path events, replaying 256 at a time for ten minutes. The client shares one EventSource but fans out full reload hints (apps/web/src/lib/files/live.ts:23).
  • Mail polls every 30 s (apps/web/src/lib/mail/MailView.svelte:227); Photos schedules reloads at 900/3500 ms (apps/web/src/lib/photos/PhotosView.svelte:742). These do not provide one cross-plugin delta contract.

Expected:

  • One monotonic sequence per User for committed changes from all plugins, including Tasks, Notes, Calendar, Mail, Money, preferences, shares and plugin enablement. Emit a wake-up with only the sequence, never content or paths. Preserve the existing Files sync-feed contract.
  • One delta endpoint with signed, scope-bound cursors; cap rows ≤100, bytes and work. Include changed IDs/revisions and deletion tombstones. Re-check current access on reads. Share revoke and disabled plugins remove affected retained items without leaking another User.
  • Define sequence retention, reconnect, cursor expiry and bounded rescan explicitly. Do not use full-corpus responses for recovery. Publish the change and its receipt/projection at one commit boundary.
  • Client coalesces wake-ups about 100 ms, drains only bounded work, updates all retained views and preserves unchanged objects by identity. Cancel work at User switch and avoid parallel refetch storms.

Tests:

  • Cross-plugin mutations; monotonic order; duplicate wake-ups; reconnect/replay; expiry; capped large deltas; concurrent writes; shared recipient access/revoke; one User cannot read or delay another User's stream. Existing Files sync clients continue to converge.
  • One real-server adversarial round with regressions for new protocol handling.
  • Production/HDD locked VM: ≥5 delta/read samples, median/p95/max, burst CPU/RSS, steady stream/heap caps, no full-list refresh on an unchanged object. Extend existing sync_feed.py and sse_storm.py with #641 completeness checks.

Reuse: Files feed and #452 isolation, #555 User lifecycle, #549 caches, #641 harness, #640 Mail readers and #642 Settings. Do not build a separate stream per Tab.

Measurement evidence: see the production/HDD matrix posted on #663. Latencies of sampled endpoints do not prove the shared contract is complete.

Locked production measurement evidence:
The Files listing five-request burst in the first phase was median/p95/max 2033.2/2034.0/2034.0 ms with 150 ms process CPU and 443,625,472 bytes RSS. That measures full listing reads, not a delta. Existing SSE pushes per-path hints with 256-event replay batches; client Mail refresh interval is 30,000 ms. No app-wide delta endpoint exists in the audited route contracts.
Runtime server source cc25c441b7, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9, shared release binary with embedded production SPA. The source audit uses the newer c4a61e8 base; no code equivalence claim. Perf VM HDD emulator uses bench/hdd-emu.sh (direct-I/O/ext4, 8 ms read/write delay, 200 IOPS and 150 MiB/s caps) and flock -w 14400 /root/perf.lock for every measured phase. First valid qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms; load 0.10/0.18/0.08 and 0.20/0.20/0.09. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty and Admin one User. No matching transport/HDD/fixture baseline exists, so no regression ratio is claimed. Full raw scopes, failure retention and endpoint baseline references are on #663.

Context: #663 instant-interaction audit. No product change is part of the audit job. Own rule 5 in DESIGN §58. This is the sole owner of the app-wide change sequence, SSE wake-up and delta protocol. Tab issues only contribute projections and adapters. Evidence at c4a61e8cf090170f35b1bed3350d9de20c83ecd5: - Files already has a durable change feed and User cursors (DESIGN §24; `crates/plugins/files/src/lib.rs:3904`). Reuse it and preserve existing sync clients. - Files SSE (`lib.rs:3834`) sends path events, replaying 256 at a time for ten minutes. The client shares one EventSource but fans out full reload hints (`apps/web/src/lib/files/live.ts:23`). - Mail polls every 30 s (`apps/web/src/lib/mail/MailView.svelte:227`); Photos schedules reloads at 900/3500 ms (`apps/web/src/lib/photos/PhotosView.svelte:742`). These do not provide one cross-plugin delta contract. Expected: - One monotonic sequence per User for committed changes from all plugins, including Tasks, Notes, Calendar, Mail, Money, preferences, shares and plugin enablement. Emit a wake-up with only the sequence, never content or paths. Preserve the existing Files sync-feed contract. - One delta endpoint with signed, scope-bound cursors; cap rows ≤100, bytes and work. Include changed IDs/revisions and deletion tombstones. Re-check current access on reads. Share revoke and disabled plugins remove affected retained items without leaking another User. - Define sequence retention, reconnect, cursor expiry and bounded rescan explicitly. Do not use full-corpus responses for recovery. Publish the change and its receipt/projection at one commit boundary. - Client coalesces wake-ups about 100 ms, drains only bounded work, updates all retained views and preserves unchanged objects by identity. Cancel work at User switch and avoid parallel refetch storms. Tests: - Cross-plugin mutations; monotonic order; duplicate wake-ups; reconnect/replay; expiry; capped large deltas; concurrent writes; shared recipient access/revoke; one User cannot read or delay another User's stream. Existing Files sync clients continue to converge. - One real-server adversarial round with regressions for new protocol handling. - Production/HDD locked VM: ≥5 delta/read samples, median/p95/max, burst CPU/RSS, steady stream/heap caps, no full-list refresh on an unchanged object. Extend existing sync_feed.py and sse_storm.py with #641 completeness checks. Reuse: Files feed and #452 isolation, #555 User lifecycle, #549 caches, #641 harness, #640 Mail readers and #642 Settings. Do not build a separate stream per Tab. Measurement evidence: see the production/HDD matrix posted on #663. Latencies of sampled endpoints do not prove the shared contract is complete. Locked production measurement evidence: The Files listing five-request burst in the first phase was median/p95/max 2033.2/2034.0/2034.0 ms with 150 ms process CPU and 443,625,472 bytes RSS. That measures full listing reads, not a delta. Existing SSE pushes per-path hints with 256-event replay batches; client Mail refresh interval is 30,000 ms. No app-wide delta endpoint exists in the audited route contracts. Runtime server source cc25c441b7a974185622a1dee853cf38686d2b67, binary SHA-256 2f3567d91c34839851247bc0acbc25a56aaacd14dca269b8f0342ddf83447ed9, shared release binary with embedded production SPA. The source audit uses the newer c4a61e8 base; no code equivalence claim. Perf VM HDD emulator uses bench/hdd-emu.sh (direct-I/O/ext4, 8 ms read/write delay, 200 IOPS and 150 MiB/s caps) and flock -w 14400 /root/perf.lock for every measured phase. First valid qualification QD1 115.3 IOPS/8.028 ms median, QD16 200.7 IOPS/96.993 ms; load 0.10/0.18/0.08 and 0.20/0.20/0.09. Fixture: 366 Daily notes, 10,980 Logs, 100 Files/Photos, 20 Notes/Tasks, three Budgets and 100 transactions; Mail empty and Admin one User. No matching transport/HDD/fixture baseline exists, so no regression ratio is claimed. Full raw scopes, failure retention and endpoint baseline references are on #663.
kayg commented 2026-10-02 09:41:06 +00:00
Author
Owner
Copy link

Started #668 on job/perf-stream-668, base c4a61e8cf090170f35b1bed3350d9de20c83ecd5. Read the repo contract, glossary, #668 and #663 matrix. Cherry-picked #663 documentation commits (local heads 4bb2aad93, 152d1248f) because §58 was absent. Scope: shared durable per-User sequence, bounded signed delta protocol and content-free SSE, with at most one proving adapter. Other Tab projections remain with their adoption owners; no cache/receipt primitive duplication. No push or deploy.

Started #668 on `job/perf-stream-668`, base `c4a61e8cf090170f35b1bed3350d9de20c83ecd5`. Read the repo contract, glossary, #668 and #663 matrix. Cherry-picked #663 documentation commits (local heads 4bb2aad93, 152d1248f) because §58 was absent. Scope: shared durable per-User sequence, bounded signed delta protocol and content-free SSE, with at most one proving adapter. Other Tab projections remain with their adoption owners; no cache/receipt primitive duplication. No push or deploy.
kayg commented 2026-10-02 10:02:11 +00:00
Author
Owner
Copy link

Shared primitive interfaces are now concrete: calternal_plugin::changes::append(&mut SqliteConnection, User ID, &Record) publishes in the projection/receipt transaction; call wake_user only after commit. invalidate_access increments a durable User authority epoch at revoke. Plugin::change_projection is fail-closed by default; adopted Plugins return current ID/revision/header or a tombstone after live access checks. Routes: /api/v1/changes and /api/v1/changes/events. Client: subscribeChanges(ChangeAdapter) in apps/web/src/lib/changeStream.ts; adapters own their retained windows and cache, reset synchronously, and rescan only a bounded window. This does not add a second cache, receipt store or snapshot primitive.

Files proving adapter reuses INSERTs into files_change_feed at their existing commit boundary. The legacy Files cursor/feed/SSE contracts remain unchanged. Its current header projection looks up (owner_id,item_id) through the existing identity index, then uses the existing live Share check. Share revoke and Plugin disable expire app cursors before content can be returned. Cursor authority binds User, credential scope, enabled Plugins and durable access epochs.

Decisions not specified by §58: 100 scanned rows / 64 KiB envelope; 10,000 retained rows per User / 14-day expiry; fixed catch-up high-water mark; four pages per client turn; 1 s safety head check after a lost wakeup; resource-scoped/App Password clients keep their existing Plugin protocols until scope adapters exist. Expiry returns 410 and requires only bounded visible-window rescans. Server proving adoption is Files; other production Tab projections and browser-view migrations remain in #669–#676/#701.

Shared primitive interfaces are now concrete: `calternal_plugin::changes::append(&mut SqliteConnection, User ID, &Record)` publishes in the projection/receipt transaction; call `wake_user` only after commit. `invalidate_access` increments a durable User authority epoch at revoke. `Plugin::change_projection` is fail-closed by default; adopted Plugins return current ID/revision/header or a tombstone after live access checks. Routes: `/api/v1/changes` and `/api/v1/changes/events`. Client: `subscribeChanges(ChangeAdapter)` in `apps/web/src/lib/changeStream.ts`; adapters own their retained windows and cache, reset synchronously, and rescan only a bounded window. This does not add a second cache, receipt store or snapshot primitive. Files proving adapter reuses INSERTs into `files_change_feed` at their existing commit boundary. The legacy Files cursor/feed/SSE contracts remain unchanged. Its current header projection looks up `(owner_id,item_id)` through the existing identity index, then uses the existing live Share check. Share revoke and Plugin disable expire app cursors before content can be returned. Cursor authority binds User, credential scope, enabled Plugins and durable access epochs. Decisions not specified by §58: 100 scanned rows / 64 KiB envelope; 10,000 retained rows per User / 14-day expiry; fixed catch-up high-water mark; four pages per client turn; 1 s safety head check after a lost wakeup; resource-scoped/App Password clients keep their existing Plugin protocols until scope adapters exist. Expiry returns 410 and requires only bounded visible-window rescans. Server proving adoption is Files; other production Tab projections and browser-view migrations remain in #669–#676/#701.
kayg commented 2026-10-02 10:31:17 +00:00
Author
Owner
Copy link

Atomic progress: shared journal 468f8255c, client coordinator ee29dbd3e, bounded header/access recovery 265738945, Files proving projection 2288f0fcf. The Files crate passed 148 tests (one pre-existing ignored test); its two added bridge/projection tests pass. The shared crate passed 32 tests. The two server protocol tests pass, including both-route credential matrix and cross-User cursor rejection. Web coordinator has nine passing tests, including failure isolation across retained views.

A generated-contract check found that the new SSE handler's default operation ID collided with the existing Files events operation. The server handlers now use explicit changes_delta / changes_events; the production/server contract is being regenerated from the rebuilt binary. No existing test expectation was changed.

Added profiles: sync_feed.py --app-wide verifies named mutation steps, IDs, complete headers and capped deltas, then five serial/five burst unchanged reads. sse_storm.py --app-wide validates 50 complete sequence-only frames and steady CPU/RSS. These check protocol completeness, not browser paint/blaze completeness. The adapter/view owners still own those UI checks. A one-round local adversarial probe is ready for the final integration step; it covers both routes, real App Password denial, concurrent writes, Share/revoke, Plugin disable, expiry and legacy Files feed convergence.

Atomic progress: shared journal `468f8255c`, client coordinator `ee29dbd3e`, bounded header/access recovery `265738945`, Files proving projection `2288f0fcf`. The Files crate passed 148 tests (one pre-existing ignored test); its two added bridge/projection tests pass. The shared crate passed 32 tests. The two server protocol tests pass, including both-route credential matrix and cross-User cursor rejection. Web coordinator has nine passing tests, including failure isolation across retained views. A generated-contract check found that the new SSE handler's default operation ID collided with the existing Files `events` operation. The server handlers now use explicit `changes_delta` / `changes_events`; the production/server contract is being regenerated from the rebuilt binary. No existing test expectation was changed. Added profiles: `sync_feed.py --app-wide` verifies named mutation steps, IDs, complete headers and capped deltas, then five serial/five burst unchanged reads. `sse_storm.py --app-wide` validates 50 complete sequence-only frames and steady CPU/RSS. These check protocol completeness, not browser paint/blaze completeness. The adapter/view owners still own those UI checks. A one-round local adversarial probe is ready for the final integration step; it covers both routes, real App Password denial, concurrent writes, Share/revoke, Plugin disable, expiry and legacy Files feed convergence.
kayg commented 2026-10-02 10:51:05 +00:00
Author
Owner
Copy link

Local round at d775b3bc1 (Rust sources unchanged from b00d42cba):

The real local server probe passed 15 cross-User / malformed-input checks. It verified 120 committed file identities and 120 shared identities, zero incomplete headers and zero missed steps. Revoke, plugin disable, sequence-only reconnect, bounded expiry and preserved legacy sync all passed. Other-User delta reads during the write storm: 32.49, 5.71, 6.09, 11.78, 5.45 ms. Local load 18.07 / 18.42 / 18.67; these timings are load evidence, not the performance claim.

The new profiles now use RUST_LOG=error because the existing server WARN setup message contains a bootstrap token. No token is retained in probe reports. The reused tests/perf/upload_scale.py ProcessSampler now adds weighted mean CPU; this small public helper addition keeps CPU measurement in one place. Unequal-interval CPU/RSS check passed (mean 50%, peak 100%).

Web final gates: svelte-check found 0 errors and 0 warnings; Test Files 154 passed (154), Tests 1062 passed (1062). Production web build passed. Rust final gates and the locked production HDD profiles are in progress. No browser layout changes; server Files is the sole proving adapter, with production view adoption left to the named follow-ups.

Local round at d775b3bc1 (Rust sources unchanged from b00d42cba): The real local server probe passed 15 cross-User / malformed-input checks. It verified 120 committed file identities and 120 shared identities, zero incomplete headers and zero missed steps. Revoke, plugin disable, sequence-only reconnect, bounded expiry and preserved legacy sync all passed. Other-User delta reads during the write storm: 32.49, 5.71, 6.09, 11.78, 5.45 ms. Local load 18.07 / 18.42 / 18.67; these timings are load evidence, not the performance claim. The new profiles now use RUST_LOG=error because the existing server WARN setup message contains a bootstrap token. No token is retained in probe reports. The reused tests/perf/upload_scale.py ProcessSampler now adds weighted mean CPU; this small public helper addition keeps CPU measurement in one place. Unequal-interval CPU/RSS check passed (mean 50%, peak 100%). Web final gates: svelte-check found 0 errors and 0 warnings; Test Files 154 passed (154), Tests 1062 passed (1062). Production web build passed. Rust final gates and the locked production HDD profiles are in progress. No browser layout changes; server Files is the sole proving adapter, with production view adoption left to the named follow-ups.
kayg commented 2026-10-02 11:15:26 +00:00
Author
Owner
Copy link

Resumed #668 at af8f0ba78 on job/perf-stream-668 after the build-host resize. Existing implementation and local round are intact. One-time git fetch origin and git merge origin/dev completed: Already up to date. Shared change migrations use the new calternal-change-stream namespace; origin/dev has no migration in that namespace. Final per-crate/web gates are now running. The shared release binary is older than this implementation and cannot measure the new routes; runtime identity checks in the profiles prevent treating it as equivalent. Checking production measurement availability next. No push or deploy.

Resumed #668 at af8f0ba78 on job/perf-stream-668 after the build-host resize. Existing implementation and local round are intact. One-time git fetch origin and git merge origin/dev completed: Already up to date. Shared change migrations use the new calternal-change-stream namespace; origin/dev has no migration in that namespace. Final per-crate/web gates are now running. The shared release binary is older than this implementation and cannot measure the new routes; runtime identity checks in the profiles prevent treating it as equivalent. Checking production measurement availability next. No push or deploy.
kayg commented 2026-10-02 11:19:53 +00:00
Author
Owner
Copy link

Resume finding: an adapter rescan can finish after session end even when its transport ignores abort. The coordinator checked generation before that await, but not after it; a new User subscription could then receive the old page. Commit 2760388ca adds the post-await fence and an unchanged-expectation regression. Ten coordinator tests pass. Web full tests also passed with 1063 tests. Commit 9460c7228 saves content-free failed benchmark reports, tests cleanup/error-detail exclusion, and measures serial/burst CPU/RSS separately. No existing test expectation changed.

Resume finding: an adapter rescan can finish after session end even when its transport ignores abort. The coordinator checked generation before that await, but not after it; a new User subscription could then receive the old page. Commit 2760388ca adds the post-await fence and an unchanged-expectation regression. Ten coordinator tests pass. Web full tests also passed with 1063 tests. Commit 9460c7228 saves content-free failed benchmark reports, tests cleanup/error-detail exclusion, and measures serial/burst CPU/RSS separately. No existing test expectation changed.
kayg commented 2026-10-02 11:31:53 +00:00
Author
Owner
Copy link

Final Files gate: clippy exit 0; tests exit 101 with 147 passed / 1 failed / 1 ignored. Failure is the existing 1,000-write reconciliation storm's unchanged five-minute wall-time bound at lib.rs:5138: writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(()). Its in-flight writer then reports atomic write 802 failed: entry not found during fixture teardown. No assertion or fixture changed. Host load at inspection was 37.73 / 30.60 / 17.00. Treating this as SLOW pending one unchanged focused diagnostic run, not declaring the full gate passed. Server gates continue separately. HDD VM qualification passed under its lock: QD1 125.024992 IOPS / 8.028160 ms p50 / 8.224768 ms p99; QD16 200.879179 IOPS / 100.139008 ms p50 / 104.333312 ms p99; load 0.01 / 0.01 / 0.06.

Final Files gate: clippy exit 0; tests exit 101 with 147 passed / 1 failed / 1 ignored. Failure is the existing 1,000-write reconciliation storm's unchanged five-minute wall-time bound at lib.rs:5138: `writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())`. Its in-flight writer then reports `atomic write 802 failed: entry not found` during fixture teardown. No assertion or fixture changed. Host load at inspection was 37.73 / 30.60 / 17.00. Treating this as SLOW pending one unchanged focused diagnostic run, not declaring the full gate passed. Server gates continue separately. HDD VM qualification passed under its lock: QD1 125.024992 IOPS / 8.028160 ms p50 / 8.224768 ms p99; QD16 200.879179 IOPS / 100.139008 ms p50 / 104.333312 ms p99; load 0.01 / 0.01 / 0.06.
kayg commented 2026-10-02 11:38:21 +00:00
Author
Owner
Copy link

The resumed real local-server protocol round passed (15 credential/malformed/cross-User matrix checks, 120 owned IDs + 120 shared IDs, 0 incomplete headers and 0 missed steps). Revoke, bounded expiry, Plugin disable, content-free reconnect and legacy Files convergence all passed. Runtime 326.59 s; host load 34.32 / 24.27 / 12.92, so local timing is not performance acceptance.

The single unchanged focused Files storm diagnostic also exceeded the same five-minute bound (0 passed / 1 failed / 148 filtered out, 313.39 s). No expectation was changed and no further rerun is planned. This is retained as SLOW host evidence; full Files tests remain failed. Server gates are now running independently.

The resumed real local-server protocol round passed (15 credential/malformed/cross-User matrix checks, 120 owned IDs + 120 shared IDs, 0 incomplete headers and 0 missed steps). Revoke, bounded expiry, Plugin disable, content-free reconnect and legacy Files convergence all passed. Runtime 326.59 s; host load 34.32 / 24.27 / 12.92, so local timing is not performance acceptance. The single unchanged focused Files storm diagnostic also exceeded the same five-minute bound (0 passed / 1 failed / 148 filtered out, 313.39 s). No expectation was changed and no further rerun is planned. This is retained as SLOW host evidence; full Files tests remain failed. Server gates are now running independently.
kayg commented 2026-10-02 11:42:47 +00:00
Author
Owner
Copy link

Locked production stream phase passed; committed evidence at 48ea1ce5c in docs/perf/change-stream.md. Runtime source 2760388ca4; executable SHA-256 fcbc0edc90c94b5d67345e7bdc3b6cd1e8138c9070add9ae6c9dd5b119c58319. Later commits change benchmarks/docs only; server and web source are unchanged. The available older shared executable lacked these routes, so the build host compiled the production executable with reused release dependencies; nothing was compiled on the VM.

Every phase holds flock -w 14400 /root/perf.lock and hdd-emu.sh run-limited (direct-I/O ext4, 8 ms read/write, 200 IOPS, 150 MiB/s). Fifty streams: 50 HTTP 200, 0 incomplete/content-bearing frames; setup median/p95/max 24.2/32.5/33.5 ms. Sixty-second steady phase (241 samples): weighted mean CPU 1.75%, peak CPU 63.88%, mean RSS 145181292 bytes, peak RSS 148299776 bytes. Load inside lock before/after 0.00/0.00/0.00 and 0.13/0.05/0.02. No matching baseline workload exists; no regression ratio is claimed. Protocol completeness is separate from client heap/browser paint/blaze acceptance. The 10,000-write, delta, warm/burst and process-cold phase is still running under its own lock.

Locked production stream phase passed; committed evidence at 48ea1ce5c in docs/perf/change-stream.md. Runtime source 2760388ca4e6920636e4b5582efbb4ff5093c3ad; executable SHA-256 fcbc0edc90c94b5d67345e7bdc3b6cd1e8138c9070add9ae6c9dd5b119c58319. Later commits change benchmarks/docs only; server and web source are unchanged. The available older shared executable lacked these routes, so the build host compiled the production executable with reused release dependencies; nothing was compiled on the VM. Every phase holds flock -w 14400 /root/perf.lock and hdd-emu.sh run-limited (direct-I/O ext4, 8 ms read/write, 200 IOPS, 150 MiB/s). Fifty streams: 50 HTTP 200, 0 incomplete/content-bearing frames; setup median/p95/max 24.2/32.5/33.5 ms. Sixty-second steady phase (241 samples): weighted mean CPU 1.75%, peak CPU 63.88%, mean RSS 145181292 bytes, peak RSS 148299776 bytes. Load inside lock before/after 0.00/0.00/0.00 and 0.13/0.05/0.02. No matching baseline workload exists; no regression ratio is claimed. Protocol completeness is separate from client heap/browser paint/blaze acceptance. The 10,000-write, delta, warm/burst and process-cold phase is still running under its own lock.
kayg commented 2026-10-02 11:50:18 +00:00
Author
Owner
Copy link

The locked HDD 10,000-Tus-write phase was time-boxed before its first 1,000-write drain checkpoint. Last operational inspection saw 110 real fixture files. Its JSON/log are retained as incomplete; no successful delta samples are counted. This phase chiefly measured the existing upload/fsync path, not shared delta reads.

Added --tombstone-journal to the existing sync_feed.py (23959705e, fixture-close fix 4b8f9cadb). It seeds exactly 10,000 deleted IDs in a disposable SQLite journal using the production append/retention trigger at one commit, then verifies every ID and drains the real production endpoint with row/byte caps. This is a synthetic deletion fixture, not 10,000 real API mutations or live headers. The regression proves 10,000 ordered rows and floor 0. This is the shared journal's largest retained data set. Local real API/header/Share correctness already passed the 120-mutation round. The locked journal phase includes five serial/five burst reads and five first reads after process restarts, with resources and timing boundaries separated.

The locked HDD 10,000-Tus-write phase was time-boxed before its first 1,000-write drain checkpoint. Last operational inspection saw 110 real fixture files. Its JSON/log are retained as incomplete; no successful delta samples are counted. This phase chiefly measured the existing upload/fsync path, not shared delta reads. Added --tombstone-journal to the existing sync_feed.py (23959705e, fixture-close fix 4b8f9cadb). It seeds exactly 10,000 deleted IDs in a disposable SQLite journal using the production append/retention trigger at one commit, then verifies every ID and drains the real production endpoint with row/byte caps. This is a synthetic deletion fixture, not 10,000 real API mutations or live headers. The regression proves 10,000 ordered rows and floor 0. This is the shared journal's largest retained data set. Local real API/header/Share correctness already passed the 120-mutation round. The locked journal phase includes five serial/five burst reads and five first reads after process restarts, with resources and timing boundaries separated.
kayg commented 2026-10-02 11:55:23 +00:00
Author
Owner
Copy link

Delivered the shared #668 primitive on job/perf-stream-668; head 7695d237f67f65c51822b32ff6ac8bd129d9cb51. No push, deploy, merge into dev, or issue closure. The one required git fetch origin / git merge origin/dev reported Already up to date. Worktree is clean. Atomic implementation commits from the previous run were retained; resume fixes and evidence are separate commits.

Built

  • Durable monotonic per-User journal, transaction-bound publication, User-local wake signals and bounded retention.
  • /api/v1/changes: signed User/authority/scope-bound cursors, at most 100 scanned rows and 64 KiB, fixed catch-up head, current access checks, deletion tombstones and bounded 410 recovery. /api/v1/changes/events: sequence-only SSE with existing stream limits.
  • Files is the sole server proving adapter. It bridges the existing committed Files feed, resolves current stable IDs/headers, reuses live Share checks, and expires retained authority on revoke/Plugin changes. Existing Files sync feed/SSE contracts remain available.
  • Shared client coordinator: 100 ms coalescing, serial four-page turns, bounded response parsing, adapter-owned retained windows, identity merge seam and session cancellation. Resume commit 2760388ca fences a late rescan after a User switch, with a regression.
  • Generated contract/client types, cross-User route tests, real-server probe and extended shared performance profiles. No second revision cache, snapshot cache, mutation receipt store or production Tab adapter was added.

Files

Core: crates/calternal-plugin/src/changes.rs, src/lib.rs, migrations/changes/{0001_changes.sql,files_bridge.sql}, Cargo.toml; crates/calternal-server/src/{changes.rs,main.rs,wire.rs}; crates/plugins/files/src/{change_stream.rs,lib.rs}.
Client/contracts: apps/web/src/lib/{changeStream.ts,changeStream.test.ts}, contracts/openapi.json, packages/api-client/src/generated.ts.
Validation: bench/{sync_feed.py,sse_storm.py}, tests/adversarial/change_stream.py, tests/perf/{upload_scale.py,test_upload_scale.py,test_change_stream.py}.
Docs/lock: CLAUDE.md, docs/DESIGN.md, docs/perf/change-stream.md, Cargo.lock. Module/function comments were re-read and stale benchmark descriptions were corrected. No existing test expectation changed.

Real local-server round

Passed 15 malformed/credential/cross-User checks, 120 owned identities and 120 shared identities, zero incomplete headers and zero missed steps. Revoke, Plugin disable, bounded expiry, sequence-only reconnect and legacy feed convergence passed. Other-User reads during the storm were 18.72, 17.60, 10.59, 14.96 and 21.25 ms. Runtime 326.59 s at host load 34.32 / 24.27 / 12.92; these are load diagnostics, not performance acceptance.

Locked production/HDD evidence

Runtime source 2760388ca4e6920636e4b5582efbb4ff5093c3ad; executable SHA-256 fcbc0edc90c94b5d67345e7bdc3b6cd1e8138c9070add9ae6c9dd5b119c58319. Later commits modify only benchmarks/docs; server and web code are unchanged. The old shared executable lacked the routes, so the build host compiled the production binary with reused release dependencies; no VM compilation.

Every phase held flock -w 14400 /root/perf.lock and bench/hdd-emu.sh run-limited, releasing the lock between phases. Direct-I/O ext4, 8 ms read/write delay, 200 IOPS and 150 MiB/s caps. Qualification QD1 125.02 IOPS / 8.03 ms median / 8.22 ms p99; QD16 200.88 IOPS / 100.14 ms median / 104.33 ms p99. Load 0.01 / 0.01 / 0.06.

Boundary Samples Median / p95 / max ms
Shared SSE setup 50 24.2 / 32.5 / 33.5
Changed tombstone page 100 1.70 / 2.43 / 3.17
Warm unchanged read 5 1.04 / 1.51 / 1.51
Unchanged burst 5 21.51 / 21.83 / 21.83
First read after process restart 5 2.25 / 2.30 / 2.30

Fifty streams returned HTTP 200 with zero incomplete/content-bearing frames. Sixty-second steady phase: 241 resource samples, weighted mean/peak CPU 1.75% / 63.88%, mean/peak RSS 145181292 / 148299776 bytes. Load before/after 0.00/0.00/0.00 and 0.13/0.05/0.02.

Largest retained journal: 10,000 synthetic deleted identities seeded in a disposable SQLite transaction with the production append trigger. All 10,000 IDs recovered through 100 capped pages; zero missed steps/content-bearing tombstones and zero full-list unchanged refreshes. This measures the production delta route, not 10,000 real API mutations/live headers. Fixture/drain mean/peak CPU 50.49% / 141.19%, mean/peak RSS 77967360 / 83972096 bytes. Serial mean/peak CPU 0% / 0%, RSS 78252032 / 78290944 bytes; burst CPU 250.17% / 250.17%, RSS 80080896 / 81870848 bytes. Short windows have only two to four resource samples; CPU ticks and startup work limit precision. Zero does not mean zero CPU cost. Journal load before/after 0.14/1.59/1.23 and 0.58/1.64/1.25.

Process restarts took 1.33–2.39 s before the read boundary. Disk cache stayed warm. The earlier 10,000-Tus-write phase was time-boxed before its first 1,000-write delta checkpoint; last operational observation was 110 files. It is retained as incomplete and contributes zero successful delta samples. Raw reports are under ignored artifacts/perf-stream-668/results/. docs/perf/baseline.json has no matching workload; no regression ratio or threshold issue is claimed. Durable measured evidence is committed in docs/perf/change-stream.md.

Gates — verbatim excerpts

cargo fmt --check and git diff --check: exit 0, no output. Per-crate clippy: all three exit 0:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 24.64s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 11s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s

cargo test -p calternal-plugin, -p calternal-plugin-files, -p calternal-server respectively:

test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.82s
test result: FAILED. 147 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 329.48s
test result: ok. 109 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 31.12s

Files exit 101: the unchanged 1,000-write/reconciliation test exceeded its five-minute bound at lib.rs:5138:

writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())

One unchanged focused diagnostic also failed the same wall-time bound:

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 148 filtered out; finished in 313.39s

This is retained as SLOW host evidence (observed load up to 53 / 42 / 26), not a passed Files gate. The first run's later missing-entry worker panic followed timeout/fixture teardown; the focused result has only the timeout. No relaxed threshold, changed assertion or further rerun.

bun run check exit 0:

$ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
User browser caches use userStorage; only documented device/public-link exceptions remain.
Text sizes and UI shape values use shared role tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/perf-stream-668/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run test exit 0, summary verbatim:

 Test Files  154 passed (154)
      Tests  1063 passed (1063)
   Start at  13:16:02
   Duration  62.92s (transform 53%, environment 20%, import 13%, tests 10%, setup 3%)

Ten coordinator tests passed, including late page and late rescan cancellation. API client: 18 tests passed. Python: five sampler tests and two profile tests passed (including readiness failure secrecy/cleanup and exact 10,000-row retention). Production web/server builds passed; regeneration of the release API contract made no diff. Full gate logs remain in ignored artifacts. Existing jsdom scrollTo diagnostics remain.

Decisions

100 scanned rows / 64 KiB envelope; 10,000 rows per User / 14 days; fixed catch-up high-water mark; four pages per client turn; 100 ms coalescing and one-second server safety checks. Expiry/access change returns 410 and clears retained authority before bounded visible-window rescans. Unrestricted account/data sessions only; App Password/resource-scoped clients keep their existing protocols until scope adapters exist. Files is the server proving adoption; other producers/views remain with #669–#676/#701. Process-cold reads keep disk cache warm. Use the maximum retained deletion fixture to isolate worst-case delta work; retain the upload-bound run as incomplete.

UX gaps closed

Late rescan completion cannot publish an old User's page to new subscribers. Duplicate wake-ups share one drain; failed reads retain current data, failed windows retry, and access expiry clears old authority before recovery. Unchanged revisions preserve identity.

UX gaps left / known gaps

Production Tab producers and retained-view adapters/poll removal await the named adoption issues. No visual layout changed, so no screenshot set was generated. Browser paint/blaze completeness, client heap, kernel-cold reads and 10,000 live headers are not established by protocol timing. The Files storm gate remains failed on its unchanged timeout. No real-Mac check was requested and the offline Mac VM was not accessed.

Cleanup: web build output, SvelteKit output and Python bytecode removed. Cargo cleanup output will be appended verbatim below.

     Removed 26572 files, 14.8GiB total
Delivered the shared #668 primitive on `job/perf-stream-668`; head `7695d237f67f65c51822b32ff6ac8bd129d9cb51`. No push, deploy, merge into dev, or issue closure. The one required `git fetch origin` / `git merge origin/dev` reported `Already up to date.` Worktree is clean. Atomic implementation commits from the previous run were retained; resume fixes and evidence are separate commits. **Built** - Durable monotonic per-User journal, transaction-bound publication, User-local wake signals and bounded retention. - `/api/v1/changes`: signed User/authority/scope-bound cursors, at most 100 scanned rows and 64 KiB, fixed catch-up head, current access checks, deletion tombstones and bounded 410 recovery. `/api/v1/changes/events`: sequence-only SSE with existing stream limits. - Files is the sole server proving adapter. It bridges the existing committed Files feed, resolves current stable IDs/headers, reuses live Share checks, and expires retained authority on revoke/Plugin changes. Existing Files sync feed/SSE contracts remain available. - Shared client coordinator: 100 ms coalescing, serial four-page turns, bounded response parsing, adapter-owned retained windows, identity merge seam and session cancellation. Resume commit 2760388ca fences a late rescan after a User switch, with a regression. - Generated contract/client types, cross-User route tests, real-server probe and extended shared performance profiles. No second revision cache, snapshot cache, mutation receipt store or production Tab adapter was added. **Files** Core: `crates/calternal-plugin/src/changes.rs`, `src/lib.rs`, `migrations/changes/{0001_changes.sql,files_bridge.sql}`, `Cargo.toml`; `crates/calternal-server/src/{changes.rs,main.rs,wire.rs}`; `crates/plugins/files/src/{change_stream.rs,lib.rs}`. Client/contracts: `apps/web/src/lib/{changeStream.ts,changeStream.test.ts}`, `contracts/openapi.json`, `packages/api-client/src/generated.ts`. Validation: `bench/{sync_feed.py,sse_storm.py}`, `tests/adversarial/change_stream.py`, `tests/perf/{upload_scale.py,test_upload_scale.py,test_change_stream.py}`. Docs/lock: `CLAUDE.md`, `docs/DESIGN.md`, `docs/perf/change-stream.md`, `Cargo.lock`. Module/function comments were re-read and stale benchmark descriptions were corrected. No existing test expectation changed. **Real local-server round** Passed 15 malformed/credential/cross-User checks, 120 owned identities and 120 shared identities, zero incomplete headers and zero missed steps. Revoke, Plugin disable, bounded expiry, sequence-only reconnect and legacy feed convergence passed. Other-User reads during the storm were 18.72, 17.60, 10.59, 14.96 and 21.25 ms. Runtime 326.59 s at host load 34.32 / 24.27 / 12.92; these are load diagnostics, not performance acceptance. **Locked production/HDD evidence** Runtime source `2760388ca4e6920636e4b5582efbb4ff5093c3ad`; executable SHA-256 `fcbc0edc90c94b5d67345e7bdc3b6cd1e8138c9070add9ae6c9dd5b119c58319`. Later commits modify only benchmarks/docs; server and web code are unchanged. The old shared executable lacked the routes, so the build host compiled the production binary with reused release dependencies; no VM compilation. Every phase held `flock -w 14400 /root/perf.lock` and `bench/hdd-emu.sh run-limited`, releasing the lock between phases. Direct-I/O ext4, 8 ms read/write delay, 200 IOPS and 150 MiB/s caps. Qualification QD1 125.02 IOPS / 8.03 ms median / 8.22 ms p99; QD16 200.88 IOPS / 100.14 ms median / 104.33 ms p99. Load 0.01 / 0.01 / 0.06. | Boundary | Samples | Median / p95 / max ms | | --- | ---: | --- | | Shared SSE setup | 50 | 24.2 / 32.5 / 33.5 | | Changed tombstone page | 100 | 1.70 / 2.43 / 3.17 | | Warm unchanged read | 5 | 1.04 / 1.51 / 1.51 | | Unchanged burst | 5 | 21.51 / 21.83 / 21.83 | | First read after process restart | 5 | 2.25 / 2.30 / 2.30 | Fifty streams returned HTTP 200 with zero incomplete/content-bearing frames. Sixty-second steady phase: 241 resource samples, weighted mean/peak CPU 1.75% / 63.88%, mean/peak RSS 145181292 / 148299776 bytes. Load before/after 0.00/0.00/0.00 and 0.13/0.05/0.02. Largest retained journal: 10,000 synthetic deleted identities seeded in a disposable SQLite transaction with the production append trigger. All 10,000 IDs recovered through 100 capped pages; zero missed steps/content-bearing tombstones and zero full-list unchanged refreshes. This measures the production delta route, not 10,000 real API mutations/live headers. Fixture/drain mean/peak CPU 50.49% / 141.19%, mean/peak RSS 77967360 / 83972096 bytes. Serial mean/peak CPU 0% / 0%, RSS 78252032 / 78290944 bytes; burst CPU 250.17% / 250.17%, RSS 80080896 / 81870848 bytes. Short windows have only two to four resource samples; CPU ticks and startup work limit precision. Zero does not mean zero CPU cost. Journal load before/after 0.14/1.59/1.23 and 0.58/1.64/1.25. Process restarts took 1.33–2.39 s before the read boundary. Disk cache stayed warm. The earlier 10,000-Tus-write phase was time-boxed before its first 1,000-write delta checkpoint; last operational observation was 110 files. It is retained as incomplete and contributes zero successful delta samples. Raw reports are under ignored `artifacts/perf-stream-668/results/`. `docs/perf/baseline.json` has no matching workload; no regression ratio or threshold issue is claimed. Durable measured evidence is committed in `docs/perf/change-stream.md`. **Gates — verbatim excerpts** `cargo fmt --check` and `git diff --check`: exit 0, no output. Per-crate clippy: all three exit 0: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 24.64s Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 11s Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s ``` `cargo test -p calternal-plugin`, `-p calternal-plugin-files`, `-p calternal-server` respectively: ```text test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.82s test result: FAILED. 147 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 329.48s test result: ok. 109 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 31.12s ``` Files exit 101: the unchanged 1,000-write/reconciliation test exceeded its five-minute bound at `lib.rs:5138`: ```text writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(()) ``` One unchanged focused diagnostic also failed the same wall-time bound: ```text test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 148 filtered out; finished in 313.39s ``` This is retained as SLOW host evidence (observed load up to 53 / 42 / 26), not a passed Files gate. The first run's later missing-entry worker panic followed timeout/fixture teardown; the focused result has only the timeout. No relaxed threshold, changed assertion or further rerun. `bun run check` exit 0: ```text $ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json User browser caches use userStorage; only documented device/public-link exceptions remain. Text sizes and UI shape values use shared role tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/perf-stream-668/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run test` exit 0, summary verbatim: ```text Test Files 154 passed (154) Tests 1063 passed (1063) Start at 13:16:02 Duration 62.92s (transform 53%, environment 20%, import 13%, tests 10%, setup 3%) ``` Ten coordinator tests passed, including late page and late rescan cancellation. API client: 18 tests passed. Python: five sampler tests and two profile tests passed (including readiness failure secrecy/cleanup and exact 10,000-row retention). Production web/server builds passed; regeneration of the release API contract made no diff. Full gate logs remain in ignored artifacts. Existing jsdom scrollTo diagnostics remain. **Decisions** 100 scanned rows / 64 KiB envelope; 10,000 rows per User / 14 days; fixed catch-up high-water mark; four pages per client turn; 100 ms coalescing and one-second server safety checks. Expiry/access change returns 410 and clears retained authority before bounded visible-window rescans. Unrestricted account/data sessions only; App Password/resource-scoped clients keep their existing protocols until scope adapters exist. Files is the server proving adoption; other producers/views remain with #669–#676/#701. Process-cold reads keep disk cache warm. Use the maximum retained deletion fixture to isolate worst-case delta work; retain the upload-bound run as incomplete. **UX gaps closed** Late rescan completion cannot publish an old User's page to new subscribers. Duplicate wake-ups share one drain; failed reads retain current data, failed windows retry, and access expiry clears old authority before recovery. Unchanged revisions preserve identity. **UX gaps left / known gaps** Production Tab producers and retained-view adapters/poll removal await the named adoption issues. No visual layout changed, so no screenshot set was generated. Browser paint/blaze completeness, client heap, kernel-cold reads and 10,000 live headers are not established by protocol timing. The Files storm gate remains failed on its unchanged timeout. No real-Mac check was requested and the offline Mac VM was not accessed. Cleanup: web build output, SvelteKit output and Python bytecode removed. Cargo cleanup output will be appended verbatim below. ```text Removed 26572 files, 14.8GiB total ```
kayg commented 2026-10-02 13:14:18 +00:00
Author
Owner
Copy link

#663 sync audit source review: base c4a61e8cf and queued job/perf-stream-668 7695d237f. Keep #668 as sole shared stream owner.

Confirmed strengths: delta scans at most 100 rows and budgets 64 KiB; signed User/scope/fixed-head cursors; count/age retention; current projection access checks; targeted wakeups; one serial client drain, 100 ms combining and four pages per turn; expiry performs bounded visible-window rescans. Preserve these.

Remaining work and evidence:

  1. Adoption is pending. git grep subscribeChanges finds only its export in changeStream.ts:192; the Files adapter is the only plugin change_projection implementation. Legacy Files SSE and polls remain. Missing producers/adopters should stay with #669–#676/#701 and #668's producer contract, not independent streams. Notifications also need an adapter: current inbox.svelte.ts:25 polls at 60,000 ms and its module docs say there is no notification event stream. Web Push is not a general SSE replacement.
  2. Legacy Files lib.rs:1653 has one process-global wake signal. Each change wakes every User's stream; each then queries its own rows (:3871). Reconnect also replays 256-path pages and live.ts:33 refreshes every listener on open. The new targeted stream fixes this only once adoption/legacy compatibility behavior is explicit. Avoid simultaneous old and new UI streams.
  3. New SSE calternal-server/src/changes.rs:247 reads the User head and calls authority on each one-second safety tick per Installation. authority does a User query, registry scope work and durable epoch query. Thus S idle streams cause about 3S reader queries/second, plus scope work. This is a reasoned operation count; no CPU measurement here. Consider one supervised tick/head+authority observation per User, fan out results, retain current access checks and bounded stale detection.
  4. Deploy reconnects remain synchronized: Notes collab.ts:221 uses fixed 400 ms exponential delays capped at 10 s without jitter. changeStream.ts retryMs doubles to 30 s without jitter; EventSource uses browser reconnect behavior. Keep exponential caps, add jitter/coordinated recovery where owned, and test many Installations disconnecting at once. Do not delay already available content.

Suggested test: many Users, several Installations each, one User changes one item. Count Index queries and delta requests, bound outstanding drains, and show unchanged objects retain identity. Repeat with a deployment reconnect and a revoked Share. These are performance findings only; no authorization exploit or crash was demonstrated.

#663 sync audit source review: base c4a61e8cf and queued job/perf-stream-668 7695d237f. Keep #668 as sole shared stream owner. Confirmed strengths: delta scans at most 100 rows and budgets 64 KiB; signed User/scope/fixed-head cursors; count/age retention; current projection access checks; targeted wakeups; one serial client drain, 100 ms combining and four pages per turn; expiry performs bounded visible-window rescans. Preserve these. Remaining work and evidence: 1. Adoption is pending. `git grep subscribeChanges` finds only its export in changeStream.ts:192; the Files adapter is the only plugin change_projection implementation. Legacy Files SSE and polls remain. Missing producers/adopters should stay with #669–#676/#701 and #668's producer contract, not independent streams. Notifications also need an adapter: current inbox.svelte.ts:25 polls at 60,000 ms and its module docs say there is no notification event stream. Web Push is not a general SSE replacement. 2. Legacy Files `lib.rs:1653` has one process-global wake signal. Each change wakes every User's stream; each then queries its own rows (:3871). Reconnect also replays 256-path pages and live.ts:33 refreshes every listener on open. The new targeted stream fixes this only once adoption/legacy compatibility behavior is explicit. Avoid simultaneous old and new UI streams. 3. New SSE `calternal-server/src/changes.rs:247` reads the User head and calls authority on each one-second safety tick per Installation. authority does a User query, registry scope work and durable epoch query. Thus S idle streams cause about 3S reader queries/second, plus scope work. This is a reasoned operation count; no CPU measurement here. Consider one supervised tick/head+authority observation per User, fan out results, retain current access checks and bounded stale detection. 4. Deploy reconnects remain synchronized: Notes collab.ts:221 uses fixed 400 ms exponential delays capped at 10 s without jitter. changeStream.ts retryMs doubles to 30 s without jitter; EventSource uses browser reconnect behavior. Keep exponential caps, add jitter/coordinated recovery where owned, and test many Installations disconnecting at once. Do not delay already available content. Suggested test: many Users, several Installations each, one User changes one item. Count Index queries and delta requests, bound outstanding drains, and show unchanged objects retain identity. Repeat with a deployment reconnect and a revoked Share. These are performance findings only; no authorization exploit or crash was demonstrated.
kayg commented 2026-10-02 13:17:43 +00:00
Author
Owner
Copy link

Memory/CPU audit #663: Files uses one Instance-wide unit signal (crates/plugins/files/src/lib.rs:3846 and :3975 on dev c4a61e8cf). Both Files SSE paths query the authenticated User feed after every global wake, including another User’s write (:3873 and :3979). One isolated change can therefore issue up to one feed query per open stream: 1,000 streams → up to 1,000 queries. This is a reasoned fan-out count, not measured throughput; bursts can coalesce or lag. Please make replacement notifications User-specific and coalesce them. Regression: two Users, many idle streams for A, one change for B; A must neither query nor receive an unchanged cursor wake. Shared broadcast rings themselves do not duplicate their entire queue per receiver.

Memory/CPU audit #663: Files uses one Instance-wide unit signal (crates/plugins/files/src/lib.rs:3846 and :3975 on dev c4a61e8cf). Both Files SSE paths query the authenticated User feed after every global wake, including another User’s write (:3873 and :3979). One isolated change can therefore issue up to one feed query per open stream: 1,000 streams → up to 1,000 queries. This is a reasoned fan-out count, not measured throughput; bursts can coalesce or lag. Please make replacement notifications User-specific and coalesce them. Regression: two Users, many idle streams for A, one change for B; A must neither query nor receive an unchanged cursor wake. Shared broadcast rings themselves do not duplicate their entire queue per receiver.
kayg commented 2026-10-02 13:26:13 +00:00
Author
Owner
Copy link

Round 2 started on job/perf-stream-668. Head: 7695d237f6. Fetched origin/dev: c4a61e8cf0.

The unchanged storm fixture applies Files, Notes, Tags and plugin registry migrations. It does not apply changes::migrations() or install_files_bridge(). Thus this failure does not execute the journal append trigger. Build two test executables on the build host, then run the exact test three times per revision under the perf VM lock. No timeout increase.

Round 2 started on job/perf-stream-668. Head: 7695d237f67f65c51822b32ff6ac8bd129d9cb51. Fetched origin/dev: c4a61e8cf090170f35b1bed3350d9de20c83ecd5. The unchanged storm fixture applies Files, Notes, Tags and plugin registry migrations. It does not apply changes::migrations() or install_files_bridge(). Thus this failure does not execute the journal append trigger. Build two test executables on the build host, then run the exact test three times per revision under the perf VM lock. No timeout increase.
kayg commented 2026-10-02 13:30:59 +00:00
Author
Owner
Copy link

Source finding for Round 2 (#668): the unchanged reconciliation test does not apply calternal_plugin::changes::migrations() or install_files_bridge(), so app_changes_files never runs. In production, reconcile_folder_locked collects changed records and calls write_records. write_records uses 64-row INSERT statements in one SQLite transaction for multiple records. app_changes_files is an AFTER INSERT trigger of the legacy feed; it executes inside that same transaction and issues no independent commit or filesystem fsync. There is no added per-file journal commit to batch here. The only added code reached by the unchanged test is the post-publication wake_user lookup.

A new locked comparison profile is committed at f395d894d. bash -n passed. A disposable executable fixture on the VM proved that all six samples remain when every head sample fails (profile exit 1, six time files). Test builds are running on the shared build host; VM binaries have not been measured yet.

Source finding for Round 2 (#668): the unchanged reconciliation test does not apply calternal_plugin::changes::migrations() or install_files_bridge(), so app_changes_files never runs. In production, reconcile_folder_locked collects changed records and calls write_records. write_records uses 64-row INSERT statements in one SQLite transaction for multiple records. app_changes_files is an AFTER INSERT trigger of the legacy feed; it executes inside that same transaction and issues no independent commit or filesystem fsync. There is no added per-file journal commit to batch here. The only added code reached by the unchanged test is the post-publication wake_user lookup. A new locked comparison profile is committed at f395d894d. bash -n passed. A disposable executable fixture on the VM proved that all six samples remain when every head sample fails (profile exit 1, six time files). Test builds are running on the shared build host; VM binaries have not been measured yet.
kayg commented 2026-10-02 13:57:51 +00:00
Author
Owner
Copy link

Round 2 build evidence: build-host load reached 124.13 / 111.22 / 92.32 during the clean test build. The shared compiler cache reported 17.13% Rust hits. The perf VM is reachable; measurements have not started. To reduce build work within the two-hour limit, stopped the cold build and copied only matching external pure Rust library cache artifacts from the read-only gates cache into this job's required target directory. calternal crates and libraries with native build scripts are excluded; Cargo validates freshness when it restarts. No target-directory override, dependency change or VM compilation. The six-run profile and its failure-retention check are committed.

Round 2 build evidence: build-host load reached 124.13 / 111.22 / 92.32 during the clean test build. The shared compiler cache reported 17.13% Rust hits. The perf VM is reachable; measurements have not started. To reduce build work within the two-hour limit, stopped the cold build and copied only matching external pure Rust library cache artifacts from the read-only gates cache into this job's required target directory. calternal crates and libraries with native build scripts are excluded; Cargo validates freshness when it restarts. No target-directory override, dependency change or VM compilation. The six-run profile and its failure-retention check are committed.
kayg commented 2026-10-02 14:53:41 +00:00
Author
Owner
Copy link

Round 2 progress at cb5dcfcb6: the comparison profile now checks GNU time before it creates results. The build host has no /usr/bin/time; the missing-tool check exits 2 and creates no result directory. On the locked perf VM, its disposable failure fixture still returns exit 1 and retains all six samples. Source and reproduction documentation are committed at 869ffccf7.

The clean build is still compiling dependencies; no real VM samples exist yet. To avoid recompiling unchanged local crates under a second source path, the comparison runner will save the head executable, build origin/dev briefly in this same isolated worktree, and restore the job branch in a finally block. Both source revisions and binary hashes remain separate. No timeout, Rust source or test expectation changed. The two-hour cap remains in force.

Round 2 progress at cb5dcfcb6: the comparison profile now checks GNU time before it creates results. The build host has no /usr/bin/time; the missing-tool check exits 2 and creates no result directory. On the locked perf VM, its disposable failure fixture still returns exit 1 and retains all six samples. Source and reproduction documentation are committed at 869ffccf7. The clean build is still compiling dependencies; no real VM samples exist yet. To avoid recompiling unchanged local crates under a second source path, the comparison runner will save the head executable, build origin/dev briefly in this same isolated worktree, and restore the job branch in a finally block. Both source revisions and binary hashes remain separate. No timeout, Rust source or test expectation changed. The two-hour cap remains in force.
kayg commented 2026-10-02 15:21:18 +00:00
Author
Owner
Copy link

Round 2 is incomplete. The requested quiet VM comparison did not run within the two-hour cap. No timings or runtime regression result is claimed.

Built and committed:

  • bench/files-reconcile-compare.sh: alternate the unchanged storm test three times per revision, under one caller-held perf VM lock. Record binary hashes, source revisions, load, wall/CPU time, peak RSS and every failed sample.
  • The profile checks GNU time before it creates results. On this host, the missing-tool check returned 2 and created no result directory. On the locked VM, a disposable failure fixture returned 1 and retained all six samples.
  • docs/perf/change-stream.md: source finding, reproduction method and incomplete-run evidence.

Head: 3cfc8a49bd. Branch: job/perf-stream-668. Atomic commits: f395d894d, 869ffccf7, cb5dcfcb6, 3cfc8a49b. Worktree is clean. No push, deployment, merge into dev or issue closure. The one required fetch and merge of origin/dev reported Already up to date. No Rust source, dependency, timeout or existing test expectation changed in this round.

Source evidence:
The unchanged storm fixture does not call changes::migrations() or install_files_bridge(), so it cannot execute the app journal append trigger. The added publication call it reaches is wake_user. Production reconciliation already writes changed rows in 64-row statements inside one folder transaction. app_changes_files runs inside that same transaction and adds no per-file commit or filesystem sync. A journal batching fix is not supported by this test's failure.

Required timings:

Revision Completed real samples Timings
origin/dev c4a61e8cf0 0/3 Not available
head cb5dcfcb6d (same Rust tree as final head) 0/3 Not available

The clean Files lib-test build was stopped in dependency compilation; it never produced the executable. Observed build-host load reached 124.13 / 111.22 / 92.32. Matching external pure Rust caches were copied read-only into the required job target directory; Cargo still validated freshness. No target override or VM compilation. The planned temporary baseline checkout was never reached, and the job branch remains checked out. Build records remain in artifacts/round2/.

Verification:

  • cargo fmt --check: exit 0, no output.
  • bash -n bench/files-reconcile-compare.sh: exit 0, no output.
  • Locked VM profile fixture output, verbatim:
failure-retention-check-passed
  • cargo clippy -p calternal-plugin-files --all-targets -- -D warnings: not run this round.
  • cargo test -p calternal-plugin-files: not run this round. The --lib --no-run prerequisite was interrupted for the time cap; it is not a passing gate.
  • cargo clean: exit 0. Output verbatim:
     Removed 4129 files, 1.8GiB total
  • Web build output was removed. Web gates were not repeated; no web files changed.

Last build output, verbatim (incomplete; no build-success claim):

   Compiling rand_chacha v0.9.0
   Compiling rfc6979 v0.4.0
   Compiling num-bigint v0.4.8
   Compiling hashify v0.2.9
   Compiling servo_arc v0.3.0
   Compiling derive_more v0.99.20
   Compiling openssl-macros v0.1.1
   Compiling num-bigint-dig v0.8.6
   Compiling webauthn-attestation-ca v0.5.5
   Compiling linkme v0.3.37
   Compiling ecdsa v0.16.9
   Compiling der-parser v9.0.0

Known gaps: all six real VM samples, runtime impact of the added wake lookup, and the full Files clippy/test gates remain unverified. The unchanged five-minute timeout has not been raised. The fixture without the bridge cannot measure production journal cost. No UI work was part of this round.

Decisions: use alternating revision pairs, fresh fixtures, the same debug profile and the native VM filesystem for this unit-test comparison. Record whole-process timing, including setup and cleanup. Keep failed samples. Do not make an unproven Rust fix or present build-host contention as measured VM evidence.

Round 2 is incomplete. The requested quiet VM comparison did not run within the two-hour cap. No timings or runtime regression result is claimed. Built and committed: - bench/files-reconcile-compare.sh: alternate the unchanged storm test three times per revision, under one caller-held perf VM lock. Record binary hashes, source revisions, load, wall/CPU time, peak RSS and every failed sample. - The profile checks GNU time before it creates results. On this host, the missing-tool check returned 2 and created no result directory. On the locked VM, a disposable failure fixture returned 1 and retained all six samples. - docs/perf/change-stream.md: source finding, reproduction method and incomplete-run evidence. Head: 3cfc8a49bd37c537ef83e0251b5102c2b27d3a81. Branch: job/perf-stream-668. Atomic commits: f395d894d, 869ffccf7, cb5dcfcb6, 3cfc8a49b. Worktree is clean. No push, deployment, merge into dev or issue closure. The one required fetch and merge of origin/dev reported `Already up to date.` No Rust source, dependency, timeout or existing test expectation changed in this round. Source evidence: The unchanged storm fixture does not call changes::migrations() or install_files_bridge(), so it cannot execute the app journal append trigger. The added publication call it reaches is wake_user. Production reconciliation already writes changed rows in 64-row statements inside one folder transaction. app_changes_files runs inside that same transaction and adds no per-file commit or filesystem sync. A journal batching fix is not supported by this test's failure. Required timings: | Revision | Completed real samples | Timings | | --- | --- | --- | | origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5 | 0/3 | Not available | | head cb5dcfcb6dbb9bc9840031f538dd55062ef8a460 (same Rust tree as final head) | 0/3 | Not available | The clean Files lib-test build was stopped in dependency compilation; it never produced the executable. Observed build-host load reached 124.13 / 111.22 / 92.32. Matching external pure Rust caches were copied read-only into the required job target directory; Cargo still validated freshness. No target override or VM compilation. The planned temporary baseline checkout was never reached, and the job branch remains checked out. Build records remain in artifacts/round2/. Verification: - cargo fmt --check: exit 0, no output. - bash -n bench/files-reconcile-compare.sh: exit 0, no output. - Locked VM profile fixture output, verbatim: ``` failure-retention-check-passed ``` - cargo clippy -p calternal-plugin-files --all-targets -- -D warnings: not run this round. - cargo test -p calternal-plugin-files: not run this round. The --lib --no-run prerequisite was interrupted for the time cap; it is not a passing gate. - cargo clean: exit 0. Output verbatim: ``` Removed 4129 files, 1.8GiB total ``` - Web build output was removed. Web gates were not repeated; no web files changed. Last build output, verbatim (incomplete; no build-success claim): ``` Compiling rand_chacha v0.9.0 Compiling rfc6979 v0.4.0 Compiling num-bigint v0.4.8 Compiling hashify v0.2.9 Compiling servo_arc v0.3.0 Compiling derive_more v0.99.20 Compiling openssl-macros v0.1.1 Compiling num-bigint-dig v0.8.6 Compiling webauthn-attestation-ca v0.5.5 Compiling linkme v0.3.37 Compiling ecdsa v0.16.9 Compiling der-parser v9.0.0 ``` Known gaps: all six real VM samples, runtime impact of the added wake lookup, and the full Files clippy/test gates remain unverified. The unchanged five-minute timeout has not been raised. The fixture without the bridge cannot measure production journal cost. No UI work was part of this round. Decisions: use alternating revision pairs, fresh fixtures, the same debug profile and the native VM filesystem for this unit-test comparison. Record whole-process timing, including setup and cleanup. Keep failed samples. Do not make an unproven Rust fix or present build-host contention as measured VM evidence.
kayg commented 2026-10-02 18:08:01 +00:00
Author
Owner
Copy link

Round 3 started on job/perf-stream-668 at 3cfc8a49bd. Merged origin/dev c4faf184df; merge head 407f25da08. Read CLAUDE.md, CONTEXT.md, DESIGN and issue history. Build-host load 47.42/45.57/51.07; perf VM load 0.00/0.00/0.25. Proceeding without a quiet-host wait. Build locked Files release lib-test binaries once per revision, with identical debug settings. Head fixture will install the journal bridge and keep an active User wake subscription; origin/dev has no shared journal. Alternate three samples each under the VM lock, retaining failures and process CPU/RSS. Then run Files gates. No push/deploy/merge into dev.

Round 3 started on job/perf-stream-668 at 3cfc8a49bd37c537ef83e0251b5102c2b27d3a81. Merged origin/dev c4faf184df726a9375ae0c13bdfb6018ac2cf57e; merge head 407f25da08f7f527bee08465c37c2928b4fd8a2b. Read CLAUDE.md, CONTEXT.md, DESIGN and issue history. Build-host load 47.42/45.57/51.07; perf VM load 0.00/0.00/0.25. Proceeding without a quiet-host wait. Build locked Files release lib-test binaries once per revision, with identical debug settings. Head fixture will install the journal bridge and keep an active User wake subscription; origin/dev has no shared journal. Alternate three samples each under the VM lock, retaining failures and process CPU/RSS. Then run Files gates. No push/deploy/merge into dev.
kayg commented 2026-10-02 18:58:52 +00:00
Author
Owner
Copy link

Round 3 checkpoint: origin/dev release build has progressed through calternal-db, calternal-fs, calternal-api and calternal-plugin, and is compiling image/protocol dependencies. No build error or real VM sample yet. One build attempt remains running; no restart or target override. Build-host load at 18:56 UTC: 29.66/31.23/36.87; target output at 18:58 UTC: 1.8 GiB. Job branch contains merge 407f25da08. Baseline is temporarily detached in the same source path, with automatic restoration to the job branch on exit; source tree is unchanged while compiling. Head fixture patch is prepared in ignored artifacts, not applied yet.

Head fixture will apply changes::migrations(), install_files_bridge(), register an active User subscription before publication, and assert journal/feed row parity and wake delivery. Existing 1,000 writes, four scans, five-minute timeout and test expectations remain. The baseline has no changes module and uses its original fixture. The issue body/history has no numeric reconciliation slowdown threshold; report pass/fail against the existing five-minute bound and explicit timing ratios.

Round 3 checkpoint: origin/dev release build has progressed through calternal-db, calternal-fs, calternal-api and calternal-plugin, and is compiling image/protocol dependencies. No build error or real VM sample yet. One build attempt remains running; no restart or target override. Build-host load at 18:56 UTC: 29.66/31.23/36.87; target output at 18:58 UTC: 1.8 GiB. Job branch contains merge 407f25da08f7f527bee08465c37c2928b4fd8a2b. Baseline is temporarily detached in the same source path, with automatic restoration to the job branch on exit; source tree is unchanged while compiling. Head fixture patch is prepared in ignored artifacts, not applied yet. Head fixture will apply changes::migrations(), install_files_bridge(), register an active User subscription before publication, and assert journal/feed row parity and wake delivery. Existing 1,000 writes, four scans, five-minute timeout and test expectations remain. The baseline has no changes module and uses its original fixture. The issue body/history has no numeric reconciliation slowdown threshold; report pass/fail against the existing five-minute bound and explicit timing ratios.
kayg commented 2026-10-02 19:39:00 +00:00
Author
Owner
Copy link

Round 3 real fixture finding: baseline release build passed (70m15s), initial head release build passed (12m27s). Locked VM comparison completed three alternating samples each. Baseline passed all three; head completed the original storm work but failed all three added wake assertions. The first VM load inside the lock was 8.34/8.81/7.05, so this is not described as quiet-host evidence. All failed samples are retained in artifacts/round3/failed-wake-fixture/.

The earlier source claim that the unchanged storm reaches wake_user was incorrect. FilesState::adopt_change commits Index/feed rows but does not call publish_inner; installing the app journal bridge adds trigger writes, not a wake. The regression exposed this fixture gap, not a production wake failure. Corrected only the fixture: call wake_user("u") after each watcher adoption commits, with an active subscription registered before writes. This follows the shared producer contract and adds no files_events write or extra transaction. Existing counts, temporary-path checks and five-minute bound stay unchanged. Production behavior is unchanged.

A second head release build is required by this focused regression fix; baseline is reused unchanged. The initial six-run comparison remains failed and will not be counted as successful samples. Clippy for the first fixture passed (exit 0); the obsolete debug test build was stopped before tests (exit 130). Corrected fixture gates and locked comparison will run once after rebuilding.

Round 3 real fixture finding: baseline release build passed (70m15s), initial head release build passed (12m27s). Locked VM comparison completed three alternating samples each. Baseline passed all three; head completed the original storm work but failed all three added wake assertions. The first VM load inside the lock was 8.34/8.81/7.05, so this is not described as quiet-host evidence. All failed samples are retained in artifacts/round3/failed-wake-fixture/. The earlier source claim that the unchanged storm reaches wake_user was incorrect. FilesState::adopt_change commits Index/feed rows but does not call publish_inner; installing the app journal bridge adds trigger writes, not a wake. The regression exposed this fixture gap, not a production wake failure. Corrected only the fixture: call wake_user("u") after each watcher adoption commits, with an active subscription registered before writes. This follows the shared producer contract and adds no files_events write or extra transaction. Existing counts, temporary-path checks and five-minute bound stay unchanged. Production behavior is unchanged. A second head release build is required by this focused regression fix; baseline is reused unchanged. The initial six-run comparison remains failed and will not be counted as successful samples. Clippy for the first fixture passed (exit 0); the obsolete debug test build was stopped before tests (exit 130). Corrected fixture gates and locked comparison will run once after rebuilding.
kayg commented 2026-10-02 19:55:26 +00:00
Author
Owner
Copy link

Corrected locked comparison completed: all three origin/dev samples and all three head samples passed the original storm checks plus head journal/wake assertions. Both source revisions use release, line-table debug information, locked dependencies, incremental off and four build jobs; no compilation on the VM.

Revision Whole-process wall samples (s) Median / p95 / max (s) Median user+system CPU (s) Peak RSS (bytes)
origin/dev c4faf184d 37.34, 38.56, 36.72 37.34 / 38.56 / 38.56 6.02 34783232
Corrected head fixture 42.11, 38.92, 45.80 42.11 / 45.80 / 45.80 6.31 35340288

Head median wall +12.8%, p95 +18.8%, median CPU +4.8%, peak RSS +1.6%. With n=3, nearest-rank p95 is max. The unchanged five-minute bound passes in both revisions; the prior timeout is not reproduced. This is a measured wall-time increase, not a zero-overhead result.

Lock was held across all six alternating runs, then released. Corrected pre-run load: dev 10.03/8.39/7.15, 4.38/6.94/6.73, 2.92/5.90/6.39; head 6.36/7.65/6.94, 3.29/6.34/6.54, 2.40/5.40/6.19. This is not quiet-VM qualification. No matching storm workload exists in docs/perf/baseline.json; #668 has no numeric storm slowdown rule. Native VM filesystem, fresh fixture per sample, warm disk cache, no HDD emulator, no HTTP/SSE/browser timing. Keep all failed/slow samples; initial failed fixture is retained separately.

Corrected executable SHA-256: 143bfa670b61016d71fadddc509da33cae25f98dbe17ee9408e19eb41b669496. Baseline SHA-256: cf7ef6ee6859096686dad1587258062775320432df77159d8b6d1df7daece667. Raw summary retains both attempts and source/fixture identities. Corrected fmt and Files clippy passed; local Files test build is running.

Corrected locked comparison completed: all three origin/dev samples and all three head samples passed the original storm checks plus head journal/wake assertions. Both source revisions use release, line-table debug information, locked dependencies, incremental off and four build jobs; no compilation on the VM. | Revision | Whole-process wall samples (s) | Median / p95 / max (s) | Median user+system CPU (s) | Peak RSS (bytes) | | --- | --- | --- | --- | --- | | origin/dev c4faf184d | 37.34, 38.56, 36.72 | 37.34 / 38.56 / 38.56 | 6.02 | 34783232 | | Corrected head fixture | 42.11, 38.92, 45.80 | 42.11 / 45.80 / 45.80 | 6.31 | 35340288 | Head median wall +12.8%, p95 +18.8%, median CPU +4.8%, peak RSS +1.6%. With n=3, nearest-rank p95 is max. The unchanged five-minute bound passes in both revisions; the prior timeout is not reproduced. This is a measured wall-time increase, not a zero-overhead result. Lock was held across all six alternating runs, then released. Corrected pre-run load: dev 10.03/8.39/7.15, 4.38/6.94/6.73, 2.92/5.90/6.39; head 6.36/7.65/6.94, 3.29/6.34/6.54, 2.40/5.40/6.19. This is not quiet-VM qualification. No matching storm workload exists in docs/perf/baseline.json; #668 has no numeric storm slowdown rule. Native VM filesystem, fresh fixture per sample, warm disk cache, no HDD emulator, no HTTP/SSE/browser timing. Keep all failed/slow samples; initial failed fixture is retained separately. Corrected executable SHA-256: 143bfa670b61016d71fadddc509da33cae25f98dbe17ee9408e19eb41b669496. Baseline SHA-256: cf7ef6ee6859096686dad1587258062775320432df77159d8b6d1df7daece667. Raw summary retains both attempts and source/fixture identities. Corrected fmt and Files clippy passed; local Files test build is running.
kayg commented 2026-10-02 20:14:34 +00:00
Author
Owner
Copy link

Round 3 measurements are recorded. Head: 25d7173df518684ad1245bc35b955a48a4cf38d9. Branch: job/perf-stream-668. Worktree is clean. Merged origin/dev c4faf184df726a9375ae0c13bdfb6018ac2cf57e once before builds/gates. No push, deployment, merge into dev or issue closure.

Built:

  • Extended the existing 1,000-write/four-scan storm fixture with the production journal migration/Files bridge, an active User subscription, explicit post-commit wakes, journal/feed row parity and wake-delivery checks.
  • Retained every existing assertion and the five-minute bound. No production behavior or dependency change.
  • Updated the existing comparison profile's documentation. Committed both attempts, source/fixture identities, binary hashes, all loads and resource samples in docs/perf/change-stream.md and docs/perf/runs/files-reconcile-668-round3.json.

Files: crates/plugins/files/src/lib.rs; bench/files-reconcile-compare.sh; docs/perf/change-stream.md; docs/perf/runs/files-reconcile-668-round3.json. Module/function comments were re-read. Atomic commits: a1dd0e477 (method), aa9428066 (fixture), de057a38c (measurements), 25d7173df (local gate result). Runtime Rust source is aa9428066e; its fixture patch hash matches the measured artifact.

Locked VM result, alternating three samples each, release with identical line-table debug settings and locked dependencies:

Revision Whole-process wall samples (s) Median / p95 / max (s) Median CPU (s) Peak RSS (bytes)
origin/dev 37.34, 38.56, 36.72 37.34 / 38.56 / 38.56 6.02 34783232
Corrected head 42.11, 38.92, 45.80 42.11 / 45.80 / 45.80 6.31 35340288

All six corrected samples passed. Median wall +12.8%, p95 +18.8%, median user+system CPU +4.8%, peak RSS +1.6%. Weighted mean CPU: dev 16.21%, head 14.74% of one core, calculated from whole-process CPU/wall time; peak CPU was not sampled. With n=3, nearest-rank p95 is max. Keep all slow samples.

Verdict: both revisions pass the unchanged five-minute storm bound; the prior timeout is not reproduced on the VM. Head has a measured wall-time increase. There is no numeric whole-process slowdown rule in #668 and no matching storm workload in baseline.json; no zero-overhead or route-budget pass is claimed. Follow-up #947 records the increase and requests low-load qualification.

Every measured run held flock -w 14400 /root/perf.lock; lock released after each comparison. Corrected pre-run loads: dev 10.03/8.39/7.15, 4.38/6.94/6.73, 2.92/5.90/6.39; head 6.36/7.65/6.94, 3.29/6.34/6.54, 2.40/5.40/6.19. This is not quiet-VM qualification. Native filesystem, fresh fixtures, warm disk cache, no HDD emulator; wall time includes setup/cleanup. No VM compilation. Baseline build: 70m15s; initial head: 12m27s; corrected head: 5m18s. Corrected executable SHA-256: 143bfa670b61016d71fadddc509da33cae25f98dbe17ee9408e19eb41b669496; baseline: cf7ef6ee6859096686dad1587258062775320432df77159d8b6d1df7daece667.

Failed attempt retained: baseline 3/3 passed; initial head 0/3 passed, all failing the added wake assertion after write/scan completion and the 1,000-row Index check. The earlier Round 2 claim that the unchanged fixture reaches wake_user was incorrect. adopt_change commits the feed/journal but does not call the Files publish path. Corrected only the fixture to wake after each adoption commits. The new assertion was kept; no extra event-table write or transaction. This regression required a second head release build and corrected focused comparison. The initial failed samples are not accepted samples. Obsolete initial debug test build stopped before tests, exit 130; logs retained.

Gates, verbatim:

cargo fmt --check: exit 0, no output.
cargo clippy --locked -p calternal-plugin-files --all-targets -- -D warnings: exit 0:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 00s

cargo test --locked -p calternal-plugin-files -- --test-threads=4: exit 101:

writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())
atomic write 595 failed: entry not found
test result: FAILED. 147 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 425.69s

The worker missing-entry panic followed timeout/fixture teardown. The local debug suite is failed, not green. Observed build-host load reached 42.00/36.66/31.56. Retain it as SLOW evidence next to the three passing focused release VM runs; profile and host differ, so no causal claim. No timeout increase, expectation change or full-suite rerun.

bash -n bench/files-reconcile-compare.sh and git diff --check: exit 0, no output. Corrected VM profile: exit 0, six passing samples. No new UI or route behavior in this round; broad web/adversarial/server suites were not repeated under the verification policy.

Cleanup: web build and SvelteKit output already absent. cargo clean: exit 0, verbatim:

     Removed 13901 files, 6.5GiB total

Known gaps: failed local Files gate; quiet-VM qualification and general >=5 samples remain in #947. This unit-test comparison does not establish HTTP/SSE/browser or HDD budgets, nor isolate wake lookup cost. Original producer/view adoption remains with its existing owners.

Decisions: baseline retains its original fixture because it has no shared journal; head models the producer wake explicitly after committed adoption. Three alternating pairs follow this job's instruction, not the general five-sample rule. Preserve both failed and corrected attempts. Judge the stated five-minute bound and report timing ratios without inventing a whole-process threshold. File measured slowdown/qualification separately (#947); do not change production code without evidence.

UX gaps closed/left: no UI feature was changed in this measurement round. The fixture coverage gap was closed: bridge installation alone did not exercise wake_user; the explicit post-commit wake now does.

Round 3 measurements are recorded. Head: `25d7173df518684ad1245bc35b955a48a4cf38d9`. Branch: `job/perf-stream-668`. Worktree is clean. Merged origin/dev `c4faf184df726a9375ae0c13bdfb6018ac2cf57e` once before builds/gates. No push, deployment, merge into dev or issue closure. Built: - Extended the existing 1,000-write/four-scan storm fixture with the production journal migration/Files bridge, an active User subscription, explicit post-commit wakes, journal/feed row parity and wake-delivery checks. - Retained every existing assertion and the five-minute bound. No production behavior or dependency change. - Updated the existing comparison profile's documentation. Committed both attempts, source/fixture identities, binary hashes, all loads and resource samples in docs/perf/change-stream.md and docs/perf/runs/files-reconcile-668-round3.json. Files: crates/plugins/files/src/lib.rs; bench/files-reconcile-compare.sh; docs/perf/change-stream.md; docs/perf/runs/files-reconcile-668-round3.json. Module/function comments were re-read. Atomic commits: a1dd0e477 (method), aa9428066 (fixture), de057a38c (measurements), 25d7173df (local gate result). Runtime Rust source is aa9428066e6b31d80b82270f7244fd3b314c63cc; its fixture patch hash matches the measured artifact. Locked VM result, alternating three samples each, release with identical line-table debug settings and locked dependencies: | Revision | Whole-process wall samples (s) | Median / p95 / max (s) | Median CPU (s) | Peak RSS (bytes) | | --- | --- | --- | --- | --- | | origin/dev | 37.34, 38.56, 36.72 | 37.34 / 38.56 / 38.56 | 6.02 | 34783232 | | Corrected head | 42.11, 38.92, 45.80 | 42.11 / 45.80 / 45.80 | 6.31 | 35340288 | All six corrected samples passed. Median wall +12.8%, p95 +18.8%, median user+system CPU +4.8%, peak RSS +1.6%. Weighted mean CPU: dev 16.21%, head 14.74% of one core, calculated from whole-process CPU/wall time; peak CPU was not sampled. With n=3, nearest-rank p95 is max. Keep all slow samples. Verdict: both revisions pass the unchanged five-minute storm bound; the prior timeout is not reproduced on the VM. Head has a measured wall-time increase. There is no numeric whole-process slowdown rule in #668 and no matching storm workload in baseline.json; no zero-overhead or route-budget pass is claimed. Follow-up #947 records the increase and requests low-load qualification. Every measured run held `flock -w 14400 /root/perf.lock`; lock released after each comparison. Corrected pre-run loads: dev 10.03/8.39/7.15, 4.38/6.94/6.73, 2.92/5.90/6.39; head 6.36/7.65/6.94, 3.29/6.34/6.54, 2.40/5.40/6.19. This is not quiet-VM qualification. Native filesystem, fresh fixtures, warm disk cache, no HDD emulator; wall time includes setup/cleanup. No VM compilation. Baseline build: 70m15s; initial head: 12m27s; corrected head: 5m18s. Corrected executable SHA-256: 143bfa670b61016d71fadddc509da33cae25f98dbe17ee9408e19eb41b669496; baseline: cf7ef6ee6859096686dad1587258062775320432df77159d8b6d1df7daece667. Failed attempt retained: baseline 3/3 passed; initial head 0/3 passed, all failing the added wake assertion after write/scan completion and the 1,000-row Index check. The earlier Round 2 claim that the unchanged fixture reaches wake_user was incorrect. `adopt_change` commits the feed/journal but does not call the Files publish path. Corrected only the fixture to wake after each adoption commits. The new assertion was kept; no extra event-table write or transaction. This regression required a second head release build and corrected focused comparison. The initial failed samples are not accepted samples. Obsolete initial debug test build stopped before tests, exit 130; logs retained. Gates, verbatim: `cargo fmt --check`: exit 0, no output. `cargo clippy --locked -p calternal-plugin-files --all-targets -- -D warnings`: exit 0: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 00s ``` `cargo test --locked -p calternal-plugin-files -- --test-threads=4`: exit 101: ```text writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(()) atomic write 595 failed: entry not found test result: FAILED. 147 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 425.69s ``` The worker missing-entry panic followed timeout/fixture teardown. The local debug suite is failed, not green. Observed build-host load reached 42.00/36.66/31.56. Retain it as SLOW evidence next to the three passing focused release VM runs; profile and host differ, so no causal claim. No timeout increase, expectation change or full-suite rerun. `bash -n bench/files-reconcile-compare.sh` and `git diff --check`: exit 0, no output. Corrected VM profile: exit 0, six passing samples. No new UI or route behavior in this round; broad web/adversarial/server suites were not repeated under the verification policy. Cleanup: web build and SvelteKit output already absent. `cargo clean`: exit 0, verbatim: ```text Removed 13901 files, 6.5GiB total ``` Known gaps: failed local Files gate; quiet-VM qualification and general >=5 samples remain in #947. This unit-test comparison does not establish HTTP/SSE/browser or HDD budgets, nor isolate wake lookup cost. Original producer/view adoption remains with its existing owners. Decisions: baseline retains its original fixture because it has no shared journal; head models the producer wake explicitly after committed adoption. Three alternating pairs follow this job's instruction, not the general five-sample rule. Preserve both failed and corrected attempts. Judge the stated five-minute bound and report timing ratios without inventing a whole-process threshold. File measured slowdown/qualification separately (#947); do not change production code without evidence. UX gaps closed/left: no UI feature was changed in this measurement round. The fixture coverage gap was closed: bridge installation alone did not exercise wake_user; the explicit post-commit wake now does.
kayg commented 2026-10-03 01:34:43 +00:00
Author
Owner
Copy link

Round 4: Files reconcile storm regression

Branch job/perf-stream-668, head d02207202 (Rust source a8d37cf68). Not pushed.

Cause

  • wake_user is not the cause. It is one mutex lock, one map lookup and one notify_waiters (about 1 ms per 1,000 calls).
  • The extra work was write amplification in the journal. Each Files feed row wrote a rowid row, a key index entry and an age index entry. Then a renumbering UPDATE moved the row from key 0 to its final key, in a second key leaf, and the head row changed. SQLite WAL frames per single-row feed commit: no journal 7.20, round 3 journal 13.26, round 4 journal 9.74.

Changes

  • d23c5e9c5: app_changes is WITHOUT ROWID and has no age index. Age eviction uses key order behind a one-seek WHEN guard. The Files bridge and append advance the head before they insert the final key. Sequence-0 producers still work through a numbering trigger. New test: unnumbered_producers_and_age_eviction.
  • cc292106d: the server watcher wakes each affected User once per drained burst, or after 64 adoptions. Before this, watcher adoption did not wake the stream at all. The 1-second head poll still repairs a missed wake.
  • a8d37cf68: the storm fixture wakes once when its queue is empty, not once per adoption.

Perf VM (release lib-test, alternating, /root/perf.lock, 18/18 passed)

Set Baseline Baseline wall (s) Median Head wall (s) Median
a origin/dev c4faf184d 46.74, 45.02, 38.75 45.02 43.87, 40.09, 39.44 40.09
b origin/dev c4faf184d 37.11, 39.34, 40.60 39.34 39.50, 39.57, 49.62 39.57
c round 3 head 51.94, 48.04, 42.75 48.04 49.41, 44.05, 48.79 48.79
  • a+b (6 samples each): origin/dev median 39.97 s, head 39.83 s (-0.35%). Median CPU 6.31 s vs 6.27 s (-0.6%). Peak RSS 33,960 vs 36,192 KiB.
  • Set a alone: -10.9%.
  • Noise: the same executable ranged 37.11 to 46.74 s in one session. This whole-process test cannot resolve less than about 10%. The round 3 +12.8% sits in that band; set c shows no clear difference between the round 3 and round 4 heads. The WAL frame reduction is the direct evidence.
  • Baseline executable: the round 3 origin/dev binary, SHA-256 cf7ef6ee… (origin/dev has not changed). Head SHA-256 409dd117…. Raw data: docs/perf/runs/files-reconcile-668-round4.json.

Gates

  • The round 3 local Files failure was the five-minute bound under build-host load 42, not a logic failure. At 25d7173df (load about 13): test result: ok. 148 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 96.37s
  • After the changes: test result: ok. 148 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 94.84s
  • cargo test -p calternal-plugin changes: test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 23 filtered out; finished in 2.75s
  • cargo test -p calternal-server changes: test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 110 filtered out; finished in 0.04s
  • cargo fmt --check: exit 0. cargo clippy -p calternal-plugin -p calternal-plugin-files -p calternal-server --all-targets -- -D warnings: Finished, no warnings.
## Round 4: Files reconcile storm regression Branch `job/perf-stream-668`, head `d02207202` (Rust source `a8d37cf68`). Not pushed. ### Cause - `wake_user` is not the cause. It is one mutex lock, one map lookup and one `notify_waiters` (about 1 ms per 1,000 calls). - The extra work was write amplification in the journal. Each Files feed row wrote a rowid row, a key index entry and an age index entry. Then a renumbering `UPDATE` moved the row from key 0 to its final key, in a second key leaf, and the head row changed. SQLite WAL frames per single-row feed commit: no journal 7.20, round 3 journal 13.26, round 4 journal 9.74. ### Changes - `d23c5e9c5`: `app_changes` is `WITHOUT ROWID` and has no age index. Age eviction uses key order behind a one-seek `WHEN` guard. The Files bridge and `append` advance the head before they insert the final key. Sequence-0 producers still work through a numbering trigger. New test: `unnumbered_producers_and_age_eviction`. - `cc292106d`: the server watcher wakes each affected User once per drained burst, or after 64 adoptions. Before this, watcher adoption did not wake the stream at all. The 1-second head poll still repairs a missed wake. - `a8d37cf68`: the storm fixture wakes once when its queue is empty, not once per adoption. ### Perf VM (release lib-test, alternating, `/root/perf.lock`, 18/18 passed) | Set | Baseline | Baseline wall (s) | Median | Head wall (s) | Median | | --- | --- | --- | --- | --- | --- | | a | origin/dev `c4faf184d` | 46.74, 45.02, 38.75 | 45.02 | 43.87, 40.09, 39.44 | 40.09 | | b | origin/dev `c4faf184d` | 37.11, 39.34, 40.60 | 39.34 | 39.50, 39.57, 49.62 | 39.57 | | c | round 3 head | 51.94, 48.04, 42.75 | 48.04 | 49.41, 44.05, 48.79 | 48.79 | - a+b (6 samples each): origin/dev median 39.97 s, head 39.83 s (**-0.35%**). Median CPU 6.31 s vs 6.27 s (-0.6%). Peak RSS 33,960 vs 36,192 KiB. - Set a alone: -10.9%. - Noise: the same executable ranged 37.11 to 46.74 s in one session. This whole-process test cannot resolve less than about 10%. The round 3 +12.8% sits in that band; set c shows no clear difference between the round 3 and round 4 heads. The WAL frame reduction is the direct evidence. - Baseline executable: the round 3 origin/dev binary, SHA-256 `cf7ef6ee…` (origin/dev has not changed). Head SHA-256 `409dd117…`. Raw data: `docs/perf/runs/files-reconcile-668-round4.json`. ### Gates - The round 3 local Files failure was the five-minute bound under build-host load 42, not a logic failure. At `25d7173df` (load about 13): `test result: ok. 148 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 96.37s` - After the changes: `test result: ok. 148 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 94.84s` - `cargo test -p calternal-plugin changes`: `test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 23 filtered out; finished in 2.75s` - `cargo test -p calternal-server changes`: `test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 110 filtered out; finished in 0.04s` - `cargo fmt --check`: exit 0. `cargo clippy -p calternal-plugin -p calternal-plugin-files -p calternal-server --all-targets -- -D warnings`: `Finished`, no warnings.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
wip/palette2-1123
wip/palette-1093
wip/onboard2-1141
wip/onboard-1141.aborted-early
wip/onboard-1141
wip/notifloop-1194
wip/nlpchip-1127
wip/morph-1104
wip/merge-round-7c5
wip/merge-round-7c4
wip/merge-round-7c3
wip/merge-round-7c2
wip/merge-round-7c
wip/mchrome-1084
wip/mailghost2-1094
wip/mailghost-1094
wip/kbpreview2-1118
wip/kbpreview-1118
wip/kanban-1092
wip/importhang-1121
wip/hiderev-1153
wip/hide4-1153
wip/hide3-1153
wip/hide2-1153
wip/hide-1153
wip/editreg-1132
wip/editorrail3-1113
wip/editorrail2-1113
wip/editorrail-1113
wip/e2e-b2-1071
wip/e2e-b-1071
wip/draw4-1101
wip/draw3-1101
wip/draw2-1101
wip/draw-1101
wip/directory-1199-r
wip/directory-1199
wip/delete-1119
wip/collabrev-1197
wip/collabloss-1197
wip/cards2-1083
wip/cards-1083
wip/canvas-visual
wip/canvasvis2-976
wip/calhdr-1112
wip/calcards-1115
wip/browserfix
wip/blocks-1125
job/onboard-1141
wip/allday-1107
wip/agenda-decks
wip/agenda-1086
wip/adv7c-1105
wip/txentry-1198
wip/trayicons2-1095
wip/trayicons-1095
wip/tagperf-1186
wip/sidebar3-1094
wip/rev2-webperf
wip/rev2-money-ident
job/adv-1202
wip/restyle-notes
wip/previewcard-1098
job/merge30
job/notifloop-1194
job/collabloss-1197
job/hide-1153
job/perf-1124
job/perf2-1124
job/tocrail-1191
job/restyle-settings
wip/restyle-settings
job/segmented-1200
job/tagperf-1186
wip/segmented-1200
job/restyle-files
job/tagdnd-1187
job/cards-1179
wip/cards2-1179
wip/cards-1179
wip/tocrail-1191
wip/tagdnd-1187
wip/restyle-files
wip/perf-1124
wip/merge30j
job/restyle-notes
job/wizchoices-1140
wip/wizchoices-1140
wip/restyle-1190
job/moneyfmt-1180
job/txentry-1198
wip/moneyfmt2-1180
wip/moneyfmt-1180-r
wip/moneyfmt-1180
job/pillglass-1189
job/flags-1181
wip/flags-1181
job/restyle-1190
job/restyle-mailmoney
job/restyle-search
job/settingsreg-1195
job/wizard-1140
site/website
wip/wizardrev2-1140
wip/wizardrev-1140
wip/wizard5-1140
wip/wizard4-1140
wip/wizard3-1140
wip/wizard2-1140
wip/wizard-1140
wip/pillglass-1189
wip/settingsreg-1195
job/merge29
job/fu-1171
wip/merge29j
wip/fu-1171
job/fu-1166
job/directory-1199
job/proflog-1204
job/txresearch-1188
wip/fu-1166
job/merge28
job/search-1066
wip/search-1066
wip/merge28j
job/gateslot-1182
job/bulkimport-1157
job/mailnet-1160
wip/mailnetrev-1160
wip/mailnet-1160
wip/bulkrev-1157
wip/bulkimport-1157
job/startup-1161
wip/startup-1161
job/merge27
job/linkcards-1151
wip/linkcards3-1151
wip/linkcards2-1151
wip/linkcards-1151
job/traydate-1144
wip/traydate3-1144
wip/traydate2-1144
wip/traydate-1144
job/draw-1101
wip/merge27j
job/blockpill-1152
wip/blockpill3-1152
wip/blockpill2-1152
wip/blockpill-1152
job/minihover-1149
wip/minihover2-1149
wip/minihover-1149
job/merge25
wip/merge25-r
wip/merge25b
wip/merge25
job/inspector-1129
job/tags-1110
wip/inspector3-1129
wip/inspector2-1129
wip/inspector-1129
wip/tagsrev-1110
wip/tags2-1110
wip/tags-1110
job/dates-1148
wip/datesrev-1148
wip/dates2-1148
wip/dates-1148
job/licence-1145
wip/licence2-1145
wip/licence-1145
job/selfhost-1156
job/merge23
wip/merge23
job/tagfilter-1109
wip/tagfilter2-1109
wip/tagfilter-1109
job/kbd-1134
wip/kbd2-1134
wip/kbd-1134
job/palfoot-1137
wip/selfhost-1156
wip/palfoot2-1137
wip/palfoot-1137
job/toggle-1158
wip/toggle-1158
job/kbpreview-1118
job/docratchet-1155
job/perflint-1133
job/devtests-1159
wip/docratchet-1155
wip/devtests-1159
job/segv-1136
wip/toast-1142
wip/segv-1136
job/toast-1142
job/blockreload-1147
wip/blockreload-1147
job/font-1150
wip/font-1150
job/importui-1120
job/minimonth-1149
wip/importui-1120
wip/minimonth-1149
job/depcheck-1146
wip/perflint-1133
wip/depcheck-1146
job/calcards-1115
job/blocks-1125
job/plus-1128
job/shift-1138
wip/plus2-1128
wip/plus-1128
wip/shift-1138
job/moneyfid-1130
job/editorrail-1113
wip/moneyrev-1130
wip/moneyfid-1130
job/noext-851
wip/noext-851
wip/noext3-851
wip/noext2-851
job/week-1135
wip/week-1135
job/editreg-1132
job/smoke-1122
wip/smoke-1122
job/docs-1143
job/palette2-1123
job/calhdr-1112
job/nlpchip-1127
job/mailghost-1094
job/reconnect-1131
wip/reconnect-1131
job/trayicons-1095
job/delete-1119
job/importhang-1121
job/cards-1083
job/palette-1093
job/mchrome-1084
job/e2e-a-1071
job/canvas-visual
job/previewcard-1098
job/allday-1107
wip/e2e-a2-1071
wip/e2e-a-1071
job/e2e-b-1071
job/adv7c-1105
job/kanban-1092
job/agenda-1086
job/merge-round-7c
job/morph-1104
wip/surfaces-p2
job/merge-round-9
wip/merge-round-9
job/7cfix-small
wip/7cfix-small
job/mailui-1078
job/merge-round-8
wip/merge-round-8
wip/mailui-1078
job/mailround-1038
job/applemail-accept
wip/settitle-1068
wip/mailround2-1038
wip/mailround-1038
wip/e2e-7b
job/crash-1069
wip/crash-1069
job/searchlost-1066
wip/searchlost-1066
job/7b-reconcile
job/flake-1065
wip/flake-1065
wip/merge-round-7b7
wip/merge-round-7b6
wip/merge-round-7b5
wip/merge-round-7b4
wip/7b-reconcile
job/appupdate-1059
job/nfd-1044
wip/appupdate-1059
job/e2e-7b
job/loop-1062
wip/loop-1062
job/pdfprev-1045
job/invtoggle-1053
wip/pdfprev-1045
wip/nfd-1044
wip/invtoggle-1053
job/7bfix-e2e
job/mailstress-b
wip/7bfix-e2e
wip/mailstress-b
job/7bfix-adv
wip/7bfix-adv
job/mailstress-a
job/stack-1054
wip/stack-1054
wip/mailstress-a
job/mailstress-1038
wip/mailstress-1038
job/upload500-1051
wip/upload500-1051
job/share-1034
wip/share-1034
job/syncerr-1037
job/7bfix-photos
wip/7bfix-photos
job/paste-1036
job/setside-1039
wip/setside-1039
wip/paste-1036
job/lease-1042
wip/syncerr-1037
wip/lease-1042
job/7bfix-data
job/passkeybind-1043
wip/apprevoke-1041
job/invite-1035
wip/invite-1035
job/merge-round-7b2
wip/merge-round-7b2
job/mailproxy-486
job/apprevoke-1041
job/rebuild-1033
job/pillborder-1029
wip/pillborder-1029
wip/mailproxy-486
wip/applemail-486
job/headless-998
wip/headless-998
job/groups-1028
wip/groups-1028
job/rebuildwarn-1016
wip/rebuildwarn-1016
job/startup-1011
wip/startup-1011
job/monthpill-1009
job/bgthumb-1025
job/sharetitle-1012
wip/monthpill-1009
wip/bgthumb-1025
wip/sharetitle-1012
job/canvas-cards-977
wip/canvas-cards-977
job/canvas-pencil-978
job/canvas-sketch-990
wip/canvas-sketch-990
wip/canvas-pencil-978
job/canvas-files-989
wip/canvas-files-989
job/canvas-collab-991
wip/canvas-collab-991
job/weekscroll-1018
wip/weekscroll-1018
wip/canvas-core-976
job/canvas-core-976
job/round-drag
wip/round-drag
job/round-settings
job/browserfix
wip/oapi-974
job/oapi-974
job/hist2-integrate
job/mailhtml-726
wip/mailhtml-726
wip/hist2-integrate
job/moneyfu-984
job/drag-1015
wip/drag-1015
job/rename-1017
wip/rename-1017
job/hist2-api
wip/hist2-api
job/oneacct-1014
wip/oneacct-1014
wip/moneyfu-984
job/hist2-bench
job/hist2-restore
wip/hist2-bench
job/hist2-write
job/hotfix-724
wip/hotfix-724
wip/hist2-write
wip/hist2-restore
job/hist2-store
job/hist2-ui
wip/hist2-ui
wip/hist2-store
job/searchstarve-965
job/shutdown-963
wip/shutdown-963
wip/pubedit-981
job/pubedit-981
job/analytics-973
wip/searchstarve-965
job/authflash-850
job/weeklane-969
job/pvtitle-1004
job/hist-975
wip/authflash-850
job/voicepill-617
wip/pvtitle-1004
job/headring-1003
wip/weeklane-969
wip/voicepill-617
wip/headring-1003
wip/analytics-973
job/agentscope-980
wip/thumbsandbox-988
job/thumbsandbox-988
wip/hist-975
job/links-856
wip/links-856
job/davetag-966
wip/davetag-966
job/filesstorm-1000
job/hoverpad-725
wip/filesstorm-1000
job/ffmpegblas-993
job/merge-round-7a
wip/hoverpad-725
wip/ffmpegblas-993
job/nowdot-1002
wip/verify-7a
job/noteid-857
wip/nowdot-1002
wip/noteid-857
wip/merge-round-7a
wip/agentscope-980
job/imapedge
job/a11yfix2
wip/imapedge-941
wip/imapedge
wip/a11yfix2
job/notetask-986
job/logheading
wip/logheading-998
job/textthumb-652
job/photolive-987
wip/photolive-987
job/davactive-983
job/savefix-985
job/tabicons-607
wip/davactive-983
wip/tabicons-607
wip/notetask-986
wip/savefix-985
job/dirid-627
job/buildspeed-1007
wip/dirid-627
job/agenda-decks
job/perfguards-impl
job/undo-a11y
wip/undo-a11y
job/mailperf
job/wal-824
wip/settings-50
job/settings-50
job/notesfilter-606
wip/notesfilter-606
job/surfaces-p2
wip/wal-824
job/maillayouts
wip/mailperf
wip/maillayouts
job/taskmeta-659
job/money-ident
wip/money-ident
wip/taskmeta-659
job/errstates
wip/perfguards-impl
job/headings-881
wip/headings-881
wip/errstates
job/voice-619
job/gaps-827
job/notesperf
wip/notesperf
wip/voice-619
job/hddsql-549
job/perf-stream-668
wip/perf-stream-668
wip/deeplinks-fix
job/deeplinks-fix
job/authfix
job/docsfix-rust
wip/docsfix-rust
job/webperf
job/docsfix-web
job/datafix2
job/webdav-lock-476
job/copyfix
wip/copyfix
wip/webperf
job/focus-658
wip/protofix
job/mediafix
job/protofix
wip/mediafix
job/agentfix
job/hhmm-724
wip/agentfix
job/undo-722
job/reuse
wip/webdav-lock-476
wip/reuse
job/scopefix
job/datafix
wip/hhmm-724
wip/undo-722
job/surfaces-p1
wip/hddsql-549
job/voicememos-618
wip/datafix2
wip/surfaces-p1
job/fix-940
wip/fix-940
job/blaze-surfaces
wip/datafix
wip/blaze-surfaces
job/taskday-655
job/linknav-639
wip/linknav-639
wip/gaps-827
job/isolation-707
job/audiophotos-720
wip/audiophotos-720
job/advfind-664
wip/voicememos-618
wip/taskday-655
wip/isolation-707
wip/advfind-664
wip/scopefix
wip/focus-658
job/testgaps
wip/testgaps
job/overscroll-718
wip/authfix
job/deps
wip/overscroll-718
job/rev2-agentfix
job/rev2-money-ident
job/rev2-mailperf
wip/deps
job/hardening-728
wip/hardening-728
job/searchgen-832
wip/searchgen-832
job/photopw-849
job/mailsql-825
wip/photopw-849
job/sharefix
wip/sharefix
job/rev2-mailhtml-726
job/rev2-perfguards
job/copyval-723
job/lightglass-r2
wip/lightglass-r2
wip/docsfix-web
job/copy-audit
job/macinterop-staging-r2
job/design-sync
job/rev2-taskmeta-659
job/rev2-webperf
job/docs-audit
job/rev2-advfind-664
job/rev2-mailproxy-486
job/states-audit
job/rev2-datafix
job/design-drift
job/test-gaps
job/rev2-voicememos-618
job/rev2-mediafix
job/rev2-deps
job/rev2-datafix2
job/licence-audit
job/issue-hygiene
job/rev2-protofix
job/rev2-voice-619
job/rev2-isolation-707
job/rev2-surfaces-p1
job/deeplink-audit2
job/rev2-audiophotos-720
wip/test-gaps
job/rev2-overscroll-718
job/rev2-undo-722
wip/states-audit
job/rev2-dropmd-719
job/rev2-linknav-639
job/merge-7b-plan
wip/merge-7b-plan
job/rev2-taskday-655
wip/mailsql-825
job/rev2-webdav-lock-476
job/rev2-browserfix
wip/design-drift
job/rev2-hddsql-549
wip/deeplink-audit2
job/rev2-scopefix
job/rev2-authfix
job/rev2-hardening-728
job/rev2-wal-824
job/rev2-sharefix
job/calsidebar-638
job/chrome-audit
job/ioperf
wip/ioperf
wip/chrome-audit
wip/calsidebar-638
job/dropmd-719
wip/dropmd-719
job/ocr-build
wip/ocr-build
job/blaze-settings
wip/copyval-723
job/toastring-721
wip/toastring-721
job/deployfix-732
wip/deployfix-732
wip/blaze-settings
job/money-import-recheck
job/rev-a11y
job/perf-arch-db
job/rev-7b-data
wip/textthumb-652
wip/perf-arch-db
job/sec-protocols
job/sidehdr-660
job/rev-7b-security
job/research-surfaces
job/rev-design-gaps
job/rev-mcp-api
wip/sidehdr-660
job/perf-arch-memory
wip/sec-protocols
job/perf-arch-bundle
job/snapedge-714
wip/rev-mcp-api
job/sec-supplychain
wip/research-surfaces
job/perf-arch-sync
job/rev-consistency
job/perf-arch-server
wip/perf-arch-server
wip/perf-arch-memory
job/perf-arch-io
job/perf-arch-client
job/sec-fs
job/sec-mcp-scopes
job/sec-sharing
job/perf-guards
job/sec-browser
job/sec-admin-deploy
job/sec-auth
wip/snapedge-714
job/bgpicker-717
wip/perf-arch-bundle
wip/money-import-recheck
job/advsetup-654
wip/bgpicker-717
wip/advsetup-654
job/burst-709
job/kbdcaps-710
job/app-pw-chooser
wip/burst-709
wip/app-pw-chooser
job/imaptest-625
wip/kbdcaps-710
job/fix-499
wip/fix-499
job/perf-mut-667
job/calimg-589
job/perf-snap-666
wip/calimg-589
wip/perf-snap-666
wip/perf-mut-667
job/perf-cache-665
wip/perf-cache-665
job/voicefiles-620
wip/voicefiles-620
job/admin-burst-705
wip/admin-burst-705
job/voicememos-review
wip/voicememos-review
wip/ryw-653
job/ryw-653
job/writeonopen-661
job/instant-663
wip/writeonopen-661
job/money-import-review
wip/money-import-review
wip/importjs-610
review/integrations-407-round6
wip/integrations-review
job/dragghost-612
wip/dragghost-612
job/integrations
wip/integrations
job/decider-656
job/merge-round-6
job/perf-rerun
wip/merge-round-6
job/integrations-review-round5
job/selalign-576
wip/selalign-576
job/mcp-events-491
job/files-631
job/cal-e2e-569
wip/cal-e2e-569
job/reload-423
wip/reload-423
wip/mcp-events-491
wip/files-631
job/notesbridge-644
wip/notesbridge-644
job/editor-series
job/calcard-series
wip/calcard-series
job/mcp-events-review-491
wip/mcp-events-review
wip/editor-series
job/quirks-546
job/integrations-recheck
job/tocrail-636
wip/tocrail-636
wip/quirks-546
wip/reminders-643
job/reminders-643
wip/davscale-573
job/davscale-573
job/integrations-review
wip/ocr-eval-584
job/ocr-eval-584
job/esc-537
wip/esc-537
job/toastname-586
wip/toastname-586
job/submenu-579
wip/submenu-579
job/tasks-mode
wip/tasks-mode
job/agentdocs-630
job/dupwrite-634
wip/agentdocs-630
wip/dupwrite-634
job/lightglass-588
wip/lightglass-588
job/tabswitch-549
job/ghosttask-623
wip/ghosttask-623
job/toaststack-616
job/weekstate-609
job/mailsync-613
wip/mailsync-613
wip/weekstate-609
job/maildup-626
wip/tabswitch-549
wip/maildup-626
wip/toaststack-616
job/motion-611
wip/motion-611
job/tlstest-601
wip/tlstest-601
job/perf-495
job/floating-sheet
wip/floating-sheet
job/remdup-585
wip/remdup-585
job/fix-502
wip/fix-502
job/attachplay-622
job/perf-batch
wip/perf-batch-563
wip/perf-495
hotfix/mail-sync-diag
job/mail-m3
wip/mail-m3
job/attach-poof-603
job/calhover-608
job/editorbar-604
job/mentions-605
job/merge-round-4
job/allday-514
wip/merge-round-4
wip/allday-514
job/merge-round-4a
wip/merge-round-4a
job/sharestack-580
job/fix-501
wip/sharestack-580
wip/fix-501
job/perf-batch-563
job/apw-cache-review
wip/apw-cache-review
job/probe-520
wip/probe-520
job/mac-393
wip/mac-393
job/header-571
job/flake-513
wip/flake-513
job/docs-thumb-547
wip/header-571
job/webcal-572
wip/webcal-572
wip/shortcuts-542
job/shortcuts-542
wip/docs-thumb-547
job/caldav-stress
wip/caldav-stress
wip/sweep-478
job/apw-cache-512
wip/apw-cache-512
job/money-empty-540
wip/restart-505
wip/money-empty-540
wip/fix-510
job/restart-505
job/fix-503
job/perf-496
wip/perf-496
job/fix-498
wip/fix-498
job/info-inspector-465
wip/info-inspector-465
job/fix-510
job/fix-507
wip/fix-507
wip/fix-503
job/fix-493
job/money-kinds
wip/money-kinds
job/hygiene-548
job/merge-round-3
wip/fix-493
job/drag-snap-536
wip/merge-round-3
wip/merge-round-0930
wip/drag-snap-536
job/align-538
wip/align-538
job/bg-flash
wip/bg-flash
job/money-import
job/search-count-544
wip/search-count-544
wip/money-import
job/settings-key-541
wip/settings-key-541
job/toast-539
job/preview-421
wip/preview-421
wip/toast-539
job/tasks-500-531
job/title-plain-526
wip/title-plain-526
wip/tasks-500-531
job/notes-bridge
wip/parity-484
job/parity-484
job/files-slow
job/crash-525
wip/notes-bridge
wip/files-slow
wip/crash-525
job/kbd-motion-527
wip/bg-422
job/analytics-504
wip/analytics-504
wip/kbd-motion-527
job/upload-pill-523
wip/upload-pill-523
wip/tray-order
job/tray-order
wip/overflow-mid
wip/merge-round-2
job/perf-494
wip/perf-494
wip/mcp-fast-492
wip/motion-477
wip/asr-ab-489
wip/theme-variants-506
wip/overflow-511
wip/week-header-508
wip/attach-427
job/dav-delete-471
job/iso-435
wip/iso-435
wip/files-sel-keys
wip/dav-delete-471
job/align-253
job/siwc-490
wip/siwc-490
job/money-kinds-review
wip/align-253
wip/money-kinds-review
job/small-bugs-3
wip/overlay-title-487
wip/multiget-500
wip/hidden-420
wip/webcal-ui
wip/webcal-431
job/perf-367
job/location
wip/small-bugs-3
wip/location
wip/perf-367
wip/admin-deny-483
job/tag-unicode-473
wip/tag-unicode-473
job/blur-436
wip/photos-470
wip/blur-436
wip/small-bugs-4
wip/hunt-20260930
wip/settings-hdr-482
wip/chips-416
job/dedup-375
wip/dedup-375
job/doc-stack
wip/doc-stack
job/tokens-literals
wip/tokens-literals
job/jobs-leftovers
wip/send-fast
wip/paste-467
wip/money-numbers
job/money-plugin
wip/money-plugin
job/break-dav
wip/merge-batch
wip/crossday-469
wip/mac-verify
wip/mail-m2
wip/break-dav
wip/money-review2
job/money-md
job/modes-424
wip/money-md
wip/jobs-leftovers
job/agenda-413
wip/agenda-413
wip/modes-424
job/recog-417
wip/recog-417
wip/bounce-425
wip/ab-384-luna
job/webdav-perf
wip/webdav-perf
job/toast-ring
wip/toast-ring
job/money-review
wip/money-review
wip/micro-motion
wip/settings-card
wip/minical
job/notes-imap-428
job/least-priv
wip/ui-small-2
wip/flaky-426
wip/drag-end-418
job/jank
wip/jank
wip/least-priv
wip/docs-site
job/agenda
job/sec-batch
wip/sec-batch
wip/per-user-index
job/area-calendars
wip/area-calendars
job/parity
wip/parity
job/documents-research
wip/documents-research
job/test-infra
job/reminders-sync
wip/small-bugs-2
wip/reminders-sync
wip/gestures
job/google-oauth
wip/tags-merge
wip/tags
job/e2e-theme
wip/e2e-theme
job/icon-align
wip/test-infra
wip/select-align
wip/editor-385
job/voice
wip/webdav
job/webdav
job/app-pw-ui
job/editor-integrity
wip/editor-integrity
wip/voice
wip/quota
wip/cal-followups
wip/icon-align
job/composer-scale
wip/composer-scale
job/jobs-page
wip/jobs-page
job/hig-type
wip/hig-type
wip/app-pw-ui
job/motion-spring
job/mcp
wip/motion-spring
wip/mcp
job/small-bugs
wip/push-hosts
job/profile-sign
wip/touch-369
wip/profile-sign
job/mobile-focus
wip/mobile-focus
wip/ui-polish-354
wip/small-bugs
wip/dup-task
job/toast-polish
job/app-pw-scopes
wip/toast-polish
wip/app-pw-scopes
wip/cli-agent
wip/selection-pills
job/preview-attach
wip/preview-attach
job/dav-proppatch
wip/dav-proppatch
wip/cal-switcher
job/atomic-race
wip/atomic-race
job/photos-shared
wip/photos-shared
wip/cal-grid
wip/note-rewrite
wip/search-rebuild
job/mail-m1
job/paperless-import
wip/paperless-import
wip/mail-m1
wip/hidden-activity
wip/search-d
wip/pricing-research
wip/cursors
wip/auto-scheme
job/single-pills
wip/single-pills
wip/xuser-matrix
wip/money-format
wip/app-pw-setup
wip/purge-dos
wip/vault-health
wip/caldav-apple
wip/xuser-audit
wip/e2e-green
wip/tabbar
wip/adv-harness
wip/maple-mono
job/search-fix
wip/search-fix
wip/search-perf-c
job/adv-harness
wip/sidebar-headers
job/glass
wip/temp-index
job/polish
wip/polish
wip/file-protocols
wip/money-research
wip/glass
wip/voice-models
wip/collab-redo
job/voice-research
wip/hunt-20260928
wip/notes-actions-research
wip/search-pad
wip/search-perf
wip/search-sticky
wip/editor-undo
wip/chrome-rules
wip/motion
wip/appearance-research
wip/appearance
wip/audit-bugs
wip/cal-glass
wip/block-actions
wip/authz-order
wip/event-stripes
wip/chrome-sidebar
wip/auth-flaky
wip/robust-2
wip/gate-fix
wip/menu-blur
wip/import-calternaljs
wip/tray-fix
job/import-calternaljs
wip/index-order
wip/audit-fixes
wip/search-chevrons
research/mail
wip/phone-chrome
wip/dedup-break
wip/csp
wip/ui-audit
wip/select-toast
wip/perf
wip/flat-layout
wip/fonts
wip/event-tint
wip/sync-converge
wip/data-split
wip/glass-audit
wip/robustness
wip/sync-chaos
wip/search-thumbs
wip/fuzz
wip/menu-icons
wip/search-pill
wip/sync-changing
wip/heading-links
wip/date-formats
wip/a11y
wip/break-editor
wip/e2e-fix
wip/settings-sections
wip/sync-root-guard
wip/search-palette
wip/share-edit
job/toasts
wip/toasts
wip/cont-analytics
wip/authz-review
wip/popovers
wip/overlay-glass
wip/change-feed
wip/editor-modes
wip/composer-align
wip/cont-agenda
wip/agenda-merge
job/agent-conventions
wip/agent-conventions
wip/backend-misc
job/route-audit
wip/route-audit
wip/ui-batch
wip/heif-hardening
wip/grid-resize
wip/ask-page
wip/webmcp
job/deeplink-audit
wip/deeplinks
wip/shortcuts
wip/cont-tz-days
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#668
No description provided.