PERF: qualify Files journal/wake storm overhead after #668 #947

New issue

Open

opened 2026-10-02 19:59:48 +00:00 by kayg · 1 comment

kayg commented

2026-10-02 19:59:48 +00:00

Owner

#668 Round 3 measured a Files storm wall-time increase. Keep this separate from shared stream adoption and the unchanged five-minute correctness bound, which passed for both revisions.

Locked perf VM, release lib-test executables, identical line-table debug settings, incremental off, locked dependencies, fresh native-filesystem fixtures, no HDD emulator or disk-cache drop. Alternating three samples per revision:

Revision	Wall samples (s)	Median / p95 / max (s)	Median CPU (s)	Peak RSS (bytes)
origin/dev `c4faf184df`	37.34, 38.56, 36.72	37.34 / 38.56 / 38.56	6.02	34783232
Corrected #668 fixture	42.11, 38.92, 45.80	42.11 / 45.80 / 45.80	6.31	35340288

Head median wall +12.8%, nearest-rank p95 +18.8%, median user+system CPU +4.8%, peak RSS +1.6%. All six samples passed. Wall time includes fixture setup and cleanup. Head installs the journal bridge and calls wake_user after each committed watcher adoption; baseline retains its original fixture. This measures added journal/wake work, not isolated wake lookup cost or HTTP/SSE/browser latency.

The first attempt failed the added wake check because adopt_change alone does not publish Files UI hints. Its failed samples remain retained, not counted as successful. Corrected binary SHA-256: 143bfa670b61016d71fadddc509da33cae25f98dbe17ee9408e19eb41b669496; baseline: cf7ef6ee6859096686dad1587258062775320432df77159d8b6d1df7daece667.

Limit: pre-run load inside the lock ranged from 2.40 to 10.03 (one-minute average). This is not quiet-VM qualification. n=3 is the job-specific count, below DESIGN §58's general five samples. No matching storm workload is in baseline.json; there is no numeric storm threshold in #668. Do not apply a route p95 budget to this whole-process test or declare a causal production regression from these samples alone.

Follow-up: use bench/files-reconcile-compare.sh under flock /root/perf.lock on a low-load VM. Record at least five samples per revision and all loads. Keep the original 1,000 writes/four scans/five-minute bound. Qualify the wall-time difference, then isolate journal trigger work and wake work if it persists. Keep failed and slow samples. Performance review is not a merge gate.

Evidence will be committed in docs/perf/change-stream.md and docs/perf/runs/files-reconcile-668-round3.json; full matrix is posted on #668. No product change is requested by this follow-up.

#668 Round 3 measured a Files storm wall-time increase. Keep this separate from shared stream adoption and the unchanged five-minute correctness bound, which passed for both revisions. Locked perf VM, release lib-test executables, identical line-table debug settings, incremental off, locked dependencies, fresh native-filesystem fixtures, no HDD emulator or disk-cache drop. Alternating three samples per revision: | Revision | Wall samples (s) | Median / p95 / max (s) | Median CPU (s) | Peak RSS (bytes) | | --- | --- | --- | --- | --- | | origin/dev c4faf184df726a9375ae0c13bdfb6018ac2cf57e | 37.34, 38.56, 36.72 | 37.34 / 38.56 / 38.56 | 6.02 | 34783232 | | Corrected #668 fixture | 42.11, 38.92, 45.80 | 42.11 / 45.80 / 45.80 | 6.31 | 35340288 | Head median wall +12.8%, nearest-rank p95 +18.8%, median user+system CPU +4.8%, peak RSS +1.6%. All six samples passed. Wall time includes fixture setup and cleanup. Head installs the journal bridge and calls wake_user after each committed watcher adoption; baseline retains its original fixture. This measures added journal/wake work, not isolated wake lookup cost or HTTP/SSE/browser latency. The first attempt failed the added wake check because adopt_change alone does not publish Files UI hints. Its failed samples remain retained, not counted as successful. Corrected binary SHA-256: 143bfa670b61016d71fadddc509da33cae25f98dbe17ee9408e19eb41b669496; baseline: cf7ef6ee6859096686dad1587258062775320432df77159d8b6d1df7daece667. Limit: pre-run load inside the lock ranged from 2.40 to 10.03 (one-minute average). This is not quiet-VM qualification. n=3 is the job-specific count, below DESIGN §58's general five samples. No matching storm workload is in baseline.json; there is no numeric storm threshold in #668. Do not apply a route p95 budget to this whole-process test or declare a causal production regression from these samples alone. Follow-up: use bench/files-reconcile-compare.sh under flock /root/perf.lock on a low-load VM. Record at least five samples per revision and all loads. Keep the original 1,000 writes/four scans/five-minute bound. Qualify the wall-time difference, then isolate journal trigger work and wake work if it persists. Keep failed and slow samples. Performance review is not a merge gate. Evidence will be committed in docs/perf/change-stream.md and docs/perf/runs/files-reconcile-668-round3.json; full matrix is posted on #668. No product change is requested by this follow-up.

kayg referenced this issue

2026-10-02 20:14:34 +00:00

PERF: one per-User change stream and capped delta endpoint (#663) #668

kayg commented

2026-10-02 20:14:35 +00:00

Author

Owner

Evidence is committed at 25d7173df5: docs/perf/change-stream.md and docs/perf/runs/files-reconcile-668-round3.json. Corrected runtime Rust source: aa9428066e. All six corrected locked release samples passed. Local Files debug suite still failed the original five-minute storm bound (147 passed / 1 failed / 1 ignored); no threshold change or rerun. Extend the existing profile's sample count for the low-load >=5-sample follow-up. Full report is on #668. No push/deploy/issue closure.

Evidence is committed at 25d7173df518684ad1245bc35b955a48a4cf38d9: docs/perf/change-stream.md and docs/perf/runs/files-reconcile-668-round3.json. Corrected runtime Rust source: aa9428066e6b31d80b82270f7244fd3b314c63cc. All six corrected locked release samples passed. Local Files debug suite still failed the original five-minute storm bound (147 passed / 1 failed / 1 ignored); no threshold change or rerun. Extend the existing profile's sample count for the low-load >=5-sample follow-up. Full report is on #668. No push/deploy/issue closure.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#947

No description provided.

Rows
Columns