User purge interferes with unrelated uploads: local p95 23.069 seconds

kayg commented

2026-10-07 23:05:58 +00:00

Owner

Found during the one bounded #1403 release-server adversarial round at head d395b6e25d. Production runtime ef0aa12b1; merged Rust runtime code is unchanged.

Uploads during user purge: p95=23.069s (20 uploads)
!! write during user purge p95: upload p95 was 23.07s; expected under 2 s

The unchanged attack2.py deletion probe uploads as another User while purge runs. Eighteen individual uploads exceed its 2-second limit (observed 2.96 to 23.07 seconds). It reports no upload status failure in this phase. The local shared host load average at 23:01 UTC was 20.49 / 21.04 / 24.17. This is context only; no cause or quiet-host regression is established.

Timing-only, non-blocking performance finding. No crash, failed upload, corruption or authorization hole is shown by these observations. Keep the existing 2-second and p95 assertions. Investigate purge lock/IO interference with unrelated User writes on a controlled benchmark run; use the required perf-run lock and prebuilt binary. Do not raise the threshold to hide this result.

Found during the one bounded #1403 release-server adversarial round at head d395b6e25da20d8c67b8b9fc1a593b813939c8a7. Production runtime ef0aa12b1; merged Rust runtime code is unchanged. ``` Uploads during user purge: p95=23.069s (20 uploads) !! write during user purge p95: upload p95 was 23.07s; expected under 2 s ``` The unchanged attack2.py deletion probe uploads as another User while purge runs. Eighteen individual uploads exceed its 2-second limit (observed 2.96 to 23.07 seconds). It reports no upload status failure in this phase. The local shared host load average at 23:01 UTC was 20.49 / 21.04 / 24.17. This is context only; no cause or quiet-host regression is established. Timing-only, non-blocking performance finding. No crash, failed upload, corruption or authorization hole is shown by these observations. Keep the existing 2-second and p95 assertions. Investigate purge lock/IO interference with unrelated User writes on a controlled benchmark run; use the required perf-run lock and prebuilt binary. Do not raise the threshold to hide this result.

purge-1420-1440-paper-files-click.png

777 KiB

purge-1420-1440-paper-people.png

216 KiB

purge-1420-1440-paper-settings-shortcut.png

186 KiB

purge-1420-1440-tokyo-night-files-click.png

689 KiB

purge-1420-1440-tokyo-night-people.png

197 KiB

purge-1420-1440-tokyo-night-settings-shortcut.png

179 KiB

purge-1420-390-paper-files-click.png

25 KiB

purge-1420-390-paper-people.png

66 KiB

purge-1420-390-paper-settings-shortcut.png

70 KiB

purge-1420-390-tokyo-night-files-click.png

30 KiB

purge-1420-390-tokyo-night-people.png

71 KiB

purge-1420-390-tokyo-night-settings-shortcut.png

73 KiB

purge-1420-820-paper-files-click.png

38 KiB

purge-1420-820-paper-people.png

132 KiB

purge-1420-820-paper-settings-shortcut.png

117 KiB

purge-1420-820-tokyo-night-files-click.png

43 KiB

purge-1420-820-tokyo-night-people.png

124 KiB

purge-1420-820-tokyo-night-settings-shortcut.png

118 KiB

purge-1420-purge-before-debug.json

852 B

purge-1420-purge-after-debug.json

861 B

purge-1420-server-regression-before-verified.log

915 B

purge-1420-server-regression-after-verified.log

959 B

purge-1420-purge-vm-before.json

861 B

purge-1420-purge-vm-after.json

874 B

kayg referenced this issue

2026-10-07 23:10:35 +00:00

Merge round 34 #1403

kayg referenced this issue

2026-10-07 23:30:43 +00:00

Merge round 34 #1403

kayg commented

2026-10-08 06:50:16 +00:00

Author

Owner

Started on job/purge-1420 at base d395b6e25d. I will trace the shared filesystem purge and run the focused real-server deletion/upload probe before and after. The existing 2-second assertions stay unchanged.

Started on job/purge-1420 at base d395b6e25da20d8c67b8b9fc1a593b813939c8a7. I will trace the shared filesystem purge and run the focused real-server deletion/upload probe before and after. The existing 2-second assertions stay unchanged.

kayg commented

2026-10-08 06:54:43 +00:00

Author

Owner

Findings: purge validation calls unpaged walk_size (a stat for each file); removal only yields the scheduler after 256 unlinks, and does not count directory descent. The deletion job also calls reconcile_all for Notes/Tasks after removal, which scans and locks unrelated Users. The fix pages both validation and removal with one stream per depth and a real 2 ms pause per 128 entries, and reconciles only the source and transfer recipient. A regression holds an unrelated User's Notes writer lock until deletion completes. Existing latency assertions remain. The focused real-server probe now reads back every unrelated upload after deletion and records p50/p95, server CPU/RSS and load. The perf VM is currently held by calternal bignote-1349-baseline-2; the initial comparison is local and labelled as such. No dependencies were added.

kayg commented

2026-10-08 06:56:52 +00:00

Author

Owner

Before-fix real-server result at d395b6e25 (prebuilt release, local): Uploads during user purge: p95=5.066s (20 uploads). Ordinary uploads: p50=0.299s, p95=0.895s (5 samples). Purge plus readback: 9.528s; server CPU 2.320s; RSS 238059520 -> 278249472 bytes; load 32.52 / 28.55 / 22.41. The existing assertions failed for 14 uploads and the p95. There were no status or byte-integrity findings. This reproduces the issue locally; it is not yet a quiet-host or controlled-VM claim.

Before-fix real-server result at d395b6e25 (prebuilt release, local): Uploads during user purge: p95=5.066s (20 uploads). Ordinary uploads: p50=0.299s, p95=0.895s (5 samples). Purge plus readback: 9.528s; server CPU 2.320s; RSS 238059520 -> 278249472 bytes; load 32.52 / 28.55 / 22.41. The existing assertions failed for 14 uploads and the p95. There were no status or byte-integrity findings. This reproduces the issue locally; it is not yet a quiet-host or controlled-VM claim.

kayg commented

2026-10-08 07:01:08 +00:00

Author

Owner

Filesystem slice committed: c18bf0f8e. Home, private Search/Voice trees and expired archives now use one journalled streaming purge owner. Validation still finishes before the first unlink; directory fsync ordering and retry completeness remain. Voice removal also retries a partial tree safely. No shared filesystem lock spans a page pause. Filesystem clippy and all tests passed. The server slice will move private-tree IO off Tokio and replace all-User reconciliation with source/transfer-recipient reconciliation. The lock regression is now building against the old reconciliation call first.

Filesystem slice committed: c18bf0f8e. Home, private Search/Voice trees and expired archives now use one journalled streaming purge owner. Validation still finishes before the first unlink; directory fsync ordering and retry completeness remain. Voice removal also retries a partial tree safely. No shared filesystem lock spans a page pause. Filesystem clippy and all tests passed. The server slice will move private-tree IO off Tokio and replace all-User reconciliation with source/transfer-recipient reconciliation. The lock regression is now building against the old reconciliation call first.

kayg commented

2026-10-08 07:10:14 +00:00

Author

Owner

Production review evidence for #1420. The fix changes the server and shared filesystem owner. The screenshots show the existing Files and People controls with real API data. Each set uses macOS platform emulation, actual CSS pixels, both themes, and the real upload after a click plus Settings opened with Cmd+,.

Width / theme	Files after click	People	Settings after Cmd+,
390 / paper	Screenshot	Screenshot	Screenshot
390 / tokyo-night	Screenshot	Screenshot	Screenshot
820 / paper	Screenshot	Screenshot	Screenshot
820 / tokyo-night	Screenshot	Screenshot	Screenshot
1440 / paper	Screenshot	Screenshot	Screenshot
1440 / tokyo-night	Screenshot	Screenshot	Screenshot

Production review evidence for #1420. The fix changes the server and shared filesystem owner. The screenshots show the existing Files and People controls with real API data. Each set uses macOS platform emulation, actual CSS pixels, both themes, and the real upload after a click plus Settings opened with Cmd+,. | Width / theme | Files after click | People | Settings after Cmd+, | | --- | --- | --- | --- | | 390 / paper | [Screenshot](https://git.kayg.org/attachments/7512165f-4b14-4831-a3ac-a566b3b7c91e) | [Screenshot](https://git.kayg.org/attachments/b8d2b200-7560-454e-a6e5-3ead6f9286f1) | [Screenshot](https://git.kayg.org/attachments/0cff55b8-8c74-496a-9f9f-c56ef5e64137) | | 390 / tokyo-night | [Screenshot](https://git.kayg.org/attachments/ea931073-601b-4e70-b6ba-ecf1154ae7cd) | [Screenshot](https://git.kayg.org/attachments/f82b3a6b-5b52-42b1-9b1f-8d8cd8d3ba9d) | [Screenshot](https://git.kayg.org/attachments/8f41e48e-c1a1-4032-a6ec-de2fc3eecd8c) | | 820 / paper | [Screenshot](https://git.kayg.org/attachments/67880771-6c44-4a50-8b5c-d6cd3aceb981) | [Screenshot](https://git.kayg.org/attachments/c7609185-d0aa-4f26-b6d5-9c1d46c6d997) | [Screenshot](https://git.kayg.org/attachments/7b6e3080-f64d-4a20-aeb3-586e918ea581) | | 820 / tokyo-night | [Screenshot](https://git.kayg.org/attachments/363d36c6-dad7-4fff-afc5-934736787276) | [Screenshot](https://git.kayg.org/attachments/670dfa48-fe0a-4dea-a15f-de391332c980) | [Screenshot](https://git.kayg.org/attachments/02cc7799-6a62-4d44-bb12-0fc1e2c50c2f) | | 1440 / paper | [Screenshot](https://git.kayg.org/attachments/92b9e382-e18e-4136-b256-15d32a67b303) | [Screenshot](https://git.kayg.org/attachments/23601f20-0f66-486c-ba50-7da848717457) | [Screenshot](https://git.kayg.org/attachments/d914ce69-26e6-42ec-9f8f-e46a7a3adf73) | | 1440 / tokyo-night | [Screenshot](https://git.kayg.org/attachments/7175e50a-c4d9-44ba-8727-ec5d5c81de3c) | [Screenshot](https://git.kayg.org/attachments/07f51b05-e7da-4054-ad1f-96f996172467) | [Screenshot](https://git.kayg.org/attachments/df22d475-9e25-453c-9c09-0ad216fc39c5) |

kayg commented

2026-10-08 08:08:26 +00:00

Author

Owner

Finding and regression for #1420: User deletion called Notes reconcile_all, which selects every User and takes each User's Notes writer lock. Deletion now reconciles only the source User and, for Transfer, the recipient under the existing per-User owner lock. Search and Voice tree removal stays on the background filesystem worker.

The real-router regression holds an unrelated User's existing Notes edit lock while the Admin submits DELETE and waits for completion. Restoring the old reconcile_all call makes the unchanged five-second completion assertion fail:

called `Result::unwrap()` on an `Err` value: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 283 filtered out; finished in 49.88s

The fixed run is in progress. The probe reuses the existing content-free CPU/RSS sampler; 23 sampler/proxy tests pass. Commits: c18bf0f8e, 963e529bc, aa2a75c8d. git fetch origin and the required git merge origin/dev completed: Already up to date. The locked perf VM baseline is queued; no failed startup attempt is counted as a measurement.

Finding and regression for #1420: User deletion called Notes `reconcile_all`, which selects every User and takes each User's Notes writer lock. Deletion now reconciles only the source User and, for Transfer, the recipient under the existing per-User owner lock. Search and Voice tree removal stays on the background filesystem worker. The real-router regression holds an unrelated User's existing Notes edit lock while the Admin submits DELETE and waits for completion. Restoring the old `reconcile_all` call makes the unchanged five-second completion assertion fail: ``` called `Result::unwrap()` on an `Err` value: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 283 filtered out; finished in 49.88s ``` The fixed run is in progress. The probe reuses the existing content-free CPU/RSS sampler; 23 sampler/proxy tests pass. Commits: c18bf0f8e, 963e529bc, aa2a75c8d. `git fetch origin` and the required `git merge origin/dev` completed: `Already up to date.` The locked perf VM baseline is queued; no failed startup attempt is counted as a measurement.

kayg commented

2026-10-08 08:20:01 +00:00

Author

Owner

Timing finding, not a READY claim: the busy-host debug comparison records p95 18.408665 s before and 26.378208 s after. The reference uploads before purge were already p95 6.620434 s and 4.023140 s respectively; load averages at sample publication were 34.49 and 39.97. This does not prove a latency improvement. The existing <2 s assertions remain unchanged and failed in both local runs. All HTTP status, exact-byte retention and purge-completeness checks passed.

The fixed regression that holds an unrelated User's real Notes edit lock passed:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 283 filtered out; finished in 58.79s

The isolation fix is committed at 0ca21f81b0. The perf VM comparison is queued behind the shared infraternal lock. No claim of improved p95 is made from the local results. Filesystem clippy and every filesystem test pass; final server clippy/test and web tests are running. bun run check completed with svelte-check found 0 errors and 4 warnings in 2 files.

Timing finding, not a READY claim: the busy-host debug comparison records p95 18.408665 s before and 26.378208 s after. The reference uploads before purge were already p95 6.620434 s and 4.023140 s respectively; load averages at sample publication were 34.49 and 39.97. This does not prove a latency improvement. The existing <2 s assertions remain unchanged and failed in both local runs. All HTTP status, exact-byte retention and purge-completeness checks passed. The fixed regression that holds an unrelated User's real Notes edit lock passed: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 283 filtered out; finished in 58.79s ``` The isolation fix is committed at 0ca21f81b073e329675a51e41a55b00448c1731d. The perf VM comparison is queued behind the shared infraternal lock. No claim of improved p95 is made from the local results. Filesystem clippy and every filesystem test pass; final server clippy/test and web tests are running. `bun run check` completed with `svelte-check found 0 errors and 4 warnings in 2 files`.

kayg referenced this issue

2026-10-08 08:35:10 +00:00

Investigate higher server RSS in the User purge profile (+25.8%) #1453

kayg commented

2026-10-08 08:36:35 +00:00

Author

Owner

Locked perf VM comparison, prebuilt debug binaries, same frontend and fixture, 50,000 files, eight upload workers, 20 unrelated uploads:

Metric	Before `d395b6e25`	After `0ca21f81b`
Upload p50 during purge	1.120951 s	0.717747 s
Upload p95 during purge	1.677853 s	1.582075 s
Ordinary upload p95 before purge	0.138630 s	0.339698 s
Process CPU in probe window	3.77 s	4.42 s
Endpoint process RSS	339.35 MiB	426.91 MiB
Purge/upload/readback window	3.469708 s	4.611795 s

Load at start: 0.45 before, 0.30 after. Each phase used its own perf-run call and released the lock afterwards. Neither phase built on the VM. Both report, verbatim:

server alive at end: True

==== ROUND 2 FINDINGS 0

==== ROUND 2 SLOW 0

All 20 unrelated uploads returned 201, retained exact bytes after deletion, and stayed below the unchanged 2 s latency assertion. Archive and Transfer checks in the same deletion phase also passed. The profile has no entry in docs/perf/baseline.json; the issue's earlier busy-host p95 was 23.069 s. Do not compare that value to these clean-host runs as a controlled improvement. The endpoint RSS increase is filed at #1453 with the CPU and elapsed-window values and the single-sample limits.

Final server gates and web tests remain in progress. No UI code changed. Screenshot links are in the earlier review comment.

Locked perf VM comparison, prebuilt debug binaries, same frontend and fixture, 50,000 files, eight upload workers, 20 unrelated uploads: | Metric | Before d395b6e25 | After 0ca21f81b | |---|---:|---:| | Upload p50 during purge | 1.120951 s | 0.717747 s | | Upload p95 during purge | 1.677853 s | 1.582075 s | | Ordinary upload p95 before purge | 0.138630 s | 0.339698 s | | Process CPU in probe window | 3.77 s | 4.42 s | | Endpoint process RSS | 339.35 MiB | 426.91 MiB | | Purge/upload/readback window | 3.469708 s | 4.611795 s | Load at start: 0.45 before, 0.30 after. Each phase used its own `perf-run` call and released the lock afterwards. Neither phase built on the VM. Both report, verbatim: ``` server alive at end: True ==== ROUND 2 FINDINGS 0 ==== ROUND 2 SLOW 0 ``` All 20 unrelated uploads returned 201, retained exact bytes after deletion, and stayed below the unchanged 2 s latency assertion. Archive and Transfer checks in the same deletion phase also passed. The profile has no entry in `docs/perf/baseline.json`; the issue's earlier busy-host p95 was 23.069 s. Do not compare that value to these clean-host runs as a controlled improvement. The endpoint RSS increase is filed at #1453 with the CPU and elapsed-window values and the single-sample limits. - [purge-vm-before.json](https://git.kayg.org/attachments/47c8cf6f-7f31-4595-bc37-432baeec608e) - [purge-vm-after.json](https://git.kayg.org/attachments/5ad6277e-d0cc-489d-9fd7-bd62bb991f20) Final server gates and web tests remain in progress. No UI code changed. Screenshot links are in the earlier review comment.

kayg commented

2026-10-08 09:02:01 +00:00

Author

Owner

#1420 final report

Built: User purge now uses the shared journalled tree owner for Home, Search, Voice and expired archives. Validation and removal run in bounded pages with IO pauses. The durable intent stays until removal completes. Search and Voice removal runs on the filesystem worker. Deletion reconciles only the source User and the Transfer recipient. It does not acquire unrelated Users' Notes writer locks.

Head: 0ca21f81b073e329675a51e41a55b00448c1731d on job/purge-1420; base d395b6e25da20d8c67b8b9fc1a593b813939c8a7. Four atomic commits: c18bf0f8e, 963e529bc, aa2a75c8d, 0ca21f81b. The required fetch and merge of origin/dev returned Already up to date.

Files:

crates/calternal-fs/src/user_homes.rs
crates/calternal-fs/src/root.rs
crates/calternal-server/src/wire.rs
tests/adversarial/attack2.py
tests/adversarial/mail_proxy.py
tests/adversarial/test_mail_proxy.py
bench/user-purge-1420.sh

Regression: the real-router test holds an unrelated User's real Notes edit lock through an Admin DELETE and deletion completion. Restoring the old reconcile_all call fails the unchanged completion deadline with Elapsed(()). The fixed call passes. The real-server deletion probe keeps all existing status and <2 s upload assertions. It now reads all 20 uploads back after deletion and checks exact bytes. The shared content-free resource sampler has 23 passing tests.

Performance: two locked perf VM runs used prebuilt debug binaries, the same frontend, 50,000 files, eight workers and 20 unrelated uploads. No build ran on the VM. The lock was released between phases.

Metric	Before	After
Upload p50 during purge	1.120951 s	0.717747 s
Upload p95 during purge	1.677853 s	1.582075 s
Server CPU in probe window	3.77 s	4.42 s
Endpoint server RSS	339.35 MiB	426.91 MiB
Purge/upload/readback window	3.469708 s	4.611795 s

Start load: 0.45 before, 0.30 after. Both runs report zero findings and zero SLOW results. All uploads returned 201, kept exact bytes, and passed the unchanged latency limit. Archive, Transfer, purge completion and server liveness checks passed. The endpoint RSS increase (+25.8%) and CPU values are filed in #1453. The endpoint sample is not a measured peak; RSS was already higher before purge. These single samples do not establish a cause.

Expected cost: two passes over the purged tree, plus one sync per removed directory. Memory and descriptors depend on depth, capped at 256. Notes reconciliation visits only affected Users. A purge has no live source Home to scan. The 50,000-file measurement above is the worst-case fixture. docs/perf/baseline.json has no User purge profile. The issue's earlier busy-host p95 was 23.069 s. Local debug p95 was 18.409 s before and 26.378 s after under load 34–40, with reference uploads already over 2 s. Those local failures are recorded, not presented as a controlled improvement.

Screenshots: 18 actual-size captures are attached in the review comment: Mac platform emulation at 390, 820 and 1440 px, light and dark, Files after a click, People, and People after the Settings shortcut. They show real API data from an isolated production-build fixture. No UI controls were added.

UX gaps closed: unrelated uploads keep their bytes; deletion completes while another User holds an edit lock; large private Derived data trees share the complete, restartable purge. UX gaps left: no additional UI flow was in scope. Busy-host upload timing remains above the target and is documented.

Decisions: DESIGN does not set the purge page size or pause. Use 128 work entries and a 2 ms pause. Count directory completion as work so directory-only trees also yield. Keep depth/name validation before the first unlink and keep the durable intent until all removal completes. Reuse the existing resource sampler and existing deletion probe.

Verification allowance: the first stock bun run test had 2113 passing tests and three Test timed out in 5000ms failures. The focused default-timeout run reproduced two timeouts. The complete retry used bun run test --maxWorkers=2 --testTimeout=20000 and passed all 2116 tests. No source assertion, fixture, status expectation or purge latency limit changed. The allowance changes only Vitest's runner timeout.

Gate excerpts, verbatim:

cargo fmt --check: exit 0, no output.

cargo clippy -p calternal-fs --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 55s

cargo test -p calternal-fs -- --test-threads=4

test result: ok. 99 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 63.62s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.21s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.48s
test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.85s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 24m 03s

cargo test -p calternal-server -- --test-threads=4

test result: ok. 272 passed; 0 failed; 12 ignored; 0 measured; 0 filtered out; finished in 384.23s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 55.55s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.93s

bun run check

perf-lint: PASS; 0 violations; 22544 scoped exceptions
svelte-check found 0 errors and 4 warnings in 2 files

bun run test --maxWorkers=2 --testTimeout=20000

 Test Files  297 passed (297)
      Tests  2116 passed (2116)

python3 -m unittest discover -s tests/adversarial -p test_mail_proxy.py

Ran 23 tests in 0.184s
OK

For the merge round:

ADVERSARIAL_SERVER_BIN="$CALTERNAL_SERVER_RELEASE_BIN" bash tests/adversarial/run.sh: run the combined API, authz, XUser and robustness matrices against the merged release server.
perf-run calternal purge-1420-release 10 -- bash bench/user-purge-1420.sh: use a prebuilt release server and an already provisioned deletion fixture on the perf VM. Export ADVERSARIAL_WORK_DIR, ADVERSARIAL_PORT, ADVERSARIAL_SERVER_PID, PURGE_PROFILE_OUTPUT and PERF_LOCATION=perf-vm first. Prove upload p95 <2 s and exact-byte retention. Compare release CPU and a sampled RSS peak for #1453.

Known gaps: the controlled timing evidence uses debug binaries. Release comparison and full matrices belong to the merge round. Endpoint RSS needs the #1453 follow-up. Stock 5 s web runner timeouts remain documented; the full functional suite passed with the stated runner allowance. bun run check has four warnings and zero errors.

Cleanup: cargo clean removed 16510 files, 11.3 GiB. Web build output and disposable benchmark binaries/fixtures were removed. Screenshots, profiles and gate logs remain under artifacts/. The worktree is clean. No secrets were displayed. No pushes, deploys or branch merges were done except the required origin/dev update. No issue was closed.

#1420 final report Built: User purge now uses the shared journalled tree owner for Home, Search, Voice and expired archives. Validation and removal run in bounded pages with IO pauses. The durable intent stays until removal completes. Search and Voice removal runs on the filesystem worker. Deletion reconciles only the source User and the Transfer recipient. It does not acquire unrelated Users' Notes writer locks. Head: `0ca21f81b073e329675a51e41a55b00448c1731d` on `job/purge-1420`; base `d395b6e25da20d8c67b8b9fc1a593b813939c8a7`. Four atomic commits: c18bf0f8e, 963e529bc, aa2a75c8d, 0ca21f81b. The required fetch and merge of origin/dev returned `Already up to date.` Files: - crates/calternal-fs/src/user_homes.rs - crates/calternal-fs/src/root.rs - crates/calternal-server/src/wire.rs - tests/adversarial/attack2.py - tests/adversarial/mail_proxy.py - tests/adversarial/test_mail_proxy.py - bench/user-purge-1420.sh Regression: the real-router test holds an unrelated User's real Notes edit lock through an Admin DELETE and deletion completion. Restoring the old `reconcile_all` call fails the unchanged completion deadline with `Elapsed(())`. The fixed call passes. The real-server deletion probe keeps all existing status and <2 s upload assertions. It now reads all 20 uploads back after deletion and checks exact bytes. The shared content-free resource sampler has 23 passing tests. Performance: two locked perf VM runs used prebuilt debug binaries, the same frontend, 50,000 files, eight workers and 20 unrelated uploads. No build ran on the VM. The lock was released between phases. | Metric | Before | After | |---|---:|---:| | Upload p50 during purge | 1.120951 s | 0.717747 s | | Upload p95 during purge | 1.677853 s | 1.582075 s | | Server CPU in probe window | 3.77 s | 4.42 s | | Endpoint server RSS | 339.35 MiB | 426.91 MiB | | Purge/upload/readback window | 3.469708 s | 4.611795 s | Start load: 0.45 before, 0.30 after. Both runs report zero findings and zero SLOW results. All uploads returned 201, kept exact bytes, and passed the unchanged latency limit. Archive, Transfer, purge completion and server liveness checks passed. The endpoint RSS increase (+25.8%) and CPU values are filed in #1453. The endpoint sample is not a measured peak; RSS was already higher before purge. These single samples do not establish a cause. Expected cost: two passes over the purged tree, plus one sync per removed directory. Memory and descriptors depend on depth, capped at 256. Notes reconciliation visits only affected Users. A purge has no live source Home to scan. The 50,000-file measurement above is the worst-case fixture. `docs/perf/baseline.json` has no User purge profile. The issue's earlier busy-host p95 was 23.069 s. Local debug p95 was 18.409 s before and 26.378 s after under load 34–40, with reference uploads already over 2 s. Those local failures are recorded, not presented as a controlled improvement. Screenshots: 18 actual-size captures are attached in the review comment: Mac platform emulation at 390, 820 and 1440 px, light and dark, Files after a click, People, and People after the Settings shortcut. They show real API data from an isolated production-build fixture. No UI controls were added. UX gaps closed: unrelated uploads keep their bytes; deletion completes while another User holds an edit lock; large private Derived data trees share the complete, restartable purge. UX gaps left: no additional UI flow was in scope. Busy-host upload timing remains above the target and is documented. Decisions: DESIGN does not set the purge page size or pause. Use 128 work entries and a 2 ms pause. Count directory completion as work so directory-only trees also yield. Keep depth/name validation before the first unlink and keep the durable intent until all removal completes. Reuse the existing resource sampler and existing deletion probe. Verification allowance: the first stock `bun run test` had 2113 passing tests and three `Test timed out in 5000ms` failures. The focused default-timeout run reproduced two timeouts. The complete retry used `bun run test --maxWorkers=2 --testTimeout=20000` and passed all 2116 tests. No source assertion, fixture, status expectation or purge latency limit changed. The allowance changes only Vitest's runner timeout. Gate excerpts, verbatim: `cargo fmt --check`: exit 0, no output. `cargo clippy -p calternal-fs --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 55s ``` `cargo test -p calternal-fs -- --test-threads=4` ``` test result: ok. 99 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 63.62s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.21s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.48s test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.85s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 24m 03s ``` `cargo test -p calternal-server -- --test-threads=4` ``` test result: ok. 272 passed; 0 failed; 12 ignored; 0 measured; 0 filtered out; finished in 384.23s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 55.55s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.93s ``` `bun run check` ``` perf-lint: PASS; 0 violations; 22544 scoped exceptions svelte-check found 0 errors and 4 warnings in 2 files ``` `bun run test --maxWorkers=2 --testTimeout=20000` ``` Test Files 297 passed (297) Tests 2116 passed (2116) ``` `python3 -m unittest discover -s tests/adversarial -p test_mail_proxy.py` ``` Ran 23 tests in 0.184s OK ``` For the merge round: - `ADVERSARIAL_SERVER_BIN="$CALTERNAL_SERVER_RELEASE_BIN" bash tests/adversarial/run.sh`: run the combined API, authz, XUser and robustness matrices against the merged release server. - `perf-run calternal purge-1420-release 10 -- bash bench/user-purge-1420.sh`: use a prebuilt release server and an already provisioned deletion fixture on the perf VM. Export ADVERSARIAL_WORK_DIR, ADVERSARIAL_PORT, ADVERSARIAL_SERVER_PID, PURGE_PROFILE_OUTPUT and PERF_LOCATION=perf-vm first. Prove upload p95 <2 s and exact-byte retention. Compare release CPU and a sampled RSS peak for #1453. Known gaps: the controlled timing evidence uses debug binaries. Release comparison and full matrices belong to the merge round. Endpoint RSS needs the #1453 follow-up. Stock 5 s web runner timeouts remain documented; the full functional suite passed with the stated runner allowance. `bun run check` has four warnings and zero errors. Cleanup: cargo clean removed 16510 files, 11.3 GiB. Web build output and disposable benchmark binaries/fixtures were removed. Screenshots, profiles and gate logs remain under artifacts/. The worktree is clean. No secrets were displayed. No pushes, deploys or branch merges were done except the required origin/dev update. No issue was closed.

kayg referenced this issue from a commit

2026-10-08 10:41:40 +00:00

Merge job/purge-1420 (#1420)

kayg referenced this issue

2026-10-08 16:42:52 +00:00

Record loaded-host upload latency during a 50000-file User purge #1476

kayg commented

2026-10-08 18:41:21 +00:00

Author

Owner

Deployed to calternal.cloud in merge round 35a (build cc1af3c12, live 2026-10-09 00:12 IST). Closing.

Deployed to calternal.cloud in merge round 35a (build cc1af3c12, live 2026-10-09 00:12 IST). Closing.