Investigate higher server RSS in the User purge profile (+25.8%) #1453

Open
opened 2026-10-08 08:35:10 +00:00 by kayg · 0 comments
Owner

The #1420 locked perf VM comparison passed the User purge correctness probe and the unchanged upload p95 <2 s assertion. It also recorded higher endpoint RSS. Investigate this sample before assigning a cause. This is a performance follow-up, not a merge gate.

Both runs used prebuilt debug binaries, the same production frontend, 50,000 files and eight upload workers with 20 uploads. Each run held the VM lock through perf-run calternal purge-1420-{before,after} 10 -- .... No build ran on the VM. Load at start: 0.45 before, 0.30 after. Each probe reported zero findings and zero SLOW results.

Metric Before d395b6e25 After 0ca21f81b
Upload p50 during purge 1.120951 s 0.717747 s
Upload p95 during purge 1.677853 s 1.582075 s
Server CPU in probe window 3.77 s 4.42 s (+17.24%)
Server RSS at window end 355835904 bytes 447651840 bytes (+25.80%)
Server RSS before purge 308203520 bytes 369741824 bytes
Purge/upload/readback window 3.469708 s 4.611795 s

The endpoint RSS change exceeds the published 15% peak-RSS comparison budget, but these are endpoint samples, not a measured peak. RSS was already higher before purge. Background jobs can affect these process counters. One sample does not prove that the purge walker caused the increase. The new two-pass walker pauses for 2 ms every 128 work entries; the longer window is expected in part from this IO pacing.

docs/perf/baseline.json has no User purge profile. The initial issue's busy-host p95 was 23.069 s. The new profile is bench/user-purge-1420.sh; it reuses tests/adversarial/attack2.py and writes JSON through PURGE_PROFILE_OUTPUT. The exact before/after JSON files are attached to #1420. The merge round should compare prebuilt release binaries with the same fixture. Measure CPU and a sampled RSS peak in addition to the endpoint values. Keep the upload and byte-retention assertions.

The #1420 locked perf VM comparison passed the User purge correctness probe and the unchanged upload p95 <2 s assertion. It also recorded higher endpoint RSS. Investigate this sample before assigning a cause. This is a performance follow-up, not a merge gate. Both runs used prebuilt debug binaries, the same production frontend, 50,000 files and eight upload workers with 20 uploads. Each run held the VM lock through `perf-run calternal purge-1420-{before,after} 10 -- ...`. No build ran on the VM. Load at start: 0.45 before, 0.30 after. Each probe reported zero findings and zero SLOW results. | Metric | Before d395b6e25 | After 0ca21f81b | |---|---:|---:| | Upload p50 during purge | 1.120951 s | 0.717747 s | | Upload p95 during purge | 1.677853 s | 1.582075 s | | Server CPU in probe window | 3.77 s | 4.42 s (+17.24%) | | Server RSS at window end | 355835904 bytes | 447651840 bytes (+25.80%) | | Server RSS before purge | 308203520 bytes | 369741824 bytes | | Purge/upload/readback window | 3.469708 s | 4.611795 s | The endpoint RSS change exceeds the published 15% peak-RSS comparison budget, but these are endpoint samples, not a measured peak. RSS was already higher before purge. Background jobs can affect these process counters. One sample does not prove that the purge walker caused the increase. The new two-pass walker pauses for 2 ms every 128 work entries; the longer window is expected in part from this IO pacing. `docs/perf/baseline.json` has no User purge profile. The initial issue's busy-host p95 was 23.069 s. The new profile is `bench/user-purge-1420.sh`; it reuses `tests/adversarial/attack2.py` and writes JSON through PURGE_PROFILE_OUTPUT. The exact before/after JSON files are attached to #1420. The merge round should compare prebuilt release binaries with the same fixture. Measure CPU and a sampled RSS peak in addition to the endpoint values. Keep the upload and byte-retention assertions.
Commenting is not possible because the repository is archived.
No labels
No milestone
No project
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal-v0#1453
No description provided.