Perf VM: disk timings not repeatable, A/B measurements blocked

kayg commented

2026-10-12 06:09:30 +00:00

Owner

The perf VM (root@10.69.69.63) cannot give stable disk timings, so Tab-switch A/B work (#1557) has produced no valid samples for three rounds.

Evidence (calact7, 2026-10-12, under perf-run): undelayed fio 4k randread QD1 direct=1 on the backing file on /dev/sdb gave p50 9.37 ms, then 0.21 ms, then 0.07 ms (about 142x spread). Load averages 1.57 1.75 1.49 during the probe on a quiet lock. Earlier runs: delayed (dm-delay 8 ms) QD1 IOPS between 65 and 99.

Goal: a perf VM whose disk timings are repeatable (undelayed p50 spread under 2x across 5 probes; delayed QD1 p50 within 8–14 ms), and an idle load average near 0 when the perf-run lock is free.
Check: what runs on the VM at idle (timers, indexers, leftover servers or browsers from jobs, journald), virtio disk cache mode and whether the host page cache serves "direct" reads, whether /dev/sdb is a real HDD, SSD or a Proxmox volume shared with other VMs, CPU steal time. Fix what is inside the VM (stop stray processes, add a quiesce step to perf-run, make the HDD emulation sit on a file/device that bypasses host caching, for example by dropping caches or using a dedicated LV). Changes on the Proxmox host are NOT allowed: list them for the owner instead with exact commands. Record before/after probe numbers on this issue.

The perf VM (root@10.69.69.63) cannot give stable disk timings, so Tab-switch A/B work (#1557) has produced no valid samples for three rounds. Evidence (calact7, 2026-10-12, under perf-run): undelayed fio 4k randread QD1 direct=1 on the backing file on /dev/sdb gave p50 9.37 ms, then 0.21 ms, then 0.07 ms (about 142x spread). Load averages 1.57 1.75 1.49 during the probe on a quiet lock. Earlier runs: delayed (dm-delay 8 ms) QD1 IOPS between 65 and 99. Goal: a perf VM whose disk timings are repeatable (undelayed p50 spread under 2x across 5 probes; delayed QD1 p50 within 8–14 ms), and an idle load average near 0 when the perf-run lock is free. Check: what runs on the VM at idle (timers, indexers, leftover servers or browsers from jobs, journald), virtio disk cache mode and whether the host page cache serves "direct" reads, whether /dev/sdb is a real HDD, SSD or a Proxmox volume shared with other VMs, CPU steal time. Fix what is inside the VM (stop stray processes, add a quiesce step to perf-run, make the HDD emulation sit on a file/device that bypasses host caching, for example by dropping caches or using a dedicated LV). Changes on the Proxmox host are NOT allowed: list them for the owner instead with exact commands. Record before/after probe numbers on this issue.

kayg commented

2026-10-12 06:10:11 +00:00

Author

Owner

Starting issue #1587 on branch job/perfvm-1587, based at 8e840962979acd2bdf58141888618ab5560f728e (origin/dev merge-base). I have read CLAUDE.md, CONTEXT.md, and DESIGN.md §59. I am inspecting the VM's idle workload, virtual disk path, and perf-run probe setup before making a small locked change.

Starting issue #1587 on branch `job/perfvm-1587`, based at `8e840962979acd2bdf58141888618ab5560f728e` (`origin/dev` merge-base). I have read CLAUDE.md, CONTEXT.md, and DESIGN.md §59. I am inspecting the VM's idle workload, virtual disk path, and perf-run probe setup before making a small locked change.

kayg commented

2026-10-12 06:18:56 +00:00

Author

Owner

Finding (diagnostic captured inside my perf-run reservation): the VM sees sdb as a 100 GiB QEMU HARDDISK with ROTA=1 and queue/write_cache=write back. Its guest mount is /mnt/hdd (/dev/sdb, ext4); the 40 GiB loop backing image is fully allocated. The delayed device uses /dev/loop0 with direct I/O and an 8 ms dm-delay table. At the diagnostic sample the load average was 2.86 2.37 1.72, vmstat showed 7% I/O wait and 1% steal. No browser or calternal server appeared in the top-process sample; dm-delay-flush-worker used 4.8% CPU, and three parentless python3 processes remain to identify. QEMU's host cache mode and physical backing medium are not visible in the guest.

Finding (diagnostic captured inside my perf-run reservation): the VM sees `sdb` as a 100 GiB `QEMU HARDDISK` with `ROTA=1` and `queue/write_cache=write back`. Its guest mount is `/mnt/hdd` (`/dev/sdb`, ext4); the 40 GiB loop backing image is fully allocated. The delayed device uses `/dev/loop0` with direct I/O and an 8 ms dm-delay table. At the diagnostic sample the load average was `2.86 2.37 1.72`, vmstat showed 7% I/O wait and 1% steal. No browser or calternal server appeared in the top-process sample; `dm-delay-flush-worker` used 4.8% CPU, and three parentless `python3` processes remain to identify. QEMU's host cache mode and physical backing medium are not visible in the guest.

kayg commented

2026-10-12 06:20:09 +00:00

Author

Owner

Finding from the five undelayed baseline samples (all inside perf-run, same 128 MiB direct QD1 fio configuration, distinct seeds): p50 was 8.094, 6.128, 0.561, 0.200 and 0.032 ms; IOPS rose from 106 to 3,932. The p50 spread is 253x. The backing-file reads become faster across runs despite direct=1, consistent with a cache effect or zero/unwritten image extents. The delayed five-sample series is still running.

kayg commented

2026-10-12 06:22:14 +00:00

Author

Owner

Baseline batch complete. The undelayed p50 values were 8.094, 6.128, 0.561, 0.200 and 0.032 ms (253x max/min; 106–3,932 IOPS). The delayed p50 values were 8.585, 8.028, 8.225, 8.094 and 8.094 ms (1.07x max/min), all within the 8–14 ms target. Every probe used direct=1, 4 KiB randread, QD1, 128 MiB, 15 seconds, and ran inside one perf-run reservation. I added inode extent and parentless Python process details to the probe's locked diagnostic for the next phase.

kayg commented

2026-10-12 06:25:19 +00:00

Author

Owner

The comparison path through the existing direct-I/O loop returned undelayed p50 values 0.081, 0.073, 0.067, 0.060 and 0.063 ms (1.35x max/min), with 5,580–14,696 IOPS. The numbers are repeatable but are too fast to treat as HDD latency. They are consistent with QEMU host caching or image zero/unwritten sectors; direct I/O in the guest does not prove the Proxmox backend bypasses its page cache. The guest reports write-cache=write back, but cannot identify the host cache mode. I will keep these values marked as non-physical and use the delayed-path results to qualify the emulated HDD.

kayg commented

2026-10-12 06:36:11 +00:00

Author

Owner

I installed the versioned perf-run helper under its existing lock. The new --quiesce option ran sync and dropped guest caches before this phase, then logged the lock-entry load average. The first two undelayed p50 values from the unchanged backing-file path were 2.277 and 0.293 ms. Guest cache quiesce did not prevent the path from becoming faster, so guest page-cache clearing alone does not fix the read source. The rest of the five-plus-five phase is still running.

I installed the versioned perf-run helper under its existing lock. The new `--quiesce` option ran `sync` and dropped guest caches before this phase, then logged the lock-entry load average. The first two undelayed p50 values from the unchanged backing-file path were 2.277 and 0.293 ms. Guest cache quiesce did not prevent the path from becoming faster, so guest page-cache clearing alone does not fix the read source. The rest of the five-plus-five phase is still running.

kayg commented

2026-10-12 06:38:11 +00:00

Author

Owner

The quiesced five-plus-five batch is complete. Undelayed p50 on the same backing file was 2.277, 0.293, 0.081, 0.065 and 0.053 ms (43.0x max/min), so sync plus guest drop_caches did not fix the instability. Delayed p50 was 8.847, 8.094, 8.028, 8.028 and 8.094 ms (1.10x max/min), all within 8–14 ms. The perf-run helper now prints lock-entry load and has an opt-in guest quiesce; I will not count the undelayed results as valid until the owner sets the Proxmox cache mode to bypass host caching.

The quiesced five-plus-five batch is complete. Undelayed p50 on the same backing file was 2.277, 0.293, 0.081, 0.065 and 0.053 ms (43.0x max/min), so `sync` plus guest `drop_caches` did not fix the instability. Delayed p50 was 8.847, 8.094, 8.028, 8.028 and 8.094 ms (1.10x max/min), all within 8–14 ms. The perf-run helper now prints lock-entry load and has an opt-in guest quiesce; I will not count the undelayed results as valid until the owner sets the Proxmox cache mode to bypass host caching.

kayg commented

2026-10-12 06:41:14 +00:00

Author

Owner

The three long-lived Python processes are inline-code process-tree monitors, inferred from their sanitized AST summaries (imports json,os,pathlib,signal,sys,time; functions stop and tree; a top-level loop). Each has PPID 1, started on Oct 8, has only the three standard descriptors on /dev/null, and owns no listening socket. They are not servers or browsers. I will send TERM to these verified stale monitors inside a short perf-run reservation and recheck the free-lock load average afterward.

The three long-lived Python processes are inline-code process-tree monitors, inferred from their sanitized AST summaries (imports `json,os,pathlib,signal,sys,time`; functions `stop` and `tree`; a top-level loop). Each has PPID 1, started on Oct 8, has only the three standard descriptors on `/dev/null`, and owns no listening socket. They are not servers or browsers. I will send TERM to these verified stale monitors inside a short perf-run reservation and recheck the free-lock load average afterward.

kayg commented

2026-10-12 06:45:26 +00:00

Author

Owner

The locked recheck after TERM showed no parentless Python processes. No browser, calternal server, running container or non-system listening socket was present. With the perf-run lock free, perf-run status reported load_average: 0.80 1.86 1.91 1/197 2616150; CPU pressure was some avg10=0.00 avg60=0.10 avg300=0.15 and full avg10=0.00 avg60=0.00 avg300=0.00. The one-minute load is falling, but the five- and fifteen-minute averages still include prior shared runs.

The locked recheck after TERM showed no parentless Python processes. No browser, calternal server, running container or non-system listening socket was present. With the perf-run lock free, `perf-run status` reported `load_average: 0.80 1.86 1.91 1/197 2616150`; CPU pressure was `some avg10=0.00 avg60=0.10 avg300=0.15` and `full avg10=0.00 avg60=0.00 avg300=0.00`. The one-minute load is falling, but the five- and fifteen-minute averages still include prior shared runs.

kayg commented

2026-10-12 06:47:50 +00:00

Author

Owner

#1587 report — branch job/perfvm-1587, head d01688eb7883928dfe6eeb94e54c942bccbccb85.

Both phases ran under the perf-run lock. Every fio sample used 4 KiB randread, QD1, direct=1, 128 MiB and 15 seconds. The after phase used the same backing-file path with perf-run --quiesce (sync plus guest cache drop).

Phase	Undelayed p50, samples 1–5 (ms)	Delayed 8 ms p50, samples 1–5 (ms)	max/min spread
Before	8.094, 6.128, 0.561, 0.200, 0.032	8.585, 8.028, 8.225, 8.094, 8.094	250x / 1.07x
After guest quiesce	2.277, 0.293, 0.081, 0.065, 0.053	8.847, 8.094, 8.028, 8.028, 8.094	43x / 1.10x

The delayed path met the 8–14 ms p50 target in all ten samples. Guest quiesce did not stabilize undelayed reads. A direct-I/O loop diagnostic was repeatable at 0.060–0.081 ms, but its 5,580–14,696 IOPS are not valid HDD latency. I did not count it as a disk result.

I added scripts/perf-run with an opt-in --quiesce flag and free-lock load reporting, plus bench/perfvm-disk-stability-1587.sh and docs/perf/2026-10-12-perfvm-1587.md. I installed the helper on the VM under the old lock; its SHA-256 is 5b16609f2b84d40f4113938a9da6fd32d61fbf1d50d70fc8e26ca6062d93fb93. I sent TERM to three parentless Python process-tree monitors started on Oct 8. A locked recheck found none. No browser, calternal server, container or non-system listener remained.

With the lock free, perf-run status reported load_average: 0.80 1.86 1.91 1/197 2616150; CPU pressure was low (some avg10=0.00 avg60=0.10 avg300=0.15). The longer load averages still include recent shared runs.

The guest sees /dev/sdb as a 100 GiB QEMU HARDDISK, ROTA=1, write_cache=write back. The Proxmox cache mode and physical storage are not visible in the guest. I made no Proxmox host changes. Owner action: resolve the VM ID and confirm the disk slot:

VMID=$(pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.name == "perf-test-69-63") | .vmid')
qm config "$VMID" | grep -E '^(scsi|virtio|sata)[0-9]+:'

If the output confirms guest /dev/sdb is scsi1, set that disk to cache=none and restart the VM if required:

DISK=$(qm config "$VMID" | sed -n 's/^scsi1: //p')
DISK=$(printf '%s\n' "$DISK" | sed -E 's/(^|,)cache=[^,]+/\1cache=none/; t; s/$/,cache=none/')
qm set "$VMID" --scsi1 "$DISK"

Then rerun five undelayed QD1 probes against the same path. The issue remains open for this host-side action. The design doc does not specify the Proxmox setting; cache=none is my inference from the cached-looking guest measurements.

Checks:

$ bash -n scripts/perf-run && bash -n bench/perfvm-disk-stability-1587.sh
(exit 0; no output)
$ git diff --check
(exit 0; no output)
$ cargo clean
     Removed 1 file, 356B total

No Rust or web files changed, so Rust/web gates were not run. No merge, push or deploy occurred.

#1587 report — branch `job/perfvm-1587`, head `d01688eb7883928dfe6eeb94e54c942bccbccb85`. Both phases ran under the perf-run lock. Every fio sample used 4 KiB randread, QD1, `direct=1`, 128 MiB and 15 seconds. The after phase used the same backing-file path with `perf-run --quiesce` (sync plus guest cache drop). | Phase | Undelayed p50, samples 1–5 (ms) | Delayed 8 ms p50, samples 1–5 (ms) | max/min spread | | --- | --- | --- | --- | | Before | 8.094, 6.128, 0.561, 0.200, 0.032 | 8.585, 8.028, 8.225, 8.094, 8.094 | 250x / 1.07x | | After guest quiesce | 2.277, 0.293, 0.081, 0.065, 0.053 | 8.847, 8.094, 8.028, 8.028, 8.094 | 43x / 1.10x | The delayed path met the 8–14 ms p50 target in all ten samples. Guest quiesce did not stabilize undelayed reads. A direct-I/O loop diagnostic was repeatable at 0.060–0.081 ms, but its 5,580–14,696 IOPS are not valid HDD latency. I did not count it as a disk result. I added `scripts/perf-run` with an opt-in `--quiesce` flag and free-lock load reporting, plus `bench/perfvm-disk-stability-1587.sh` and `docs/perf/2026-10-12-perfvm-1587.md`. I installed the helper on the VM under the old lock; its SHA-256 is `5b16609f2b84d40f4113938a9da6fd32d61fbf1d50d70fc8e26ca6062d93fb93`. I sent TERM to three parentless Python process-tree monitors started on Oct 8. A locked recheck found none. No browser, calternal server, container or non-system listener remained. With the lock free, `perf-run status` reported `load_average: 0.80 1.86 1.91 1/197 2616150`; CPU pressure was low (`some avg10=0.00 avg60=0.10 avg300=0.15`). The longer load averages still include recent shared runs. The guest sees `/dev/sdb` as a 100 GiB `QEMU HARDDISK`, `ROTA=1`, `write_cache=write back`. The Proxmox cache mode and physical storage are not visible in the guest. I made no Proxmox host changes. Owner action: resolve the VM ID and confirm the disk slot: ```sh VMID=$(pvesh get /cluster/resources --type vm --output-format json | jq -r '.[] | select(.name == "perf-test-69-63") | .vmid') qm config "$VMID" | grep -E '^(scsi|virtio|sata)[0-9]+:' ``` If the output confirms guest `/dev/sdb` is `scsi1`, set that disk to `cache=none` and restart the VM if required: ```sh DISK=$(qm config "$VMID" | sed -n 's/^scsi1: //p') DISK=$(printf '%s\n' "$DISK" | sed -E 's/(^|,)cache=[^,]+/\1cache=none/; t; s/$/,cache=none/') qm set "$VMID" --scsi1 "$DISK" ``` Then rerun five undelayed QD1 probes against the same path. The issue remains open for this host-side action. The design doc does not specify the Proxmox setting; `cache=none` is my inference from the cached-looking guest measurements. Checks: ```text $ bash -n scripts/perf-run && bash -n bench/perfvm-disk-stability-1587.sh (exit 0; no output) $ git diff --check (exit 0; no output) $ cargo clean Removed 1 file, 356B total ``` No Rust or web files changed, so Rust/web gates were not run. No merge, push or deploy occurred.

kayg referenced this issue

2026-10-12 06:58:13 +00:00

Reduce client work during Calendar Tab activation #1557

kayg referenced this issue

2026-10-12 06:58:35 +00:00

Reduce client work during Calendar Tab activation #1557

kayg referenced this issue

2026-10-12 08:27:34 +00:00

Reduce client work during Calendar Tab activation #1557

Rows
Columns

Perf VM: disk timings not repeatable, A/B measurements blocked #1587