Browser timing checks run on the perf VM

kayg commented

2026-10-12 08:00:31 +00:00

Owner

Move browser timing checks from the loaded build host to the dedicated performance VM. Add idempotent VM setup and a run/A-B script that builds locally, transfers only required artifacts, locks and records the VM environment, runs Playwright specs with their existing RTT settings, returns reports and traces, and cleans up. Document usage and prove the three production-red checks tracked by #176 with an interleaved 3-run comparison of 1f77a449 and origin/release-prod-3w. Parent: #176.

Move browser timing checks from the loaded build host to the dedicated performance VM. Add idempotent VM setup and a run/A-B script that builds locally, transfers only required artifacts, locks and records the VM environment, runs Playwright specs with their existing RTT settings, returns reports and traces, and cleans up. Document usage and prove the three production-red checks tracked by #176 with an interleaved 3-run comparison of 1f77a449 and origin/release-prod-3w. Parent: #176.

kayg commented

2026-10-12 08:01:42 +00:00

Author

Owner

Starting on perf-vm-browser, base 638e4f1e9114969ed431feb5aa9f8498f2225dc7 (origin/dev). Read #184 and parent #176 history; no earlier #184 work. #176 acceptance remains unmet with theme long-task, prediction/resume, and access first-paint failures; limits and sample counts stay unchanged. Building the VM browser setup/runner, then the prescribed base 1f77a449 vs origin/release-prod-3w three-run A/B and a usage note. No deployment.

Starting on `perf-vm-browser`, base `638e4f1e9114969ed431feb5aa9f8498f2225dc7` (`origin/dev`). Read #184 and parent #176 history; no earlier #184 work. #176 acceptance remains unmet with theme long-task, prediction/resume, and access first-paint failures; limits and sample counts stay unchanged. Building the VM browser setup/runner, then the prescribed base `1f77a449` vs `origin/release-prod-3w` three-run A/B and a usage note. No deployment.

kayg commented

2026-10-12 08:05:47 +00:00

Author

Owner

Perf VM finding: strict NetBird SSH to root@10.69.69.208 succeeded; under /var/lock/perf-bench.lock, the host reports 4 vCPUs, Debian 13, kernel 6.12.69+deb13-amd64, load 0.13 0.22 0.27, and 91 GB free. bun, node, and rsync are absent; fc-list is present. Local Bun is 1.4.2, and apps/tracer/bun.lock plus package.json are unchanged between 1f77a449 and origin/release-prod-3w. apps/tracer/playwright.config.ts:43 currently rebuilds in webServer; the runner must serve the locally built dist. The existing config intentionally keeps arbitrary browser traces off because auth ceremonies can contain capabilities; the runner will preserve that and return only test-produced result artifacts.

Perf VM finding: strict NetBird SSH to `root@10.69.69.208` succeeded; under `/var/lock/perf-bench.lock`, the host reports 4 vCPUs, Debian 13, kernel `6.12.69+deb13-amd64`, load `0.13 0.22 0.27`, and 91 GB free. `bun`, `node`, and `rsync` are absent; `fc-list` is present. Local Bun is 1.4.2, and `apps/tracer/bun.lock` plus `package.json` are unchanged between `1f77a449` and `origin/release-prod-3w`. `apps/tracer/playwright.config.ts:43` currently rebuilds in `webServer`; the runner must serve the locally built `dist`. The existing config intentionally keeps arbitrary browser traces off because auth ceremonies can contain capabilities; the runner will preserve that and return only test-produced result artifacts.

kayg commented

2026-10-12 08:17:06 +00:00

Author

Owner

TypeScript check setup finding: the initial bun run check failed with TS2688 because bun-types and node type definitions were absent from this worktree's node_modules. Shell syntax and Python AST checks passed. Installing the repository lock with bun install --frozen-lockfile before retrying; no source change is implied by the missing dependency tree.

TypeScript check setup finding: the initial `bun run check` failed with TS2688 because `bun-types` and `node` type definitions were absent from this worktree's `node_modules`. Shell syntax and Python AST checks passed. Installing the repository lock with `bun install --frozen-lockfile` before retrying; no source change is implied by the missing dependency tree.

kayg commented

2026-10-12 08:23:37 +00:00

Author

Owner

Perf VM transfer finding: the one-time apt step installed rsync 3.5.0 and the Noto/Liberation/DejaVu font packages. The first rsync -e 'netbird ssh ...' failed before transfer because rsync supplied -l root <peer>, which NetBird parsed as its log-level option: failed parsing log-level root: not a valid logrus Level: "root". Direct netbird ssh --no-browser --strict-host-key-checking root@10.69.69.208 remains successful; no host key was accepted or changed. Adding a narrow rsync shell adapter for the named peer.

Perf VM transfer finding: the one-time apt step installed rsync 3.5.0 and the Noto/Liberation/DejaVu font packages. The first `rsync -e 'netbird ssh ...'` failed before transfer because rsync supplied `-l root <peer>`, which NetBird parsed as its log-level option: `failed parsing log-level root: not a valid logrus Level: "root"`. Direct `netbird ssh --no-browser --strict-host-key-checking root@10.69.69.208` remains successful; no host key was accepted or changed. Adding a narrow rsync shell adapter for the named peer.

kayg commented

2026-10-12 08:25:49 +00:00

Author

Owner

Remote-shell finding: invoking bash -lc through the NetBird CLI did not preserve the nested command argument; setup ended with chmod: missing operand. The login shell also emitted its startup environment before command output. I replaced nested login shells with a single remote command string and non-login Bash for staged scripts, avoiding startup output and keeping the exact remote operation explicit.

Remote-shell finding: invoking `bash -lc` through the NetBird CLI did not preserve the nested command argument; setup ended with `chmod: missing operand`. The login shell also emitted its startup environment before command output. I replaced nested login shells with a single remote command string and non-login Bash for staged scripts, avoiding startup output and keeping the exact remote operation explicit.

kayg commented

2026-10-12 08:26:31 +00:00

Author

Owner

Setup finding: the pinned Bun 1.4.2 archive downloaded successfully (34.9 MB), then installation stopped because the destination toolchain/bun-1.4.2/bin/ had not been created. Setup now creates the versioned bin directory before installation and removes an incomplete download on retry. VM packages are installed; rerunning the idempotent setup.

Setup finding: the pinned Bun 1.4.2 archive downloaded successfully (34.9 MB), then installation stopped because the destination `toolchain/bun-1.4.2/bin/` had not been created. Setup now creates the versioned `bin` directory before installation and removes an incomplete download on retry. VM packages are installed; rerunning the idempotent setup.

kayg commented

2026-10-12 08:30:51 +00:00

Author

Owner

VM setup completed from branch perf-vm-browser at 638e4f1e9114969ed431feb5aa9f8498f2225dc7. Under /var/lock/perf-bench.lock, installed Bun 1.4.2, Playwright 1.61.1, Chromium 149.0.7827.55 and WebKit 26.5; Debian 13 font package versions and matches are recorded at evidence/perf-vm-browser/setup-versions.txt. During local dry review, corrected a staging argument error before any measurement; the run now passes each selected spec once to the staging walker.

VM setup completed from branch `perf-vm-browser` at `638e4f1e9114969ed431feb5aa9f8498f2225dc7`. Under `/var/lock/perf-bench.lock`, installed Bun 1.4.2, Playwright 1.61.1, Chromium 149.0.7827.55 and WebKit 26.5; Debian 13 font package versions and matches are recorded at `evidence/perf-vm-browser/setup-versions.txt`. During local dry review, corrected a staging argument error before any measurement; the run now passes each selected spec once to the staging walker.

kayg commented

2026-10-12 08:33:27 +00:00

Author

Owner

Harness finding: the first A/B invocation completed all six slots but no test specs ran. Each Playwright run exited 1 after 1.3–1.4 s because the historical configs ran bun run build on the VM; the staged payload intentionally contains dist and selected tests, not the build source. playwright.json records webServer exit 1 with empty suites for all six runs under evidence/perf-vm-browser/20261012T083058Z-219211621-2051692/. Updated the staging adapter to enable prebuilt-bundle serving for older configs and fixed EXIT cleanup to capture the run ID. The failed harness attempt is excluded from the comparison; its VM payload was removed under the lock.

Harness finding: the first A/B invocation completed all six slots but no test specs ran. Each Playwright run exited 1 after 1.3–1.4 s because the historical configs ran `bun run build` on the VM; the staged payload intentionally contains `dist` and selected tests, not the build source. `playwright.json` records `webServer` exit 1 with empty `suites` for all six runs under `evidence/perf-vm-browser/20261012T083058Z-219211621-2051692/`. Updated the staging adapter to enable prebuilt-bundle serving for older configs and fixed EXIT cleanup to capture the run ID. The failed harness attempt is excluded from the comparison; its VM payload was removed under the lock.

kayg commented

2026-10-12 08:41:31 +00:00

Author

Owner

BASE-1 completed on the 4-vCPU VM at 1f77a449; pre-run load was 0.21 / 0.36 / 0.35 and duration was 284.638 s. The theme check passed: p99 1.000 ms, 0 long tasks over 50 ms. terminal.spec.ts reported returning sign-in p99 5013.40 ms / 5000 budget, reconnect p99 1015.20 ms / 900 budget, predicted glyph p99 13.90 ms / 17 budget, and prediction CPU p99 2.200 ms / 2 budget; the access screen render p99 was 1973.90 ms for setup and 2196.10 ms for sign-in against 1000 ms. These are first-slot observations; A/B slots remain in progress. Playwright encodes test stdout as base64 buffers in its JSON report, so I updated the summary parser to decode those measurements.

BASE-1 completed on the 4-vCPU VM at 1f77a449; pre-run load was 0.21 / 0.36 / 0.35 and duration was 284.638 s. The theme check passed: p99 1.000 ms, 0 long tasks over 50 ms. `terminal.spec.ts` reported returning sign-in p99 5013.40 ms / 5000 budget, reconnect p99 1015.20 ms / 900 budget, predicted glyph p99 13.90 ms / 17 budget, and prediction CPU p99 2.200 ms / 2 budget; the access screen render p99 was 1973.90 ms for setup and 2196.10 ms for sign-in against 1000 ms. These are first-slot observations; A/B slots remain in progress. Playwright encodes test stdout as base64 buffers in its JSON report, so I updated the summary parser to decode those measurements.

kayg commented

2026-10-12 08:45:32 +00:00

Author

Owner

HEAD-1 completed on the 4-vCPU VM at 4855e033; pre-run load was 0.60 / 0.63 / 0.47 and duration was 276.462 s. Theme p99 was 1.600 ms with 0 long tasks over 50 ms. Returning sign-in p99 was 4010.40 / 5000 ms (pass); reconnect p99 was 996.60 / 900 ms (fail); predicted glyph p99 was 14.70 / 17 ms (pass); prediction CPU p99 was 0.600 / 2 ms (pass). Access render p99 was 1655.30 ms for setup and 2238.20 ms for sign-in (both over 1000 ms). Four of six slots remain.

HEAD-1 completed on the 4-vCPU VM at 4855e033; pre-run load was 0.60 / 0.63 / 0.47 and duration was 276.462 s. Theme p99 was 1.600 ms with 0 long tasks over 50 ms. Returning sign-in p99 was 4010.40 / 5000 ms (pass); reconnect p99 was 996.60 / 900 ms (fail); predicted glyph p99 was 14.70 / 17 ms (pass); prediction CPU p99 was 0.600 / 2 ms (pass). Access render p99 was 1655.30 ms for setup and 2238.20 ms for sign-in (both over 1000 ms). Four of six slots remain.

kayg commented

2026-10-12 08:48:55 +00:00

Author

Owner

BASE-2 completed in 287.216 s on the VM; pre-run load was 0.56 / 0.62 / 0.52. Theme remained green (p99 2.000 ms, 0 long tasks >50 ms). Reconnect remained red at 973.00 / 900 ms; setup/sign-in render p99s were 2279.30 / 2213.10 ms against 1000 ms. Returning sign-in was 5016.70 / 5000 ms; predicted glyph was 27.40 / 17 ms; prediction CPU was 4.400 / 2 ms. One base and one head slot remain. These additional terminal metrics vary between slots, while reconnect and access render are consistently over budget so far.

kayg commented

2026-10-12 08:53:45 +00:00

Author

Owner

HEAD-2 completed in 284.435 s; pre-run load was 0.50 / 0.82 / 0.67. Theme p99 1.900 ms, no long tasks >50 ms. Reconnect p99 1011.60 / 900 ms failed; setup/sign-in render p99 2256.30 / 2150.30 ms failed against 1000 ms. Returning sign-in p99 5015.00 / 5000 ms also failed; predicted glyph and prediction CPU passed at 13.90 / 17 ms and 1.400 / 2 ms. Two final slots remain.

kayg commented

2026-10-12 08:57:35 +00:00

Author

Owner

BASE-3 completed in 152.387 s; pre-run load was 0.21 / 0.51 / 0.58. Theme passed at p99 2.000 ms with no long tasks >50 ms. The first terminal test failed before emitting login/resume metrics: window.__tracerTestDouble.isConnected() did not become true within its 5 s predicate timeout. Access render p99 remained red at 2704.00 ms (setup) and 2273.70 ms (sign-in) against 1000 ms. The terminal connection failure is retained as a run failure; its missing resume metric is not counted as a pass. One HEAD slot remains.

BASE-3 completed in 152.387 s; pre-run load was 0.21 / 0.51 / 0.58. Theme passed at p99 2.000 ms with no long tasks >50 ms. The first terminal test failed before emitting login/resume metrics: `window.__tracerTestDouble.isConnected()` did not become true within its 5 s predicate timeout. Access render p99 remained red at 2704.00 ms (setup) and 2273.70 ms (sign-in) against 1000 ms. The terminal connection failure is retained as a run failure; its missing resume metric is not counted as a pass. One HEAD slot remains.

kayg commented

2026-10-12 09:02:42 +00:00

Author

Owner

HEAD-3 completed in 298.462 s; pre-run load was 0.89 / 0.59 / 0.60. Theme p99 2.500 ms, zero long tasks >50 ms. Reconnect p99 1012.50 / 900 ms failed; setup/sign-in render p99s were 2926.30 / 2322.70 ms against 1000 ms. Returning sign-in p99 passed at 4966.70 / 5000 ms, while predicted glyph p99 failed at 26.50 / 17 ms. All six requested slots are complete; summary and raw results are under evidence/perf-vm-browser/20261012T083333Z-726800496-2080513/.

HEAD-3 completed in 298.462 s; pre-run load was 0.89 / 0.59 / 0.60. Theme p99 2.500 ms, zero long tasks >50 ms. Reconnect p99 1012.50 / 900 ms failed; setup/sign-in render p99s were 2926.30 / 2322.70 ms against 1000 ms. Returning sign-in p99 passed at 4966.70 / 5000 ms, while predicted glyph p99 failed at 26.50 / 17 ms. All six requested slots are complete; summary and raw results are under `evidence/perf-vm-browser/20261012T083333Z-726800496-2080513/`.

kayg commented

2026-10-12 09:11:44 +00:00

Author

Owner

Repeated scripts/perf-vm setup completed successfully under the VM lock. Bun reported all 38 locked installs already present with no changes; Playwright's dependency install reported 0 upgraded/new/removed packages; browser versions remained Chromium 149.0.7827.55 and WebKit 26.5. The recorded setup file was refreshed at evidence/perf-vm-browser/setup-versions.txt.

Repeated `scripts/perf-vm setup` completed successfully under the VM lock. Bun reported all 38 locked installs already present with no changes; Playwright's dependency install reported 0 upgraded/new/removed packages; browser versions remained Chromium 149.0.7827.55 and WebKit 26.5. The recorded setup file was refreshed at `evidence/perf-vm-browser/setup-versions.txt`.

kayg commented

2026-10-12 09:37:42 +00:00

Author

Owner

Real-server harness finding: the first local static binary build stopped before any VM run because the pinned musl wrapper expected Zig at the revision-specific scratch target's auth-tools/zig/zig. The runner had not yet called scripts/auth-toolchain. Evidence: evidence/perf-vm-browser/20261012T093327Z-2805033779-2818866/build.log reports that exact missing path. No real-server timing was measured. I am adding the repository toolchain bootstrap before the local binary build and will retry.

Real-server harness finding: the first local static binary build stopped before any VM run because the pinned musl wrapper expected Zig at the revision-specific scratch target's `auth-tools/zig/zig`. The runner had not yet called `scripts/auth-toolchain`. Evidence: `evidence/perf-vm-browser/20261012T093327Z-2805033779-2818866/build.log` reports that exact missing path. No real-server timing was measured. I am adding the repository toolchain bootstrap before the local binary build and will retry.

kayg commented

2026-10-12 10:15:29 +00:00

Author

Owner

Optional real-mode validation built and transferred the 23,655,526-byte static binary, then ran real/host-facts.spec.ts on the VM (revision 4855e033, 24.429 s, exit 1, pre-run load 0.14/0.04/0.06). Both assertions failed: the Host popover had no Kernel row; the low-memory toast reported 6,807 MiB free and 0 sessions, while the fixture expects 1 session. Raw JSON and environment are in evidence/perf-vm-browser/20261012T093952Z-3236422064-2875539/runs/run-1/. This spec is outside the required A/B; no product change was made.

Optional real-mode validation built and transferred the 23,655,526-byte static binary, then ran real/host-facts.spec.ts on the VM (revision 4855e033, 24.429 s, exit 1, pre-run load 0.14/0.04/0.06). Both assertions failed: the Host popover had no Kernel row; the low-memory toast reported 6,807 MiB free and 0 sessions, while the fixture expects 1 session. Raw JSON and environment are in evidence/perf-vm-browser/20261012T093952Z-3236422064-2875539/runs/run-1/. This spec is outside the required A/B; no product change was made.

kayg commented

2026-10-12 10:18:42 +00:00

Author

Owner

Repeated scripts/perf-vm setup after the dependency-presence guard change: exit 0; Bun 1.4.2; Playwright 1.61.1; Chromium 149.0.7827.55; WebKit 26.5; Bun reports 38 locked packages unchanged. Updated record: evidence/perf-vm-browser/setup-versions.txt.

kayg commented

2026-10-12 10:22:04 +00:00

Author

Owner

Transport module committed and pushed: 402130633d. Gate: bash -n scripts/perf-vm-rsh exited 0.

Transport module committed and pushed: 402130633d700cf1db4438f1eaad147e81d3fbc9. Gate: `bash -n scripts/perf-vm-rsh` exited 0.

kayg commented

2026-10-12 10:22:30 +00:00

Author

Owner

Selected-input staging module committed and pushed: 3fb91c2c2f. Gate: Python AST parse and git diff --check exited 0.

Selected-input staging module committed and pushed: 3fb91c2c2f4d219ed25356c45eb32312f663e78f. Gate: Python AST parse and `git diff --check` exited 0.

kayg commented

2026-10-12 10:22:54 +00:00

Author

Owner

A/B reporting module committed and pushed: 8f9e32e8f0. Gate: Python AST parse and git diff --check exited 0. The regenerated six-slot report records theme p99 1.0–2.5 ms and 0 long tasks; it retains the reconnect and access-screen failures with their per-run load records.

A/B reporting module committed and pushed: 8f9e32e8f088b2a743fe5c87e523686be4f29500. Gate: Python AST parse and `git diff --check` exited 0. The regenerated six-slot report records theme p99 1.0–2.5 ms and 0 long tasks; it retains the reconnect and access-screen failures with their per-run load records.

kayg commented

2026-10-12 10:23:20 +00:00

Author

Owner

Pinned VM runtime and remote runner module committed and pushed: 553c2089f1. Gate: bash -n scripts/perf-vm-remote.sh and git diff --check exited 0. Repeated setup verified 38 locked packages unchanged and the recorded browser/font versions.

Pinned VM runtime and remote runner module committed and pushed: 553c2089f1b6619f18d4725e5715403121130098. Gate: `bash -n scripts/perf-vm-remote.sh` and `git diff --check` exited 0. Repeated setup verified 38 locked packages unchanged and the recorded browser/font versions.

kayg commented

2026-10-12 10:24:37 +00:00

Author

Owner

Runner integration and the staged-bundle Playwright config committed and pushed: 00e8a490d3. Gates: bun run check exited 0; shell syntax and git diff --check exited 0. Functional evidence: the required six-slot A/B completed and the single theme run passed on the VM.

Runner integration and the staged-bundle Playwright config committed and pushed: 00e8a490d386b344989764c6eac15cf0d40afe94. Gates: `bun run check` exited 0; shell syntax and `git diff --check` exited 0. Functional evidence: the required six-slot A/B completed and the single theme run passed on the VM.

kayg commented

2026-10-12 10:24:59 +00:00

Author

Owner

Performance-process documentation committed and pushed: 03bd476a27. Gate: git diff --check exited 0.

Performance-process documentation committed and pushed: 03bd476a27805cdecb603cf55fa036dd15192689. Gate: `git diff --check` exited 0.

kayg commented

2026-10-12 10:27:35 +00:00

Author

Owner

Perf-VM evidence and author checklist committed and pushed: c716b75a99. The evidence commit contains 421 files (16,232 text lines plus browser captures); git diff --cached --check exited 0. The six-slot A/B records theme p99 1.0–2.5 ms with zero >50 ms tasks and six setup/sign-in upper-bound failures; raw VM records and captures are under evidence/perf-vm-browser/20261012T083333Z-726800496-2080513/.

Perf-VM evidence and author checklist committed and pushed: c716b75a99f4ac5c66e39593283cd968e7fb8855. The evidence commit contains 421 files (16,232 text lines plus browser captures); `git diff --cached --check` exited 0. The six-slot A/B records theme p99 1.0–2.5 ms with zero >50 ms tasks and six setup/sign-in upper-bound failures; raw VM records and captures are under evidence/perf-vm-browser/20261012T083333Z-726800496-2080513/.

kayg commented

2026-10-12 10:48:11 +00:00

Author

Owner

Resuming after the AI service sign-in expired. Branch perf-vm-browser is clean at c716b75a99f4ac5c66e39593283cd968e7fb8855, based on 638e4f1e9114969ed431feb5aa9f8498f2225dc7 (origin/dev). Read #184 and parent #176 histories. The setup, runner, docs, required six-slot A/B, and evidence are already committed; the A/B table is on #176. Completing review and the final branch-head gate.

Resuming after the AI service sign-in expired. Branch `perf-vm-browser` is clean at `c716b75a99f4ac5c66e39593283cd968e7fb8855`, based on `638e4f1e9114969ed431feb5aa9f8498f2225dc7` (`origin/dev`). Read #184 and parent #176 histories. The setup, runner, docs, required six-slot A/B, and evidence are already committed; the A/B table is on #176. Completing review and the final branch-head gate.

kayg commented

2026-10-12 12:45:39 +00:00

Author

Owner

Finished review on perf-vm-browser at head c716b75a99f4ac5c66e39593283cd968e7fb8855 (clean; matches origin/perf-vm-browser). The setup, runner, process documentation, setup record, A/B raw reports, and author checklist are committed. The required three-pair comparison (1f77a449 vs 4855e033, test/terminal-theme.spec.ts and test/terminal.spec.ts) is posted on #176; theme passed all six slots at p99 1.0–2.5 ms with zero >50 ms long tasks, reconnect exceeded budget in all five measured slots, and setup/sign-in render exceeded budget in all six. Raw evidence: evidence/perf-vm-browser/20261012T083333Z-726800496-2080513/.

Final checks: shell syntax, Python AST, apps/tracer bun run check, and git diff --check passed. scripts/verify fast exited 1. Final summary verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

The test row failure is real_binary_preserves_input_and_output_across_24_abrupt_stops: its build-host idle-RSS assertion (idle < 15 * 1024) failed. The static binary was 21,709,776 bytes against the existing 16,404,816-byte cap. This change touches no Rust sources or build configuration; the memory result was not used as performance evidence. I did not waive either failure.

Finished review on `perf-vm-browser` at head `c716b75a99f4ac5c66e39593283cd968e7fb8855` (clean; matches `origin/perf-vm-browser`). The setup, runner, process documentation, setup record, A/B raw reports, and author checklist are committed. The required three-pair comparison (`1f77a449` vs `4855e033`, `test/terminal-theme.spec.ts` and `test/terminal.spec.ts`) is posted on #176; theme passed all six slots at p99 1.0–2.5 ms with zero >50 ms long tasks, reconnect exceeded budget in all five measured slots, and setup/sign-in render exceeded budget in all six. Raw evidence: `evidence/perf-vm-browser/20261012T083333Z-726800496-2080513/`. Final checks: shell syntax, Python AST, `apps/tracer` `bun run check`, and `git diff --check` passed. `scripts/verify fast` exited 1. Final summary verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` The test row failure is `real_binary_preserves_input_and_output_across_24_abrupt_stops`: its build-host idle-RSS assertion (`idle < 15 * 1024`) failed. The static binary was 21,709,776 bytes against the existing 16,404,816-byte cap. This change touches no Rust sources or build configuration; the memory result was not used as performance evidence. I did not waive either failure.

kayg commented

2026-10-12 12:50:23 +00:00

Author

Owner

Starting the requested perf-VM remeasurement on perf-remeasure, base c0d01d2285b92ea8e452f10a3a0d8b479fa1bff8 (origin/dev). Read #184 and parent #176 histories. The prior #176 acceptance run failed; its pushed evidence head is 2562ee88, with production code frozen at 9cf26b07. This run adds the requested #176, #182 and #169/#160/#168 comparisons on the perf VM; no product code changes.

Starting the requested perf-VM remeasurement on `perf-remeasure`, base `c0d01d2285b92ea8e452f10a3a0d8b479fa1bff8` (`origin/dev`). Read #184 and parent #176 histories. The prior #176 acceptance run failed; its pushed evidence head is `2562ee88`, with production code frozen at `9cf26b07`. This run adds the requested #176, #182 and #169/#160/#168 comparisons on the perf VM; no product code changes.

kayg commented

2026-10-12 12:54:51 +00:00

Author

Owner

Runner integration completed: merged origin/perf-vm-browser at local merge SHA 113adcde. The VM setup command is still waiting without output; no timing run has started.

Runner integration completed: merged `origin/perf-vm-browser` at local merge SHA `113adcde`. The VM setup command is still waiting without output; no timing run has started.

kayg commented

2026-10-12 13:00:40 +00:00

Author

Owner

Input review: ab.md records three specs, while the task explicitly adds sessions-canvas; all four named specs exist at both batch-4 refs. I will run the four explicit specs: terminal, dead-session, sessions-canvas, and theme. Perf VM setup is still waiting without output; no timing slot has started.

Input review: `ab.md` records three specs, while the task explicitly adds `sessions-canvas`; all four named specs exist at both batch-4 refs. I will run the four explicit specs: terminal, dead-session, sessions-canvas, and theme. Perf VM setup is still waiting without output; no timing slot has started.

kayg commented

2026-10-12 13:05:06 +00:00

Author

Owner

Finding: the perf-VM nonblocking lock probe (netbird ssh --no-browser --strict-host-key-checking root@10.69.69.208 'flock -n /var/lock/perf-bench.lock -c true') returned exit 1 with no output. scripts/perf-vm setup remains queued; no measurement has started. I will continue waiting for the shared lock.

Finding: the perf-VM nonblocking lock probe (`netbird ssh --no-browser --strict-host-key-checking root@10.69.69.208 'flock -n /var/lock/perf-bench.lock -c true'`) returned exit 1 with no output. `scripts/perf-vm setup` remains queued; no measurement has started. I will continue waiting for the shared lock.

kayg commented

2026-10-12 13:06:19 +00:00

Author

Owner

Cursor probe prepared in temporary clean overlays: base 1f77a449 + harness commit 5d2d64a7, head 4855e033 + harness commit 9625471a. Each overlay changes only Playwright project selection and the new 30-second measurement spec; app source, package files and dependencies are byte-for-byte unchanged. The VM lock remains occupied; no cursor slot has run.

Cursor probe prepared in temporary clean overlays: base `1f77a449` + harness commit `5d2d64a7`, head `4855e033` + harness commit `9625471a`. Each overlay changes only Playwright project selection and the new 30-second measurement spec; app source, package files and dependencies are byte-for-byte unchanged. The VM lock remains occupied; no cursor slot has run.

kayg commented

2026-10-12 13:45:37 +00:00

Author

Owner

#176 perf-VM A/B complete: 1f77a449 vs 2562ee88 (origin/prod-perf-fix), 3 interleaved pairs on Debian 13 / 4 vCPU, Playwright 1.61.1, Chromium 149.0.7827.55. Theme p99 was 1.4–5.1 ms with zero long tasks >50 ms in all six slots (PASS). Terminal resume p99 was 981.6–1018.2 ms vs 900 ms budget on both revisions (REAL PRE-EXISTING FAILURE); sign-in-to-terminal p99 and glyph visibility passed. Setup-screen paint p99 was 1192.5–1952.4 ms vs 1000 ms across all slots (REAL PRE-EXISTING FAILURE). Sign-in-screen paint failed on base (p99 2127.7–2156.9 ms) and passed on head (p99 778.6–975.3 ms); FCP p99 was 2127.7–2156.9 ms base vs 714.0–872.4 ms head. Raw run summary and per-slot environment/results: evidence/perf-vm-browser/20261012T130928Z-1162600825-1174652/summary.md and runs/.

#176 perf-VM A/B complete: `1f77a449` vs `2562ee88` (`origin/prod-perf-fix`), 3 interleaved pairs on Debian 13 / 4 vCPU, Playwright 1.61.1, Chromium 149.0.7827.55. Theme p99 was 1.4–5.1 ms with zero long tasks >50 ms in all six slots (PASS). Terminal resume p99 was 981.6–1018.2 ms vs 900 ms budget on both revisions (REAL PRE-EXISTING FAILURE); sign-in-to-terminal p99 and glyph visibility passed. Setup-screen paint p99 was 1192.5–1952.4 ms vs 1000 ms across all slots (REAL PRE-EXISTING FAILURE). Sign-in-screen paint failed on base (p99 2127.7–2156.9 ms) and passed on head (p99 778.6–975.3 ms); FCP p99 was 2127.7–2156.9 ms base vs 714.0–872.4 ms head. Raw run summary and per-slot environment/results: `evidence/perf-vm-browser/20261012T130928Z-1162600825-1174652/summary.md` and `runs/`.

kayg commented

2026-10-12 14:04:24 +00:00

Author

Owner

#182 probe first attempt invalid: all three release-prod-3w head slots exited 1 after 40.5–42.4 s with expected cursor layer animation is missing; all three base slots passed in 39.8–41.4 s. The harness had not focused the browser tab and terminal, so the cursor layer correctly stayed stopped. I updated only the temporary probe overlays (base c0172d1f, head f53f46d8) to bring the page forward, focus the terminal, and start the blink window with input. No product source or dependency changed. Rerunning Chromium before WebKit.

#182 probe first attempt invalid: all three release-prod-3w head slots exited 1 after 40.5–42.4 s with `expected cursor layer animation is missing`; all three base slots passed in 39.8–41.4 s. The harness had not focused the browser tab and terminal, so the cursor layer correctly stayed stopped. I updated only the temporary probe overlays (base `c0172d1f`, head `f53f46d8`) to bring the page forward, focus the terminal, and start the blink window with input. No product source or dependency changed. Rerunning Chromium before WebKit.

kayg commented

2026-10-12 14:19:06 +00:00

Author

Owner

#182 WebKit first series produced no timing data. All six slots (both revisions, 3 pairs) exited 1 after 36.4–38.0 s; each timed out after 30 s waiting for __tracerTestDouble.isConnected() at cursor-blink-measure.spec.ts:234. This is a matched browser-harness failure, not yet a product finding. Captured report: evidence/perf-vm-browser/20261012T141320Z-945728314-1993590/summary.md. Adding page diagnostics on the perf VM before retrying WebKit.

#182 WebKit first series produced no timing data. All six slots (both revisions, 3 pairs) exited 1 after 36.4–38.0 s; each timed out after 30 s waiting for `__tracerTestDouble.isConnected()` at `cursor-blink-measure.spec.ts:234`. This is a matched browser-harness failure, not yet a product finding. Captured report: `evidence/perf-vm-browser/20261012T141320Z-945728314-1993590/summary.md`. Adding page diagnostics on the perf VM before retrying WebKit.

kayg commented

2026-10-12 14:25:23 +00:00

Author

Owner

WebKit diagnostic evidence: Playwright reports browser 26.5 and page response 200; both mock setup API requests returned 200, but the page reported no test double and a 401 on /auth/session in both revisions (.../20261012T142242Z-1082004773-2139527/runs/base-1/playwright.json). The BrowserContext API setup did not produce a session visible to the WebKit page. I’m switching the test-only setup to same-origin page fetches so the browser stores its own mock cookie; no product code changes.

WebKit diagnostic evidence: Playwright reports browser 26.5 and page response 200; both mock setup API requests returned 200, but the page reported no test double and a 401 on `/auth/session` in both revisions (`.../20261012T142242Z-1082004773-2139527/runs/base-1/playwright.json`). The BrowserContext API setup did not produce a session visible to the WebKit page. I’m switching the test-only setup to same-origin page fetches so the browser stores its own mock cookie; no product code changes.

kayg commented

2026-10-12 14:41:38 +00:00

Author

Owner

WebKit fixture diagnosis: the original temporary WebKit project inherited Chromium's --disable-renderer-backgrounding launch argument. The perf-VM stderr shows WebKit 26.5 rejecting that flag (Cannot parse arguments: Unknown option --disable-renderer-backgrounding) for all six cursor specs. I cleared launch args only for the temporary WebKit project overlay (head commit b91df0ee) and reran the existing cursor spec; it exited 1 after 184.5 s (load 0.29/0.46/0.78). Its report is queued for lock-protected retrieval at evidence/perf-vm-browser/20261012T142939Z-3430700029-2200409/. This is harness diagnosis, not timing evidence.

WebKit fixture diagnosis: the original temporary WebKit project inherited Chromium's `--disable-renderer-backgrounding` launch argument. The perf-VM stderr shows WebKit 26.5 rejecting that flag (`Cannot parse arguments: Unknown option --disable-renderer-backgrounding`) for all six cursor specs. I cleared launch args only for the temporary WebKit project overlay (head commit `b91df0ee`) and reran the existing cursor spec; it exited 1 after 184.5 s (load 0.29/0.46/0.78). Its report is queued for lock-protected retrieval at `evidence/perf-vm-browser/20261012T142939Z-3430700029-2200409/`. This is harness diagnosis, not timing evidence.

kayg commented

2026-10-12 15:09:12 +00:00

Author

Owner

#182 WebKit cursor blink preliminary (first interleaved pair; n=1, not final): browser version 26.5. release-prod-3 blink/steady process CPU was 0.37% / 0.27% of one core; release-prod-3w was 4.60% / 0.27%. Both passed the 30-second probe with zero long tasks. The head showed 58 animation frames and 58 render passes during blink; steady had zero render passes. Base cursor layer was absent in this harness. Full three-pair sequence is running, so classification waits for replication. Evidence: evidence/perf-vm-browser/20261012T145852Z-800732755-2493630/ (BASE 1 / HEAD 1); pre-run loads were 0.09/0.25/0.43 and 0.11/0.24/0.41.

#182 WebKit cursor blink preliminary (first interleaved pair; n=1, not final): browser version 26.5. `release-prod-3` blink/steady process CPU was 0.37% / 0.27% of one core; `release-prod-3w` was 4.60% / 0.27%. Both passed the 30-second probe with zero long tasks. The head showed 58 animation frames and 58 render passes during blink; steady had zero render passes. Base cursor layer was absent in this harness. Full three-pair sequence is running, so classification waits for replication. Evidence: `evidence/perf-vm-browser/20261012T145852Z-800732755-2493630/` (BASE 1 / HEAD 1); pre-run loads were 0.09/0.25/0.43 and 0.11/0.24/0.41.

kayg commented

2026-10-12 15:15:54 +00:00

Author

Owner

#182 harness correction: the initial cursor probe tried to make steady state by finishing .terminal-cursor-layer, which does not exist on release-prod-3; its steady toggle was a no-op. Those cursor numbers are superseded. I updated only the temporary test overlay: both refs now use the shared freezeCursorForTest() hook for steady state, and the BASE overlay holds the 700 ms renderer-idle callback through the 30 s active window so its renderer cursor continues blinking. No product source changed. I’m running one Chromium and WebKit smoke pair before the full three-pair sequences.

#182 harness correction: the initial cursor probe tried to make steady state by finishing `.terminal-cursor-layer`, which does not exist on `release-prod-3`; its steady toggle was a no-op. Those cursor numbers are superseded. I updated only the temporary test overlay: both refs now use the shared `freezeCursorForTest()` hook for steady state, and the BASE overlay holds the 700 ms renderer-idle callback through the 30 s active window so its renderer cursor continues blinking. No product source changed. I’m running one Chromium and WebKit smoke pair before the full three-pair sequences.

kayg commented

2026-10-12 15:17:22 +00:00

Author

Owner

The corrected Chromium smoke pair reached the probe but collected no timing: both slots exited before measurement because /proc/<pid>/comm did not match the harness's Chromium process-name allowlist (no chromium browser processes found). This is a test-harness filter failure on the VM, not a product result. I’m widening the name match and will repeat the smoke pair before collecting the three runs.

The corrected Chromium smoke pair reached the probe but collected no timing: both slots exited before measurement because `/proc/<pid>/comm` did not match the harness's Chromium process-name allowlist (`no chromium browser processes found`). This is a test-harness filter failure on the VM, not a product result. I’m widening the name match and will repeat the smoke pair before collecting the three runs.

kayg commented

2026-10-12 15:37:12 +00:00

Author

Owner

#182 corrected-state smoke pair (n=1, preliminary; full three-pair runs pending). The BASE timer was held for the active window and freezeCursorForTest() was used for both steady controls.

Engine / measure	release-prod-3	release-prod-3w
Chromium active browser CPU (% one core)	40.23	2.90
Chromium active main-thread TaskDuration (ms / 30 s)	1885.5	70.8
Chromium active ScriptDuration (ms / 30 s)	512.7	7.4
Chromium steady browser CPU (% one core)	0.33	0.33
WebKit active browser CPU (% one core)	5.03	4.70
WebKit active page rAF callbacks / 30 s	1861	63
WebKit steady browser CPU (% one core)	0.27	0.30

All four runs passed with zero long tasks. WebKit's active CPU delta is small and its BASE slot ran at load 3.24/3.64/2.01 versus HEAD 1.39/3.02/1.92, so treat the one-pair values as preliminary. Evidence: Chromium evidence/perf-vm-browser/20261012T151733Z-3306175936-2714921/; WebKit evidence/perf-vm-browser/20261012T152356Z-4077536085-2770889/. Earlier cursor numbers from the selector-only steady probe are superseded by this matched-state probe.

#182 corrected-state smoke pair (n=1, preliminary; full three-pair runs pending). The BASE timer was held for the active window and `freezeCursorForTest()` was used for both steady controls. | Engine / measure | release-prod-3 | release-prod-3w | | --- | ---: | ---: | | Chromium active browser CPU (% one core) | 40.23 | 2.90 | | Chromium active main-thread TaskDuration (ms / 30 s) | 1885.5 | 70.8 | | Chromium active ScriptDuration (ms / 30 s) | 512.7 | 7.4 | | Chromium steady browser CPU (% one core) | 0.33 | 0.33 | | WebKit active browser CPU (% one core) | 5.03 | 4.70 | | WebKit active page rAF callbacks / 30 s | 1861 | 63 | | WebKit steady browser CPU (% one core) | 0.27 | 0.30 | All four runs passed with zero long tasks. WebKit's active CPU delta is small and its BASE slot ran at load 3.24/3.64/2.01 versus HEAD 1.39/3.02/1.92, so treat the one-pair values as preliminary. Evidence: Chromium `evidence/perf-vm-browser/20261012T151733Z-3306175936-2714921/`; WebKit `evidence/perf-vm-browser/20261012T152356Z-4077536085-2770889/`. Earlier cursor numbers from the selector-only steady probe are superseded by this matched-state probe.

kayg commented

2026-10-12 15:48:03 +00:00

Author

Owner

#182 corrected Chromium result (3 interleaved pairs; browser 149.0.7827.55, Playwright 1.61.1). All six probes passed; loads ranged 0.11–0.29 / 0.62–1.49 / 1.12–1.55 (1/5/15 min); ports 4823–4828.

Measure (30 s)	release-prod-3 median	release-prod-3w median	Change
Active browser-process CPU (% one core)	41.03	2.70	-93.4%
Active Chromium TaskDuration (ms)	1952.9	73.6	-96.2%
Active ScriptDuration (ms)	520.3	7.9	-98.5%
Active page rAF callbacks	1800	62	-96.6%
Steady browser-process CPU (% one core)	0.37	0.40	+0.03 pp
Long tasks >50 ms	0	0	—

Classification: PASS for the cursor CPU/activity check. BASE active windows deferred the renderer's 700 ms idle callback; steady windows froze the cursor on both refs. Evidence: evidence/perf-vm-browser/20261012T153720Z-2170831494-2867709/ (slots/loads/ports and raw reports).

#182 corrected Chromium result (3 interleaved pairs; browser 149.0.7827.55, Playwright 1.61.1). All six probes passed; loads ranged 0.11–0.29 / 0.62–1.49 / 1.12–1.55 (1/5/15 min); ports 4823–4828. | Measure (30 s) | release-prod-3 median | release-prod-3w median | Change | | --- | ---: | ---: | ---: | | Active browser-process CPU (% one core) | 41.03 | 2.70 | -93.4% | | Active Chromium TaskDuration (ms) | 1952.9 | 73.6 | -96.2% | | Active ScriptDuration (ms) | 520.3 | 7.9 | -98.5% | | Active page rAF callbacks | 1800 | 62 | -96.6% | | Steady browser-process CPU (% one core) | 0.37 | 0.40 | +0.03 pp | | Long tasks >50 ms | 0 | 0 | — | Classification: PASS for the cursor CPU/activity check. BASE active windows deferred the renderer's 700 ms idle callback; steady windows froze the cursor on both refs. Evidence: `evidence/perf-vm-browser/20261012T153720Z-2170831494-2867709/` (slots/loads/ports and raw reports).

kayg commented

2026-10-12 16:01:00 +00:00

Author

Owner

#182 corrected WebKit result (3 interleaved pairs; WebKit 26.5, Playwright 1.61.1). All six probes passed; loads ranged 0.19–1.01 / 0.31–0.81 / 0.83–0.93 (1/5/15 min); ports 4423–4428.

Measure (30 s)	release-prod-3 median	release-prod-3w median	Change
Active MiniBrowser CPU (% one core)	4.97	4.63	-6.8%
Active page rAF callbacks	1859	63	-96.6%
Active interval callbacks	56	0	-100%
Steady MiniBrowser CPU (% one core)	0.30	0.30	no change
Long tasks >50 ms	0	0	—

Classification: PASS. Together with the Chromium result, the matched-state probe shows lower active CPU/activity and steady CPU is unchanged; no active-window long tasks were recorded. BASE deferred its 700 ms renderer-idle callback only for the active window; steady windows froze both refs. Evidence: evidence/perf-vm-browser/20261012T154810Z-1444956166-2970883/.

#182 corrected WebKit result (3 interleaved pairs; WebKit 26.5, Playwright 1.61.1). All six probes passed; loads ranged 0.19–1.01 / 0.31–0.81 / 0.83–0.93 (1/5/15 min); ports 4423–4428. | Measure (30 s) | release-prod-3 median | release-prod-3w median | Change | | --- | ---: | ---: | ---: | | Active MiniBrowser CPU (% one core) | 4.97 | 4.63 | -6.8% | | Active page rAF callbacks | 1859 | 63 | -96.6% | | Active interval callbacks | 56 | 0 | -100% | | Steady MiniBrowser CPU (% one core) | 0.30 | 0.30 | no change | | Long tasks >50 ms | 0 | 0 | — | Classification: PASS. Together with the Chromium result, the matched-state probe shows lower active CPU/activity and steady CPU is unchanged; no active-window long tasks were recorded. BASE deferred its 700 ms renderer-idle callback only for the active window; steady windows froze both refs. Evidence: `evidence/perf-vm-browser/20261012T154810Z-1444956166-2970883/`.

kayg commented

2026-10-12 16:15:32 +00:00

Author

Owner

Batch 4 BASE 1 (1f77a449) exited 1 after 530,916 ms at load 0.64 / 0.69 / 0.84. This is one of three baseline controls; test-level failures are not yet classified. HEAD 1 and the other four slots are still running in interleaved order. Evidence is being retrieved under evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

Batch 4 BASE 1 (`1f77a449`) exited 1 after 530,916 ms at load 0.64 / 0.69 / 0.84. This is one of three baseline controls; test-level failures are not yet classified. HEAD 1 and the other four slots are still running in interleaved order. Evidence is being retrieved under `evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/`.

kayg commented

2026-10-12 16:18:18 +00:00

Author

Owner

Batch 4 pair 1 completed. Both refs exited 1: BASE 1f77a449 530,916 ms at load 0.64/0.69/0.84; HEAD 4d1b1950 463,708 ms at load 0.16/0.44/0.66. This pair alone cannot classify the failures; three more BASE/HEAD pairs are pending. Evidence: evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/ (slot logs will be synced when the full sequence finishes).

Batch 4 pair 1 completed. Both refs exited 1: BASE `1f77a449` 530,916 ms at load 0.64/0.69/0.84; HEAD `4d1b1950` 463,708 ms at load 0.16/0.44/0.66. This pair alone cannot classify the failures; three more BASE/HEAD pairs are pending. Evidence: `evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/` (slot logs will be synced when the full sequence finishes).

kayg commented

2026-10-12 16:27:36 +00:00

Author

Owner

Batch 4 BASE 2 (1f77a449) also exited 1, after 551,953 ms at load 0.43 / 0.48 / 0.58. This confirms a repeated BASE suite failure but not which checks fail; test-level classification waits for the reports and all three HEAD controls. BASE 1/2 evidence is under evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

Batch 4 BASE 2 (`1f77a449`) also exited 1, after 551,953 ms at load 0.43 / 0.48 / 0.58. This confirms a repeated BASE suite failure but not which checks fail; test-level classification waits for the reports and all three HEAD controls. BASE 1/2 evidence is under `evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/`.

kayg commented

2026-10-12 16:35:20 +00:00

Author

Owner

Batch 4 pair 2 completed. BASE (1f77a449) and HEAD (4d1b1950) both exited 1; BASE 2 took 551,953 ms at load 0.43/0.48/0.58, HEAD 2 took 472,650 ms at load 0.44/0.61/0.63. Failures remain unclassified until test-level reports and the final pair are available. Evidence: evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

Batch 4 pair 2 completed. BASE (`1f77a449`) and HEAD (`4d1b1950`) both exited 1; BASE 2 took 551,953 ms at load 0.43/0.48/0.58, HEAD 2 took 472,650 ms at load 0.44/0.61/0.63. Failures remain unclassified until test-level reports and the final pair are available. Evidence: `evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/`.

kayg commented

2026-10-12 16:44:41 +00:00

Author

Owner

Batch 4 BASE 3 (1f77a449) also exited 1 after 538,524 ms at load 1.21 / 0.73 / 0.66. All three BASE suite invocations now fail; failed test names and metrics will be classified from the raw reports after HEAD 3 completes. Evidence: evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

Batch 4 BASE 3 (`1f77a449`) also exited 1 after 538,524 ms at load 1.21 / 0.73 / 0.66. All three BASE suite invocations now fail; failed test names and metrics will be classified from the raw reports after HEAD 3 completes. Evidence: `evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/`.

kayg commented

2026-10-12 16:53:52 +00:00

Author

Owner

Batch 4 HEAD 3 (4d1b1950) exited 1 after 533,746 ms at load 0.92 / 0.85 / 0.74. All six suite invocations (three per ref) exited 1. Test-level failures and timing values remain pending raw report retrieval; I’ll classify them after the runner returns its artifacts. Evidence directory: evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

Batch 4 HEAD 3 (`4d1b1950`) exited 1 after 533,746 ms at load 0.92 / 0.85 / 0.74. All six suite invocations (three per ref) exited 1. Test-level failures and timing values remain pending raw report retrieval; I’ll classify them after the runner returns its artifacts. Evidence directory: `evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/`.

kayg commented

2026-10-12 17:12:06 +00:00

Author

Owner

Batch 4 assertion-level classification from the six VM runs (evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/):

Theme: PASS. All six theme tests passed. Switch p99 was 0.8–8.0 ms (budget 16.67 ms); all six recorded zero long tasks >50 ms.
Dead-session: PASS on HEAD. All 12 HEAD dead-session tests passed. BASE 1 alone failed the older attach-count assertion (expected 2, received 3); BASE 2/3 and all HEAD cases passed.
Sessions canvas retry: REAL REGRESSION. “The empty, loading and error states keep the product's rhythm” passed BASE 1/3, failed BASE 2/3, and failed HEAD 3/3. The timeout snapshot shows the error state still loading after Retry, waiting for six rows. Every other sessions-canvas check passed in all six slots, including the 200/130 ms popover assertion.
Resume after drop: REAL PRE-EXISTING FAILURE. Budget 900 ms; BASE p99 was 945.6 / 1012.9 / 1005.2 ms; the one completed HEAD measurement was 978.3 ms. HEAD 1/3 stopped earlier, before this metric.
Access-screen first paint: REAL PRE-EXISTING FAILURE. Budget 1000 ms. Setup render p99 was 2586.8 / 2770.1 / 2759.4 ms on BASE; 1732.5 / no measurement / 2202.6 ms on HEAD. Sign-in render p99 was 2136.6 / 2233.1 / 2150.9 ms on BASE; 2388.8 / no measurement / 2323.6 ms on HEAD. HEAD 2 hit a mock-server socket hang-up before sampling.
Returning sign-in: PASS where measured. All five completed samples had p99 3979.4–4009.6 ms against a 5000 ms budget. Predicted glyph visibility also passed every completed sample (max p99 14.8 ms against 17 ms).
Terminal interaction: REAL REGRESSION signals on HEAD. HEAD 1/3 timed out waiting for the terminal test double to connect after Open infraternal (5 s); HEAD 3/3 timed out with predictive pending text still empty where the probe expected ok (terminal.spec.ts:243). Neither assertion failed on BASE. HEAD 2 completed the interaction checks and then hit the pre-existing 978.3 ms resume budget failure.

All slot durations, load averages, ports, summaries and raw Playwright JSON are in the evidence directory. No product code was changed in this remeasurement.

Batch 4 assertion-level classification from the six VM runs (`evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/`): - **Theme: PASS.** All six theme tests passed. Switch p99 was 0.8–8.0 ms (budget 16.67 ms); all six recorded zero long tasks >50 ms. - **Dead-session: PASS on HEAD.** All 12 HEAD dead-session tests passed. BASE 1 alone failed the older attach-count assertion (expected 2, received 3); BASE 2/3 and all HEAD cases passed. - **Sessions canvas retry: REAL REGRESSION.** “The empty, loading and error states keep the product's rhythm” passed BASE 1/3, failed BASE 2/3, and failed HEAD 3/3. The timeout snapshot shows the error state still loading after Retry, waiting for six rows. Every other sessions-canvas check passed in all six slots, including the 200/130 ms popover assertion. - **Resume after drop: REAL PRE-EXISTING FAILURE.** Budget 900 ms; BASE p99 was 945.6 / 1012.9 / 1005.2 ms; the one completed HEAD measurement was 978.3 ms. HEAD 1/3 stopped earlier, before this metric. - **Access-screen first paint: REAL PRE-EXISTING FAILURE.** Budget 1000 ms. Setup render p99 was 2586.8 / 2770.1 / 2759.4 ms on BASE; 1732.5 / no measurement / 2202.6 ms on HEAD. Sign-in render p99 was 2136.6 / 2233.1 / 2150.9 ms on BASE; 2388.8 / no measurement / 2323.6 ms on HEAD. HEAD 2 hit a mock-server socket hang-up before sampling. - **Returning sign-in: PASS where measured.** All five completed samples had p99 3979.4–4009.6 ms against a 5000 ms budget. Predicted glyph visibility also passed every completed sample (max p99 14.8 ms against 17 ms). - **Terminal interaction: REAL REGRESSION signals on HEAD.** HEAD 1/3 timed out waiting for the terminal test double to connect after Open infraternal (5 s); HEAD 3/3 timed out with predictive pending text still empty where the probe expected `ok` (`terminal.spec.ts:243`). Neither assertion failed on BASE. HEAD 2 completed the interaction checks and then hit the pre-existing 978.3 ms resume budget failure. All slot durations, load averages, ports, summaries and raw Playwright JSON are in the evidence directory. No product code was changed in this remeasurement.

kayg commented

2026-10-12 17:19:25 +00:00

Author

Owner

#176 measurement step committed as e89cd78b. On three interleaved perf-VM pairs, theme-switch p99 was 1.4–5.1 ms with zero >50 ms long tasks; resume p99 was 981.6–1018.2 ms against 900 ms on both refs; setup paint exceeded 1000 ms on both refs; sign-in render p99 improved from 2179.2–2228.7 ms on BASE to 778.6–975.3 ms on HEAD. Full table and raw reports: docs/verification/perf-vm-remeasure.md and evidence/perf-vm-browser/20261012T130928Z-1162600825-1174652/.

#176 measurement step committed as e89cd78b. On three interleaved perf-VM pairs, theme-switch p99 was 1.4–5.1 ms with zero >50 ms long tasks; resume p99 was 981.6–1018.2 ms against 900 ms on both refs; setup paint exceeded 1000 ms on both refs; sign-in render p99 improved from 2179.2–2228.7 ms on BASE to 778.6–975.3 ms on HEAD. Full table and raw reports: docs/verification/perf-vm-remeasure.md and evidence/perf-vm-browser/20261012T130928Z-1162600825-1174652/.

kayg commented

2026-10-12 17:20:13 +00:00

Author

Owner

#182 measurement step committed as cba7fc26. Across three pairs per engine, Chromium median active CPU was 41.03%→2.70%, TaskDuration 1952.9→73.6 ms, and ScriptDuration 520.3→7.9 ms; WebKit MiniBrowser CPU was 4.97%→4.63%, with active rAF callbacks 1859→63 and interval callbacks 56→0. Steady CPU was similar and long tasks were zero. The exact-ref comparison includes changes beyond D249. Tables and raw JSON are in docs/verification/perf-vm-remeasure.md and evidence/perf-vm-browser/20261012T153720Z-2170831494-2867709/ plus 20261012T154810Z-1444956166-2970883/.

#182 measurement step committed as cba7fc26. Across three pairs per engine, Chromium median active CPU was 41.03%→2.70%, TaskDuration 1952.9→73.6 ms, and ScriptDuration 520.3→7.9 ms; WebKit MiniBrowser CPU was 4.97%→4.63%, with active rAF callbacks 1859→63 and interval callbacks 56→0. Steady CPU was similar and long tasks were zero. The exact-ref comparison includes changes beyond D249. Tables and raw JSON are in docs/verification/perf-vm-remeasure.md and evidence/perf-vm-browser/20261012T153720Z-2170831494-2867709/ plus 20261012T154810Z-1444956166-2970883/.

kayg commented

2026-10-12 17:20:54 +00:00

Author

Owner

Batch 4 measurement step committed as e8b98754. Theme p99 was 0.8–8.0 ms with zero >50 ms long tasks; all current HEAD dead-session tests passed. The sessions-canvas retry failed BASE 1/3 and HEAD 3/3 (real regression). Two terminal interaction assertions failed only on HEAD. Resume and access-screen render budgets also failed on BASE, so those remain pre-existing where measured. Full per-run tables and Playwright artifacts: docs/verification/perf-vm-remeasure.md and evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

Batch 4 measurement step committed as e8b98754. Theme p99 was 0.8–8.0 ms with zero >50 ms long tasks; all current HEAD dead-session tests passed. The sessions-canvas retry failed BASE 1/3 and HEAD 3/3 (real regression). Two terminal interaction assertions failed only on HEAD. Resume and access-screen render budgets also failed on BASE, so those remain pre-existing where measured. Full per-run tables and Playwright artifacts: docs/verification/perf-vm-remeasure.md and evidence/perf-vm-browser/20261012T160109Z-1915616797-3103943/.

kayg commented

2026-10-12 17:50:38 +00:00

Author

Owner

Finding: the required scripts/verify fast run on branch head e8b98754 failed after the cache-targeted rerun. Static builds produced a 21,857,264-byte binary against the 16,404,816-byte budget (binary exceeds ...). The Rust test gate returned exit code 101. I am capturing the failing test row and checking whether the gate failures are unrelated to this docs-only change; no product source was changed.

Finding: the required `scripts/verify fast` run on branch head e8b98754 failed after the cache-targeted rerun. Static builds produced a 21,857,264-byte binary against the 16,404,816-byte budget (`binary exceeds ...`). The Rust `test` gate returned exit code 101. I am capturing the failing test row and checking whether the gate failures are unrelated to this docs-only change; no product source was changed.

kayg commented

2026-10-12 18:03:22 +00:00

Author

Owner

Final cache-targeted scripts/verify fast rerun on e8b98754: tests pass after keeping the scratch target reachable at the path expected by the daemon fixture; no allowed #118 gate test failed. The only remaining Rust gate failure is the static release binary size. Product source is unchanged from pre-measurement HEAD 113adcde (empty diff under crates/, apps/, and packages/). ts_verify_fast was not reached because the Rust gate returned nonzero.

Gate output (verbatim):

/home/kayg/.cache/codex-scratch/perf-remeasure/verify-target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped
PT_INTERP: absent; size: 21857264 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 21857264 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Final cache-targeted `scripts/verify fast` rerun on e8b98754: tests pass after keeping the scratch target reachable at the path expected by the daemon fixture; no allowed #118 gate test failed. The only remaining Rust gate failure is the static release binary size. Product source is unchanged from pre-measurement HEAD 113adcde (empty diff under crates/, apps/, and packages/). `ts_verify_fast` was not reached because the Rust gate returned nonzero. Gate output (verbatim): ``` /home/kayg/.cache/codex-scratch/perf-remeasure/verify-target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped PT_INTERP: absent; size: 21857264 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 21857264 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg commented

2026-10-12 18:04:52 +00:00

Author

Owner

Finished the remeasurement; report and raw run evidence are in docs/verification/perf-vm-remeasure.md and the linked evidence directories. The three commits are e89cd78b, cba7fc26, and e8b98754. Head e8b98754 is pushed to origin/perf-remeasure. The final scripts/verify fast run passed tests, formatting, Clippy, and cargo-deny; it failed only the static binary size budget. Rust returned nonzero before ts_verify_fast ran. The worktree contains no product-source changes.

Final gate output (verbatim):

/home/kayg/.cache/codex-scratch/perf-remeasure/verify-target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped
PT_INTERP: absent; size: 21857264 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 21857264 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Finished the remeasurement; report and raw run evidence are in `docs/verification/perf-vm-remeasure.md` and the linked evidence directories. The three commits are e89cd78b, cba7fc26, and e8b98754. Head e8b98754 is pushed to `origin/perf-remeasure`. The final `scripts/verify fast` run passed tests, formatting, Clippy, and cargo-deny; it failed only the static binary size budget. Rust returned nonzero before `ts_verify_fast` ran. The worktree contains no product-source changes. Final gate output (verbatim): ``` /home/kayg/.cache/codex-scratch/perf-remeasure/verify-target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped PT_INTERP: absent; size: 21857264 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 21857264 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg referenced this issue

2026-10-12 22:49:44 +00:00

Production browser checks red: theme long tasks, terminal resume/latency probe, access-screen first paint #176

~~kayg referenced this issue 2026-10-28 17:53:56 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

Rows
Columns

Browser timing checks run on the perf VM #184