CI: a web e2e job that passes when the code is right, busy machine or not (#11)

kayg commented

2026-10-14 05:46:53 +00:00

Owner

Part of #11. The web e2e job failed on every PR, the same tests on a commit that had been green. This makes a correct suite pass on CI while the machine is busy, and keeps real failures failing. No retries, no longer timeouts.

What was wrong (measured, details on #11)

CI had no priority. Job containers ran in system.slice at the same weight as all the agents together. The agents' CPUWeight=20 sits on user-1001.slice, the only slice under user.slice, so it changed nothing. A web job got a median 3.8 cores and waited for the CPU 45% of the time.
WebKit compiles shaders on every run. The glass (backdrop blur) makes WebKit composite with Mesa's software GL (llvmpipe) in the container. Each shader is compiled on first use, on the compositor's single thread. With a cold cache a page's first frame took ~9 s at 1440 (~4 s at 390), however many cores it had. Every step waiting for a frame waited for that. This isn't a WebKit bug in the app, and real Safari uses the GPU. It is starvation-shaped, though: six workers made each compile take longer still.
Too many browsers for the share. Playwright's default of 6 workers, plus the CPU-throttled (×4, ×6) layout checks running alongside everything else.
Tests sharing state. The settings tests flip the theme and backdrop of the one shared person, and that repaints every other test's open pages, live. Today's note was written by four shortcut tests at once: two log entries merged on one line.
Test bugs that only show under load. The shortcut helper press() could read a modifier's answer (⌘⌥↑ is three keydowns) or an earlier typed key's answer. Fixed sleeps were used before measuring.
Fixed ports. The web, primitives and mockup configs used fixed ports, and two of them reused whatever server was already listening, which could be another agent's code.

What changed

Host (done on calternal-dev, documented in ci/README.md): ci/host/ci.slice (CPUWeight 1000), runner container.options: --cgroup-parent=ci.slice, new allowed volume calternal-mesa-cache.
ci.yml: the web job keeps the Mesa shader cache in that volume.
playwright.config.ts: 4 workers on CI, WebKit 1440 at most 2, @throttled tests run alone after the rest (Chromium 1440, then 390). The settings tests get their own server and person (prefs-* projects, one at a time). All ports come from the OS, and there's one app build per run instead of four.
Tests: still() replaces fixed sleeps. press() reads only its own key. ⌘⇧; is checked in a day note of the test's own, and Bold is hovered once the bar is still.
Primitives and mockup Playwright configs: OS-picked ports, never reuse an existing server.

Verified

The same suite, in the CI image, on this machine while agents were building (host load 70–105). Each row changes one thing from the row above:

run	failed / passed	wall
as CI was (system.slice, 6 workers)	29 / 107	11.5 min
ci.slice + 4 workers + isolation fixes	3–8 / 126–131	4.9–5.8 min
+ warm shader cache	1–5 / 129–133	4.7–5.2 min
+ WebKit 1440 at 2, press() fix	0 / 150	5.6 min

CI runs on this PR and their numbers go on #11.

Found, not fixed here

links.spec "Info shows … plain mentions": once, with one worker on an idle machine, the plain mention never showed. The search index updates after the note-changed event the Info sheet re-reads on, so Info can read a stale index and never re-read. That looks like a real race, so the test is right to fail. It needs its own issue.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Part of #11. The web e2e job failed on every PR, the same tests on a commit that had been green. This makes a correct suite pass on CI while the machine is busy, and keeps real failures failing. No retries, no longer timeouts. ## What was wrong (measured, details on #11) 1. **CI had no priority.** Job containers ran in `system.slice` at the same weight as all the agents together. The agents' `CPUWeight=20` sits on `user-1001.slice`, the only slice under `user.slice`, so it changed nothing. A web job got a median 3.8 cores and waited for the CPU 45% of the time. 2. **WebKit compiles shaders on every run.** The glass (backdrop blur) makes WebKit composite with Mesa's software GL (llvmpipe) in the container. Each shader is compiled on first use, on the compositor's single thread. With a cold cache a page's first frame took ~9 s at 1440 (~4 s at 390), however many cores it had. Every step waiting for a frame waited for that. This isn't a WebKit bug in the app, and real Safari uses the GPU. It is starvation-shaped, though: six workers made each compile take longer still. 3. **Too many browsers for the share.** Playwright's default of 6 workers, plus the CPU-throttled (×4, ×6) layout checks running alongside everything else. 4. **Tests sharing state.** The settings tests flip the theme and backdrop of the one shared person, and that repaints every other test's open pages, live. Today's note was written by four shortcut tests at once: two log entries merged on one line. 5. **Test bugs that only show under load.** The shortcut helper `press()` could read a modifier's answer (⌘⌥↑ is three keydowns) or an earlier typed key's answer. Fixed sleeps were used before measuring. 6. **Fixed ports.** The web, primitives and mockup configs used fixed ports, and two of them reused whatever server was already listening, which could be another agent's code. ## What changed - Host (done on calternal-dev, documented in `ci/README.md`): `ci/host/ci.slice` (CPUWeight 1000), runner `container.options: --cgroup-parent=ci.slice`, new allowed volume `calternal-mesa-cache`. - `ci.yml`: the web job keeps the Mesa shader cache in that volume. - `playwright.config.ts`: 4 workers on CI, WebKit 1440 at most 2, `@throttled` tests run alone after the rest (Chromium 1440, then 390). The settings tests get their own server and person (`prefs-*` projects, one at a time). All ports come from the OS, and there's one app build per run instead of four. - Tests: `still()` replaces fixed sleeps. `press()` reads only its own key. ⌘⇧; is checked in a day note of the test's own, and Bold is hovered once the bar is still. - Primitives and mockup Playwright configs: OS-picked ports, never reuse an existing server. ## Verified The same suite, in the CI image, on this machine while agents were building (host load 70–105). Each row changes one thing from the row above: | run | failed / passed | wall | |---|---|---| | as CI was (system.slice, 6 workers) | 29 / 107 | 11.5 min | | ci.slice + 4 workers + isolation fixes | 3–8 / 126–131 | 4.9–5.8 min | | + warm shader cache | 1–5 / 129–133 | 4.7–5.2 min | | + WebKit 1440 at 2, press() fix | **0 / 150** | 5.6 min | CI runs on this PR and their numbers go on #11. ## Found, not fixed here - `links.spec` "Info shows … plain mentions": once, with one worker on an idle machine, the plain mention never showed. The search index updates after the note-changed event the Info sheet re-reads on, so Info can read a stale index and never re-read. That looks like a real race, so the test is right to fail. It needs its own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 1 commit

2026-10-14 05:46:53 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11 )

CI / rust (pull_request) Failing after 27s

Details

CI / web (pull_request) Failing after 1m46s

Details

c33b2cb1fa

The browser tests failed on every PR, the same tests on a commit that
was green before. Measured causes, and what changed:

- CI had no priority. Job containers sat in system.slice at the same
  weight as all the agents (user-1001's CPUWeight=20 is the only slice
  under user.slice, so it changed nothing). Now every job container is
  in ci.slice (weight 1000, ci/host/ci.slice; the runner passes
  --cgroup-parent=ci.slice).
- WebKit composites the glass with Mesa's software GL in the container.
  Each shader is compiled on first use on the compositor's one thread:
  a page's first frame took ~9 s at 1440, whatever the cores. The shader
  cache now lives in a volume kept between runs (~2.5 s warm).
- Four workers on CI instead of six, WebKit 1440 at two at a time, and
  the CPU-throttled layout checks (@throttled) alone after the rest.
- The settings tests get their own server and person: flipping the
  theme of the shared person repainted every other test's pages.
- Test bugs a busy machine exposed: the shortcut helper read a
  modifier's or an earlier key's answer; ⌘⇧; was checked in today's
  note, which other tests log into too; fixed sleeps before measuring
  are now still(), which waits for animations to end.
- Every server (web, primitives, mockup) listens on a port the system
  picks, so runs side by side never collide or reuse another run's
  server; the app is built once per run instead of four times at once.

ci/README.md has the host setup and the numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-14 06:25:28 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #159

kayg commented

2026-10-14 06:25:34 +00:00

Author

Owner

Replaced by #159: same work plus a runner of its own for the web job (this PR's second run, 107, failed when two web jobs overlapped). The forge ignores pushes to an open PR.

kayg closed this pull request

2026-10-14 06:25:36 +00:00

kayg referenced this pull request

2026-10-14 07:18:37 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #175

kayg referenced this pull request

2026-10-14 07:20:17 +00:00

UX gap sweep: dead buttons, wrong action inputs, non-live views #107

kayg referenced this pull request

2026-10-14 08:14:06 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #204

kayg referenced this pull request