CI: a web e2e job that passes when the code is right, busy machine or not (#11)

kayg commented

2026-10-14 06:25:28 +00:00

Owner

Part of #11. Replaces #147 (the forge ignores pushes to an open PR). The web e2e job failed on every PR, the same tests on a commit that had been green. This makes a correct suite pass on CI while the machine is busy, and keeps real failures failing. No retries, no longer timeouts.

What was wrong (measured, details on #11)

CI had no priority. Job containers ran in system.slice at the same weight as all the agents together. The agents' CPUWeight=20 sits on user-1001.slice, the only slice under user.slice, so it changed nothing. A web job got a median 3.8 cores and waited for the CPU 45% of the time.
WebKit compiles shaders on every run. The glass (backdrop blur) makes WebKit composite with Mesa's software GL (llvmpipe) in the container. Each shader is compiled on first use, on the compositor's single thread. With a cold cache a page's first frame took ~9 s at 1440 (~4 s at 390), however many cores it had. Every step waiting for a frame waited for that. This isn't a WebKit bug in the app, and real Safari uses the GPU. It is starvation-shaped, though: six workers made each compile take longer still.
Too many browsers for the share. Playwright's default of 6 workers, plus the CPU-throttled (×4, ×6) layout checks running alongside everything else.
Tests sharing state. The settings tests flip the theme and backdrop of the one shared person, and that repaints every other test's open pages, live. Today's note was written by four shortcut tests at once: two log entries merged on one line.
Test bugs that only show under load. The shortcut helper press() could read a modifier's answer (⌘⌥↑ is three keydowns) or an earlier typed key's answer. Fixed sleeps were used before measuring.
Fixed ports. The web, primitives and mockup configs used fixed ports, and two of them reused whatever server was already listening, which could be another agent's code.

What changed

Host (done on calternal-dev, documented in ci/README.md): ci/host/ci.slice (CPUWeight 1000), runner container.options: --cgroup-parent=ci.slice, new allowed volume calternal-mesa-cache.
Host: a second runner, forgejo-runner-web (label web, capacity 1, ci/host/forgejo-runner-web.service), takes only the web job, so two web jobs never share CI's slice. Rust jobs stay on the main runner, so a web job never waits for a rust job that can't start. #147's second run failed because of exactly that overlap: two web jobs at once, and a WebKit toast in the primitives tests ran out of time.
ci.yml: the web job runs-on: web and keeps the Mesa shader cache in that volume.
playwright.config.ts: 4 workers on CI, WebKit 1440 at most 2, @throttled tests run alone after the rest (Chromium 1440, then 390). The settings tests get their own server and person (prefs-* projects, one at a time). All ports come from the OS, and there's one app build per run instead of four.
Tests: still() replaces fixed sleeps. press() reads only its own key. ⌘⇧; is checked in a day note of the test's own, and Bold is hovered once the bar is still.
Primitives and mockup Playwright configs: OS-picked ports, never reuse an existing server.

Verified

The same suite, in the CI image, on this machine while agents were building (host load 70–105). Each row changes one thing from the row above:

run	failed / passed	wall
as CI was (system.slice, 6 workers)	29 / 107	11.5 min
ci.slice + 4 workers + isolation fixes	3–8 / 126–131	4.9–5.8 min
+ warm shader cache	1–5 / 129–133	4.7–5.2 min
+ WebKit 1440 at 2, press() fix	0 / 150	5.6 min

On CI (#147, run 103, alongside another web job still on the old config): web job green, 150 passed / 0 failed in 7.3 min. The runs of this PR and their numbers go on #11.

Found, not fixed here

The rust job is red on dev for every PR right now: cargo fmt wants mod trash; before mod writers; in crates/notes/src/tests.rs. That came from another merge and is left for whoever owns it.
links.spec "Info shows … plain mentions": once, with one worker on an idle machine, the plain mention never showed. The search index updates after the note-changed event the Info sheet re-reads on, so Info can read a stale index and never re-read. That looks like a real race, so the test is right to fail. It needs its own issue.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Part of #11. Replaces #147 (the forge ignores pushes to an open PR). The web e2e job failed on every PR, the same tests on a commit that had been green. This makes a correct suite pass on CI while the machine is busy, and keeps real failures failing. No retries, no longer timeouts. ## What was wrong (measured, details on #11) 1. **CI had no priority.** Job containers ran in `system.slice` at the same weight as all the agents together. The agents' `CPUWeight=20` sits on `user-1001.slice`, the only slice under `user.slice`, so it changed nothing. A web job got a median 3.8 cores and waited for the CPU 45% of the time. 2. **WebKit compiles shaders on every run.** The glass (backdrop blur) makes WebKit composite with Mesa's software GL (llvmpipe) in the container. Each shader is compiled on first use, on the compositor's single thread. With a cold cache a page's first frame took ~9 s at 1440 (~4 s at 390), however many cores it had. Every step waiting for a frame waited for that. This isn't a WebKit bug in the app, and real Safari uses the GPU. It is starvation-shaped, though: six workers made each compile take longer still. 3. **Too many browsers for the share.** Playwright's default of 6 workers, plus the CPU-throttled (×4, ×6) layout checks running alongside everything else. 4. **Tests sharing state.** The settings tests flip the theme and backdrop of the one shared person, and that repaints every other test's open pages, live. Today's note was written by four shortcut tests at once: two log entries merged on one line. 5. **Test bugs that only show under load.** The shortcut helper `press()` could read a modifier's answer (⌘⌥↑ is three keydowns) or an earlier typed key's answer. Fixed sleeps were used before measuring. 6. **Fixed ports.** The web, primitives and mockup configs used fixed ports, and two of them reused whatever server was already listening, which could be another agent's code. ## What changed - Host (done on calternal-dev, documented in `ci/README.md`): `ci/host/ci.slice` (CPUWeight 1000), runner `container.options: --cgroup-parent=ci.slice`, new allowed volume `calternal-mesa-cache`. - Host: a second runner, `forgejo-runner-web` (label `web`, capacity 1, `ci/host/forgejo-runner-web.service`), takes only the web job, so two web jobs never share CI's slice. Rust jobs stay on the main runner, so a web job never waits for a rust job that can't start. #147's second run failed because of exactly that overlap: two web jobs at once, and a WebKit toast in the primitives tests ran out of time. - `ci.yml`: the web job `runs-on: web` and keeps the Mesa shader cache in that volume. - `playwright.config.ts`: 4 workers on CI, WebKit 1440 at most 2, `@throttled` tests run alone after the rest (Chromium 1440, then 390). The settings tests get their own server and person (`prefs-*` projects, one at a time). All ports come from the OS, and there's one app build per run instead of four. - Tests: `still()` replaces fixed sleeps. `press()` reads only its own key. ⌘⇧; is checked in a day note of the test's own, and Bold is hovered once the bar is still. - Primitives and mockup Playwright configs: OS-picked ports, never reuse an existing server. ## Verified The same suite, in the CI image, on this machine while agents were building (host load 70–105). Each row changes one thing from the row above: | run | failed / passed | wall | |---|---|---| | as CI was (system.slice, 6 workers) | 29 / 107 | 11.5 min | | ci.slice + 4 workers + isolation fixes | 3–8 / 126–131 | 4.9–5.8 min | | + warm shader cache | 1–5 / 129–133 | 4.7–5.2 min | | + WebKit 1440 at 2, press() fix | **0 / 150** | 5.6 min | On CI (#147, run 103, alongside another web job still on the old config): web job green, 150 passed / 0 failed in 7.3 min. The runs of this PR and their numbers go on #11. ## Found, not fixed here - The rust job is red on dev for every PR right now: `cargo fmt` wants `mod trash;` before `mod writers;` in `crates/notes/src/tests.rs`. That came from another merge and is left for whoever owns it. - `links.spec` "Info shows … plain mentions": once, with one worker on an idle machine, the plain mention never showed. The search index updates after the note-changed event the Info sheet re-reads on, so Info can read a stale index and never re-read. That looks like a real race, so the test is right to fail. It needs its own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 3 commits

2026-10-14 06:25:28 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11 ) 10128ac3bd

The browser tests failed on every PR, the same tests on a commit that
was green before. Measured causes, and what changed:

- CI had no priority. Job containers sat in system.slice at the same
  weight as all the agents (user-1001's CPUWeight=20 is the only slice
  under user.slice, so it changed nothing). Now every job container is
  in ci.slice (weight 1000, ci/host/ci.slice; the runner passes
  --cgroup-parent=ci.slice).
- WebKit composites the glass with Mesa's software GL in the container.
  Each shader is compiled on first use on the compositor's one thread:
  a page's first frame took ~9 s at 1440, whatever the cores. The shader
  cache now lives in a volume kept between runs (~2.5 s warm).
- Four workers on CI instead of six, WebKit 1440 at two at a time, and
  the CPU-throttled layout checks (@throttled) alone after the rest.
- The settings tests get their own server and person: flipping the
  theme of the shared person repainted every other test's pages.
- Test bugs a busy machine exposed: the shortcut helper read a
  modifier's or an earlier key's answer; ⌘⇧; was checked in today's
  note, which other tests log into too; fixed sleeps before measuring
  are now still(), which waits for animations to end.
- Every server (web, primitives, mockup) listens on a port the system
  picks, so runs side by side never collide or reuse another run's
  server; the app is built once per run instead of four times at once.

ci/README.md has the host setup and the numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

CI: the web job on a runner of its own, one at a time (#11 ) 99ab37ddc1

Two web jobs at once split CI's share between eight browsers, and
WebKit's software drawing slowed until toasts timed out before they
could be clicked (run 107 of #147). A second runner with the label web
and capacity 1 takes only the web job; rust stays on the main runner, so
a web job never waits for a rust job that can't start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Web e2e: the contrast check waits for a still screen, not 250 ms (#11 )

CI / rust (pull_request) Failing after 24s

Details

CI / web (pull_request) Failing after 11m3s

Details

08da50e230

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-14 06:25:34 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #147

kayg referenced this pull request

2026-10-14 07:18:37 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #175

kayg commented

2026-10-14 07:18:48 +00:00

Author

Owner

Replaced by #175 (same work, rebased on dev, plus: contrast check on its own server, WebKit 1440 one at a time, a copy of the build for the production-like server). The forge ignores pushes to an open PR.

kayg closed this pull request

2026-10-14 07:18:49 +00:00

kayg referenced this pull request

2026-10-14 08:14:06 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #204

kayg referenced this pull request

2026-10-14 09:06:36 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #218

kayg referenced this pull request

2026-10-14 10:18:48 +00:00

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #236

kayg referenced this pull request