kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

CI: a web e2e job that passes when the code is right, busy machine or not (#11) #236

Closed
kayg wants to merge 0 commits from issue/11-trustworthy-web-e2e-6 into dev
pull from: issue/11-trustworthy-web-e2e-6
merge into: kayg:dev
Conversation 1 Commits - Files changed -
kayg commented 2026-10-14 10:18:48 +00:00
Owner
Copy link

Part of #11. Replaces #147, #159, #175, #204 and #218 (the forge ignores pushes to an open PR). The web e2e job failed on every PR, the same tests on a commit that had been green. This makes a correct suite pass on CI while the machine is busy, and keeps real failures failing. No retries, no longer timeouts.

What was wrong (measured, details on #11)

  1. CI had no priority. Job containers ran in system.slice at the same weight as all the agents together. The agents' CPUWeight=20 sits on user-1001.slice, the only slice under user.slice, so it changed nothing. A web job got a median 3.8 cores and waited for the CPU 45% of the time.
  2. WebKit compiles shaders on every run. The glass (backdrop blur) makes WebKit composite with Mesa's software GL (llvmpipe) in the container. Each shader is compiled on first use, on the compositor's single thread. With a cold cache a page's first frame took ~9 s at 1440 (~4 s at 390), however many cores it had. Every step waiting for a frame waited for that. This isn't a WebKit bug in the app, and real Safari uses the GPU. It is starvation-shaped, though: six workers made each compile take longer still.
  3. Too many browsers for the share. Playwright's default of 6 workers, plus the CPU-throttled (×4, ×6) layout checks running alongside everything else.
  4. Tests sharing state. The settings tests flip the theme and backdrop of the one shared person, and that repaints every other test's open pages, live. The contrast check photographs the lists while other tests add notes to them (#159 run 110: ratios of 1.4 for plain titles). Today's note was written by four shortcut tests at once: two log entries merged on one line.
  5. Test bugs that only show under load. The shortcut helper press() could read a modifier's answer (⌘⌥↑ is three keydowns) or an earlier typed key's answer. Fixed sleeps were used before measuring.
  6. Fixed ports. The web, primitives and mockup configs used fixed ports, and two of them reused whatever server was already listening, which could be another agent's code.

What changed

  • Host (done on calternal-dev, documented in ci/README.md): ci/host/ci.slice (CPUWeight 1000), runner container.options: --cgroup-parent=ci.slice, new allowed volume calternal-mesa-cache.
  • Host: a second runner, forgejo-runner-web (label web, capacity 1, ci/host/forgejo-runner-web.service), takes only the web job, so two web jobs never share CI's slice. Rust jobs stay on the main runner, two at a time, so a run's rust job can always start. Inside ci.slice the web container outranks a rust build (--cpu-shares=8192 on the web runner: weight 532 to 100). Both runners let running jobs finish on a restart (ci/host/graceful.conf, since systemd's 90 s default killed them). #147's second run failed because of exactly that overlap: two web jobs at once, and a WebKit toast in the primitives tests ran out of time.
  • ci.yml: the web job runs-on: web and keeps the Mesa shader cache in that volume. A small server job builds the binary and the web job needs: server. Waiting inside the web job held the web runner while the forge gave the free docker slots to other runs, and two proof runs gave up after 40 minutes with the binary never built. The browser tests no longer depend on format, lint and the Rust tests passing. Every job has a timeout: a linker deadlocked on a futex once held the shared cargo lock for 45 minutes, and with it everyone's queue, until I killed it.
  • playwright.config.ts: 4 workers on CI. WebKit runs one test at a time per width (and one in the primitives): its frames with the glass open take 1–3 s each, and more at once starved each other (WebKit 1440 tests averaged 16 s two at a time, 9 s one at a time; 390 went from 9 s to 5 s). @throttled tests run alone after the rest (Chromium 1440, then 390). The settings tests and the contrast check each get a server and a person of their own (prefs-* and contrast projects, one test at a time). All ports come from the OS. There's one app build per run instead of four, plus a copy for the production-like server, whose tests rewrite its sw.js.
  • Tests: still() replaces fixed sleeps, including those in specs merged meanwhile (contrast, safe areas, tooltips, selector fit, tags). hidden-cost.test.ts measures CPU time, not the wall clock: on a busy machine the wall clock also counts waiting for a core. press() reads only its own key. ⌘⇧; is checked in a day note of the test's own, and Bold is hovered once the bar is still.
  • Primitives and mockup Playwright configs: OS-picked ports, never reuse an existing server.

Verified

The same suite, in the CI image, on this machine while agents were building (host load 70–105). Each row changes one thing from the row above:

run failed / passed wall
as CI was (system.slice, 6 workers) 29 / 107 11.5 min
ci.slice + 4 workers + isolation fixes 3–8 / 126–131 4.9–5.8 min
+ warm shader cache 1–5 / 129–133 4.7–5.2 min
+ WebKit 1440 at 2, press() fix 0 / 150 5.6 min
+ contrast on its own server, WebKit 1440 at 1 (dev at dff1833) 0 / 151 8.6 min
+ WebKit one per width (dev at dff1833, host load 107) 0 / 151 7.9 min

On CI (#147, run 103, alongside another web job still on the old config): web job green, 150 passed / 0 failed in 7.3 min. The proof of three green runs in a row is on a branch of the same commits on dev at dff1833 (proof PR #237, which replaces #177, #205 and #219; numbers on #11). It's a separate branch because dev has gone red since then for reasons outside this PR (below).

Found, not fixed here

  • pwa-update.spec.ts "a reload after a deploy shows the new version" fails every time on dev (from #164). It failed in that PR's own CI runs, and here alone on one worker: after the reload the old shell-… cache is still there 60 s later. This is a real failure, so the suite is right to stay red until #132's owner looks at it. This PR only gives that server its own copy of the build, so the faked deploy can't reach the other servers.

  • The rust job is red on dev for every PR right now: cargo fmt wants mod trash; before mod writers; in crates/notes/src/tests.rs. That came from another merge and is left for whoever owns it.

  • links.spec "Info shows … plain mentions": once, with one worker on an idle machine, the plain mention never showed. The search index updates after the note-changed event the Info sheet re-reads on, so Info can read a stale index and never re-read. That looks like a real race, so the test is right to fail. It needs its own issue.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Part of #11. Replaces #147, #159, #175, #204 and #218 (the forge ignores pushes to an open PR). The web e2e job failed on every PR, the same tests on a commit that had been green. This makes a correct suite pass on CI while the machine is busy, and keeps real failures failing. No retries, no longer timeouts. ## What was wrong (measured, details on #11) 1. **CI had no priority.** Job containers ran in `system.slice` at the same weight as all the agents together. The agents' `CPUWeight=20` sits on `user-1001.slice`, the only slice under `user.slice`, so it changed nothing. A web job got a median 3.8 cores and waited for the CPU 45% of the time. 2. **WebKit compiles shaders on every run.** The glass (backdrop blur) makes WebKit composite with Mesa's software GL (llvmpipe) in the container. Each shader is compiled on first use, on the compositor's single thread. With a cold cache a page's first frame took ~9 s at 1440 (~4 s at 390), however many cores it had. Every step waiting for a frame waited for that. This isn't a WebKit bug in the app, and real Safari uses the GPU. It is starvation-shaped, though: six workers made each compile take longer still. 3. **Too many browsers for the share.** Playwright's default of 6 workers, plus the CPU-throttled (×4, ×6) layout checks running alongside everything else. 4. **Tests sharing state.** The settings tests flip the theme and backdrop of the one shared person, and that repaints every other test's open pages, live. The contrast check photographs the lists while other tests add notes to them (#159 run 110: ratios of 1.4 for plain titles). Today's note was written by four shortcut tests at once: two log entries merged on one line. 5. **Test bugs that only show under load.** The shortcut helper `press()` could read a modifier's answer (⌘⌥↑ is three keydowns) or an earlier typed key's answer. Fixed sleeps were used before measuring. 6. **Fixed ports.** The web, primitives and mockup configs used fixed ports, and two of them reused whatever server was already listening, which could be another agent's code. ## What changed - Host (done on calternal-dev, documented in `ci/README.md`): `ci/host/ci.slice` (CPUWeight 1000), runner `container.options: --cgroup-parent=ci.slice`, new allowed volume `calternal-mesa-cache`. - Host: a second runner, `forgejo-runner-web` (label `web`, capacity 1, `ci/host/forgejo-runner-web.service`), takes only the web job, so two web jobs never share CI's slice. Rust jobs stay on the main runner, two at a time, so a run's rust job can always start. Inside `ci.slice` the web container outranks a rust build (`--cpu-shares=8192` on the web runner: weight 532 to 100). Both runners let running jobs finish on a restart (`ci/host/graceful.conf`, since systemd's 90 s default killed them). #147's second run failed because of exactly that overlap: two web jobs at once, and a WebKit toast in the primitives tests ran out of time. - `ci.yml`: the web job `runs-on: web` and keeps the Mesa shader cache in that volume. A small `server` job builds the binary and the web job `needs: server`. Waiting inside the web job held the web runner while the forge gave the free docker slots to other runs, and two proof runs gave up after 40 minutes with the binary never built. The browser tests no longer depend on format, lint and the Rust tests passing. Every job has a timeout: a linker deadlocked on a futex once held the shared cargo lock for 45 minutes, and with it everyone's queue, until I killed it. - `playwright.config.ts`: 4 workers on CI. WebKit runs one test at a time per width (and one in the primitives): its frames with the glass open take 1–3 s each, and more at once starved each other (WebKit 1440 tests averaged 16 s two at a time, 9 s one at a time; 390 went from 9 s to 5 s). `@throttled` tests run alone after the rest (Chromium 1440, then 390). The settings tests and the contrast check each get a server and a person of their own (`prefs-*` and `contrast` projects, one test at a time). All ports come from the OS. There's one app build per run instead of four, plus a copy for the production-like server, whose tests rewrite its `sw.js`. - Tests: `still()` replaces fixed sleeps, including those in specs merged meanwhile (contrast, safe areas, tooltips, selector fit, tags). `hidden-cost.test.ts` measures CPU time, not the wall clock: on a busy machine the wall clock also counts waiting for a core. `press()` reads only its own key. ⌘⇧; is checked in a day note of the test's own, and Bold is hovered once the bar is still. - Primitives and mockup Playwright configs: OS-picked ports, never reuse an existing server. ## Verified The same suite, in the CI image, on this machine while agents were building (host load 70–105). Each row changes one thing from the row above: | run | failed / passed | wall | |---|---|---| | as CI was (system.slice, 6 workers) | 29 / 107 | 11.5 min | | ci.slice + 4 workers + isolation fixes | 3–8 / 126–131 | 4.9–5.8 min | | + warm shader cache | 1–5 / 129–133 | 4.7–5.2 min | | + WebKit 1440 at 2, press() fix | **0 / 150** | 5.6 min | | + contrast on its own server, WebKit 1440 at 1 (dev at dff1833) | **0 / 151** | 8.6 min | | + WebKit one per width (dev at dff1833, host load 107) | **0 / 151** | 7.9 min | On CI (#147, run 103, alongside another web job still on the old config): web job green, 150 passed / 0 failed in 7.3 min. The proof of three green runs in a row is on a branch of the same commits on dev at dff1833 (proof PR #237, which replaces #177, #205 and #219; numbers on #11). It's a separate branch because dev has gone red since then for reasons outside this PR (below). ## Found, not fixed here - **`pwa-update.spec.ts` "a reload after a deploy shows the new version" fails every time on dev (from #164).** It failed in that PR's own CI runs, and here alone on one worker: after the reload the old `shell-…` cache is still there 60 s later. This is a real failure, so the suite is right to stay red until #132's owner looks at it. This PR only gives that server its own copy of the build, so the faked deploy can't reach the other servers. - The rust job is red on dev for every PR right now: `cargo fmt` wants `mod trash;` before `mod writers;` in `crates/notes/src/tests.rs`. That came from another merge and is left for whoever owns it. - `links.spec` "Info shows … plain mentions": once, with one worker on an idle machine, the plain mention never showed. The search index updates after the note-changed event the Info sheet re-reads on, so Info can read a stale index and never re-read. That looks like a real race, so the test is right to fail. It needs its own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg added 10 commits 2026-10-14 10:18:48 +00:00 
The browser tests failed on every PR, the same tests on a commit that
was green before. Measured causes, and what changed:

- CI had no priority. Job containers sat in system.slice at the same
  weight as all the agents (user-1001's CPUWeight=20 is the only slice
  under user.slice, so it changed nothing). Now every job container is
  in ci.slice (weight 1000, ci/host/ci.slice; the runner passes
  --cgroup-parent=ci.slice).
- WebKit composites the glass with Mesa's software GL in the container.
  Each shader is compiled on first use on the compositor's one thread:
  a page's first frame took ~9 s at 1440, whatever the cores. The shader
  cache now lives in a volume kept between runs (~2.5 s warm).
- Four workers on CI instead of six, WebKit 1440 at two at a time, and
  the CPU-throttled layout checks (@throttled) alone after the rest.
- The settings tests get their own server and person: flipping the
  theme of the shared person repainted every other test's pages.
- Test bugs a busy machine exposed: the shortcut helper read a
  modifier's or an earlier key's answer; ⌘⇧; was checked in today's
  note, which other tests log into too; fixed sleeps before measuring
  are now still(), which waits for animations to end.
- Every server (web, primitives, mockup) listens on a port the system
  picks, so runs side by side never collide or reuse another run's
  server; the app is built once per run instead of four times at once.

ci/README.md has the host setup and the numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Two web jobs at once split CI's share between eight browsers, and
WebKit's software drawing slowed until toasts timed out before they
could be clicked (run 107 of #147). A second runner with the label web
and capacity 1 takes only the web job; rust stays on the main runner, so
a web job never waits for a rust job that can't start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
The contrast check photographs the lists, and other tests adding notes
moved the rows between finding the words and taking the picture (#159,
run 110: ratios of 1.4 for ordinary titles). It now runs on its own
server, like the settings tests; the server comes from the project's
metadata. WebKit 1440 frames with the glass open took 1-3 s each on a
busy machine, and two at once starved each other: one at a time, a test
averages 10 s instead of 16 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
pwa-update.spec.ts fakes a deploy by rewriting sw.js in the served
build; with one build for every server that would have reached all of
them. Also still() for two new fixed sleeps in safe-areas.spec.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
WebKit 1440 alone still met three WebKit 390 workers: four software-drawn
browsers at once, and the first WebKit 1440 test ran out of time twice
(#177). Now one per width, and one in the primitives. hidden-cost.test.ts
timed buildHidden by the wall clock, which on a busy machine also counts
waiting for a core; it now measures CPU time. The main runner takes one
job at a time, and both runners finish running jobs on a restart
(ci/host/graceful.conf).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Two proof runs failed waiting 15 minutes for a rust job still queued
behind other runs' jobs. The web job now waits up to 40 minutes, and the
rust job leaves a build-failed mark so a broken build ends the wait at
once. The web job's container outranks the rust build inside ci.slice
(--cpu-shares on the web runner, weight 532 to 100); the main runner
stays at two jobs so a run's rust job can always start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
Web e2e: still() for the fixed sleeps in the new tooltip, selector and tag specs (#11)
Some checks failed
CI / web (pull_request) Failing after 43m7s
Details
CI / rust (pull_request) Failing after 1m53s
Details
475d854fff 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
A rust job's linker deadlocked on a futex and held the shared target
lock for 45 minutes; every other run's rust job, and so its web job,
waited behind it. Rust jobs now stop after 30 minutes, web jobs after 60
(40 of which may be waiting for the binary).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
CI: a server job the web job needs, instead of a wait inside the web job (#11)
Some checks failed
CI / server (pull_request) Successful in 22s
Details
CI / rust (pull_request) Successful in 1m47s
Details
CI / web (pull_request) Has been cancelled
Details
ff7ea090ac 
The web job started at once on its own runner and waited inside for the
binary; meanwhile the forge gave the free docker slots to other runs'
jobs, and the wait ran out with the binary never built (#218, #219). A
small server job now builds the binary; the web job needs it, so the web
runner only takes web jobs that can run, and the browser tests no longer
depend on format, lint and the Rust tests passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T
kayg commented 2026-10-14 11:54:38 +00:00
Author
Owner
Copy link

Replaced by #259: needs never schedules the web job on this forge, so the web job waits for the server job's binary again. The forge ignores pushes to an open PR.

Replaced by #259: `needs` never schedules the web job on this forge, so the web job waits for the server job's binary again. The forge ignores pushes to an open PR.
kayg closed this pull request 2026-10-14 11:54:38 +00:00
Some checks failed
CI / server (pull_request) Successful in 22s
Details
CI / rust (pull_request) Successful in 1m47s
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!236
No description provided.