kayg/infraternal

Watch

Fork 0

browser suite: 34 spec files fail on dev (roll-up) #482

New issue

Open

opened 2026-11-10 04:41:37 +00:00 by kayg · 2 comments

kayg commented

2026-11-10 04:41:37 +00:00

Owner

Nightly browser suite on dev (D373, #443).

34 new spec files failed out of 165 that ran. That is more than 10 new issues or a mostly red suite, so it is filed once, not per spec: look for a dead test server or a broken build first.

Source: 8c2b8f5dd166eee270c5aa5484572dd94b3a1c76
Evidence: /var/lib/infraternal-browser/runs/20261109T234323Z-8c2b8f5/evidence on infraternal-dev-02-std (the newest seven nights are kept)
Full log: /var/lib/infraternal-browser/runs/20261109T234323Z-8c2b8f5/suite.log

Failing spec files (names as Playwright reported them)

access-level.spec.ts
addhost-pill-stream.spec.ts
addhost-pill.spec.ts
ca-unlock.spec.ts
canvas.spec.ts
clusters-bench.spec.ts
clusters-close-centre.spec.ts
clusters-keyboard-split.spec.ts
clusters-swipes-both.spec.ts
clusters-swipes.spec.ts
clusters.spec.ts
core-screen.spec.ts
font-metrics.spec.ts
font-picker.spec.ts
host-removal-confirm.spec.ts
hosts-new-window-review.spec.ts
inbox.spec.ts
keytray-modifiers.spec.ts
machine-pill-flicker.spec.ts
machine-sections.spec.ts
mesh-rename.spec.ts
mesh-view-overlap.spec.ts
mesh.spec.ts
motion-audit.spec.ts
palette-navigation.spec.ts
perf-hud.spec.ts
phone-polish.spec.ts
phone-window-width.spec.ts
recovery-ui-hold.spec.ts
sessions-canvas.spec.ts
settings-rows-of-windows.spec.ts
startup-demo.spec.ts
terminal-theme.spec.ts
wheel-scroll.spec.ts

Nightly browser suite on dev (D373, #443). 34 new spec files failed out of 165 that ran. That is more than 10 new issues or a mostly red suite, so it is filed once, not per spec: look for a dead test server or a broken build first. Source: `8c2b8f5dd166eee270c5aa5484572dd94b3a1c76` Evidence: `/var/lib/infraternal-browser/runs/20261109T234323Z-8c2b8f5/evidence` on infraternal-dev-02-std (the newest seven nights are kept) Full log: `/var/lib/infraternal-browser/runs/20261109T234323Z-8c2b8f5/suite.log` ### Failing spec files (names as Playwright reported them) ``` access-level.spec.ts addhost-pill-stream.spec.ts addhost-pill.spec.ts ca-unlock.spec.ts canvas.spec.ts clusters-bench.spec.ts clusters-close-centre.spec.ts clusters-keyboard-split.spec.ts clusters-swipes-both.spec.ts clusters-swipes.spec.ts clusters.spec.ts core-screen.spec.ts font-metrics.spec.ts font-picker.spec.ts host-removal-confirm.spec.ts hosts-new-window-review.spec.ts inbox.spec.ts keytray-modifiers.spec.ts machine-pill-flicker.spec.ts machine-sections.spec.ts mesh-rename.spec.ts mesh-view-overlap.spec.ts mesh.spec.ts motion-audit.spec.ts palette-navigation.spec.ts perf-hud.spec.ts phone-polish.spec.ts phone-window-width.spec.ts recovery-ui-hold.spec.ts sessions-canvas.spec.ts settings-rows-of-windows.spec.ts startup-demo.spec.ts terminal-theme.spec.ts wheel-scroll.spec.ts ```

kayg referenced this issue

2026-11-10 04:41:57 +00:00

Nightly full browser suite on dev with per-spec issues (D373) #443

kayg referenced this issue

2026-11-10 05:08:16 +00:00

Nightly full browser suite on dev with per-spec issues (D373) #443

kayg commented

2026-11-10 05:08:16 +00:00

Author

Owner

Request for the Maintainer (via the orchestrator), for the #482 triage: /var/lib/infraternal-browser/runs/20261109T234323Z-8c2b8f5/ is 0700, so claude cannot read suite.log, results.jsonl or the evidence. Please copy suite.log and results.jsonl (and, if small, the evidence/ folder) to /home/claude/j406-ev/r8c2b8f5/ with a+r, originals untouched. Meanwhile I am triaging from my own runs on the same specs and from fresh reruns on current dev (3c0b20252); the comment on #482 will say which parts rest on those and which on the 8c2b8f5 logs.

Request for the Maintainer (via the orchestrator), for the #482 triage: `/var/lib/infraternal-browser/runs/20261109T234323Z-8c2b8f5/` is 0700, so `claude` cannot read `suite.log`, `results.jsonl` or the evidence. Please copy `suite.log` and `results.jsonl` (and, if small, the `evidence/` folder) to `/home/claude/j406-ev/r8c2b8f5/` with `a+r`, originals untouched. Meanwhile I am triaging from my own runs on the same specs and from fresh reruns on current dev (3c0b20252); the comment on #482 will say which parts rest on those and which on the 8c2b8f5 logs.

kayg commented

2026-11-10 09:43:09 +00:00

Author

Owner

Triage of the 34 red spec files (run `20261109T234323Z-8c2b8f5`) — J406

Verdict: this is not a dead or slow test server. It is 34 files of mostly spec/product drift on dev, deterministic, and most of it is already in an open branch.

Evidence base.

The 8c2b8f5 run itself (copied by the Maintainer to /home/claude/j406-ev/r8c2b8f5/ on dev-02, copy of suite.log and results.json on the control host in ~/.cache/codex-scratch/j406/evidence/tri/): single worker, 4.24 h, 1,240 passed, 80 failed, 195 skipped. In suite.log there are 0 ECONNRESET/ECONNREFUSED/socket hang up lines and 0 90 s timeouts, and every server-dependent spec outside these 34 passed. So no server death, no slow server.
Rerun on current dev (3c0b20252) of 33 of the 34 files (all except clusters-bench, a 4-minute frame benchmark), two parallel Playwright runs on dev-02, ports 5741 and 5751, one build slot each, nothing else changed: 344 passed, 77 failed against 342 passed, 79 failed for the same files at 8c2b8f5. Same errors in 31 of 33 files; only keytray-modifiers went green and sessions-canvas went from 3 to 2 red tests (logs ~/j406-ev/tri1/, tri2/ on dev-02; results copied next to the 8c2b8f5 ones).
Earlier runs on other commits and a 2-shard run agree: the 10 files that were red only in my 2-shard run (access-level, clusters-keyboard-split, core-screen, font-metrics, font-picker, forget-retired-host, machine-sections, palette-navigation, phone-window-width, settings-rows-of-windows) were rerun without shards on the same head and failed the same way (21 failed tests, ~/j406-ev/rerun10.log), so they are dev drift, not sharding artefacts. (forget-retired-host was not red in the 8c2b8f5 run; it is not one of the 34.)

#	Group	Specs (tests red)	Cause	Evidence	Fix owner
1	Stale specs after D336 stacked toasts	ca-unlock (7), inbox (1), machine-pill-flicker (1), mesh-rename (1), mesh (1), startup-demo (6), sessions-canvas (3, now 2)	A second toast is on screen, so `locator('.ui-toast')` hits "strict mode violation: resolved to 2 elements"	error text in `results.json` of both runs, same on current dev	J386, #434, branch `fix-compute01-browser-baseline` (head `a9a68b702`, not merged): it edits exactly these specs. Merge it, then re-run
2	Stale specs: confirm tip name / setup dialog / backdrop	addhost-pill-stream (8), addhost-pill (1), core-screen (1)	The confirm tip is now named by its message (#391); `getByRole('dialog', { name: 'Setup of o2' })` not found; the backdrop `--mesh-image` is a gradient, not `blob:`	errors above	J386 branch (same one; `core-screen.spec.ts` and `src/core-screen.ts` are in it)
3	Fixed in J386's branch (product or baseline change)	access-level (1), canvas (2), wheel-scroll (4), recovery-ui-hold (4)	access-level: the consent card leaves no room under its buttons, so the tooltip opens above and covers the field (real placement bug); canvas: palette height assertions; wheel-scroll: Settings/docs ids changed; recovery-ui-hold: 0-pixel sign-in baselines differ by CPU architecture (this host is arm64)	J386's comments on #434; branch commit `a9a68b702` ("leave room under the consent buttons") and the per-architecture PNGs	J386 branch. Their own focused run still had `canvas.spec:871` (palette `::before` inset) red
4	Perf and frame-budget specs on a shared host	clusters-bench (1, 240 s timeout), clusters-keyboard-split (1: 29 to 38 dropped frames, limit 4), motion-audit (1: "0.18s" vs "0.2s" at 4x CPU)	Speed measurements, not functional checks; they belong on the perf VM (the owner's rule)	`clusters-bench` took 4 min and failed in both full runs; the other two fail identically at `8c2b8f5` and now	J386 tags the last two `@perf` with `grepInvert` in `playwright.config.ts`; my branch `nightly-suite-shards` (head `a2b674871`, #443) tags `clusters-bench` `@perf` and passes `--grep-invert=@perf`. No product work
5	Account appearance sync (#400, D352/D366)	font-picker (1), machine-sections (2), settings-rows-of-windows (1), terminal-theme (1, likely)	Specs still read `localStorage` (`tracer.extended-info`, the saved font) that the synced settings no longer write that way, and expect the Themes window's section list without the new "Interface font" section (commit `469165a10` updated one spec, not this one). `terminal-theme` (selection background after choosing a theme) looks like the same family but I did not trace it	font-picker: `saved` is `null`; machine-sections: key `tracer.extended-info` is `null` after the click; settings-rows: extra row "Interface font" in the expected list	The #400 job (commits `7fbd176cc`, `22a981116`, `469165a10` are on dev). No open branch covers these: propose a new job "update the specs that read localStorage after #400, and check that sync writes it in the mock". `latency-targets-120hz` touches `terminal-theme.spec.ts`; its owner should say whether that fixes it
6	Swipe stops and tap-to-step	clusters-swipes (2), clusters-swipes-both (2), clusters (1)	The settings stop count is 5, not 9 (`stops` 5 vs >= 8); after swipes `canvasLock.index` is 0 where 6 is expected; tapping a peeking neighbour does not move	errors above, identical now	`swipe-content-first-d374` (D374) touches the swipe behaviour but not these specs: its owner to say whether these specs are stale or the behaviour regressed
7	Font loading	font-metrics (10 tests: Iosevka, Recursive Mono, Hack, Inconsolata, Lilex, each at DPR 1 and 2)	No face of those five fonts reports `loaded` (`mapleLoaded` is false); the other 20 fonts pass. Fonts added in D329 are "fetched only when chosen"	files are present and non-empty in `apps/tracer/fonts/`; same at `8c2b8f5` and now	Needs an owner: the D329 fonts job. I did not find the cause
8	Layout and design changes with no owner yet	phone-window-width (2: a lone window is 349 px wide on a 393 px phone), phone-polish (2: the host name pill is cut), host-removal-confirm (6: text behind the card is legible, correlation 1.0, but the spec wants < 0.15), hosts-new-window-review (1: no "Open shell on review-offline-host" option), mesh-view-overlap (1: "containers" overlaps "mesh" at 600x900), palette-navigation (1: "Palette has no selected option"), clusters-close-centre (1: close X off by 0.75 px, limit 0.5)	host-removal-confirm is probably D380 ("confirmation cards are glass", merged 2026-11-09): the card is now see-through by design, so the spec is stale. The others look like real layout or behaviour changes or tolerance limits; I did not bisect them	errors above	host-removal-confirm: the D380 job. The rest: propose one job that bisects each to its commit and decides stale spec vs regression (7 specs, small)
9	Flakes	keytray-modifiers (1, green now), perf-hud (1: the injected stall was recorded 0 times at `8c2b8f5`, and the next error is about script attribution now)	Timing-dependent; `perf-hud` records an injected 200 ms stall and depends on the host's load	keytray-modifiers passed in the rerun; perf-hud fails differently on the two runs	`perf-hud`: J386 judged it not a frame budget and did not tag it; it needs a look by whoever owns the HUD. `keytray-modifiers`: leave it, watch it

All 34 files are in exactly one row (7+3+4+3+4+3+1+7+2).

Counts. Covered by an open branch (rows 1, 2, 3 and the @perf part of 4): 17 files. Need a new job or an owner's answer: rows 5 to 9, 17 files. I changed no product code and did not close this issue.

What this means for the nightly. About half of this roll-up goes away when #434 merges. The reporter already files per spec once the cap allows; the open-issue budget is 30, so no new issues are filed for these until the roll-up is resolved. When #434 lands, the next nights show what is left.

## Triage of the 34 red spec files (run `20261109T234323Z-8c2b8f5`) — J406 **Verdict: this is not a dead or slow test server. It is 34 files of mostly spec/product drift on dev, deterministic, and most of it is already in an open branch.** **Evidence base.** - **The 8c2b8f5 run itself** (copied by the Maintainer to `/home/claude/j406-ev/r8c2b8f5/` on dev-02, copy of `suite.log` and `results.json` on the control host in `~/.cache/codex-scratch/j406/evidence/tri/`): single worker, 4.24 h, 1,240 passed, 80 failed, 195 skipped. In `suite.log` there are **0** `ECONNRESET`/`ECONNREFUSED`/`socket hang up` lines and **0** 90 s timeouts, and every server-dependent spec outside these 34 passed. So no server death, no slow server. - **Rerun on current dev (3c0b20252)** of 33 of the 34 files (all except `clusters-bench`, a 4-minute frame benchmark), two parallel Playwright runs on dev-02, ports 5741 and 5751, one build slot each, nothing else changed: **344 passed, 77 failed** against 342 passed, 79 failed for the same files at 8c2b8f5. Same errors in 31 of 33 files; only `keytray-modifiers` went green and `sessions-canvas` went from 3 to 2 red tests (logs `~/j406-ev/tri1/`, `tri2/` on dev-02; results copied next to the 8c2b8f5 ones). - Earlier runs on other commits and a 2-shard run agree: the 10 files that were red only in my 2-shard run (`access-level`, `clusters-keyboard-split`, `core-screen`, `font-metrics`, `font-picker`, `forget-retired-host`, `machine-sections`, `palette-navigation`, `phone-window-width`, `settings-rows-of-windows`) were rerun without shards on the same head and failed the same way (21 failed tests, `~/j406-ev/rerun10.log`), so they are dev drift, not sharding artefacts. (`forget-retired-host` was not red in the 8c2b8f5 run; it is not one of the 34.) | # | Group | Specs (tests red) | Cause | Evidence | Fix owner | |--:|---|---|---|---|---| | 1 | **Stale specs after D336 stacked toasts** | ca-unlock (7), inbox (1), machine-pill-flicker (1), mesh-rename (1), mesh (1), startup-demo (6), sessions-canvas (3, now 2) | A second toast is on screen, so `locator('.ui-toast')` hits "strict mode violation: resolved to 2 elements" | error text in `results.json` of both runs, same on current dev | **J386, #434**, branch `fix-compute01-browser-baseline` (head a9a68b702, not merged): it edits exactly these specs. Merge it, then re-run | | 2 | **Stale specs: confirm tip name / setup dialog / backdrop** | addhost-pill-stream (8), addhost-pill (1), core-screen (1) | The confirm tip is now named by its message (#391); `getByRole('dialog', { name: 'Setup of o2' })` not found; the backdrop `--mesh-image` is a gradient, not `blob:` | errors above | J386 branch (same one; `core-screen.spec.ts` and `src/core-screen.ts` are in it) | | 3 | **Fixed in J386's branch (product or baseline change)** | access-level (1), canvas (2), wheel-scroll (4), recovery-ui-hold (4) | access-level: the consent card leaves no room under its buttons, so the tooltip opens above and covers the field (real placement bug); canvas: palette height assertions; wheel-scroll: Settings/docs ids changed; recovery-ui-hold: 0-pixel sign-in baselines differ by CPU architecture (this host is arm64) | J386's comments on #434; branch commit a9a68b702 ("leave room under the consent buttons") and the per-architecture PNGs | J386 branch. Their own focused run still had `canvas.spec:871` (palette `::before` inset) red | | 4 | **Perf and frame-budget specs on a shared host** | clusters-bench (1, 240 s timeout), clusters-keyboard-split (1: 29 to 38 dropped frames, limit 4), motion-audit (1: "0.18s" vs "0.2s" at 4x CPU) | Speed measurements, not functional checks; they belong on the perf VM (the owner's rule) | `clusters-bench` took 4 min and failed in both full runs; the other two fail identically at 8c2b8f5 and now | J386 tags the last two `@perf` with `grepInvert` in `playwright.config.ts`; my branch `nightly-suite-shards` (head a2b674871, #443) tags `clusters-bench` `@perf` and passes `--grep-invert=@perf`. No product work | | 5 | **Account appearance sync (#400, D352/D366)** | font-picker (1), machine-sections (2), settings-rows-of-windows (1), terminal-theme (1, likely) | Specs still read `localStorage` (`tracer.extended-info`, the saved font) that the synced settings no longer write that way, and expect the Themes window's section list without the new "Interface font" section (commit 469165a10 updated one spec, not this one). `terminal-theme` (selection background after choosing a theme) looks like the same family but I did not trace it | font-picker: `saved` is `null`; machine-sections: key `tracer.extended-info` is `null` after the click; settings-rows: extra row "Interface font" in the expected list | The #400 job (commits 7fbd176cc, 22a981116, 469165a10 are on dev). **No open branch covers these: propose a new job "update the specs that read localStorage after #400, and check that sync writes it in the mock"**. `latency-targets-120hz` touches `terminal-theme.spec.ts`; its owner should say whether that fixes it | | 6 | **Swipe stops and tap-to-step** | clusters-swipes (2), clusters-swipes-both (2), clusters (1) | The settings stop count is 5, not 9 (`stops` 5 vs >= 8); after swipes `canvasLock.index` is 0 where 6 is expected; tapping a peeking neighbour does not move | errors above, identical now | `swipe-content-first-d374` (D374) touches the swipe behaviour but not these specs: its owner to say whether these specs are stale or the behaviour regressed | | 7 | **Font loading** | font-metrics (10 tests: Iosevka, Recursive Mono, Hack, Inconsolata, Lilex, each at DPR 1 and 2) | No face of those five fonts reports `loaded` (`mapleLoaded` is false); the other 20 fonts pass. Fonts added in D329 are "fetched only when chosen" | files are present and non-empty in `apps/tracer/fonts/`; same at 8c2b8f5 and now | **Needs an owner**: the D329 fonts job. I did not find the cause | | 8 | **Layout and design changes with no owner yet** | phone-window-width (2: a lone window is 349 px wide on a 393 px phone), phone-polish (2: the host name pill is cut), host-removal-confirm (6: text behind the card is legible, correlation 1.0, but the spec wants < 0.15), hosts-new-window-review (1: no "Open shell on review-offline-host" option), mesh-view-overlap (1: "containers" overlaps "mesh" at 600x900), palette-navigation (1: "Palette has no selected option"), clusters-close-centre (1: close X off by 0.75 px, limit 0.5) | host-removal-confirm is probably D380 ("confirmation cards are glass", merged 2026-11-09): the card is now see-through by design, so the spec is stale. The others look like real layout or behaviour changes or tolerance limits; I did not bisect them | errors above | host-removal-confirm: the D380 job. The rest: **propose one job that bisects each to its commit and decides stale spec vs regression** (7 specs, small) | | 9 | **Flakes** | keytray-modifiers (1, green now), perf-hud (1: the injected stall was recorded 0 times at 8c2b8f5, and the next error is about script attribution now) | Timing-dependent; `perf-hud` records an injected 200 ms stall and depends on the host's load | keytray-modifiers passed in the rerun; perf-hud fails differently on the two runs | `perf-hud`: J386 judged it not a frame budget and did not tag it; it needs a look by whoever owns the HUD. `keytray-modifiers`: leave it, watch it | All 34 files are in exactly one row (7+3+4+3+4+3+1+7+2). **Counts.** Covered by an open branch (rows 1, 2, 3 and the `@perf` part of 4): 17 files. Need a new job or an owner's answer: rows 5 to 9, 17 files. I changed no product code and did not close this issue. **What this means for the nightly.** About half of this roll-up goes away when #434 merges. The reporter already files per spec once the cap allows; the open-issue budget is 30, so no new issues are filed for these until the roll-up is resolved. When #434 lands, the next nights show what is left.

No labels

launch-qa

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal#482

No description provided.

Rows
Columns

browser suite: 34 spec files fail on dev (roll-up) #482

Failing spec files (names as Playwright reported them)

Triage of the 34 red spec files (run 20261109T234323Z-8c2b8f5) — J406

Triage of the 34 red spec files (run `20261109T234323Z-8c2b8f5`) — J406