TEST: visible-effect check of every Settings control on every merge (fail-closed) + weekly AI exploratory pass #593

Open
opened 2026-10-01 08:01:27 +00:00 by kayg · 0 comments
Owner

Owner request (2026-09-30)

"this still does not work. it says 'Adding' and disappears? we need a subagent (later) to test every setting in the UI!" The owner also asked whether the OpenAI Decisions API or a computer-use model would be cheaper for this.

Status when this issue was filed (2026-10-01)

#422 (closed) added settings-effects.mjs, a visible-effect sweep. Its own report says it covers only Appearance and the Calendar date/time formats. It does not cover the other Settings sections, every theme and font option, or Unsplash. The orchestrator proposed two parts on 2026-09-30 and asked to file them; no issue was filed.

Expected behaviour

  1. Scripted sweep on every merge (fail-closed). A Playwright run on a production build opens every Settings section in the §50 layout, changes every control the way a User does, and asserts the visible effect on the real page (pixels, colours, fonts, the background shows, the list changes), not only that a request was sent. A new Settings control without a check fails the gate.
  2. AI exploratory pass (weekly, next to the consistency hunt). A computer-use agent uses the real app like the owner does, including Safari on the macOS VM (cua-driver). Each finding becomes a scripted test in part 1.
  3. Tool choice: the Decisions API classifies content and cannot see or click a screen, so it does not fit part 2. Record the choice and its cost model in the issue (no prices in the repo).

Acceptance test

  • The sweep covers 100% of the Settings controls listed by the Settings registry; the guard test fails when a control has no check.
  • One exploratory run report is attached, with each finding filed as an issue or a scripted test.

Related: #422, #407 and DESIGN §50 (Better Settings), the self-driven visual QA rule.

## Owner request (2026-09-30) "this still does not work. it says 'Adding' and disappears? we need a subagent (later) to test every setting in the UI!" The owner also asked whether the OpenAI Decisions API or a computer-use model would be cheaper for this. ## Status when this issue was filed (2026-10-01) #422 (closed) added `settings-effects.mjs`, a visible-effect sweep. Its own report says it covers only Appearance and the Calendar date/time formats. It does not cover the other Settings sections, every theme and font option, or Unsplash. The orchestrator proposed two parts on 2026-09-30 and asked to file them; no issue was filed. ## Expected behaviour 1. **Scripted sweep on every merge (fail-closed).** A Playwright run on a production build opens every Settings section in the §50 layout, changes every control the way a User does, and asserts the **visible** effect on the real page (pixels, colours, fonts, the background shows, the list changes), not only that a request was sent. A new Settings control without a check fails the gate. 2. **AI exploratory pass (weekly, next to the consistency hunt).** A computer-use agent uses the real app like the owner does, including Safari on the macOS VM (cua-driver). Each finding becomes a scripted test in part 1. 3. Tool choice: the Decisions API classifies content and cannot see or click a screen, so it does not fit part 2. Record the choice and its cost model in the issue (no prices in the repo). ## Acceptance test - The sweep covers 100% of the Settings controls listed by the Settings registry; the guard test fails when a control has no check. - One exploratory run report is attached, with each finding filed as an issue or a scripted test. Related: #422, #407 and DESIGN §50 (Better Settings), the self-driven visual QA rule.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#593
No description provided.