kayg/calternal-v0

Archived

Watch

Notes editor: randomized editing fuzz with invariant checks #1335

Open

opened 2026-10-07 06:50:11 +00:00 by kayg · 6 comments

kayg commented

2026-10-07 06:50:11 +00:00

Owner

Owner request 2026-10-07: find Notes block editor breakages by automated randomized editing in a real browser (sequences of real keyboard, mouse, paste, drag, undo/redo actions) with invariant checks after each step, then shrink each failure to a minimal repro. Discovery only: one Forgejo issue per distinct bug with the minimal action sequence. No product code changes in this issue's job; fixes are assigned after triage.

Owner request 2026-10-07: find Notes block editor breakages by automated randomized editing in a real browser (sequences of real keyboard, mouse, paste, drag, undo/redo actions) with invariant checks after each step, then shrink each failure to a minimal repro. Discovery only: one Forgejo issue per distinct bug with the minimal action sequence. No product code changes in this issue's job; fixes are assigned after triage.

kayg commented

2026-10-07 06:51:55 +00:00

Author

Owner

Starting #1335 on branch job/editfuzz-1335, based at 3601719b80a39f712daf07b37879b22929dfffa0 (origin/dev). I have read CLAUDE.md, CONTEXT.md, docs/DESIGN.md, and the issue. I will build an opt-in real-browser fuzz harness that reuses existing e2e setup, then run it against the production web build with Chromium and WebKit. Discovery only: no product code changes.

Starting #1335 on branch `job/editfuzz-1335`, based at `3601719b80a39f712daf07b37879b22929dfffa0` (origin/dev). I have read CLAUDE.md, CONTEXT.md, docs/DESIGN.md, and the issue. I will build an opt-in real-browser fuzz harness that reuses existing e2e setup, then run it against the production web build with Chromium and WebKit. Discovery only: no product code changes.

kayg referenced this issue from a commit

2026-10-07 08:32:42 +00:00

test(web): add Notes editor randomized browser fuzzing (#1335)

kayg referenced this issue from a commit

2026-10-07 08:32:42 +00:00

test(web): stabilize Notes fuzz action targets (#1335)

kayg referenced this issue from a commit

2026-10-07 08:32:42 +00:00

test(web): wait for editor selection before fuzz typing (#1335)

kayg referenced this issue from a commit

2026-10-07 08:58:43 +00:00

test(web): stabilize live Notes fuzz targets (#1335)

kayg referenced this issue from a commit

2026-10-07 08:58:43 +00:00

test(web): settle mark selection before shortcut fuzzing (#1335)

kayg referenced this issue from a commit

2026-10-07 08:58:43 +00:00

test(web): keep Notes fuzz seeds running after route startup errors (#1335)

kayg commented

2026-10-07 09:26:30 +00:00

Author

Owner

Confirmed a deterministic Notes editor save-state mismatch and filed it as #1344: pasted multiline text in a table header creates body cells in the visible model, while the saved Markdown reparses those cells as headers. It failed invariant 4 with the same signature on 3/3 replays; delta debugging reduced it to one paste action.

The screenshot is attached to #1344. Local evidence: artifacts/editfuzz/chromium-seed-15-sequence.png and artifacts/editfuzz/chromium-seed-15-minimal.json.

Confirmed a deterministic Notes editor save-state mismatch and filed it as #1344: pasted multiline text in a table header creates body cells in the visible model, while the saved Markdown reparses those cells as headers. It failed invariant 4 with the same signature on 3/3 replays; delta debugging reduced it to one paste action. The screenshot is attached to #1344. Local evidence: `artifacts/editfuzz/chromium-seed-15-sequence.png` and `artifacts/editfuzz/chromium-seed-15-minimal.json`.

kayg commented

2026-10-07 11:27:46 +00:00

Author

Owner

Finished — editfuzz-1335

Built and committed the opt-in Notes editor fuzz harness. It uses real Playwright input, real API-created Notes, macOS platform emulation, Chromium and WebKit, invariant checks after each action, JSONL action logs, screenshots and delta shrinking. It is not part of the default bun run test gate.

Findings

Issue	Symptom	Severity	Engine	Invariant
#1344	Pasted Markdown changes table header cell roles after save and reload	Major	Chromium	4: saved state differs from visible state

The confirmed repro, exact action sequence and screenshot are in artifacts/editfuzz/issue-1344-minimal.json and artifacts/editfuzz/issue-1344.png. The screenshot is attached to #1344. No other issue met the 3/3 repeat requirement. The final Chromium smoke had a link-removal candidate in 2/3 replays; it was not filed because it was not deterministic.

Runs and latency

Engine	Broad exploratory run before harness correction	Final post-fix smoke
Chromium	Seeds 1–8 attempted; 136 action attempts recorded (135 completed, one in progress when stopped); no full 50-action sequence completed	Seed 4 stopped after 28 actions at the nondeterministic link-removal candidate; p50 286.2 ms, p95 559.6 ms across 27 action samples
WebKit	The broad run did not reach this engine	Seed 4 completed 50 random actions, 12 undo/redo actions, and save/reload; p50 1,380 ms, p95 2,177 ms across 60 action samples

The post-fix run was local under shared host load. All measured action samples exceeded 100 ms. These timings are not a performance benchmark. The requested 200 seeds per engine were not reached within the time box.

Rerun

From the repository root, with a release server and production web build already present:

TMPDIR="$PWD/target/tmp" CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/release/calternal-server" bun apps/web/tests/editfuzz-1335.mjs --engine all --seeds 200 --start-seed 1 --min-actions 50 --max-actions 200

The harness writes action logs, summaries, screenshots and minimized repros to artifacts/editfuzz/.

Files and commits

apps/web/tests/editfuzz-1335.mjs
artifacts/editfuzz/findings.md and the ignored repro artifacts listed above
Head: a44cde1e2 (Document Notes fuzz harness invariants)

Gate output

cd apps/web && bun run check exited 0. The perf linter emitted its existing long DEBT inventory; these are the concise gate result lines, verbatim:

NOTE Bun skipped 1 lock package(s) not installed in this production environment
PASS production dependency licences: 759 locked package releases across apps/web, apps/docs and packages
perf-lint: parse Rust product sources
perf-lint: parse browser product sources
perf-lint: validate coverage and architecture
perf-lint: PASS; 0 violations; 22548 scoped exceptions
User browser caches use userStorage; only documented device/public-link exceptions remain.
Glass alpha, blur and backdrop-filter roles use packages/ui/src/tokens.css.
Text sizes and UI shape values use shared role tokens.
Keyboard focus rings use the shared focus tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/editfuzz-1335/apps/web
Getting Svelte diagnostics...

/home/kayg/Developer/calternal-wt/editfuzz-1335/apps/web/../../packages/ui/src/components/calendar/AttachmentDeck.svelte:1055:3
Warn: Do not use empty rulesets (css)
  }
  .attachment-row:focus {

====================================
svelte-check found 0 errors and 1 warning in 1 file

Gaps and decisions

UX gaps closed: none; this was a discovery-only job and no product UI code changed.
UX gaps left: the #1344 save/reload mismatch remains for a product fix. The 200-seed target remains outstanding.
Decisions beyond DESIGN: hidden, inactive App Tab editor DOM does not count as rendered content; caret expectations use the visible editor containing the real selection and a fresh schema snapshot after pointer placement.
For the merge round: run cd apps/web && bun run test against the combined branch. No Rust code changed.

## Finished — editfuzz-1335 Built and committed the opt-in Notes editor fuzz harness. It uses real Playwright input, real API-created Notes, macOS platform emulation, Chromium and WebKit, invariant checks after each action, JSONL action logs, screenshots and delta shrinking. It is not part of the default `bun run test` gate. ### Findings | Issue | Symptom | Severity | Engine | Invariant | | --- | --- | --- | --- | --- | | [#1344](https://git.kayg.org/kayg/calternal/issues/1344) | Pasted Markdown changes table header cell roles after save and reload | Major | Chromium | 4: saved state differs from visible state | The confirmed repro, exact action sequence and screenshot are in `artifacts/editfuzz/issue-1344-minimal.json` and `artifacts/editfuzz/issue-1344.png`. The screenshot is attached to #1344. No other issue met the 3/3 repeat requirement. The final Chromium smoke had a link-removal candidate in 2/3 replays; it was not filed because it was not deterministic. ### Runs and latency | Engine | Broad exploratory run before harness correction | Final post-fix smoke | | --- | --- | --- | | Chromium | Seeds 1–8 attempted; 136 action attempts recorded (135 completed, one in progress when stopped); no full 50-action sequence completed | Seed 4 stopped after 28 actions at the nondeterministic link-removal candidate; p50 286.2 ms, p95 559.6 ms across 27 action samples | | WebKit | The broad run did not reach this engine | Seed 4 completed 50 random actions, 12 undo/redo actions, and save/reload; p50 1,380 ms, p95 2,177 ms across 60 action samples | The post-fix run was local under shared host load. All measured action samples exceeded 100 ms. These timings are not a performance benchmark. The requested 200 seeds per engine were not reached within the time box. ### Rerun From the repository root, with a release server and production web build already present: ```sh TMPDIR="$PWD/target/tmp" CALTERNAL_SERVER_BIN="$CARGO_TARGET_DIR/release/calternal-server" bun apps/web/tests/editfuzz-1335.mjs --engine all --seeds 200 --start-seed 1 --min-actions 50 --max-actions 200 ``` The harness writes action logs, summaries, screenshots and minimized repros to `artifacts/editfuzz/`. ### Files and commits - `apps/web/tests/editfuzz-1335.mjs` - `artifacts/editfuzz/findings.md` and the ignored repro artifacts listed above - Head: `a44cde1e2` (`Document Notes fuzz harness invariants`) ### Gate output `cd apps/web && bun run check` exited 0. The perf linter emitted its existing long `DEBT` inventory; these are the concise gate result lines, verbatim: ```text NOTE Bun skipped 1 lock package(s) not installed in this production environment PASS production dependency licences: 759 locked package releases across apps/web, apps/docs and packages perf-lint: parse Rust product sources perf-lint: parse browser product sources perf-lint: validate coverage and architecture perf-lint: PASS; 0 violations; 22548 scoped exceptions User browser caches use userStorage; only documented device/public-link exceptions remain. Glass alpha, blur and backdrop-filter roles use packages/ui/src/tokens.css. Text sizes and UI shape values use shared role tokens. Keyboard focus rings use the shared focus tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/editfuzz-1335/apps/web Getting Svelte diagnostics... /home/kayg/Developer/calternal-wt/editfuzz-1335/apps/web/../../packages/ui/src/components/calendar/AttachmentDeck.svelte:1055:3 Warn: Do not use empty rulesets (css) } .attachment-row:focus { ==================================== svelte-check found 0 errors and 1 warning in 1 file ``` ### Gaps and decisions - UX gaps closed: none; this was a discovery-only job and no product UI code changed. - UX gaps left: the #1344 save/reload mismatch remains for a product fix. The 200-seed target remains outstanding. - Decisions beyond DESIGN: hidden, inactive App Tab editor DOM does not count as rendered content; caret expectations use the visible editor containing the real selection and a fresh schema snapshot after pointer placement. - For the merge round: run `cd apps/web && bun run test` against the combined branch. No Rust code changed.

kayg commented

2026-10-07 11:29:25 +00:00

Author

Owner

Started discovery on branch job/editfuzz-1335 after merging origin/dev.
Starting job head: a44cde1e22e52a8ef53e7741051025f8adc6824a.
Merged base: ; working head: .
Round 2 will run the existing real-browser edit fuzz harness. Discovery only; no product code changes.

Started discovery on branch `job/editfuzz-1335` after merging `origin/dev`. Starting job head: `a44cde1e22e52a8ef53e7741051025f8adc6824a`. Merged base: ; working head: . Round 2 will run the existing real-browser edit fuzz harness. Discovery only; no product code changes.

kayg commented

2026-10-07 11:29:39 +00:00

Author

Owner

Start metadata for round 2 (corrected):
Branch: job/editfuzz-1335.
Starting job head: a44cde1e22e52a8ef53e7741051025f8adc6824a.
Fetched and merged base: 0419170fe456b58614befcb12d9c9d255118f1c0.
Working head after merge: e40046713fc58d28a082deb51b21dc2f81652a50.
Discovery only; no product code changes.

Start metadata for round 2 (corrected): Branch: `job/editfuzz-1335`. Starting job head: `a44cde1e22e52a8ef53e7741051025f8adc6824a`. Fetched and merged base: `0419170fe456b58614befcb12d9c9d255118f1c0`. Working head after merge: `e40046713fc58d28a082deb51b21dc2f81652a50`. Discovery only; no product code changes.

kayg commented

2026-10-07 14:24:37 +00:00

Author

Owner

editfuzz2 final report

Branch: job/editfuzz-1335
Head: 3027b98afc56bc92385ed9bc55c1344b4b7a8037

Work

Updated apps/web/tests/editfuzz-1335.mjs and committed four small harness changes: parallel isolated Users, link timing/replay actions, action-bound handling, and opt-in --fast-discovery to save raw candidates without expensive automatic replay/shrinking.
No product code changed.
Release server build and production web build succeeded. Build output has been cleaned.
The final bun run check passed. Exact output excerpts:

perf-lint: PASS; 0 violations; 22544 scoped exceptions
User browser caches use userStorage; only documented device/public-link exceptions remain.
Glass alpha, blur and backdrop-filter roles use packages/ui/src/tokens.css.
Text sizes and UI shape values use shared role tokens.
Keyboard focus rings use the shared focus tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
svelte-check found 0 errors and 1 warning in 1 file

The one warning is the existing empty CSS rule at AttachmentDeck.svelte:1055:3.

The web test suite also passed:

Test Files  287 passed (287)
     Tests  2036 passed (2036)
  Duration  452.35s

Campaign

The requested campaign attempted 60 sequences per engine with three isolated Users/contexts. It did not complete the target workload: WebKit workers stalled on link-related actions, so I interrupted the run. No sequence completed its save, reload and history checks, and the campaign did not write its final actions-per-second summary.

Engine	Attempts	Seeds that reached actions	Recorded actions	Completed sequences	Actions/sec
Chromium	60	18	197 action-prefix records	0	unavailable; campaign incomplete
WebKit	60	31	372 action-start records	0	unavailable; campaign incomplete

The remaining 42 Chromium seeds were setup failures after the browser pages had closed. WebKit's remaining 29 seeds were setup failures after the interrupted run. The WebKit contexts were stuck at link-remove, link-remove-toolbar, and type-at-link-edge; these cancellation and setup failures are not filed as product bugs. A WebKit drag screenshot was blank. Local preview API access-control/404 diagnostics and a highlight color:null fingerprint mismatch were also not treated as product findings.

Link-removal investigation

The candidate did not meet the required 3/3 reproduction bar, so no new Forgejo issue was filed. The immediate and save-settled removal variants passed. The variant with a fixed 600 ms delay failed once and passed twice (1/3). The unmodified earlier sequence failed nondeterministically at typed text before reaching the final removal. The saved candidates and screenshots remain under ignored artifacts/editfuzz/ in the worktree.

Rerun

Rebuild the release server and production web app, then run:

CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 TMPDIR="$PWD/target/tmp" CARGO_TARGET_DIR=/mnt/hdd/targets/jobs/editfuzz-1335 cargo build --release -p calternal-server
(cd apps/web && bun run build)
CALTERNAL_SERVER_BIN=/mnt/hdd/targets/jobs/editfuzz-1335/release/calternal-server TMPDIR="$PWD/target/tmp" bun apps/web/tests/editfuzz-1335.mjs --engine all --seeds 60 --workers 3 --start-seed 133600 --min-actions 40 --max-actions 80 --fast-discovery

Decisions and gaps

The design docs do not define a fast discovery mode. I made --fast-discovery opt-in; it preserves the failing action prefix and skips automatic replay/shrinking so a throughput run can continue. Focused replays still use the default confirmation behavior.
The campaign was incomplete. We do not have representative per-engine throughput numbers or completed history/save-reload coverage for this round. No confirmed editor bug was filed.
Gates: bun run check and bun run test passed; no Rust source changed, so no Rust crate gates were run.

# editfuzz2 final report Branch: `job/editfuzz-1335` Head: `3027b98afc56bc92385ed9bc55c1344b4b7a8037` ## Work - Updated `apps/web/tests/editfuzz-1335.mjs` and committed four small harness changes: parallel isolated Users, link timing/replay actions, action-bound handling, and opt-in `--fast-discovery` to save raw candidates without expensive automatic replay/shrinking. - No product code changed. - Release server build and production web build succeeded. Build output has been cleaned. - The final `bun run check` passed. Exact output excerpts: ```text perf-lint: PASS; 0 violations; 22544 scoped exceptions User browser caches use userStorage; only documented device/public-link exceptions remain. Glass alpha, blur and backdrop-filter roles use packages/ui/src/tokens.css. Text sizes and UI shape values use shared role tokens. Keyboard focus rings use the shared focus tokens. UI transitions and animation options use shared motion tokens or documented exceptions. svelte-check found 0 errors and 1 warning in 1 file ``` The one warning is the existing empty CSS rule at `AttachmentDeck.svelte:1055:3`. The web test suite also passed: ```text Test Files 287 passed (287) Tests 2036 passed (2036) Duration 452.35s ``` ## Campaign The requested campaign attempted 60 sequences per engine with three isolated Users/contexts. It did not complete the target workload: WebKit workers stalled on link-related actions, so I interrupted the run. No sequence completed its save, reload and history checks, and the campaign did not write its final actions-per-second summary. | Engine | Attempts | Seeds that reached actions | Recorded actions | Completed sequences | Actions/sec | | --- | ---: | ---: | ---: | ---: | --- | | Chromium | 60 | 18 | 197 action-prefix records | 0 | unavailable; campaign incomplete | | WebKit | 60 | 31 | 372 action-start records | 0 | unavailable; campaign incomplete | The remaining 42 Chromium seeds were setup failures after the browser pages had closed. WebKit's remaining 29 seeds were setup failures after the interrupted run. The WebKit contexts were stuck at `link-remove`, `link-remove-toolbar`, and `type-at-link-edge`; these cancellation and setup failures are not filed as product bugs. A WebKit drag screenshot was blank. Local preview API access-control/404 diagnostics and a highlight `color:null` fingerprint mismatch were also not treated as product findings. ## Link-removal investigation The candidate did not meet the required 3/3 reproduction bar, so no new Forgejo issue was filed. The immediate and save-settled removal variants passed. The variant with a fixed 600 ms delay failed once and passed twice (1/3). The unmodified earlier sequence failed nondeterministically at typed text before reaching the final removal. The saved candidates and screenshots remain under ignored `artifacts/editfuzz/` in the worktree. ## Rerun Rebuild the release server and production web app, then run: ```sh CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 TMPDIR="$PWD/target/tmp" CARGO_TARGET_DIR=/mnt/hdd/targets/jobs/editfuzz-1335 cargo build --release -p calternal-server (cd apps/web && bun run build) CALTERNAL_SERVER_BIN=/mnt/hdd/targets/jobs/editfuzz-1335/release/calternal-server TMPDIR="$PWD/target/tmp" bun apps/web/tests/editfuzz-1335.mjs --engine all --seeds 60 --workers 3 --start-seed 133600 --min-actions 40 --max-actions 80 --fast-discovery ``` ## Decisions and gaps - The design docs do not define a fast discovery mode. I made `--fast-discovery` opt-in; it preserves the failing action prefix and skips automatic replay/shrinking so a throughput run can continue. Focused replays still use the default confirmation behavior. - The campaign was incomplete. We do not have representative per-engine throughput numbers or completed history/save-reload coverage for this round. No confirmed editor bug was filed. - Gates: `bun run check` and `bun run test` passed; no Rust source changed, so no Rust crate gates were run.

kayg referenced this issue

2026-10-07 14:26:25 +00:00

Notes editor: WebKit stalls on link actions during fuzzing (hang or harness?) #1371

Commenting is not possible because the repository is archived.

No labels

No milestone

No project

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal-v0#1335

No description provided.