Investigate editor undo, conflict propagation, and restart persistence adversarial failures

kayg commented

2026-09-29 02:47:48 +00:00

Owner

The editor adversarial probes reported three non-SLOW failures during the #373 round:

After 186 undo/redo interactions, the resulting text differed from the expected text by inserted spaces (Anchor heading parent... versus Anchor headingparent...).
A concurrent delete/edit probe timed out waiting for the conflict to reach both peers.
A server-restart probe timed out waiting for the edited content to be persisted exactly once.

These symptoms need reproduction and triage in the editor collaboration and persistence code. The #373 Voice change does not touch that code. The adversarial round was interrupted before a complete run, so this report records only the failures observed before interruption.

The editor adversarial probes reported three non-SLOW failures during the #373 round: - After 186 undo/redo interactions, the resulting text differed from the expected text by inserted spaces (`Anchor heading parent...` versus `Anchor headingparent...`). - A concurrent delete/edit probe timed out waiting for the conflict to reach both peers. - A server-restart probe timed out waiting for the edited content to be persisted exactly once. These symptoms need reproduction and triage in the editor collaboration and persistence code. The #373 Voice change does not touch that code. The adversarial round was interrupted before a complete run, so this report records only the failures observed before interruption.

editor-1440-dark.png

617 KiB

editor-1440-light.png

688 KiB

editor-390-dark.png

230 KiB

editor-390-light.png

233 KiB

editor-820-dark.png

392 KiB

editor-820-light.png

414 KiB

kayg referenced this issue

2026-09-29 02:48:15 +00:00

VOICE: local transcription (Parakeet v2 → S1-mini by Superwhisper), background model download, inline mic morph #373

kayg referenced this issue

2026-09-29 04:20:55 +00:00

VOICE: local transcription (Parakeet v2 → S1-mini by Superwhisper), background model download, inline mic morph #373

kayg commented

2026-09-29 04:36:16 +00:00

Author

Owner

Started investigation on branch job/editor-385 at base 00015e652a. Reproducing the three editor failures before changes.

Started investigation on branch job/editor-385 at base 00015e652a153485d9e7adda1c5ee61bc17300db. Reproducing the three editor failures before changes.

kayg commented

2026-09-29 04:39:33 +00:00

Author

Owner

Finding during reproduction: tests/adversarial/editor.mjs edit-first waits until both peers already contain the edit before deleting the block. That is a serial delete after delivery, so the later assertion requiring a conflict update to echo to both peers can time out when the server correctly emits no conflict. The server also intentionally suppresses a sender's own relay. I am checking the live probe before changing the race fixture.

kayg commented

2026-09-29 05:10:22 +00:00

Author

Owner

The edit-first adversarial fixture waited for both peers to contain the edit before deleting, so it exercised a serial delete and then required a conflict relay that the server does not emit for that order. I changed the fixture to hold the edit relay at the deleting peer until after its stale delete reaches the server. The restart fixture also signaled the runner without verifying that text entered the live editor; a single click can leave the Note in block selection mode. I added an edit-mode transition and an assertion on the live ProseMirror text before signaling restart. A focused Rust regression covers edit reaching the server before a stale peer delete. Validation is in progress.

kayg commented

2026-09-29 05:35:55 +00:00

Author

Owner

Validation update after merging dev once (merge commit 4c610bf1): the focused Rust stale-delete regression passes (1 passed); cargo fmt --check passes; web bun run check reports 0 errors and 0 warnings; web bun run test reports 123 files and 787 tests passed; editor bun run check reports 0 errors and 0 warnings; editor bun run test reports 18 files and 401 tests passed. The production server build for the editor adversarial campaign is still running on the shared host. Full Rust gates and live probe results remain pending.

Validation update after merging dev once (merge commit 4c610bf1): the focused Rust stale-delete regression passes (1 passed); `cargo fmt --check` passes; web `bun run check` reports 0 errors and 0 warnings; web `bun run test` reports 123 files and 787 tests passed; editor `bun run check` reports 0 errors and 0 warnings; editor `bun run test` reports 18 files and 401 tests passed. The production server build for the editor adversarial campaign is still running on the shared host. Full Rust gates and live probe results remain pending.

kayg commented

2026-09-29 05:53:17 +00:00

Author

Owner

Build status: the production web build and all web/editor checks and tests passed. The one time-boxed editor adversarial run is still in its Rust server build on the shared host, with no compiler error so far; it has not reached the browser probes. The branch has no uncommitted changes. I will report the live results and full Rust gate output when available.

kayg commented

2026-09-29 06:08:08 +00:00

Author

Owner

The first 45-minute adversarial command ended with exit 124 during a cold Rust build (the probe had not started; no browser finding). I am building only calternal-server from the cached dependencies, then will run the same editor campaign with the completed binary and already-built production web assets. This is a continuation of the one planned campaign, not a second probe round.

kayg commented

2026-09-29 06:25:00 +00:00

Author

Owner

The cold server build remains in progress on the shared host, now compiling the original feature set after the first campaign timed out before probes. No Rust compiler error has appeared. I also committed 06179f93, adding an exactly-once saved Note assertion to the delete/edit adversarial case; its JavaScript syntax and diff checks pass. The branch is clean. The browser campaign and full Rust gates are still pending.

The cold server build remains in progress on the shared host, now compiling the original feature set after the first campaign timed out before probes. No Rust compiler error has appeared. I also committed `06179f93`, adding an exactly-once saved Note assertion to the delete/edit adversarial case; its JavaScript syntax and diff checks pass. The branch is clean. The browser campaign and full Rust gates are still pending.

kayg commented

2026-09-29 06:49:53 +00:00

Author

Owner

Live editor campaign result: the isolated 500-step browser history storm passed, both delete/edit orders passed, and the live server restart persisted the marker exactly once. The full sweep initially reported one repeatable browser-area deep-link timeout. Diagnostics showed the test's own history typing inserted its markers into the previously copied heading, changing Anchor heading to history-storm-186 history-round-trip-186Anchor heading; the copied slug then correctly had no target. The saved Note contained the same edited heading. I changed the probe to focus its intended paragraph before each history marker and to assert the heading stays named Anchor heading. The focused browser area now passes, including both heading and block deep links (PASS editor browser editor paste, Unicode, and block actions ...). No runtime editor defect was reproduced by these three #385 cases. Six 390/820/1440 light/dark production screenshots are in the ignored artifacts/editor-385/ directory.

Live editor campaign result: the isolated 500-step browser history storm passed, both delete/edit orders passed, and the live server restart persisted the marker exactly once. The full sweep initially reported one repeatable browser-area deep-link timeout. Diagnostics showed the test's own history typing inserted its markers into the previously copied heading, changing `Anchor heading` to `history-storm-186 history-round-trip-186Anchor heading`; the copied slug then correctly had no target. The saved Note contained the same edited heading. I changed the probe to focus its intended paragraph before each history marker and to assert the heading stays named `Anchor heading`. The focused browser area now passes, including both heading and block deep links (`PASS editor browser editor paste, Unicode, and block actions ...`). No runtime editor defect was reproduced by these three #385 cases. Six 390/820/1440 light/dark production screenshots are in the ignored `artifacts/editor-385/` directory.

kayg commented

2026-09-29 06:54:29 +00:00

Author

Owner

Production Note editor screenshots (real server, theme asserted before capture):

Width	Light	Dark
390 px	light	dark
820 px	light	dark
1440 px	light	dark

Production Note editor screenshots (real server, theme asserted before capture): | Width | Light | Dark | | --- | --- | --- | | 390 px | [light](https://git.kayg.org/attachments/2efc57b0-444b-48a9-bd18-992785ab0e4e) | [dark](https://git.kayg.org/attachments/5463b265-0116-409c-91d7-460fd3321c92) | | 820 px | [light](https://git.kayg.org/attachments/cd1476c8-0d7f-4481-955f-ca318e983bf1) | [dark](https://git.kayg.org/attachments/31c79d22-b8d7-40ec-8e24-18528b988537) | | 1440 px | [light](https://git.kayg.org/attachments/4d34049d-90b5-4e49-b258-34535080df1d) | [dark](https://git.kayg.org/attachments/2c241d60-d325-46f3-92a2-c16af58e7f14) |

kayg commented

2026-09-29 07:09:44 +00:00

Author

Owner

At head d34e6266, the full workspace cargo test gate failed in crates/calternal-collab/tests/two_clients.rs at crash_before_debounce_reloads_only_complete_markdown: the child exited 101 instead of the expected 137. The test's child sleeps a fixed 900 ms, asserts that a Markdown write has completed, then calls process::exit(137). A direct rerun of the already-built test binary passed both two_clients tests (2 passed; 0 failed), so the failure is load-sensitive in this run. job/small-bugs-2 does not change collab files or this expectation. The gate remains reported as failed; I did not edit the assertion.

At head `d34e6266`, the full workspace `cargo test` gate failed in `crates/calternal-collab/tests/two_clients.rs` at `crash_before_debounce_reloads_only_complete_markdown`: the child exited 101 instead of the expected 137. The test's child sleeps a fixed 900 ms, asserts that a Markdown write has completed, then calls `process::exit(137)`. A direct rerun of the already-built test binary passed both `two_clients` tests (`2 passed; 0 failed`), so the failure is load-sensitive in this run. `job/small-bugs-2` does not change collab files or this expectation. The gate remains reported as failed; I did not edit the assertion.

kayg commented

2026-09-29 07:11:37 +00:00

Author

Owner

Final-gate progress: cargo fmt --check passed (exit 0, no output). Web bun run check and bun run test passed (0 diagnostics; 123 files / 787 tests). Editor bun run check and bun run test passed (0 diagnostics; 18 files / 401 tests). The fresh workspace clippy cache advanced without warnings but compiled slowly under shared-host load; I interrupted that attempt to run the workspace cargo test gate while there is time left in the four-hour job limit. The full test command is compiling now. I will resume clippy afterward and report exact outcomes. No uncommitted files.

Final-gate progress: `cargo fmt --check` passed (exit 0, no output). Web `bun run check` and `bun run test` passed (0 diagnostics; 123 files / 787 tests). Editor `bun run check` and `bun run test` passed (0 diagnostics; 18 files / 401 tests). The fresh workspace clippy cache advanced without warnings but compiled slowly under shared-host load; I interrupted that attempt to run the workspace `cargo test` gate while there is time left in the four-hour job limit. The full test command is compiling now. I will resume clippy afterward and report exact outcomes. No uncommitted files.

kayg commented

2026-09-29 07:24:29 +00:00

Author

Owner

Workspace Rust gate status: cargo test is still compiling and linking the application test targets. It has reached calternal-collab and calternal-server with no compiler or test error so far. I have not marked the gate passed before its process exits. The prior focused collaboration test passed, and the corrected real-server editor campaign passed the three #385 cases.

Workspace Rust gate status: `cargo test` is still compiling and linking the application test targets. It has reached calternal-collab and calternal-server with no compiler or test error so far. I have not marked the gate passed before its process exits. The prior focused collaboration test passed, and the corrected real-server editor campaign passed the three #385 cases.

kayg commented

2026-09-29 07:38:14 +00:00

Author

Owner

Workspace cargo test reached the collaboration integration tests and failed one timing-sensitive case: crash_before_debounce_reloads_only_complete_markdown expected child exit 137 but got 101. The child sleeps 900 ms for a 750 ms debounce, then asserts the durable write before intentionally exiting 137. On this shared host that 150 ms scheduler margin is not reliable. The focused crate test passed immediately (1 passed), and the exact binary built by the failed workspace run also passed when rerun alone (1 passed, 1.56 s). This is a SLOW-only gate finding under load; no data corruption or restart failure was observed. I am keeping the original assertion and reporting the failed full run verbatim rather than changing its expectation.

Workspace `cargo test` reached the collaboration integration tests and failed one timing-sensitive case: `crash_before_debounce_reloads_only_complete_markdown` expected child exit 137 but got 101. The child sleeps 900 ms for a 750 ms debounce, then asserts the durable write before intentionally exiting 137. On this shared host that 150 ms scheduler margin is not reliable. The focused crate test passed immediately (1 passed), and the exact binary built by the failed workspace run also passed when rerun alone (1 passed, 1.56 s). This is a SLOW-only gate finding under load; no data corruption or restart failure was observed. I am keeping the original assertion and reporting the failed full run verbatim rather than changing its expectation.

kayg referenced this issue

2026-09-29 07:52:15 +00:00

Files: names with ZWJ/ZWNJ are refused (emoji sequences, Persian words), and the upload toast gives no reason #142

kayg commented

2026-09-29 07:55:39 +00:00

Author

Owner

Final-gate status at the shared-host compile stage: workspace cargo test exited 101 only because the crash-before-debounce child exited 101 instead of the test's intentional 137 after a 900 ms wait for a 750 ms debounce. Both the focused crate test and the exact workspace-built test binary passed alone; no content or restart mismatch was seen. Full workspace cargo clippy --all-targets -- -D warnings is still compiling dependencies without diagnostics. All web/editor gates and cargo fmt --check passed. The branch has no uncommitted files and the six screenshots are attached above.

Final-gate status at the shared-host compile stage: workspace `cargo test` exited 101 only because the crash-before-debounce child exited 101 instead of the test's intentional 137 after a 900 ms wait for a 750 ms debounce. Both the focused crate test and the exact workspace-built test binary passed alone; no content or restart mismatch was seen. Full workspace `cargo clippy --all-targets -- -D warnings` is still compiling dependencies without diagnostics. All web/editor gates and `cargo fmt --check` passed. The branch has no uncommitted files and the six screenshots are attached above.

kayg commented

2026-09-29 08:07:07 +00:00

Author

Owner

Clippy progress: the full workspace cargo clippy --all-targets -- -D warnings run is still active, now checking calternal-notes-core after SQLx/rustls dependencies. No diagnostic has appeared. The job remains within its four-hour limit; I will leave enough time for cleanup and a final report if this shared-host gate does not finish.

Clippy progress: the full workspace `cargo clippy --all-targets -- -D warnings` run is still active, now checking `calternal-notes-core` after SQLx/rustls dependencies. No diagnostic has appeared. The job remains within its four-hour limit; I will leave enough time for cleanup and a final report if this shared-host gate does not finish.

kayg commented

2026-09-29 08:19:46 +00:00

Author

Owner

Clippy progress: the resumed full workspace cargo clippy --all-targets -- -D warnings run is still active and has reached calternal-plugin plus OpenSSL dependencies. No warning or error has appeared. I am continuing until it exits or the four-hour job cutoff, then will clean the Cargo target and web build output and post the final report with exact gate outcomes.

Clippy progress: the resumed full workspace `cargo clippy --all-targets -- -D warnings` run is still active and has reached `calternal-plugin` plus OpenSSL dependencies. No warning or error has appeared. I am continuing until it exits or the four-hour job cutoff, then will clean the Cargo target and web build output and post the final report with exact gate outcomes.

kayg commented

2026-09-29 08:37:33 +00:00

Author

Owner

Four-hour cutoff planning: full workspace clippy is still compiling dependencies without diagnostics after roughly an hour of resumed runtime; the shared host load is again above 30 on 8 cores. I will stop the unfinished full run at about 10:45–10:50 local time if necessary, run clippy for the touched calternal-collab crate, then clean the target and web build output and post the final report before the job exceeds four hours. This is a gate capacity limit, not a code warning.

kayg commented

2026-09-29 08:56:36 +00:00

Author

Owner

Final report — job/editor-385, head 6ece2a3935d21caa1ca8da463646de134d27ecc2 (pushed; worktree clean)

Built: deterministic edit-first/stale-delete Rust regression; editor adversarial probe now holds the edit relay so the deleting peer is actually stale, asserts the saved survivor exactly once, confirms restart text entered the live editor before restarting, and refocuses the intended paragraph during undo history so the copied heading link remains valid. No product runtime change was needed: the corrected real-server probes pass all three #385 cases. Merged dev once at 4c610bf1.

Files: crates/calternal-collab/src/session.rs, tests/adversarial/editor.mjs, packages/editor/src/collaborationUndo.test.ts.

Real-server evidence (verbatim):

PASS editor isolated editor paste history storm seed=25608414 ms=22014
PASS editor delete and edit of one block seed=25608414 ms=11855
PASS editor server restart with a live editor seed=25608414 ms=8744
PASS editor browser editor paste, Unicode, and block actions seed=25608414 ms=21800

Six production screenshots at 390/820/1440 px in light and dark are attached to this issue in the table above. Theme was asserted before capture.

Gate output (verbatim excerpts):

cargo fmt --check: exit 0, no output.
bun run --cwd apps/web check:

svelte-check found 0 errors and 0 warnings

bun run --cwd apps/web test:

 Test Files  123 passed (123)
      Tests  787 passed (787)

bun run --cwd packages/editor check:

svelte-check found 0 errors and 0 warnings

bun run --cwd packages/editor test:

 Test Files  18 passed (18)
      Tests  401 passed (401)

Focused Rust regression:

test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 23 filtered out; finished in 0.00s

Full workspace cargo test exited 101 under shared-host load:

test crash_before_debounce_reloads_only_complete_markdown ... FAILED
assertion `left == right` failed: 
  left: Some(101)
 right: Some(137)
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.08s
error: test failed, to rerun pass `-p calternal-collab --test two_clients`

The child test uses a fixed 900 ms wait for a 750 ms debounce. The focused crate test passed, and the exact workspace-built binary also passed alone in 1.56 s. This is classified SLOW-only; no data mismatch was observed. I kept the existing assertion and fixture.

Full workspace cargo clippy --all-targets -- -D warnings hit its 60-minute build limit (exit 124) before completion, with no diagnostic. Its final output was:

    Checking serde_plain v0.1.2
    Checking reqwest v0.13.5

Focused calternal-collab clippy hit a 15-minute build limit (exit 124) before completion, with no diagnostic. Its final output was:

    Checking regex-automata v0.4.18
    Checking cssparser v0.31.2

Known gaps: the full Rust test and clippy gates are not green for the reasons above. Run them again on a quiet host before merge. No non-SLOW adversarial finding remains. cargo clean completed (Removed 14571 files, 15.2GiB total); web build output was deleted. Screenshots remain in ignored artifacts/editor-385/ and are attached to this issue.

Decisions not covered by DESIGN: no product design decision. The probe defers only Yjs update relays, preserving epoch/handshake traffic, and requires exactly-once saved content.

Final report — job/editor-385, head `6ece2a3935d21caa1ca8da463646de134d27ecc2` (pushed; worktree clean) Built: deterministic edit-first/stale-delete Rust regression; editor adversarial probe now holds the edit relay so the deleting peer is actually stale, asserts the saved survivor exactly once, confirms restart text entered the live editor before restarting, and refocuses the intended paragraph during undo history so the copied heading link remains valid. No product runtime change was needed: the corrected real-server probes pass all three #385 cases. Merged `dev` once at `4c610bf1`. Files: `crates/calternal-collab/src/session.rs`, `tests/adversarial/editor.mjs`, `packages/editor/src/collaborationUndo.test.ts`. Real-server evidence (verbatim): ``` PASS editor isolated editor paste history storm seed=25608414 ms=22014 PASS editor delete and edit of one block seed=25608414 ms=11855 PASS editor server restart with a live editor seed=25608414 ms=8744 PASS editor browser editor paste, Unicode, and block actions seed=25608414 ms=21800 ``` Six production screenshots at 390/820/1440 px in light and dark are attached to this issue in the table above. Theme was asserted before capture. Gate output (verbatim excerpts): - `cargo fmt --check`: exit 0, no output. - `bun run --cwd apps/web check`: ``` svelte-check found 0 errors and 0 warnings ``` - `bun run --cwd apps/web test`: ``` Test Files 123 passed (123) Tests 787 passed (787) ``` - `bun run --cwd packages/editor check`: ``` svelte-check found 0 errors and 0 warnings ``` - `bun run --cwd packages/editor test`: ``` Test Files 18 passed (18) Tests 401 passed (401) ``` - Focused Rust regression: ``` test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 23 filtered out; finished in 0.00s ``` - Full workspace `cargo test` exited 101 under shared-host load: ``` test crash_before_debounce_reloads_only_complete_markdown ... FAILED assertion `left == right` failed: left: Some(101) right: Some(137) test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.08s error: test failed, to rerun pass `-p calternal-collab --test two_clients` ``` The child test uses a fixed 900 ms wait for a 750 ms debounce. The focused crate test passed, and the exact workspace-built binary also passed alone in 1.56 s. This is classified SLOW-only; no data mismatch was observed. I kept the existing assertion and fixture. - Full workspace `cargo clippy --all-targets -- -D warnings` hit its 60-minute build limit (exit 124) before completion, with no diagnostic. Its final output was: ``` Checking serde_plain v0.1.2 Checking reqwest v0.13.5 ``` - Focused `calternal-collab` clippy hit a 15-minute build limit (exit 124) before completion, with no diagnostic. Its final output was: ``` Checking regex-automata v0.4.18 Checking cssparser v0.31.2 ``` Known gaps: the full Rust test and clippy gates are not green for the reasons above. Run them again on a quiet host before merge. No non-SLOW adversarial finding remains. `cargo clean` completed (`Removed 14571 files, 15.2GiB total`); web build output was deleted. Screenshots remain in ignored `artifacts/editor-385/` and are attached to this issue. Decisions not covered by DESIGN: no product design decision. The probe defers only Yjs update relays, preserving epoch/handshake traffic, and requires exactly-once saved content.

kayg referenced this issue from a commit

2026-09-29 09:03:36 +00:00

Merge job/editor-385 (#385: editor probes corrected; stale-delete conflict regression test; no runtime change needed)

kayg referenced this issue

2026-09-29 10:38:01 +00:00

Flaky: calternal-collab crash simulation exits 101 instead of 137 under load #426

kayg referenced this issue

2026-09-29 12:49:47 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-29 18:20:51 +00:00

Author