kayg/calternal

Watch

Fork 0

Collab: restart_epoch test flaked once in a full workspace run #84

New issue

Closed

opened 2026-09-25 10:08:24 +00:00 by kayg · 6 comments

kayg commented

2026-09-25 10:08:24 +00:00

Owner

The photos-incremental job (#79) saw one transient failure of a crates/calternal-collab/tests/restart_epoch.rs test during a full cargo test --workspace run on dev (af185c5 + the photos branch). It passed when run alone and on a full rerun.

The test restarts a Hub on the same data while a Yrs client is connected; it guards the Note duplication fix (0a62831), so a flake here can hide a real regression.

Do: run cargo test -p calternal-collab --test restart_epoch in a loop (e.g. 200 runs, and also under cargo test --workspace load). Find the timing assumption (sleep, save debounce, socket close ordering) and replace it with a wait on a real condition. Acceptance: 200/200 passes under load.

The photos-incremental job (#79) saw one transient failure of a crates/calternal-collab/tests/restart_epoch.rs test during a full `cargo test --workspace` run on dev (af185c5 + the photos branch). It passed when run alone and on a full rerun. The test restarts a Hub on the same data while a Yrs client is connected; it guards the Note duplication fix (0a62831), so a flake here can hide a real regression. Do: run `cargo test -p calternal-collab --test restart_epoch` in a loop (e.g. 200 runs, and also under `cargo test --workspace` load). Find the timing assumption (sleep, save debounce, socket close ordering) and replace it with a wait on a real condition. Acceptance: 200/200 passes under load.

kayg commented

2026-09-25 10:11:42 +00:00

Author

Owner

Starting investigation on branch job/collab-flake-2, based on dev at 6d71754cfe. The worktree is clean. I am beginning standalone reproduction runs of the restart_epoch integration test before changing its timing assumptions.

Starting investigation on branch job/collab-flake-2, based on dev at 6d71754cfe94bafa0efa60a66ca6b5ebb903d133. The worktree is clean. I am beginning standalone reproduction runs of the restart_epoch integration test before changing its timing assumptions.

kayg commented

2026-09-25 10:13:40 +00:00

Author

Owner

Baseline reproduction: the first standalone cargo test -p calternal-collab --test restart_epoch failed in graceful_restart_keeps_the_epoch_and_offline_edits at restart_epoch.rs:355. The restarted Hub sent epoch ebc9d796-ac6c-4433-bfcb-8f82c8c26f90, while the test expected the saved epoch 3653f86b-d05b-48a6-9b85-595cb58319cb; the other four tests passed. This test calls Hub::shutdown() before dropping the listener task and the WebSocket. I am inspecting stored-state and close ordering.

Baseline reproduction: the first standalone `cargo test -p calternal-collab --test restart_epoch` failed in `graceful_restart_keeps_the_epoch_and_offline_edits` at restart_epoch.rs:355. The restarted Hub sent epoch `ebc9d796-ac6c-4433-bfcb-8f82c8c26f90`, while the test expected the saved epoch `3653f86b-d05b-48a6-9b85-595cb58319cb`; the other four tests passed. This test calls `Hub::shutdown()` before dropping the listener task and the WebSocket. I am inspecting stored-state and close ordering.

kayg commented

2026-09-25 10:26:04 +00:00

Author

Owner

Additional evidence from the first verification loop: test run 3 failed at the new post-shutdown cache assertion (stored_epoch was None, expected the active epoch). The initial fix marked the room dirty inside the document lock, but schedule_flush() marked it again after flush() cleared the flag; shutdown then correctly skipped storing a dirty room. I moved dirty marking to all three mutation sites (WebSocket update, agent edit, external change), so the debounce scheduler only arms a timer. One full restart_epoch run now passes; I am restarting acceptance loops on this version.

Additional evidence from the first verification loop: test run 3 failed at the new post-shutdown cache assertion (`stored_epoch` was `None`, expected the active epoch). The initial fix marked the room dirty inside the document lock, but `schedule_flush()` marked it again after `flush()` cleared the flag; shutdown then correctly skipped storing a dirty room. I moved dirty marking to all three mutation sites (WebSocket update, agent edit, external change), so the debounce scheduler only arms a timer. One full `restart_epoch` run now passes; I am restarting acceptance loops on this version.

kayg commented

2026-09-25 10:31:08 +00:00

Author

Owner

A further diagnostic reproduced the race in targeted run 10: shutdown reported flushed=true but dirty_since became nonzero before the document snapshot, and the store skipped the room. flush_pending_locked() had cleared the flag before locking the document. I moved the reset into the same awareness-lock section as the snapshot, so an update cannot fall between them. The targeted graceful-restart test then passed 10/10 runs.

A further diagnostic reproduced the race in targeted run 10: shutdown reported `flushed=true` but `dirty_since` became nonzero before the document snapshot, and the store skipped the room. `flush_pending_locked()` had cleared the flag before locking the document. I moved the reset into the same awareness-lock section as the snapshot, so an update cannot fall between them. The targeted graceful-restart test then passed 10/10 runs.

kayg commented

2026-09-25 10:36:01 +00:00

Author

Owner

The final mutation/store ordering fix passed the requested standalone loop: cargo test -p calternal-collab --test restart_epoch — 200/200 passed, 0 failed. I am now running the 100-run loop concurrently with cargo test --workspace as the load check.

The final mutation/store ordering fix passed the requested standalone loop: `cargo test -p calternal-collab --test restart_epoch` — 200/200 passed, 0 failed. I am now running the 100-run loop concurrently with `cargo test --workspace` as the load check.

kayg commented

2026-09-25 11:03:32 +00:00

Author

Owner

Finished on branch job/collab-flake-2, HEAD 3c6c3324dd68d72017706f11e23745529aaee021.

Root cause: the WebSocket handler acknowledged an accepted update before the debounce scheduler marked it dirty. Shutdown could race that mark. Also, flush reset dirty_since before it took the document snapshot, so an update could be included in the snapshot but leave the room dirty and make the stored epoch skip. Dirty marking now happens under the document lock; flush pairs the reset with its snapshot. Shutdown gates new room loads and document edits, then holds the room flush lock through its final flush and cache store.

The restart tests now wait for room unload after the rejected socket leaves instead of sleeping past MAX_FLUSH_WAIT. The graceful restart case drains the listener, checks the stored epoch, and waits for the old socket's last-leave save before reconnecting.

Acceptance output:

Standalone loop: 200/200 passed, 0 failed
Under-load loop: 100/100 passed, 0 failed
Parallel cargo test --workspace exit code: 0

Gates:

cargo fmt --check
(no stdout; exit code 0)

cargo clippy --workspace --all-targets -- -D warnings
    Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/collab-flake-2/crates/calternal-server)
    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/collab-flake-2/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.20s
(exit code 0)

cargo test --workspace
running 5 tests
test stale_epoch_is_told_the_new_epoch_and_closed ... ok
test tab_open_across_a_restart_does_not_duplicate_the_note ... ok
test unload_then_reconnect_continues_the_same_room ... ok
test graceful_restart_keeps_the_epoch_and_offline_edits ... ok
test out_of_band_change_starts_a_new_epoch ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.15s
(exit code 0; 60 test-result summaries totaled 989 passed, 0 failed, 9 ignored)

Decision not specified in DESIGN.md: Hub shutdown is a write barrier. It returns HTTP 503 for new room loads and rejects new WebSocket and agent document edits while final saves run.

The workspace test required a local web build because RustEmbed expects apps/web/build/; after bun run build, the workspace gate passed. No build files or lockfiles were committed.

Finished on branch `job/collab-flake-2`, HEAD `3c6c3324dd68d72017706f11e23745529aaee021`. Root cause: the WebSocket handler acknowledged an accepted update before the debounce scheduler marked it dirty. Shutdown could race that mark. Also, flush reset `dirty_since` before it took the document snapshot, so an update could be included in the snapshot but leave the room dirty and make the stored epoch skip. Dirty marking now happens under the document lock; flush pairs the reset with its snapshot. Shutdown gates new room loads and document edits, then holds the room flush lock through its final flush and cache store. The restart tests now wait for room unload after the rejected socket leaves instead of sleeping past `MAX_FLUSH_WAIT`. The graceful restart case drains the listener, checks the stored epoch, and waits for the old socket's last-leave save before reconnecting. Acceptance output: ```text Standalone loop: 200/200 passed, 0 failed Under-load loop: 100/100 passed, 0 failed Parallel cargo test --workspace exit code: 0 ``` Gates: ```text cargo fmt --check (no stdout; exit code 0) cargo clippy --workspace --all-targets -- -D warnings Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/collab-flake-2/crates/calternal-server) Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/collab-flake-2/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.20s (exit code 0) cargo test --workspace running 5 tests test stale_epoch_is_told_the_new_epoch_and_closed ... ok test tab_open_across_a_restart_does_not_duplicate_the_note ... ok test unload_then_reconnect_continues_the_same_room ... ok test graceful_restart_keeps_the_epoch_and_offline_edits ... ok test out_of_band_change_starts_a_new_epoch ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.15s (exit code 0; 60 test-result summaries totaled 989 passed, 0 failed, 9 ignored) ``` Decision not specified in DESIGN.md: Hub shutdown is a write barrier. It returns HTTP 503 for new room loads and rejects new WebSocket and agent document edits while final saves run. The workspace test required a local web build because `RustEmbed` expects `apps/web/build/`; after `bun run build`, the workspace gate passed. No build files or lockfiles were committed.

kayg referenced this issue from a commit

2026-09-25 11:13:30 +00:00

Merge job/collab-flake-2: serialize shutdown room snapshots; restart test waits on real conditions (#84)

kayg closed this issue

2026-09-25 11:13:30 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#84

No description provided.

Rows
Columns