Collab: hostile_clients 'room keeps saving' test intermittently fails (possible lost edits after a rejected update)

kayg commented

2026-09-25 04:08:49 +00:00

Owner

crates/calternal-collab/tests/hostile_clients.rs::unrepresentable_update_is_rejected_and_room_keeps_saving fails intermittently under load (assertion failed: eventually_loaded(&fixture, None).await, line ~268). Seen on main at 696ad35 and on several job branches, sometimes on retry too. It is the same family as #77 (shared-note timing), which fixed shared_notes with a short stale-fingerprint retry.

Do: find what eventually_loaded waits for and why it sometimes never happens (a room that stops saving after rejecting an unrepresentable update would be a real bug: the owner's later edits would be lost). Decide whether it's a test timing issue or a product bug, fix it, and prove it with 50 consecutive passes under load.

`crates/calternal-collab/tests/hostile_clients.rs::unrepresentable_update_is_rejected_and_room_keeps_saving` fails intermittently under load (`assertion failed: eventually_loaded(&fixture, None).await`, line ~268). Seen on main at `696ad35` and on several job branches, sometimes on retry too. It is the same family as #77 (shared-note timing), which fixed `shared_notes` with a short stale-fingerprint retry. Do: find what `eventually_loaded` waits for and why it sometimes never happens (a room that stops saving after rejecting an unrepresentable update would be a real bug: the owner's later edits would be lost). Decide whether it's a test timing issue or a product bug, fix it, and prove it with 50 consecutive passes under load.

kayg commented

2026-09-25 04:11:39 +00:00

Author

Owner

Starting investigation on branch job/collab-hostile, base f8e93b7e7fa9531d1ce98049fbf9d06284812044. I am tracing the rejected-update path and the room unload assertion in #81, including the hostile-client test and the #77 fix, before deciding whether this is data loss or a timing race.

Starting investigation on branch `job/collab-hostile`, base `f8e93b7e7fa9531d1ce98049fbf9d06284812044`. I am tracing the rejected-update path and the room unload assertion in #81, including the hostile-client test and the #77 fix, before deciding whether this is data loss or a timing race.

kayg commented

2026-09-25 04:16:37 +00:00

Author

Owner

Finding: the targeted baseline command cargo test -p calternal-collab --test hostile_clients unrepresentable_update_is_rejected_and_room_keeps_saving -- --nocapture failed at hostile_clients.rs:268, the eventually_loaded(&fixture, None) assertion. The preceding assertion that the later edit reached disk passed. I am investigating the final disconnect flush/unload path; this is evidence of a stuck room after a rejected update, not just a save wait expiring.

Finding: the targeted baseline command `cargo test -p calternal-collab --test hostile_clients unrepresentable_update_is_rejected_and_room_keeps_saving -- --nocapture` failed at `hostile_clients.rs:268`, the `eventually_loaded(&fixture, None)` assertion. The preceding assertion that the later edit reached disk passed. I am investigating the final disconnect flush/unload path; this is evidence of a stuck room after a rejected update, not just a save wait expiring.

kayg commented

2026-09-25 04:22:19 +00:00

Author

Owner

Confirmed cause: hostile_clients::fixture applied only calternal_plugin_notes::migrations(). A live Note flush writes the Markdown file, then store::index calls tasks_store::index_source, which stages the shared Tags projection. Without calternal_tags::migrations(), the Index stage fails after the file replacement; the room has zero clients but remains loaded while its retry task sees the old persisted ETag. This is test setup, not a production room failure after a rejected update: the real plugin migration set includes Notes and Tags, as do Notes' own test fixtures. I updated the collab fixture to apply both sets and strengthened the regression to require two successive post-attack edits to reach disk. The targeted test now passes (1 passed, 0 failed, 7 filtered out; 4.39s).

Confirmed cause: `hostile_clients::fixture` applied only `calternal_plugin_notes::migrations()`. A live Note flush writes the Markdown file, then `store::index` calls `tasks_store::index_source`, which stages the shared Tags projection. Without `calternal_tags::migrations()`, the Index stage fails after the file replacement; the room has zero clients but remains loaded while its retry task sees the old persisted ETag. This is test setup, not a production room failure after a rejected update: the real plugin migration set includes Notes and Tags, as do Notes' own test fixtures. I updated the collab fixture to apply both sets and strengthened the regression to require two successive post-attack edits to reach disk. The targeted test now passes (1 passed, 0 failed, 7 filtered out; 4.39s).

kayg commented

2026-09-25 05:00:14 +00:00

Author

Owner

Finished on job/collab-hostile. HEAD: 92fcd89f587fd7a8f0ede50bff8c826e172d7a4b.

Built:

The hostile-client fixture now applies both Notes and Tags migration sets, matching the production save path. This prevents Index projection failure after the Markdown replacement.
The rejected-update regression now verifies two later saves reach disk, preserves the existing Note body, and checks the room unloads.

Gate output:

cargo fmt --check: no output; exit 0.
cargo clippy --all-targets -- -D warnings: Finished dev profile [unoptimized + debuginfo] target(s) in 11.56s
50-run loop: hostile_clients consecutive passes: 50/50

cargo test --workspace: failed in the out-of-scope calternal-server live-app test. Relevant output:

assertion `left == right` failed
  left: 401
 right: 204
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 15 filtered out; finished in 4.82s
test result: FAILED. 13 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.30s
error: test failed, to rerun pass `-p calternal-server --bin calternal-server`

cargo clean: Removed 15332 files, 9.8GiB total

No product behavior changed. Decision not specified in DESIGN.md: the collab test fixture uses the same Notes + Tags migration sets as production because a live Note save updates both projections.

Finished on `job/collab-hostile`. HEAD: `92fcd89f587fd7a8f0ede50bff8c826e172d7a4b`. Built: - The hostile-client fixture now applies both Notes and Tags migration sets, matching the production save path. This prevents Index projection failure after the Markdown replacement. - The rejected-update regression now verifies two later saves reach disk, preserves the existing Note body, and checks the room unloads. Gate output: - `cargo fmt --check`: no output; exit 0. - `cargo clippy --all-targets -- -D warnings`: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.56s` - 50-run loop: `hostile_clients consecutive passes: 50/50` - `cargo test --workspace`: failed in the out-of-scope calternal-server live-app test. Relevant output: ``` assertion `left == right` failed left: 401 right: 204 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 15 filtered out; finished in 4.82s test result: FAILED. 13 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.30s error: test failed, to rerun pass `-p calternal-server --bin calternal-server` ``` - `cargo clean`: `Removed 15332 files, 9.8GiB total` No product behavior changed. Decision not specified in DESIGN.md: the collab test fixture uses the same Notes + Tags migration sets as production because a live Note save updates both projections.

kayg referenced this issue from a commit

2026-09-25 05:23:20 +00:00

Merge job/collab-hostile: collab test fixture applies Tags migrations; later edits reach disk (#81)

kayg commented

2026-09-27 12:36:29 +00:00

Author

Owner

Evidence from the #148 robustness workspace gate on branch job/robustness:

cargo test --workspace --no-fail-fast -- --test-threads=1 ran calternal-collab --test hostile_clients with 8 passed and 3 failed under heavy shared-host load. The failures were:

reconnect_storm_does_not_cancel_pending_save: expected the pending-save content, but read only start.
unrepresentable_update_is_rejected_and_room_keeps_saving: the follow-up edit was not saved in the asserted window.
wiki_embeds_open_live_and_save_unchanged: byte-stability assertion failed.

An earlier workspace run passed all 11 hostile-client tests. The retry also showed large timing inflation in other crates and several DB fixture pool timeouts. I did not change Collab code because this evidence is intermittent and load-correlated; please use the failures above as a regression lead.

Evidence from the #148 robustness workspace gate on branch `job/robustness`: `cargo test --workspace --no-fail-fast -- --test-threads=1` ran `calternal-collab --test hostile_clients` with 8 passed and 3 failed under heavy shared-host load. The failures were: - `reconnect_storm_does_not_cancel_pending_save`: expected the pending-save content, but read only `start`. - `unrepresentable_update_is_rejected_and_room_keeps_saving`: the follow-up edit was not saved in the asserted window. - `wiki_embeds_open_live_and_save_unchanged`: byte-stability assertion failed. An earlier workspace run passed all 11 hostile-client tests. The retry also showed large timing inflation in other crates and several DB fixture pool timeouts. I did not change Collab code because this evidence is intermittent and load-correlated; please use the failures above as a regression lead.

kayg referenced this issue

2026-09-27 12:39:28 +00:00

Files: HTTP 500 'files Index update failed' under heavy load (adversarial, load 50-72) #148

kayg referenced this issue

2026-09-28 11:28:04 +00:00

COLLAB: concurrent delete and edit can lose an edit to the deleted block #332

kayg referenced this issue

2026-10-03 11:04:20 +00:00

SECURITY: public links still allow guest editing; §54 says public links are view only #981

kayg referenced this issue

2026-10-03 11:16:24 +00:00

SECURITY: public links still allow guest editing; §54 says public links are view only #981

kayg referenced this issue

2026-10-03 11:20:08 +00:00

SECURITY: public links still allow guest editing; §54 says public links are view only #981

kayg commented

2026-10-03 11:55:26 +00:00

Author

Owner

Fixed in 4bbedae355 (origin/dev); hostile_clients::unrepresentable_update_is_rejected_and_room_keeps_saving covers the reported collaboration test.

Fixed in 4bbedae355 (origin/dev); `hostile_clients::unrepresentable_update_is_rejected_and_room_keeps_saving` covers the reported collaboration test.

kayg closed this issue

2026-10-03 11:55:27 +00:00

kayg referenced this issue

2026-10-03 12:04:41 +00:00

Issue hygiene sweep 2026-10-03 #1008

kayg referenced this issue

2026-10-03 13:47:24 +00:00

SECURITY: public links still allow guest editing; §54 says public links are view only #981

kayg referenced this issue

2026-10-03 13:56:03 +00:00

SECURITY: public links still allow guest editing; §54 says public links are view only #981