FILES: atomic write reconcile storm can time out with entry-not-found under load #337

Closed
opened 2026-09-28 12:48:15 +00:00 by kayg · 2 comments
Owner

The workspace test gate failed on job/single-pills at merged head 46497b6a7c03eb02a7cbbbfac3f36b563c906af7 while other Cargo builds were active on the shared host.

Command: cargo test (with CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4, and worktree-local TMPDIR).

Failure:

  • calternal-plugin-files::tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm exceeded its 300-second bound while writing 1,000 files, running four reconcile scans, and adopting watcher events.
  • The test also reported atomic write 828 failed: entry not found.
  • Files tests before the failing case passed, including the earlier collaboration, API, and storage suites. Cargo stopped at this failing test binary, so later workspace test binaries did not run.

The host had concurrent Cargo builds and other adversarial jobs. This may be load-sensitive, but the entry not found write result may indicate a real race. The gate was not rerun. Please reproduce with lower host contention and check atomic-write and watcher/reconcile interaction. Keep the existing test expectation and timeout unchanged until the cause is known.

The workspace test gate failed on `job/single-pills` at merged head `46497b6a7c03eb02a7cbbbfac3f36b563c906af7` while other Cargo builds were active on the shared host. Command: `cargo test` (with `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4`, and worktree-local `TMPDIR`). Failure: - `calternal-plugin-files::tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm` exceeded its 300-second bound while writing 1,000 files, running four reconcile scans, and adopting watcher events. - The test also reported `atomic write 828 failed: entry not found`. - Files tests before the failing case passed, including the earlier collaboration, API, and storage suites. Cargo stopped at this failing test binary, so later workspace test binaries did not run. The host had concurrent Cargo builds and other adversarial jobs. This may be load-sensitive, but the `entry not found` write result may indicate a real race. The gate was not rerun. Please reproduce with lower host contention and check atomic-write and watcher/reconcile interaction. Keep the existing test expectation and timeout unchanged until the cause is known.
Author
Owner

Duplicate candidate: #343 reports the same internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm test and the same atomic-write entry-not-found failure during reconcile and watcher activity. #343 adds a deterministic interleaving regression requirement. Recommend linking this run to #343 and keeping #343 as the tracking issue; do not close either issue as part of this audit.

Duplicate candidate: #343 reports the same internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm test and the same atomic-write entry-not-found failure during reconcile and watcher activity. #343 adds a deterministic interleaving regression requirement. Recommend linking this run to #343 and keeping #343 as the tracking issue; do not close either issue as part of this audit.
Author
Owner

Duplicate of #343: both report the same atomic-write entry-not-found race during Files reconciliation. Keep #343 as the active tracker.

Duplicate of #343: both report the same atomic-write entry-not-found race during Files reconciliation. Keep #343 as the active tracker.
kayg closed this issue 2026-10-03 11:55:45 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#337
No description provided.