Files storm test times out on shared host and races fixture cleanup

kayg commented

2026-10-03 09:06:39 +00:00

Owner

Found during #988 on the shared build host. The unchanged Files regression internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm exceeded its five-minute completion budget in the full crate run and one focused run. No test assertion or timeout was changed. Load average after the focused run: 37.75 39.96 39.70 on the shared host.

The focused run reached atomic write 761 before cleanup. After the timeout, aborting its tasks does not cancel an in-flight blocking filesystem write; the fixture directory is then removed and that write reports entry not found. This is a fixture-cleanup consequence, not evidence that #988 changes data writes. #988 changes native media diagnostics and thumbnail failure handling, not atomic writes or reconcile.

Reproduce with cargo test -p calternal-plugin-files internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm -- --test-threads=1. The merge round should distinguish host-load delay from a regression and make cleanup safe without weakening the existing expectations.

Focused output:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 15.18s
     Running unittests src/lib.rs (/mnt/hdd/targets/jobs/thumbsandbox-988/debug/deps/calternal_plugin_files-81ce52810bf7fc1a)

running 1 test
test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... FAILED

failures:

---- tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm stdout ----

thread 'tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm' (635426) panicked at crates/plugins/files/src/lib.rs:5114:14:
writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'tokio-rt-worker' (635429) panicked at crates/plugins/files/src/lib.rs:5088:45:
atomic write 761 failed: entry not found


failures:
    tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 152 filtered out; finished in 305.37s

error: test failed, to rerun pass `-p calternal-plugin-files --lib`

Found during #988 on the shared build host. The unchanged Files regression `internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm` exceeded its five-minute completion budget in the full crate run and one focused run. No test assertion or timeout was changed. Load average after the focused run: `37.75 39.96 39.70` on the shared host. The focused run reached atomic write 761 before cleanup. After the timeout, aborting its tasks does not cancel an in-flight blocking filesystem write; the fixture directory is then removed and that write reports `entry not found`. This is a fixture-cleanup consequence, not evidence that #988 changes data writes. #988 changes native media diagnostics and thumbnail failure handling, not atomic writes or reconcile. Reproduce with `cargo test -p calternal-plugin-files internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm -- --test-threads=1`. The merge round should distinguish host-load delay from a regression and make cleanup safe without weakening the existing expectations. Focused output: ``` Finished `test` profile [unoptimized + debuginfo] target(s) in 15.18s Running unittests src/lib.rs (/mnt/hdd/targets/jobs/thumbsandbox-988/debug/deps/calternal_plugin_files-81ce52810bf7fc1a) running 1 test test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... FAILED failures: ---- tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm stdout ---- thread 'tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm' (635426) panicked at crates/plugins/files/src/lib.rs:5114:14: writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(()) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'tokio-rt-worker' (635429) panicked at crates/plugins/files/src/lib.rs:5088:45: atomic write 761 failed: entry not found failures: tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 152 filtered out; finished in 305.37s error: test failed, to rerun pass `-p calternal-plugin-files --lib` ```

kayg referenced this issue

2026-10-03 09:19:52 +00:00

Thumbnails fail on staging and the build host: media sandbox cannot create namespaces, failure is silent #988

kayg referenced this issue

2026-10-03 11:21:20 +00:00

Thumbnails fail on staging and the build host: media sandbox cannot create namespaces, failure is silent #988

kayg referenced this issue

2026-10-03 11:24:17 +00:00

Assign stable identity to imported Markdown Notes #857

kayg commented

2026-10-03 11:36:34 +00:00

Author

Owner

Starting filesstorm-1000 from origin/dev at d4e7188810.

Starting filesstorm-1000 from origin/dev at d4e7188810a89fb0e8e6b162279917f7e23989f9.

kayg commented

2026-10-03 11:42:32 +00:00

Author

Owner

Finding: the existing storm test aborts its writer, scanner and watcher on timeout, then unwinds without joining them. The Files writer performs handle-relative filesystem calls such as temp-file writes, fsync and install synchronously between async reads (crates/calternal-fs/src/write.rs, Root::write); abort cannot preempt a call already running in a Tokio task. That leaves the test's TempDir free to remove the fixture while a write is still using it. I will keep the five-minute budget and all index assertions, and wait for aborted tasks to finish before fixture cleanup.

Finding: the existing storm test aborts its writer, scanner and watcher on timeout, then unwinds without joining them. The Files writer performs handle-relative filesystem calls such as temp-file writes, fsync and install synchronously between async reads (`crates/calternal-fs/src/write.rs`, `Root::write`); abort cannot preempt a call already running in a Tokio task. That leaves the test's TempDir free to remove the fixture while a write is still using it. I will keep the five-minute budget and all index assertions, and wait for aborted tasks to finish before fixture cleanup.

kayg referenced this issue

2026-10-03 12:04:19 +00:00

Assign stable identity to imported Markdown Notes #857

kayg referenced this issue

2026-10-03 12:12:16 +00:00

Thumbnails fail on staging and the build host: media sandbox cannot create namespaces, failure is silent #988

kayg referenced this issue

2026-10-03 12:21:35 +00:00

Assign stable identity to imported Markdown Notes #857

kayg commented

2026-10-03 12:34:32 +00:00

Author

Owner

The focused reproduction passed unchanged: cargo test -p calternal-plugin-files internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm -- --test-threads=1 → test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 295.64s. The five-minute budget was nearly reached, consistent with shared-host contention, but cleanup now joins unfinished writer/scanner/watcher tasks before fixture teardown on any failure.

The focused reproduction passed unchanged: `cargo test -p calternal-plugin-files internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm -- --test-threads=1` → `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 295.64s`. The five-minute budget was nearly reached, consistent with shared-host contention, but cleanup now joins unfinished writer/scanner/watcher tasks before fixture teardown on any failure.

kayg commented

2026-10-03 12:44:42 +00:00

Author

Owner

Rechecked origin/dev d4e7188810. The storm test still aborts writer, scanner and watcher tasks on timeout, then unwinds without joining them. The reported cleanup change and unchanged focused pass are on job/filesstorm-1000 only. Keep open until that change is on origin/dev.

Rechecked origin/dev d4e7188810a89fb0e8e6b162279917f7e23989f9. The storm test still aborts writer, scanner and watcher tasks on timeout, then unwinds without joining them. The reported cleanup change and unchanged focused pass are on job/filesstorm-1000 only. Keep open until that change is on origin/dev.

kayg referenced this issue

2026-10-03 13:10:49 +00:00

Thumbnails fail on staging and the build host: media sandbox cannot create namespaces, failure is silent #988

kayg referenced this issue

2026-10-03 13:13:42 +00:00

Issue hygiene sweep 2026-10-03 #1008

kayg commented

2026-10-03 13:18:06 +00:00

Author

Owner

Host finding during the full crate gate: cargo reported sccache: warning: The server looks like it shut down unexpectedly, compiling locally instead. The test command remains active. This affects verification time on the shared host; it does not change the test assertions or implementation.

Host finding during the full crate gate: cargo reported `sccache: warning: The server looks like it shut down unexpectedly, compiling locally instead`. The test command remains active. This affects verification time on the shared host; it does not change the test assertions or implementation.

kayg commented

2026-10-03 13:26:47 +00:00

Author

Owner

Completed filesstorm-1000.

Built: the Files storm test now aborts and joins unfinished writer/scanner tasks before fixture cleanup on failure, and always aborts and joins the watcher. The five-minute timeout and existing Index assertions are unchanged.

Files: crates/plugins/files/src/lib.rs

Commit: d009fb351ff003d45b8c946c2d4a314be72cf69b (fix(files): join storm tasks before fixture cleanup)

Gate output:

cargo fmt --check

(no output; exit status 0)

cargo clippy -p calternal-plugin-files --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 40m 48s

Focused regression:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 44m 07s
     Running unittests src/lib.rs (/mnt/hdd/targets/jobs/filesstorm-1000/debug/deps/calternal_plugin_files-81ce52810bf7fc1a)

running 1 test
test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 295.64s

cargo test -p calternal-plugin-files

test result: ok. 146 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 339.53s

   Doc-tests calternal_plugin_files

running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clean

     Removed 8665 files, 3.3GiB total

Known gaps: the timeout cleanup branch was not forced directly; both focused and full storm runs completed successfully. During the full test gate, cargo reported sccache: warning: The server looks like it shut down unexpectedly, compiling locally instead; the gate continued and passed.

Decisions: DESIGN §2 defines atomic writes, but does not define async test-fixture cleanup. I kept the timeout and expectations unchanged and waited for unfinished task polls to stop before TempDir cleanup, so an active filesystem call cannot race fixture removal.

Completed filesstorm-1000. Built: the Files storm test now aborts and joins unfinished writer/scanner tasks before fixture cleanup on failure, and always aborts and joins the watcher. The five-minute timeout and existing Index assertions are unchanged. Files: `crates/plugins/files/src/lib.rs` Commit: `d009fb351ff003d45b8c946c2d4a314be72cf69b` (`fix(files): join storm tasks before fixture cleanup`) Gate output: `cargo fmt --check` ``` (no output; exit status 0) ``` `cargo clippy -p calternal-plugin-files --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 40m 48s ``` Focused regression: ``` Finished `test` profile [unoptimized + debuginfo] target(s) in 44m 07s Running unittests src/lib.rs (/mnt/hdd/targets/jobs/filesstorm-1000/debug/deps/calternal_plugin_files-81ce52810bf7fc1a) running 1 test test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 295.64s ``` `cargo test -p calternal-plugin-files` ``` test result: ok. 146 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 339.53s Doc-tests calternal_plugin_files running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clean` ``` Removed 8665 files, 3.3GiB total ``` Known gaps: the timeout cleanup branch was not forced directly; both focused and full storm runs completed successfully. During the full test gate, cargo reported `sccache: warning: The server looks like it shut down unexpectedly, compiling locally instead`; the gate continued and passed. Decisions: DESIGN §2 defines atomic writes, but does not define async test-fixture cleanup. I kept the timeout and expectations unchanged and waited for unfinished task polls to stop before TempDir cleanup, so an active filesystem call cannot race fixture removal.

Rows
Columns

Files storm test times out on shared host and races fixture cleanup #1000