DATA: Files Index lags storage after concurrent upload + rename (repaired only by a later listing)

kayg commented

2026-09-27 18:40:02 +00:00

Owner

Found by the #234 fonts job's adversarial round (2026-09-27), filed by the orchestrator.

Finding

The post-dev adversarial round found an Index and storage mismatch in the Files upload path.

tests/adversarial/attack.py ran eight concurrent 64 KiB TUS uploads and renames while 16 requests listed the same directory and CPU workers kept the host busy. The uploads and renames completed. The probe read the raw SQLite Index before listing could repair it. For all eight files (uploaded-0.bin through uploaded-7.bin), the Index row existed but the installed file bytes did not match the upload payload.

The server stayed alive. This is a data consistency finding, not a SLOW latency result. Please reproduce it on a quiet host, fix the install/Index ordering, and keep a regression test for concurrent upload, rename and listing.

Probe: tests/adversarial/attack.py, Files contention section. The round ran on job/fonts after merging dev at ac941215.

Required

Root-cause the install/Index ordering in the Files upload path (TUS finalize → calternal-fs install → Index record) and the rename path, under concurrency with listings. The Index must be consistent with storage at the moment the API returns success (single writer: the server commits the Index change in the same logical operation as the storage change, or journals it so a crash replays it).
Coordinate with #258 (search/index reliability): the search index must follow the Files Index, so a lagging Files Index also means stale search results. Reuse its integrity checker if it lands first; do not build a second one.
Deterministic regression test (concurrent uploads + renames + listings, read the raw Index right after each success) and extend the adversarial Files contention section to assert zero mismatches.
Data-consistency class: blocks merges touching this path.

Found by the #234 fonts job's adversarial round (2026-09-27), filed by the orchestrator. ## Finding The post-`dev` adversarial round found an Index and storage mismatch in the Files upload path. `tests/adversarial/attack.py` ran eight concurrent 64 KiB TUS uploads and renames while 16 requests listed the same directory and CPU workers kept the host busy. The uploads and renames completed. The probe read the raw SQLite Index before listing could repair it. For all eight files (`uploaded-0.bin` through `uploaded-7.bin`), the Index row existed but the installed file bytes did not match the upload payload. The server stayed alive. This is a data consistency finding, not a SLOW latency result. Please reproduce it on a quiet host, fix the install/Index ordering, and keep a regression test for concurrent upload, rename and listing. Probe: `tests/adversarial/attack.py`, Files contention section. The round ran on `job/fonts` after merging `dev` at `ac941215`. ## Required - Root-cause the install/Index ordering in the Files upload path (TUS finalize → calternal-fs install → Index record) and the rename path, under concurrency with listings. The Index must be consistent with storage at the moment the API returns success (single writer: the server commits the Index change in the same logical operation as the storage change, or journals it so a crash replays it). - Coordinate with #258 (search/index reliability): the search index must follow the Files Index, so a lagging Files Index also means stale search results. Reuse its integrity checker if it lands first; do not build a second one. - Deterministic regression test (concurrent uploads + renames + listings, read the raw Index right after each success) and extend the adversarial Files contention section to assert zero mismatches. Data-consistency class: blocks merges touching this path.

kayg commented

2026-09-27 18:41:22 +00:00

Author

Owner

Starting #259 on job/index-order.

Base (dev): 19e65b2243453ac53ee1c374b74f499d708dda1b
Scope: trace Files upload/rename/list ordering, add a deterministic concurrency regression and adversarial assertion, then run the requested gates and one adversarial round.
I read #258 and its comments. It is implementing a separate search integrity checker; I will not duplicate it. The Files Index race remains in this issue's storage/index mutation path.

Starting #259 on `job/index-order`. - Base (`dev`): `19e65b2243453ac53ee1c374b74f499d708dda1b` - Scope: trace Files upload/rename/list ordering, add a deterministic concurrency regression and adversarial assertion, then run the requested gates and one adversarial round. - I read #258 and its comments. It is implementing a separate search integrity checker; I will not duplicate it. The Files Index race remains in this issue's storage/index mutation path.

kayg commented

2026-09-27 18:59:43 +00:00

Author

Owner

Finding during the first regression build: the shared sccache wrapper failed while compiling futures-channel because it could not find /home/kayg/Developer/calternal-wt/fonts/target/tmp/sccacheE9NUrm/deps.d. Rust also reported missing dep-info paths under that worktree for cmake and dunce. The target directory for this job is local to index-order; I am rerunning with RUSTC_WRAPPER cleared to avoid the shared temporary-file race.

Finding during the first regression build: the shared `sccache` wrapper failed while compiling `futures-channel` because it could not find `/home/kayg/Developer/calternal-wt/fonts/target/tmp/sccacheE9NUrm/deps.d`. Rust also reported missing dep-info paths under that worktree for `cmake` and `dunce`. The target directory for this job is local to `index-order`; I am rerunning with `RUSTC_WRAPPER` cleared to avoid the shared temporary-file race.

kayg commented

2026-09-27 19:21:04 +00:00

Author

Owner

Root-cause finding from the upload path: finish persists staged_hash as files_uploads.install_hash, then write_checked returns a separate hash. complete_install trusts that returned hash, commits the Files Index row, and deletes the recovery intent without checking that the committed row still matches the installed inode and staged hash. The existing concurrency regression passes with the current mutation locks and delayed change consumer, so I am adding a deterministic failure test for this unchecked postcondition before changing it.

Root-cause finding from the upload path: `finish` persists `staged_hash` as `files_uploads.install_hash`, then `write_checked` returns a separate hash. `complete_install` trusts that returned hash, commits the Files Index row, and deletes the recovery intent without checking that the committed row still matches the installed inode and staged hash. The existing concurrency regression passes with the current mutation locks and delayed change consumer, so I am adding a deterministic failure test for this unchecked postcondition before changing it.

kayg commented

2026-09-27 19:24:14 +00:00

Author

Owner

The deterministic fault test is red against the current upload path. It installs the real file, uses a SQLite trigger to change that upload's Files Index hash to wrong, and observed PATCH return 204 (expected 500). This proves finalization can report success without validating the committed row against the installed fingerprint and upload hash. The test also checks that a failed attempt retains the install intent and can complete on retry after the Index write is available.

The deterministic fault test is red against the current upload path. It installs the real file, uses a SQLite trigger to change that upload's Files Index hash to `wrong`, and observed `PATCH` return `204` (`expected 500`). This proves finalization can report success without validating the committed row against the installed fingerprint and upload hash. The test also checks that a failed attempt retains the install intent and can complete on retry after the Index write is available.

kayg referenced this issue

2026-09-27 20:44:02 +00:00

[authz] Non-admin config writes return 422 instead of 403 #268

kayg commented

2026-09-27 21:57:37 +00:00

Author

Owner

Complete

Fixed the Tus install commit point so the Files Index cannot lag behind a successful upload. The server now compares the installed file with the durable upload hash, refreshes the index from that installed file, verifies the index row's stable identity/fingerprint and content hash, and only then clears the recovery intent. If verification fails, the request fails and the intent remains available for retry/recovery.

Added a deterministic concurrent upload/rename/listing regression and an adversarial Files assertion. The regression checks the raw Index row and installed bytes after success while a delayed watcher consumer and concurrent listings run. A second regression corrupts the Index hash and confirms the API returns 500 while retaining the install intent; a retry succeeds after removing the injected corruption.

Files

crates/plugins/files/src/index.rs
crates/plugins/files/src/uploads.rs
crates/plugins/files/src/lib.rs
crates/plugins/files/DATA-SAFETY.md
tests/adversarial/attack.py

Commits

5e4e0adcfb80bee026bfa935236e5ecddf8740cb — fix Files upload index verification
418fcdfce8fc1300eddb71fc6e1301d5afdedece — merge dev once before final gates
Pushed branch: job/index-order

Gates (after the one `git merge dev`)

cargo fmt --check: exit 0; no output.

cargo clippy --all-targets -- -D warnings (exit 0):

    Blocking waiting for file lock on package cache
   Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/index-order/crates/calternal-server)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 20.64s

cargo test (exit 0), Files crate and new regression output:

running 112 tests
test tests::concurrent_upload_rename_and_listing_keep_index_and_bytes_aligned ... ok
test tests::tus_keeps_install_intent_until_the_index_matches_the_installed_file ... ok
test result: ok. 112 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 75.85s

bun run --cwd apps/web check (exit 0):

$ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/index-order/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run --cwd apps/web test (exit 0):

Test Files  86 passed (86)
     Tests  604 passed (604)
   Start at  22:23:20
   Duration  132.48s (transform 62%, environment 15%, import 12%, tests 9%, setup 2%)

Production web build also succeeded: ✓ built in 46.60s.

Adversarial round

One full real local-server round completed. The Files upload/rename/listing assertion produced no finding; hostile-byte probes reported ==== HOSTILE BYTES FINDINGS 0; the server remained alive; the live Note restart probe reported restart probe: 0 findings.

The round command exited 1. Its summaries were:

==== FINDINGS 130
==== ROUND 2 FINDINGS 79
==== ROUND 2 SLOW 139
EXIT_CODE=1

The non-SLOW DAV timeouts were recorded on #264 (DAV initial sync and Reminders discovery) and #267 (Calendar Event from Log). A standard-user admin config write returned 422 where the probe expects 403; I filed #268. The late dedup failures came after the full runner deleted/revoked its B–D fixtures and did not coordinate restart barriers 4–6; I added this run's evidence to existing issue #272. The eventual scrub status marked the affected paths repaired. The SLOW-only responses were load.

Decision

The design docs do not specify the Tus install-intent commit point. I kept the durable install intent until the installed file and Index row both match the upload's durable hash. This preserves recovery information when indexing does not commit the exact installed bytes. I read #258 and did not duplicate its integrity checker.

Known gaps

The unrelated #264/#267 timeouts and #268 authorization-status mismatch remain open. The full adversarial invocation's late dedup and crash/restart results remain inconclusive until #272 is fixed. No Files Index consistency mismatch was observed.

## Complete Fixed the Tus install commit point so the Files Index cannot lag behind a successful upload. The server now compares the installed file with the durable upload hash, refreshes the index from that installed file, verifies the index row's stable identity/fingerprint and content hash, and only then clears the recovery intent. If verification fails, the request fails and the intent remains available for retry/recovery. Added a deterministic concurrent upload/rename/listing regression and an adversarial Files assertion. The regression checks the raw Index row and installed bytes after success while a delayed watcher consumer and concurrent listings run. A second regression corrupts the Index hash and confirms the API returns 500 while retaining the install intent; a retry succeeds after removing the injected corruption. ### Files - `crates/plugins/files/src/index.rs` - `crates/plugins/files/src/uploads.rs` - `crates/plugins/files/src/lib.rs` - `crates/plugins/files/DATA-SAFETY.md` - `tests/adversarial/attack.py` ### Commits - `5e4e0adcfb80bee026bfa935236e5ecddf8740cb` — fix Files upload index verification - `418fcdfce8fc1300eddb71fc6e1301d5afdedece` — merge `dev` once before final gates - Pushed branch: `job/index-order` ### Gates (after the one `git merge dev`) `cargo fmt --check`: exit 0; no output. `cargo clippy --all-targets -- -D warnings` (exit 0): ``` Blocking waiting for file lock on package cache Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/index-order/crates/calternal-server) Finished `dev` profile [unoptimized + debuginfo] target(s) in 20.64s ``` `cargo test` (exit 0), Files crate and new regression output: ``` running 112 tests test tests::concurrent_upload_rename_and_listing_keep_index_and_bytes_aligned ... ok test tests::tus_keeps_install_intent_until_the_index_matches_the_installed_file ... ok test result: ok. 112 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 75.85s ``` `bun run --cwd apps/web check` (exit 0): ``` $ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/index-order/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run --cwd apps/web test` (exit 0): ``` Test Files 86 passed (86) Tests 604 passed (604) Start at 22:23:20 Duration 132.48s (transform 62%, environment 15%, import 12%, tests 9%, setup 2%) ``` Production web build also succeeded: `✓ built in 46.60s`. ### Adversarial round One full real local-server round completed. The Files upload/rename/listing assertion produced no finding; hostile-byte probes reported `==== HOSTILE BYTES FINDINGS 0`; the server remained alive; the live Note restart probe reported `restart probe: 0 findings`. The round command exited 1. Its summaries were: ``` ==== FINDINGS 130 ==== ROUND 2 FINDINGS 79 ==== ROUND 2 SLOW 139 EXIT_CODE=1 ``` The non-SLOW DAV timeouts were recorded on #264 (DAV initial sync and Reminders discovery) and #267 (Calendar Event from Log). A standard-user admin config write returned 422 where the probe expects 403; I filed #268. The late dedup failures came after the full runner deleted/revoked its B–D fixtures and did not coordinate restart barriers 4–6; I added this run's evidence to existing issue #272. The eventual scrub status marked the affected paths repaired. The SLOW-only responses were load. ### Decision The design docs do not specify the Tus install-intent commit point. I kept the durable install intent until the installed file and Index row both match the upload's durable hash. This preserves recovery information when indexing does not commit the exact installed bytes. I read #258 and did not duplicate its integrity checker. ### Known gaps The unrelated #264/#267 timeouts and #268 authorization-status mismatch remain open. The full adversarial invocation's late dedup and crash/restart results remain inconclusive until #272 is fixed. No Files Index consistency mismatch was observed.

kayg commented

2026-09-27 22:00:20 +00:00

Author

Owner

Merged in 6a6ffd5a. Tus install intent kept until installed file + Index row match the durable hash; concurrent upload/rename/listing regression green.

Merged in 6a6ffd5a. Tus install intent kept until installed file + Index row match the durable hash; concurrent upload/rename/listing regression green.

kayg closed this issue

2026-09-27 22:00:21 +00:00

kayg referenced this issue from a commit

2026-09-27 22:00:21 +00:00

Merge job/index-order (#259)

kayg referenced this issue

2026-09-28 00:16:05 +00:00

Chrome: sidebar and H1 never repeat each other; Files breadcrumb; Pinned first with drag-to-pin; calm H1 transitions #156

kayg referenced this issue

2026-09-28 01:22:16 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg referenced this issue