Files restore returns 500 when Tags scans non-UTF-8 Markdown

kayg commented

2026-10-01 12:18:46 +00:00

Owner

Found in the bounded local round for #623, on job/ghosttask-623 after merging origin/dev.

Files can upload a file named Notes/binary.md with bytes FF 00 80. Files Trash returns 200. Files restore returns 500, although the bytes have moved out of Trash.

The existing restore path calls calternal_tags::restore_trash before Notes adoption. That function calls index::reconcile_user. markdown_item_from_file in crates/calternal-tags/src/index.rs decodes every .md file with String::from_utf8 and returns InvalidMarkdown. This error propagates from the restored file to HTTP 500. The same scan can fail because of another malformed Markdown file in the Home.

Expected: preserve arbitrary File bytes and complete Files restore without 5xx. Skip invalid Markdown projections with a warning. Do not rewrite those bytes.

Evidence: tests/adversarial/task-trash-623.mjs retains the status 200 assertion and reports the failure. Notes adoption has separate passing regression coverage for invalid UTF-8 and removes derived Task rows. Tags behavior is outside the files owned by #623, so this finding is filed rather than changed here.

Found in the bounded local round for #623, on job/ghosttask-623 after merging origin/dev. Files can upload a file named Notes/binary.md with bytes FF 00 80. Files Trash returns 200. Files restore returns 500, although the bytes have moved out of Trash. The existing restore path calls calternal_tags::restore_trash before Notes adoption. That function calls index::reconcile_user. markdown_item_from_file in crates/calternal-tags/src/index.rs decodes every .md file with String::from_utf8 and returns InvalidMarkdown. This error propagates from the restored file to HTTP 500. The same scan can fail because of another malformed Markdown file in the Home. Expected: preserve arbitrary File bytes and complete Files restore without 5xx. Skip invalid Markdown projections with a warning. Do not rewrite those bytes. Evidence: tests/adversarial/task-trash-623.mjs retains the status 200 assertion and reports the failure. Notes adoption has separate passing regression coverage for invalid UTF-8 and removes derived Task rows. Tags behavior is outside the files owned by #623, so this finding is filed rather than changed here.

kayg referenced this issue

2026-10-01 12:19:49 +00:00

Deleted task note still shows as a title-less task in Calendar (stale projection/cache); delete must remove it everywhere #623

kayg referenced this issue

2026-10-01 12:44:32 +00:00

Deleted task note still shows as a title-less task in Calendar (stale projection/cache); delete must remove it everywhere #623

kayg commented

2026-10-01 21:43:19 +00:00

Author

Owner

Starting #631 and #627 on branch job/files-631, based on 687ff70313 (origin/dev). I am tracing Tags Markdown projection failures and Files directory identity changes during tus uploads, then I will add the requested regressions and run the crate gates plus the adversarial and Calendar e2e checks.

Starting #631 and #627 on branch job/files-631, based on 687ff703136e71e89f8dfba139e93cd0788b25c1 (origin/dev). I am tracing Tags Markdown projection failures and Files directory identity changes during tus uploads, then I will add the requested regressions and run the crate gates plus the adversarial and Calendar e2e checks.

kayg referenced this issue

2026-10-01 21:43:20 +00:00

Files: directory re-index invalidates an open tus destination after a sibling Note write #627

kayg commented

2026-10-01 21:52:03 +00:00

Author

Owner

The failure path is confirmed: markdown_item_from_file maps String::from_utf8 failure to InvalidMarkdown, and reconcile_user propagates it to restore and other callers. I will skip only the invalid Markdown projection, emit a warning with the item ID and byte count, and leave the source bytes untouched. The requested adversarial script is present on the #623 job branch but not in this branch, so I will restore its existing 200 assertion as the regression.

The failure path is confirmed: `markdown_item_from_file` maps `String::from_utf8` failure to `InvalidMarkdown`, and `reconcile_user` propagates it to restore and other callers. I will skip only the invalid Markdown projection, emit a warning with the item ID and byte count, and leave the source bytes untouched. The requested adversarial script is present on the #623 job branch but not in this branch, so I will restore its existing 200 assertion as the regression.

kayg referenced this issue

2026-10-01 23:15:28 +00:00

Deleted task note still shows as a title-less task in Calendar (stale projection/cache); delete must remove it everywhere #623

kayg commented

2026-10-02 00:03:13 +00:00

Author

Owner

Built

#631: Tags now skips invalid UTF-8 Markdown projections, logs the stable Files item ID and byte count without source bytes, and leaves the source unchanged. Tag rename also skips malformed Markdown.
#627: Files stores each directory's direct parent fingerprint. A directory keeps its ID across child writes when its device/inode match and the saved parent fingerprint is unchanged. A changed parent token is replacement evidence. Target fingerprint, grants, stale-path checks, upload hash preconditions, and parent-rename behavior remain covered.
Added the Files directory-identity profile to bench/run.sh and its report integration.

Files

crates/calternal-tags/src/index.rs, crates/calternal-tags/src/rename.rs, crates/calternal-tags/src/lib.rs, crates/plugins/files/src/index.rs, crates/plugins/files/src/lib.rs, crates/plugins/files/migrations/0019_directory_parent_fingerprint.sql, tests/adversarial/task-trash-623.mjs, bench/files-directory-identity-627.py, bench/files-directory-identity-627.mjs, bench/run.sh, bench/record.py.

Commits

877124ccb, 970820261, bd6433b0b, d97a3c60f, e5e03a5b8, 0122162b0.

Head: 0122162b0126f7f7518468d1c6a3a3c54d78e26f.

Final gate output

cargo fmt --check passed with no output.

    Checking calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/files-631/crates/calternal-tags)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 14.23s

test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.39s

   Doc-tests calternal_tags

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/files-631)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 29.70s

test result: ok. 149 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 128.66s

   Doc-tests calternal_plugin_files

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 19s

test wire::tests::live_apps_run_in_separate_processes ... ok

test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 15.46s

Probes

tests/adversarial/task-trash-623.mjs: the non-UTF-8 Markdown restore assertion returned HTTP 200. The later Task projection assertion found the separate #623 bug: after Files Trash, /api/v1/notes/tasks/by-id still returned 200 where the probe expected 404. I preserved the expectation and reported it on #623.
Cross-user matrix passed: Two-User OpenAPI matrix: 335 operations classified; 159 operations replayed; 733 A-ID vs missing-ID comparisons across B, C, D and anonymous; 22 identifier routes classified with no local fixture factory; median absolute timing delta 1.5 ms; Job/Mail/quota ownership checks: 97 comparisons; 0 denial failures.
calendar.mjs passed beyond line 1117, then failed at line 1401: AssertionError [ERR_ASSERTION]: the send carries the snapshot: 01:20 nFrozen snapshot. No Calendar UI code changed in this job.
The local debug upload profile ran with load average [26.7744140625, 21.46923828125, 17.32568359375]: 50 serial uploads p50/p95 283.08/3039.73 ms; 16 concurrent uploads p50/p95 5724.78/8084.86 ms; burst peak RSS 444096512 bytes. The published files.entries read baseline is p50/p95 1.8/3.1 ms; it is not a directly comparable tus-write baseline. Treat the local numbers as noisy.

Decisions

FileFingerprint has no birth time. Migration 0019 stores the direct parent fingerprint so a changed directory token retains identity only when the inode still matches and the parent fingerprint is unchanged. origin/dev had Files migrations through 0018, so 0019 was free. git merge origin/dev reported Already up to date.
The final commit after the Rust gates changes documentation comments only.

Cargo cleanup output: Removed 18629 files, 10.2GiB total.

## Built - #631: Tags now skips invalid UTF-8 Markdown projections, logs the stable Files item ID and byte count without source bytes, and leaves the source unchanged. Tag rename also skips malformed Markdown. - #627: Files stores each directory's direct parent fingerprint. A directory keeps its ID across child writes when its device/inode match and the saved parent fingerprint is unchanged. A changed parent token is replacement evidence. Target fingerprint, grants, stale-path checks, upload hash preconditions, and parent-rename behavior remain covered. - Added the Files directory-identity profile to `bench/run.sh` and its report integration. ## Files `crates/calternal-tags/src/index.rs`, `crates/calternal-tags/src/rename.rs`, `crates/calternal-tags/src/lib.rs`, `crates/plugins/files/src/index.rs`, `crates/plugins/files/src/lib.rs`, `crates/plugins/files/migrations/0019_directory_parent_fingerprint.sql`, `tests/adversarial/task-trash-623.mjs`, `bench/files-directory-identity-627.py`, `bench/files-directory-identity-627.mjs`, `bench/run.sh`, `bench/record.py`. ## Commits `877124ccb`, `970820261`, `bd6433b0b`, `d97a3c60f`, `e5e03a5b8`, `0122162b0`. Head: `0122162b0126f7f7518468d1c6a3a3c54d78e26f`. ## Final gate output `cargo fmt --check` passed with no output. ```text Checking calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/files-631/crates/calternal-tags) Finished `dev` profile [unoptimized + debuginfo] target(s) in 14.23s ``` ```text test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.39s Doc-tests calternal_tags running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ```text Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/files-631) Finished `dev` profile [unoptimized + debuginfo] target(s) in 29.70s ``` ```text test result: ok. 149 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 128.66s Doc-tests calternal_plugin_files running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 19s ``` ```text test wire::tests::live_apps_run_in_separate_processes ... ok test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 15.46s ``` ## Probes - `tests/adversarial/task-trash-623.mjs`: the non-UTF-8 Markdown restore assertion returned HTTP 200. The later Task projection assertion found the separate #623 bug: after Files Trash, `/api/v1/notes/tasks/by-id` still returned 200 where the probe expected 404. I preserved the expectation and reported it on #623. - Cross-user matrix passed: `Two-User OpenAPI matrix: 335 operations classified; 159 operations replayed; 733 A-ID vs missing-ID comparisons across B, C, D and anonymous; 22 identifier routes classified with no local fixture factory; median absolute timing delta 1.5 ms`; `Job/Mail/quota ownership checks: 97 comparisons; 0 denial failures`. - `calendar.mjs` passed beyond line 1117, then failed at line 1401: `AssertionError [ERR_ASSERTION]: the send carries the snapshot: 01:20 nFrozen snapshot`. No Calendar UI code changed in this job. - The local debug upload profile ran with load average `[26.7744140625, 21.46923828125, 17.32568359375]`: 50 serial uploads p50/p95 `283.08/3039.73 ms`; 16 concurrent uploads p50/p95 `5724.78/8084.86 ms`; burst peak RSS `444096512` bytes. The published `files.entries` read baseline is p50/p95 `1.8/3.1 ms`; it is not a directly comparable tus-write baseline. Treat the local numbers as noisy. ## Decisions - `FileFingerprint` has no birth time. Migration 0019 stores the direct parent fingerprint so a changed directory token retains identity only when the inode still matches and the parent fingerprint is unchanged. `origin/dev` had Files migrations through 0018, so 0019 was free. `git merge origin/dev` reported `Already up to date.` - The final commit after the Rust gates changes documentation comments only. Cargo cleanup output: `Removed 18629 files, 10.2GiB total`.

kayg referenced this issue

2026-10-02 00:04:21 +00:00

Files: directory re-index invalidates an open tus destination after a sibling Note write #627

kayg commented

2026-10-02 00:04:29 +00:00

Author

Owner

Correction to the #631 final report: the verbatim Files clippy check line is:

    Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/files-631/crates/plugins/files)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 29.70s

The #627 identity rule also applies when the directory's own fingerprint changes: Files then retains the ID only with matching device/inode and an unchanged saved parent fingerprint. A parent fingerprint change is replacement evidence in that changed-target case.

Correction to the #631 final report: the verbatim Files clippy check line is: ```text Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/files-631/crates/plugins/files) Finished `dev` profile [unoptimized + debuginfo] target(s) in 29.70s ``` The #627 identity rule also applies when the directory's own fingerprint changes: Files then retains the ID only with matching device/inode and an unchanged saved parent fingerprint. A parent fingerprint change is replacement evidence in that changed-target case.

kayg commented

2026-10-02 00:07:12 +00:00

Author

Owner

Correction to the #631 final report: the current head is 1752147b9f782acdb94b1df1ebbeb897430a34fd. The extra commit after 0122162b0 documents the conservative legacy-directory migration decision for #627; it changes comments only, so the gate results and issue findings are unchanged.

Correction to the #631 final report: the current head is `1752147b9f782acdb94b1df1ebbeb897430a34fd`. The extra commit after `0122162b0` documents the conservative legacy-directory migration decision for #627; it changes comments only, so the gate results and issue findings are unchanged.

kayg referenced this issue

2026-10-02 11:23:54 +00:00

Moving in by dropping Markdown folders: find and fix what breaks (replaces the dedicated importer) #719

kayg referenced this issue

2026-10-03 12:51:32 +00:00

Issue hygiene sweep 2026-10-03 #1008