SYNC/INDEX: atomic-write temp files (.calternal-tmp-*) get indexed and pushed to the change feed; stale rows show as 'files saved'; hide dot-files

kayg commented

2026-09-28 07:07:46 +00:00

Owner

Owner report (2026-09-28, production calternal.cloud 58818bc3): the Calendar day shows '2 files saved' at 12:29: .calternal-tmp-1-6 and .calternal-tmp-1-7. The owner saved no files. Owner: 'what are these tmp files? are they not being purged automatically and why are hidden files (files starting with dot) even showing up?'

Evidence (orchestrator, read-only snapshot of /srv/calternal/data/.system/index.sqlite):

On disk: 0 files named .calternal-tmp-* (the atomic writes did rename them; nothing leaked on disk).
files_index: two stale rows: Notes/.calternal-tmp-1-6 and Notes/.calternal-tmp-1-7 (mime application/octet-stream, size 632, modified 1790578748), so Calendar (crates/plugins/calendar/src/view.rs file activity) counts them.
files_events: 4 rows for the same temp paths.
files_change_feed: cursor 14/15 create temp-1-6/1-7 (12:28:24), 18/19 delete (12:29:03), 20 (+21?) create again (12:29:08) with no later delete. Sync clients received creates for files that never existed as user files → a sync-collision class bug (merge blocker per CLAUDE.md).
Timing: the owner logged a composer entry around 12:28; the writer is the day-file/notes write path creating temps in Notes/ (crates/calternal-fs/src/root.rs:887 names them .calternal-tmp-{a}-{b}; thumbnails.rs and journal.rs use the same prefix).

Root cause to confirm: the watcher/reconcile scan observes a temp file between its create and its rename. It indexes it and appends it to the change feed. The rename is then seen as a new file (or missed), so the temp row is never removed.

Fix:

Temp files are invisible to everything: the watcher, the reconcile scan, files_index, files_events, the change feed, search, calendar activity and the Files listing. Filter them at the single entry point in calternal-fs/calternal-sync (one predicate, e.g. calternal_path::is_internal_temp, used everywhere; reuse the existing reserved-name list in calternal-path lib.rs:54). Better still: create temps where no watcher looks (e.g. O_TMPFILE + linkat on Linux, or a .system/tmp dir on the same filesystem, then renameat into place), so no scan can ever see them. Check that openat2 RESOLVE_BENEATH rules still hold.
A rename of temp → final is reported as ONE create/modify of the final path in the change feed, never create(temp) + create(final).
Repair migration: delete the index, events and change-feed-derived rows for any .calternal-tmp-* path, and emit a compensating delete to sync clients only if a create was ever published (so a client that recreated a temp can remove it). Idempotent.
Hidden files (dot-files), owner question: proposal: dot-files are stored and synced as they are ('file over app': e.g. .obsidian/), but they are not shown in Files, Calendar activity, Photos or Search by default. Files gets a 'Show hidden files' view toggle (⌘⇧. like Finder, persisted per user). Internal names (.calternal*, temps) never show, even with the toggle.
Tests: a concurrent test that hammers atomic writes while the watcher and reconcile run (1,000 writes, 0 temp rows in the index or the feed); a regression test for the rename-seen-as-create race; a sync-client test showing no temp path is ever downloaded. Add a case to tests/adversarial: rapid composer logs plus a scan storm.
Also check: does this race explain the owner's other report, where a composer log entry did not appear in the Calendar until a reload (#298)? If the day-file write is reported as a temp create, the Calendar live update may miss the real file. Coordinate on #298.

Owner report (2026-09-28, production calternal.cloud 58818bc3): the Calendar day shows '2 files saved' at 12:29: `.calternal-tmp-1-6` and `.calternal-tmp-1-7`. The owner saved no files. Owner: 'what are these tmp files? are they not being purged automatically and why are hidden files (files starting with dot) even showing up?' **Evidence (orchestrator, read-only snapshot of /srv/calternal/data/.system/index.sqlite):** - On disk: 0 files named `.calternal-tmp-*` (the atomic writes did rename them; nothing leaked on disk). - `files_index`: two **stale rows**: `Notes/.calternal-tmp-1-6` and `Notes/.calternal-tmp-1-7` (mime application/octet-stream, size 632, modified 1790578748), so Calendar (crates/plugins/calendar/src/view.rs file activity) counts them. - `files_events`: 4 rows for the same temp paths. - `files_change_feed`: cursor 14/15 **create** temp-1-6/1-7 (12:28:24), 18/19 **delete** (12:29:03), 20 (+21?) **create** again (12:29:08) with no later delete. **Sync clients received creates for files that never existed as user files** → a sync-collision class bug (merge blocker per CLAUDE.md). - Timing: the owner logged a composer entry around 12:28; the writer is the day-file/notes write path creating temps in `Notes/` (crates/calternal-fs/src/root.rs:887 names them `.calternal-tmp-{a}-{b}`; thumbnails.rs and journal.rs use the same prefix). **Root cause to confirm:** the watcher/reconcile scan observes a temp file between its create and its rename. It indexes it and appends it to the change feed. The rename is then seen as a new file (or missed), so the temp row is never removed. **Fix:** 1. Temp files are invisible to everything: the watcher, the reconcile scan, files_index, files_events, the change feed, search, calendar activity and the Files listing. Filter them at the single entry point in calternal-fs/calternal-sync (one predicate, e.g. `calternal_path::is_internal_temp`, used everywhere; reuse the existing reserved-name list in calternal-path lib.rs:54). Better still: create temps where no watcher looks (e.g. `O_TMPFILE` + `linkat` on Linux, or a `.system/tmp` dir on the same filesystem, then renameat into place), so no scan can ever see them. Check that openat2 RESOLVE_BENEATH rules still hold. 2. A rename of temp → final is reported as ONE create/modify of the final path in the change feed, never create(temp) + create(final). 3. **Repair migration:** delete the index, events and change-feed-derived rows for any `.calternal-tmp-*` path, and emit a compensating delete to sync clients only if a create was ever published (so a client that recreated a temp can remove it). Idempotent. 4. **Hidden files (dot-files), owner question:** proposal: dot-files are stored and synced as they are ('file over app': e.g. `.obsidian/`), but they are **not shown** in Files, Calendar activity, Photos or Search by default. Files gets a 'Show hidden files' view toggle (⌘⇧. like Finder, persisted per user). Internal names (`.calternal*`, temps) never show, even with the toggle. 5. Tests: a concurrent test that hammers atomic writes while the watcher and reconcile run (1,000 writes, 0 temp rows in the index or the feed); a regression test for the rename-seen-as-create race; a sync-client test showing no temp path is ever downloaded. Add a case to tests/adversarial: rapid composer logs plus a scan storm. 6. Also check: does this race explain the owner's other report, where a composer log entry did not appear in the Calendar until a reload (#298)? If the day-file write is reported as a temp create, the Calendar live update may miss the real file. Coordinate on #298.

kayg commented

2026-09-28 07:09:18 +00:00

Author

Owner

Starting work on job/temp-index at base/head c9ec6aff84a9ec810124a2cfbdd07c189eea1b9c (current dev). I am tracing temp-file visibility through calternal-fs, calternal-sync, the Files Index/feed, and the repair migration. I will also check the reported Calendar live-update race on #298.

Starting work on `job/temp-index` at base/head `c9ec6aff84a9ec810124a2cfbdd07c189eea1b9c` (current `dev`). I am tracing temp-file visibility through calternal-fs, calternal-sync, the Files Index/feed, and the repair migration. I will also check the reported Calendar live-update race on #298.

kayg commented

2026-09-28 07:14:03 +00:00

Author

Owner

Confirmed root cause: Root::temp first tries O_TMPFILE, then links the inode into the destination folder as .calternal-tmp-<pid>-<counter> while bytes are written. Files startup/folder reconciliation enumerates that folder through Root::list; that list currently filters is_reserved only, and .calternal-tmp-* is not reserved. Root::write_inner emits an FsChange only for the final path after the journal install, so the temp records are from a reconciliation scan that overlaps the write. I will add one shared path predicate, filter the scan/list and reject direct access, and repair the stale Index/event/feed state in a migration.

Confirmed root cause: `Root::temp` first tries `O_TMPFILE`, then links the inode into the destination folder as `.calternal-tmp-<pid>-<counter>` while bytes are written. `Files` startup/folder reconciliation enumerates that folder through `Root::list`; that list currently filters `is_reserved` only, and `.calternal-tmp-*` is not reserved. `Root::write_inner` emits an `FsChange` only for the final path after the journal install, so the temp records are from a reconciliation scan that overlaps the write. I will add one shared path predicate, filter the scan/list and reject direct access, and repair the stale Index/event/feed state in a migration.

kayg referenced this issue

2026-09-28 07:14:45 +00:00

RESEARCH: standard file-transfer protocols for Home (WebDAV first; SFTP/FTPS/S3/others) — rootless, PhotoSync-ready (NO implementation) #306

kayg referenced this issue

2026-09-28 07:15:55 +00:00

E2E: get the gates e2e green — public note share heading, mode pill border, calendar log block (fix real regressions first) #298

kayg commented

2026-09-28 08:28:14 +00:00

Author

Owner

Confirmed the collision path and added the compatibility fix. Atomic Root::write links .calternal-tmp-* into the destination directory while it reads the payload, then publishes only the final path change. Files reconciliation previously listed that name; the local sync scanner could also inventory it.

The fix reserves the temp namespace in shared path validation and filters it from Filesystem lists, sync scans, and sync watcher events. Migration 14 removes stale Index/SSE rows and feed rows, then emits ordered compensating deletes for previously published identities. Sync accepts that reserved path only on delete operations and moves a legacy local copy to OS Trash before clearing its baseline, so failed cleanup retries.

The focused path and sync suites pass. The Files suite initially exposed two regressions during review: invalid traversal returned 404 before path validation, and an existing owner test needed to request show_hidden=true. Both are fixed without changing expected statuses or hidden-share rules. The new hidden-file API flag defaults to hidden and is ignored for Shared listings. The Files UI toggle remains a follow-up because this job owns only the server filter/API flag.

Confirmed the collision path and added the compatibility fix. Atomic `Root::write` links `.calternal-tmp-*` into the destination directory while it reads the payload, then publishes only the final path change. Files reconciliation previously listed that name; the local sync scanner could also inventory it. The fix reserves the temp namespace in shared path validation and filters it from Filesystem lists, sync scans, and sync watcher events. Migration 14 removes stale Index/SSE rows and feed rows, then emits ordered compensating deletes for previously published identities. Sync accepts that reserved path only on delete operations and moves a legacy local copy to OS Trash before clearing its baseline, so failed cleanup retries. The focused path and sync suites pass. The Files suite initially exposed two regressions during review: invalid traversal returned 404 before path validation, and an existing owner test needed to request `show_hidden=true`. Both are fixed without changing expected statuses or hidden-share rules. The new hidden-file API flag defaults to hidden and is ignored for Shared listings. The Files UI toggle remains a follow-up because this job owns only the server filter/API flag.

kayg commented

2026-09-28 08:50:43 +00:00

Author

Owner

Migration review found a feed-history edge and the repair now covers it. A temp may be represented by write, restore, a Shared path, or a current Index identity after older create rows were pruned. A move away from a temp also loses the visible destination when its old feed row is scrubbed. Migration 14 now snapshots all current/upsert identities, including active Share recipients, emits deletes for both old and temp move identities, and republishes a move destination as a create only when the current Index still contains that item there. I added regression coverage for these cases.

Migration review found a feed-history edge and the repair now covers it. A temp may be represented by `write`, `restore`, a Shared path, or a current Index identity after older create rows were pruned. A `move` away from a temp also loses the visible destination when its old feed row is scrubbed. Migration 14 now snapshots all current/upsert identities, including active Share recipients, emits deletes for both old and temp move identities, and republishes a move destination as a create only when the current Index still contains that item there. I added regression coverage for these cases.

kayg commented

2026-09-28 09:38:26 +00:00

Author

Owner

Completed Forgejo #305 on job/temp-index.

Head: 4c2102bfc21190e9107eeb0ad5c8f0c70e42ef33. Merged local dev once at a1207dcb. git push origin job/temp-index returned Everything up-to-date because the remote already had this head.

Built

Reserved .calternal-tmp-* paths as internal in calternal-path; filtered them from filesystem listings, sync scans/watch events, and Files index, SSE event, and durable feed publication.
Added the owner-only show_hidden=true Files listing API flag. It reveals normal hidden entries but never internal write temps; shared listings always omit hidden entries. The cursor carries the filter state.
Added Files migration 14 to remove legacy temp-derived rows and compensate published create/write/restore/share/move identities. If a temp-backed move still exists at a visible destination, the migration publishes the current destination identity.
Added handle-relative sync cleanup for legacy local temp copies, routed through the OS Trash. Sync only accepts an internal temp path for a feed delete.
Added a concurrent composer-write and Files-listing adversarial probe, plus migration, path, filesystem, and sync regressions.

Main files: crates/calternal-path/src/lib.rs, crates/calternal-fs/src/lib.rs, crates/calternal-fs/tests/storage.rs, crates/calternal-sync/src/{config.rs,engine.rs,local.rs}, crates/plugins/files/migrations/0014_internal_temporary_paths.sql, crates/plugins/files/src/{lib.rs,listing.rs,tests/hidden_entries.rs}, and tests/adversarial/attack2.py.

Gates

All final gates exited 0. Output excerpts below are copied verbatim.

cargo fmt --check
[no output]

cargo clippy --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 34s

cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 12s
test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 165.47s
test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.39s

bun run check
svelte-check found 0 errors and 0 warnings

bun run test
 Test Files  110 passed (110)
      Tests  709 passed (709)
   Duration  89.40s (transform 59%, environment 17%, import 14%, tests 7%, setup 3%)

Adversarial round: composer section only, against the local server. Exact summary:

server alive at end: True

==== ROUND 2 FINDINGS 0

==== ROUND 2 SLOW 0

The 1,000-write watcher/reconcile stress test passed in the Files suite. Findings for the related #298 calendar live-update report were posted to #298; that bug is separate from the temp-path race.

Gap and decisions

The Files UI hidden-file toggle remains a follow-up, as requested; the server filter and API flag are implemented. I treated the atomic write temp basename as reserved internal state even when show_hidden=true, and made legacy sync cleanup eligible only for server feed deletes. These choices fill gaps not specified in the design text.

Completed Forgejo #305 on `job/temp-index`. Head: `4c2102bfc21190e9107eeb0ad5c8f0c70e42ef33`. Merged local `dev` once at `a1207dcb`. `git push origin job/temp-index` returned `Everything up-to-date` because the remote already had this head. ## Built - Reserved `.calternal-tmp-*` paths as internal in `calternal-path`; filtered them from filesystem listings, sync scans/watch events, and Files index, SSE event, and durable feed publication. - Added the owner-only `show_hidden=true` Files listing API flag. It reveals normal hidden entries but never internal write temps; shared listings always omit hidden entries. The cursor carries the filter state. - Added Files migration 14 to remove legacy temp-derived rows and compensate published create/write/restore/share/move identities. If a temp-backed move still exists at a visible destination, the migration publishes the current destination identity. - Added handle-relative sync cleanup for legacy local temp copies, routed through the OS Trash. Sync only accepts an internal temp path for a feed delete. - Added a concurrent composer-write and Files-listing adversarial probe, plus migration, path, filesystem, and sync regressions. Main files: `crates/calternal-path/src/lib.rs`, `crates/calternal-fs/src/lib.rs`, `crates/calternal-fs/tests/storage.rs`, `crates/calternal-sync/src/{config.rs,engine.rs,local.rs}`, `crates/plugins/files/migrations/0014_internal_temporary_paths.sql`, `crates/plugins/files/src/{lib.rs,listing.rs,tests/hidden_entries.rs}`, and `tests/adversarial/attack2.py`. ## Gates All final gates exited 0. Output excerpts below are copied verbatim. ```text cargo fmt --check [no output] cargo clippy --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 34s cargo test Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 12s test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 165.47s test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.39s bun run check svelte-check found 0 errors and 0 warnings bun run test Test Files 110 passed (110) Tests 709 passed (709) Duration 89.40s (transform 59%, environment 17%, import 14%, tests 7%, setup 3%) ``` Adversarial round: `composer` section only, against the local server. Exact summary: ```text server alive at end: True ==== ROUND 2 FINDINGS 0 ==== ROUND 2 SLOW 0 ``` The 1,000-write watcher/reconcile stress test passed in the Files suite. Findings for the related #298 calendar live-update report were posted to #298; that bug is separate from the temp-path race. ## Gap and decisions The Files UI hidden-file toggle remains a follow-up, as requested; the server filter and API flag are implemented. I treated the atomic write temp basename as reserved internal state even when `show_hidden=true`, and made legacy sync cleanup eligible only for server feed deletes. These choices fill gaps not specified in the design text.

kayg referenced this issue from a commit

2026-09-28 09:41:24 +00:00

Merge job/temp-index (#305: atomic-write temps never indexed or synced; repair migration)

kayg referenced this issue from a commit

2026-09-28 09:54:50 +00:00

chore(api): regenerate contract for Files show_hidden (#305)

kayg closed this issue

2026-09-28 09:54:52 +00:00

kayg referenced this issue

2026-09-28 13:13:43 +00:00

WEBDAV: file WebDAV for Home (RFC 4918 class 1) — Finder, iOS Files, PhotoSync, rclone #339

kayg referenced this issue