Claim storage cleanup files privately before verification and removal

kayg commented

2026-10-28 19:22:31 +00:00

Owner

Replaces #740 for #642. This new branch continues the saved quarantine work at 90d8834635, including the interrupted rename and independent journal-proof fixes carried by that PR. The forge ignores pushes to an open PR branch, so the saved branch is preserved and this PR uses fix/storage-exact-file-quarantine-642-3b. Target: dev, recorded base d83e54dbe5.

Recovery previously checked an inode and then removed a public pathname. Another program could replace that pathname after the check, causing cleanup to remove unjournaled bytes. Cleanup now moves the entry into a fresh private directory on the source filesystem before verifying and removing it. Device/inode, length, read lease and byte/journal proof guard disposal. Unknown versions restore exclusively or remain as reported conflict copies. Upload, guarded-save, temporary-write and abandoned-directory cleanup use the same fsio quarantine interface.

This finishes the saved tests with cfg(test) channel pauses in the real cleanup paths. A worker blocks at the selected boundary while another thread atomically replaces the public name, then signals the worker to continue. Missing pauses fail the tests; there is no completion fallback. Both C fixtures and all preload use are removed. Hooks compile out of production builds. Ordinary-binary canonical before/after tests remain.

Change	Test
Move staging into a fresh private mode-0700 folder; remove through its pinned descriptor after proof	16 concurrent cases: original/recovery basename, equal/different length, before-check/before-move/after-move/after-check; replacement survives a second recovery
Reject a different inode between the earlier metadata check and the move	6 concurrent cancellation cases, including equal-length replacement
Claim attachment staging through its pinned parent before removal	6 concurrent sweep cases before move, after move and after proof
Restore exclusively or retain both versions on collision	staging_failed_proof_and_restore_collision_preserve_both_versions
Recover a crash-left quarantine without staging leftovers	staging_crash_quarantine_preserves_destination_and_journal_proof; upload_marker_still_finds_an_image_after_a_crash_during_publication
Keep independent journal proof; canonical equality cannot authorize deletion	Notes staging and ordinary-binary canonical before/after tests
Keep displaced versions during guarded save and rollback	displaced_replacement_and_rollback_replacement_keep_external_versions
Preserve acknowledged edits across process death and settle cleanup	Fresh 1,000 stops: 250 saves, 250 renames, 250 Trash operations, 250 restores; all acknowledged Unicode markers checked exactly once after every restart; no staging leftovers after explicit save barriers

All requested gates passed at ccc7a2b790. Gate output, verbatim:

fmt-write exit=0
fmt exit=0
fsio exit=0
clippy exit=0
workspace exit=0
earlier-races exit=0
canonical-races exit=0
binary exit=0
durability-1000 exit=0
final-fmt exit=0
final-fsio exit=0
final-clippy exit=0
final-workspace exit=0

Settled loop output, verbatim:

Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage.

Every local heavy command ran through build-slot.sh, one job at a time, with cargo -j3. Existing ignored tests remain ignored. The loop tests process death on local storage, not block-device power loss. The recovery pause was tightened to follow the final identity and byte checks; final-fmt, final-fsio, final-clippy and final-workspace revalidate that exact final test placement. No new latency claim is made.

Only product code and permanent tests are in the branch. Evidence and run scripts remain untracked in the job folder and in issue comments. No merge, deployment or issue closure.

Replaces #740 for #642. This new branch continues the saved quarantine work at 90d883463580bdd5b46dc8447a3317e8ea4c2fae, including the interrupted rename and independent journal-proof fixes carried by that PR. The forge ignores pushes to an open PR branch, so the saved branch is preserved and this PR uses fix/storage-exact-file-quarantine-642-3b. Target: dev, recorded base d83e54dbe571b625504001a8b6add90d44ca6161. Recovery previously checked an inode and then removed a public pathname. Another program could replace that pathname after the check, causing cleanup to remove unjournaled bytes. Cleanup now moves the entry into a fresh private directory on the source filesystem before verifying and removing it. Device/inode, length, read lease and byte/journal proof guard disposal. Unknown versions restore exclusively or remain as reported conflict copies. Upload, guarded-save, temporary-write and abandoned-directory cleanup use the same fsio quarantine interface. This finishes the saved tests with cfg(test) channel pauses in the real cleanup paths. A worker blocks at the selected boundary while another thread atomically replaces the public name, then signals the worker to continue. Missing pauses fail the tests; there is no completion fallback. Both C fixtures and all preload use are removed. Hooks compile out of production builds. Ordinary-binary canonical before/after tests remain. | Change | Test | | --- | --- | | Move staging into a fresh private mode-0700 folder; remove through its pinned descriptor after proof | 16 concurrent cases: original/recovery basename, equal/different length, before-check/before-move/after-move/after-check; replacement survives a second recovery | | Reject a different inode between the earlier metadata check and the move | 6 concurrent cancellation cases, including equal-length replacement | | Claim attachment staging through its pinned parent before removal | 6 concurrent sweep cases before move, after move and after proof | | Restore exclusively or retain both versions on collision | staging_failed_proof_and_restore_collision_preserve_both_versions | | Recover a crash-left quarantine without staging leftovers | staging_crash_quarantine_preserves_destination_and_journal_proof; upload_marker_still_finds_an_image_after_a_crash_during_publication | | Keep independent journal proof; canonical equality cannot authorize deletion | Notes staging and ordinary-binary canonical before/after tests | | Keep displaced versions during guarded save and rollback | displaced_replacement_and_rollback_replacement_keep_external_versions | | Preserve acknowledged edits across process death and settle cleanup | Fresh 1,000 stops: 250 saves, 250 renames, 250 Trash operations, 250 restores; all acknowledged Unicode markers checked exactly once after every restart; no staging leftovers after explicit save barriers | All requested gates passed at ccc7a2b790890d661de8d16d6ca1f8632e93efde. Gate output, verbatim: ```text fmt-write exit=0 fmt exit=0 fsio exit=0 clippy exit=0 workspace exit=0 earlier-races exit=0 canonical-races exit=0 binary exit=0 durability-1000 exit=0 final-fmt exit=0 final-fsio exit=0 final-clippy exit=0 final-workspace exit=0 ``` Settled loop output, verbatim: ```text Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage. ``` Every local heavy command ran through build-slot.sh, one job at a time, with cargo -j3. Existing ignored tests remain ignored. The loop tests process death on local storage, not block-device power loss. The recovery pause was tightened to follow the final identity and byte checks; final-fmt, final-fsio, final-clippy and final-workspace revalidate that exact final test placement. No new latency claim is made. Only product code and permanent tests are in the branch. Evidence and run scripts remain untracked in the job folder and in issue comments. No merge, deployment or issue closure.

kayg added 4 commits

2026-10-28 19:22:32 +00:00

fix: recover interrupted note renames and staged saves

CI / server (pull_request) Successful in 57s

Details

CI / rust (pull_request) Successful in 20m26s

Details

CI / web (pull_request) Failing after 2h59m10s

Details

66f27ce3cd

Preserve unjournaled staging bytes across canonical file races

CI / server (pull_request) Successful in 1m18s

Details

CI / rust (pull_request) Successful in 15m56s

Details

CI / web (pull_request) Has been cancelled

Details

e46ce1c306

wip(storage): move-then-verify quarantine for staging cleanup (unfinished) 90d8834635

Saved by the coordinator after the job stopped early; gates not yet run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Cj2Q1hCskP8Frsk6X7EHw

test(storage): verify quarantine cleanup at concurrent replacement boundaries

CI / server (pull_request) Successful in 1m18s

Details

CI / rust (pull_request) Successful in 21m5s

Details

CI / web (pull_request) Has been cancelled

Details

ccc7a2b790

kayg referenced this pull request

2026-10-28 19:22:34 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg referenced this pull request

2026-10-28 19:22:34 +00:00

Recover interrupted storage operations without discarding external staging bytes #740

kayg referenced this pull request

2026-10-28 19:23:45 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg referenced this pull request

2026-10-28 19:33:10 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg commented

2026-10-28 19:34:26 +00:00

Author

Owner

Verdict: fix first. Independent review of PR #742 at ccc7a2b790, into dev. Two reproducible ways to lose completed external writes remain, plus one unmet restoration requirement. No product edits, commits, pushes, merges or deployments remain from this review.

P1: a crash during attachment cleanup loses the original identity proof; the next startup deletes an external replacement. In fsio/src/beneath.rs:145-149, cleanup records the original inode, then claims the public staging name. An independent Python process atomically replaced that name with fsynced external bytes at BeforeMove. I SIGKILLed cleanup at AfterMove, before it could reject the changed inode. On restart, remove_staging recursively visits the crash-left quarantine (:160-165), records the replacement as a fresh baseline and discards it using the always-true byte predicate. The pending marker contains only a pathname, so the original authorization is not recoverable. This was reproduced through the ordinary server binary's real startup sweeper as well as the unit helper. No writer touched private metadata.

Reproduction: create a pending upload marker naming Documents/2026/2026-10-28/.calternal-tmp-742-Note.md; replace that public file with an external inode at BeforeMove; kill cleanup at AfterMove; start the ordinary binary on the same home/state. Before restart the sole retained file contains the external bytes. After restart, verbatim: Real startup sweeper: marker remains= False ; external-byte survivors= []. The preservation assertion exits 1. The unit reproduction reports restart removed=true, surviving images=0.
P1: equal-length in-place external rewrites are deleted even without a crash. The attachment reaper, TempFile::Drop and failed TempFile::persist pass |_| true as the content proof (beneath.rs:149, :272 and :252). Pause after capturing metadata but before the move. Another process opens the original public inode with r+b, writes outside-newer over journal-known (both 13 bytes), fsyncs and closes. Resume. Device, inode and length still match, and the newly acquired read lease only freezes the already changed bytes. All three upload cleanup paths delete those bytes. The guarded note cancellation and journal-recovery controls correctly retain them.

Verbatim observations: equal-length rewrite attachment: surviving images=0; equal-length rewrite upload-drop: surviving images=0; equal-length rewrite upload-reject: surviving images=0. The independent test fails with cleanup deleted completed, fsynced outside rewrites in ["attachment", "upload-drop", "upload-reject"]. Recheck exact original byte authorization, rather than accepting all bytes on the same inode; conservatively retain files when that proof is unavailable.
P2: unknown crash-left note images never return from the hidden private folder. Kill recover_staging at AfterMove with unknown bytes. On next start, enumerate the moved image and reject its journal proof. staging.rs:123 restores it to its input path, which is already inside the crash-left quarantine. The bytes survive, but remain at .calternal-tmp-<random>-quarantine/.calternal-tmp-742-Note.md rather than returning to an adjacent recovery/conflict path. Collisions likewise retain versions in random private directories. This fails the job's explicit next-start restoration and predictable conflict-copy requirements; it is not a byte-loss claim. Twelve colliding versions survived three subsequent recovery passes at stable retained paths and did not overwrite each other.

The independent expanded unit suite finished 10 passed, 3 failed. All failures are preservation assertions for the three ranked findings, not fixture startup failures. The tests use cfg(test) pauses and separate-process writers, with a required handshake before each SIGKILL. No preloaded libraries, syscall interposition or fixed sleeps were used.

Checks that passed: 18 public atomic-replacement cases across recovery, guarded cancellation, attachment reaping, upload drop, failed upload publication and accepted guarded replacement; nine known-image deaths at BeforeMove/AfterMove/AfterCheck; both upload-publication move boundaries; first and second rollback exchange boundaries; import-directory replacement boundaries; exclusive rename, named bulk, anonymous bulk, guarded-create and directory publication; private cancellation; repeated conflict preservation. The original fsio suite passed all 49 tests, note staging passed 3, rename crash boundaries passed 3, split-save acknowledged-byte recovery passed both prepare/place phases, and anchors passed 6.

The ordinary binary excluded quarantine content from note lists, settled search, journal catch-up, notes.get and live WebSocket note-room joins. Both ordinary markdown basenames and hidden staging basenames were checked.

A fresh 1,000/1,000 stop loop also passed: 250 saves, 250 renames, 250 Trash operations and 250 restores. All 1,000 acknowledged Unicode markers survived exactly once after every restart; same-key retry jobs completed; auxiliary path/title/body checks passed; final canonical bytes equaled notes.get; no staging files or directories remained after restart and explicit save barriers in any round. This loop kills at the after-send boundary; it does not prove which kernel syscall was interrupted. The independent pause tests provide that separate coverage. The passing loop does not discharge the external-writer failures.

Every local heavy command ran sequentially under build-slot.sh, with cargo -j3. The audit inventory distinguishes guarded note writes from explicitly unconditional application-metadata replacement and caller-authorized deletion. No claim is made about hostile writes into private metadata, non-Linux cleanup or block-device power failure. Access to the earlier reviewer host failed (DNS/refused connection/timeout); earlier reviewer findings were obtained from Forgejo comments instead of those remote files.

Evidence and exact reproduction instructions: /home/agent/wt/verify-742/findings.md and .verification/REPRODUCE.md. The temporary test patch is .verification/review-tests.patch; it applies cleanly to the reviewed head. Tracked files have been restored byte for byte; only uncommitted review artifacts remain.

Head: ccc7a2b790. Local review branch: verify/742-independent. Gate output, verbatim:

fsio-original exit=0
notes-staging exit=0
notes-renames exit=0
notes-storage-crash exit=0
binary exit=0
visibility exit=0
upload-startup-preservation exit=1
durability-smoke exit=0
independent-initial exit=101
durability-1000 exit=0
independent-expanded exit=101
fsio-final exit=0
anchors exit=0
tracked-tree-clean exit=0
reproduction-patch-check exit=0

independent-initial exit=101 is the first pair of preservation failures. independent-expanded exit=101 is the final three preservation failures. upload-startup-preservation exit=1 is the actual external-byte loss through the ordinary binary. None of these are safety passes. No issue was closed.

Verdict: **fix first**. Independent review of PR #742 at ccc7a2b790890d661de8d16d6ca1f8632e93efde, into dev. Two reproducible ways to lose completed external writes remain, plus one unmet restoration requirement. No product edits, commits, pushes, merges or deployments remain from this review. 1. **P1: a crash during attachment cleanup loses the original identity proof; the next startup deletes an external replacement.** In fsio/src/beneath.rs:145-149, cleanup records the original inode, then claims the public staging name. An independent Python process atomically replaced that name with fsynced external bytes at BeforeMove. I SIGKILLed cleanup at AfterMove, before it could reject the changed inode. On restart, remove_staging recursively visits the crash-left quarantine (:160-165), records the replacement as a fresh baseline and discards it using the always-true byte predicate. The pending marker contains only a pathname, so the original authorization is not recoverable. This was reproduced through the ordinary server binary's real startup sweeper as well as the unit helper. No writer touched private metadata. Reproduction: create a pending upload marker naming Documents/2026/2026-10-28/.calternal-tmp-742-Note.md; replace that public file with an external inode at BeforeMove; kill cleanup at AfterMove; start the ordinary binary on the same home/state. Before restart the sole retained file contains the external bytes. After restart, verbatim: `Real startup sweeper: marker remains= False ; external-byte survivors= []`. The preservation assertion exits 1. The unit reproduction reports `restart removed=true, surviving images=0`. 2. **P1: equal-length in-place external rewrites are deleted even without a crash.** The attachment reaper, TempFile::Drop and failed TempFile::persist pass `|_| true` as the content proof (beneath.rs:149, :272 and :252). Pause after capturing metadata but before the move. Another process opens the original public inode with r+b, writes `outside-newer` over `journal-known` (both 13 bytes), fsyncs and closes. Resume. Device, inode and length still match, and the newly acquired read lease only freezes the already changed bytes. All three upload cleanup paths delete those bytes. The guarded note cancellation and journal-recovery controls correctly retain them. Verbatim observations: `equal-length rewrite attachment: surviving images=0`; `equal-length rewrite upload-drop: surviving images=0`; `equal-length rewrite upload-reject: surviving images=0`. The independent test fails with `cleanup deleted completed, fsynced outside rewrites in ["attachment", "upload-drop", "upload-reject"]`. Recheck exact original byte authorization, rather than accepting all bytes on the same inode; conservatively retain files when that proof is unavailable. 3. **P2: unknown crash-left note images never return from the hidden private folder.** Kill recover_staging at AfterMove with unknown bytes. On next start, enumerate the moved image and reject its journal proof. staging.rs:123 restores it to its input path, which is already inside the crash-left quarantine. The bytes survive, but remain at `.calternal-tmp-<random>-quarantine/.calternal-tmp-742-Note.md` rather than returning to an adjacent recovery/conflict path. Collisions likewise retain versions in random private directories. This fails the job's explicit next-start restoration and predictable conflict-copy requirements; it is not a byte-loss claim. Twelve colliding versions survived three subsequent recovery passes at stable retained paths and did not overwrite each other. The independent expanded unit suite finished **10 passed, 3 failed**. All failures are preservation assertions for the three ranked findings, not fixture startup failures. The tests use cfg(test) pauses and separate-process writers, with a required handshake before each SIGKILL. No preloaded libraries, syscall interposition or fixed sleeps were used. Checks that passed: 18 public atomic-replacement cases across recovery, guarded cancellation, attachment reaping, upload drop, failed upload publication and accepted guarded replacement; nine known-image deaths at BeforeMove/AfterMove/AfterCheck; both upload-publication move boundaries; first and second rollback exchange boundaries; import-directory replacement boundaries; exclusive rename, named bulk, anonymous bulk, guarded-create and directory publication; private cancellation; repeated conflict preservation. The original fsio suite passed all 49 tests, note staging passed 3, rename crash boundaries passed 3, split-save acknowledged-byte recovery passed both prepare/place phases, and anchors passed 6. The ordinary binary excluded quarantine content from note lists, settled search, journal catch-up, notes.get and live WebSocket note-room joins. Both ordinary markdown basenames and hidden staging basenames were checked. A fresh **1,000/1,000** stop loop also passed: 250 saves, 250 renames, 250 Trash operations and 250 restores. All 1,000 acknowledged Unicode markers survived exactly once after every restart; same-key retry jobs completed; auxiliary path/title/body checks passed; final canonical bytes equaled notes.get; no staging files or directories remained after restart and explicit save barriers in any round. This loop kills at the after-send boundary; it does not prove which kernel syscall was interrupted. The independent pause tests provide that separate coverage. The passing loop does not discharge the external-writer failures. Every local heavy command ran sequentially under build-slot.sh, with cargo -j3. The audit inventory distinguishes guarded note writes from explicitly unconditional application-metadata replacement and caller-authorized deletion. No claim is made about hostile writes into private metadata, non-Linux cleanup or block-device power failure. Access to the earlier reviewer host failed (DNS/refused connection/timeout); earlier reviewer findings were obtained from Forgejo comments instead of those remote files. Evidence and exact reproduction instructions: /home/agent/wt/verify-742/findings.md and .verification/REPRODUCE.md. The temporary test patch is .verification/review-tests.patch; it applies cleanly to the reviewed head. Tracked files have been restored byte for byte; only uncommitted review artifacts remain. Head: ccc7a2b790890d661de8d16d6ca1f8632e93efde. Local review branch: verify/742-independent. Gate output, verbatim: ```text fsio-original exit=0 notes-staging exit=0 notes-renames exit=0 notes-storage-crash exit=0 binary exit=0 visibility exit=0 upload-startup-preservation exit=1 durability-smoke exit=0 independent-initial exit=101 durability-1000 exit=0 independent-expanded exit=101 fsio-final exit=0 anchors exit=0 tracked-tree-clean exit=0 reproduction-patch-check exit=0 ``` `independent-initial exit=101` is the first pair of preservation failures. `independent-expanded exit=101` is the final three preservation failures. `upload-startup-preservation exit=1` is the actual external-byte loss through the ordinary binary. None of these are safety passes. No issue was closed.

kayg referenced this pull request

2026-10-28 19:34:27 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg commented

2026-10-28 19:36:14 +00:00

Author

Owner

Starting storage-fix-4 on branch fix/storage-exact-file-quarantine-642-4, base ccc7a2b790. Fetched origin first. The assigned unsuffixed branch already has a published PR; using a new branch as required because this forge ignores pushes to open PR branches. Scope: one durable ownership and move-then-verify cleanup primitive, all cleanup callers, permanent regressions and required gates. Evidence stays in issue comments.

Starting storage-fix-4 on branch fix/storage-exact-file-quarantine-642-4, base ccc7a2b790890d661de8d16d6ca1f8632e93efde. Fetched origin first. The assigned unsuffixed branch already has a published PR; using a new branch as required because this forge ignores pushes to open PR branches. Scope: one durable ownership and move-then-verify cleanup primitive, all cleanup callers, permanent regressions and required gates. Evidence stays in issue comments.

kayg commented

2026-10-28 19:38:58 +00:00

Author

Owner

Finding: cleanup had no durable creation record. Folder::remove_staging re-baselined current inode metadata on each restart, and both TempFile cancellation paths accepted every byte on that inode. Existing fsio::remove_file, Folder::remove, metadata replacement, and recursive import-cache disposal bypassed even those checks. Evidence: beneath.rs passed |_| true at all three upload cleanup sites; lib.rs directly called std::fs::remove_file and std::fs::rename; directory.rs used remove_dir_all. The new shared check requires a creator-written inode receipt (device, inode, size, blake3) as well as a read lease. Streaming writes maintain their own incremental hash rather than reading current public bytes to authorize deletion. Unknown crash-left private entries are returned to exclusive, predictable conflict names. Reviewer host SSH timed out; exact reviewer evidence was read from comment 68130.

kayg referenced this pull request

2026-10-28 19:45:43 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg commented

2026-10-28 19:45:43 +00:00

Author

Owner

Additional finding: creation proof must stay tied to its public name. An inode receipt alone can follow a different app-created file moved in by another program. Receipts now include that binding, and only fsio exclusive moves/exchanges can update it; public cleanup still verifies device, inode, size and content hash under a lease. The new shared concurrency test and all three #742 regressions passed before this strengthening; they are being rerun with it. Performance measurement is blocked: ssh to the designated 10by32-01-std returns "Could not resolve hostname 10by32-01-std: No address associated with hostname". No performance command ran on any other VM or production. Required local gates continue through build-slot.sh.

kayg commented

2026-10-28 19:54:38 +00:00

Author

Owner

Gate finding: the initial centralized verifier loaded every moved file into a Vec. The existing bounded-memory upload regression correctly rejected that implementation. Verbatim: "peak above idle: 32 MiB upload 34920 KiB, 128 MiB upload 132212 KiB"; workspace exit=101 at attachments_crash::a_big_upload_streams_in_bounded_memory. Receipt-only cleanup and upload publication now use a streaming hash with a fixed 64 KiB buffer; note journal predicates retain their bounded note image check. Missing or identity/name-mismatched receipts are rejected before allocating content. The preservation, abrupt-stop, three reviewer repro and ordinary-binary startup gates all passed before this change; affected checks will be repeated. This failure is not reported as a safety pass.

kayg commented

2026-10-28 20:00:29 +00:00

Author

Owner

Settled-loop finding: restart 790 failed with a remaining staging file at Notes/quarantine.conflict-27/.calternal-tmp-24d9af18e2c9d87f4aa7d6-Durable text.md. All acknowledged markers had survived to that point; the failure was the no-hidden-staging assertion. The shared startup drain correctly skipped a locked nested transaction during recursion but then treated that same directory as an unknown entry in its second pass and moved it. That invalidated the active recovery worker's path. A deterministic two-worker regression is being added before the central fix; the full 1,000-stop loop will restart from a fresh Home. The 790-stop attempt is a failed gate, not a pass.

kayg commented

2026-10-28 20:01:55 +00:00

Author

Owner

Central fix verified: the drain now leaves managed nested transaction directories to their lock-holding recovery worker instead of moving them as unknown objects. The new startup_drain_does_not_reparent_an_active_nested_recovery test failed on the prior code with NotFound and passes with the fix. All 59 fsio tests and both shared 14-path properties pass. The streaming verifier also passes the existing ordinary-binary upload-memory test. Verbatim: "peak above idle: 32 MiB upload 2668 KiB, 128 MiB upload 432 KiB". The prior failing implementation measured 34920 KiB and 132212 KiB. Final workspace tests and a fresh 1,000-stop loop are running sequentially in one build slot. Current gate output, verbatim:

fmt-write exit=0
fmt exit=0
clippy exit=0
nested-recovery exit=0
fsio exit=0
property exit=0
reviewer-repros exit=0
binary-repros exit=0
upload-memory exit=0

Central fix verified: the drain now leaves managed nested transaction directories to their lock-holding recovery worker instead of moving them as unknown objects. The new startup_drain_does_not_reparent_an_active_nested_recovery test failed on the prior code with NotFound and passes with the fix. All 59 fsio tests and both shared 14-path properties pass. The streaming verifier also passes the existing ordinary-binary upload-memory test. Verbatim: "peak above idle: 32 MiB upload 2668 KiB, 128 MiB upload 432 KiB". The prior failing implementation measured 34920 KiB and 132212 KiB. Final workspace tests and a fresh 1,000-stop loop are running sequentially in one build slot. Current gate output, verbatim: ```text fmt-write exit=0 fmt exit=0 clippy exit=0 nested-recovery exit=0 fsio exit=0 property exit=0 reviewer-repros exit=0 binary-repros exit=0 upload-memory exit=0 ```

kayg commented

2026-10-28 20:03:07 +00:00

Author

Owner

Workspace fixture finding: kill_nine_mid_upload_leaves_nothing_half_written rejected a visible 4,321,381-byte conflict prefix from an earlier interrupted round: Photos/2026/2026-10-17/calternal-tmp-740c954db6214182bad2b0307a7e0a64.conflict-1. A process can die between a streaming write and its receipt update; the exact prefix then has no valid durable hash and must be preserved under the requested invariant. The fixture is being corrected to exclude explicit cleanup conflict copies from its completed-attachment assertion, while continuing to forbid publication under an attachment name and requiring the hidden staging/marker cleanup to settle. No product check is being weakened to delete those unproven bytes. Gate output for that run: workspace exit=101. The streaming-memory check and 59 fsio tests passed, including the new nested recovery regression.

kayg commented

2026-10-28 20:12:29 +00:00

Author

Owner

Execution note: during early investigation I started a targeted fsio/clippy job before the first workspace run had returned. Both commands were wrapped in build-slot.sh, but this briefly held two slots, contrary to the one-slot limit. I corrected the orchestration; the final gate scripts hold one slot and run their commands sequentially. No unrelated process was stopped, and no production or staging VM was touched.

kayg commented

2026-10-28 20:18:42 +00:00

Author

Owner

Fresh settled loop passed 1,000/1,000 stops after the nested-recovery fix: 250 saves, 250 renames, 250 Trash operations and 250 restores. All 1,000 acknowledged Unicode markers survived exactly once after every restart; same-key retries finished, auxiliary title/path/body checks passed, final canonical bytes equal notes.get, and no staging files or directories remained after explicit save barriers in any round. This is after-send randomized SIGKILL coverage; the product-pause property provides the exact cleanup-boundary coverage. A final run now checks the added after-check alias breakers, the receipt flush, lint and workspace. Verbatim loop gate output:

fmt-write exit=0
fmt exit=0
clippy exit=0
nested-recovery exit=0
fsio exit=0
property exit=0
reviewer-repros exit=0
binary-repros exit=0
upload-memory exit=0
workspace exit=0
binary exit=0
durability-1000 exit=0
diff-check exit=0

Fresh settled loop passed 1,000/1,000 stops after the nested-recovery fix: 250 saves, 250 renames, 250 Trash operations and 250 restores. All 1,000 acknowledged Unicode markers survived exactly once after every restart; same-key retries finished, auxiliary title/path/body checks passed, final canonical bytes equal notes.get, and no staging files or directories remained after explicit save barriers in any round. This is after-send randomized SIGKILL coverage; the product-pause property provides the exact cleanup-boundary coverage. A final run now checks the added after-check alias breakers, the receipt flush, lint and workspace. Verbatim loop gate output: ```text fmt-write exit=0 fmt exit=0 clippy exit=0 nested-recovery exit=0 fsio exit=0 property exit=0 reviewer-repros exit=0 binary-repros exit=0 upload-memory exit=0 workspace exit=0 binary exit=0 durability-1000 exit=0 diff-check exit=0 ```

kayg commented

2026-10-28 20:21:35 +00:00

Author

Owner

Expanded-property finding: all foreign bytes survived, but AtomicReplace / AfterMove / in-place rewrite returned the conflict as .calternal-tmp-742-Note.md.conflict-1, which is still hidden. The conflict formatter stripped dots in its fallback branch but did not strip a dot inherited from staging_destination. The deterministic expanded property failed with that exact path. The shared formatter now strips leading dots in both cases, and a permanent hidden-metadata replacement test covers .prefs.json as well. Final gates and a new fresh-Home 1,000-stop loop are being repeated. No unproven bytes are deleted to satisfy the visibility assertion.

kayg commented

2026-10-28 20:24:12 +00:00

Author

Owner

The expanded 14-path property now passes with after-check alias breakers and guaranteed equal-length payloads. Workspace found one more old interrupted-upload assertion that I missed: its final files(data) equality still expected only whole.jpg and no retained conflicts. The preserved 6,017,125- and 6,439,141-byte prefixes are required when no exact durable receipt survived. The test now separately checks that whole.jpg is the only published attachment, every visible conflict is an exact prefix of the submitted 8 MiB input, no hidden staging remains, and the completed attachment is byte-for-byte correct. This is a fixture correction, not permission to delete unproven prefixes. The previous workspace gate exited 101 at that assertion; the final run is restarting.

kayg referenced this pull request

2026-10-28 20:40:33 +00:00

Require creator receipts for all storage cleanup (#642) #744

kayg referenced this pull request

2026-10-28 20:40:34 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg commented

2026-10-28 20:40:35 +00:00

Author

Owner

Finished storage-fix-4. Head c19aef3072 on fix/storage-exact-file-quarantine-642-4. New PR: #744. The new PR is required because pushes to the existing #742 branch are ignored by this forge.

Call sites now using the shared removal and replacement boundary:

fsio: write_with error cleanup and metadata replacement; write_bulk_temp error cleanup; replace_many replacement and error cleanup; remove_file; exclusive moves; Prepared::Drop; guarded-create alias cleanup; after_swap displaced-image cleanup and rollback; Folder::remove; Folder::remove_staging; TempFile::Drop and failed persist; upload publication; PreparedDirectory::Drop; failed directory preparation; clean_prepared_directories; recover_staging; empty staging-directory cleanup.
Notes: Trash note/details/link purge and restore detail cleanup through the existing remove_file entry point; deletion::unlink through that entry point; trash::links::remember, discard and repair_file; bulk::creates::place rollback; MetaDb::open_with cache reset. Trash details now use a recorded atomic writer rather than a plain write.
Attachments: Pending::Drop, sweep_at marker cleanup, remove_temp through Folder::remove_staging. The existing startup Home enumeration also drains abandoned private transactions without adopting a new identity, including when a pending marker is absent.
Other callers covered by the workspace-wide ban: Cards::remove_many, backdrop Cache::trim, search schema::open cache reset, InviteKey::open temporary-key cleanup and admin::start_inner stale socket cleanup. Nonregular or unrecorded entries are retained as conflicts rather than unlinked.

Creation proofs are written by RecordedFile, including the upload Receiver's bounded chunks, and by the anonymous bulk and lock-file creation paths. Device, inode, size, BLAKE3 and the public-name binding travel with the inode in a durable xattr. Exclusive app moves can change the binding; an external move cannot transfer deletion authority. A held read lease covers the final verification and unlink. The verifier flushes an existing creator receipt before using it, including cancellation before the writer reached its final flush. Unknown bytes return to exclusively numbered visible conflict names. Locked live private directories are skipped by startup recovery.

The generic preservation property covers 14 entry points at BeforeCheck, BeforeMove, AfterMove and AfterCheck, with independent Python writers, generated binary payloads, inode replacement and completed in-place rewrites. In-place aliases are tested both before acquisition and as lease breakers at the final check; the alias is removed after the writer finishes so it cannot conceal a missing survivor. Every path also uses an explicit equal-length payload. The same 14-entry table also runs abrupt child-process kills at all four pauses. Tests separately cover an app-created replacement carrying another valid receipt, unknown import-cache bytes, live prepared-image exclusion, all three #742 repros and the ordinary startup binary with and without markers.

The retained private namespaces stay on the source filesystem inside each Home. Unsupported receipt or lease operations fail conservatively. These Linux tests cover process death and ordinary outside writers, not hostile changes to private application metadata or block-device power cuts. The designated performance VM cannot be resolved through either its direct or NetBird hostname, so there is no before/after latency claim.

Gate output, verbatim:

fmt-write exit=0
fmt exit=0
clippy exit=0
nested-recovery exit=0
fsio exit=0
property exit=0
reviewer-repros exit=0
binary-repros exit=0
upload-memory exit=0
workspace exit=0
binary exit=0
durability-1000 exit=0
diff-check exit=0

Stop-loop result, verbatim:

Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage.

The final pass includes the memory correction, nested active-recovery regression, visible conflict naming and the interrupted-upload prefix checks. Earlier failed gates are recorded in the preceding comments. All final gates passed. No evidence or artifacts were committed; no issue was closed; no merge or deployment was performed.

Finished storage-fix-4. Head c19aef30723397e355adf1d3e82c9e24e862466c on fix/storage-exact-file-quarantine-642-4. New PR: https://git.kayg.org/kayg/calternal/pulls/744. The new PR is required because pushes to the existing #742 branch are ignored by this forge. Call sites now using the shared removal and replacement boundary: - fsio: write_with error cleanup and metadata replacement; write_bulk_temp error cleanup; replace_many replacement and error cleanup; remove_file; exclusive moves; Prepared::Drop; guarded-create alias cleanup; after_swap displaced-image cleanup and rollback; Folder::remove; Folder::remove_staging; TempFile::Drop and failed persist; upload publication; PreparedDirectory::Drop; failed directory preparation; clean_prepared_directories; recover_staging; empty staging-directory cleanup. - Notes: Trash note/details/link purge and restore detail cleanup through the existing remove_file entry point; deletion::unlink through that entry point; trash::links::remember, discard and repair_file; bulk::creates::place rollback; MetaDb::open_with cache reset. Trash details now use a recorded atomic writer rather than a plain write. - Attachments: Pending::Drop, sweep_at marker cleanup, remove_temp through Folder::remove_staging. The existing startup Home enumeration also drains abandoned private transactions without adopting a new identity, including when a pending marker is absent. - Other callers covered by the workspace-wide ban: Cards::remove_many, backdrop Cache::trim, search schema::open cache reset, InviteKey::open temporary-key cleanup and admin::start_inner stale socket cleanup. Nonregular or unrecorded entries are retained as conflicts rather than unlinked. Creation proofs are written by RecordedFile, including the upload Receiver's bounded chunks, and by the anonymous bulk and lock-file creation paths. Device, inode, size, BLAKE3 and the public-name binding travel with the inode in a durable xattr. Exclusive app moves can change the binding; an external move cannot transfer deletion authority. A held read lease covers the final verification and unlink. The verifier flushes an existing creator receipt before using it, including cancellation before the writer reached its final flush. Unknown bytes return to exclusively numbered visible conflict names. Locked live private directories are skipped by startup recovery. The generic preservation property covers 14 entry points at BeforeCheck, BeforeMove, AfterMove and AfterCheck, with independent Python writers, generated binary payloads, inode replacement and completed in-place rewrites. In-place aliases are tested both before acquisition and as lease breakers at the final check; the alias is removed after the writer finishes so it cannot conceal a missing survivor. Every path also uses an explicit equal-length payload. The same 14-entry table also runs abrupt child-process kills at all four pauses. Tests separately cover an app-created replacement carrying another valid receipt, unknown import-cache bytes, live prepared-image exclusion, all three #742 repros and the ordinary startup binary with and without markers. The retained private namespaces stay on the source filesystem inside each Home. Unsupported receipt or lease operations fail conservatively. These Linux tests cover process death and ordinary outside writers, not hostile changes to private application metadata or block-device power cuts. The designated performance VM cannot be resolved through either its direct or NetBird hostname, so there is no before/after latency claim. Gate output, verbatim: ```text fmt-write exit=0 fmt exit=0 clippy exit=0 nested-recovery exit=0 fsio exit=0 property exit=0 reviewer-repros exit=0 binary-repros exit=0 upload-memory exit=0 workspace exit=0 binary exit=0 durability-1000 exit=0 diff-check exit=0 ``` Stop-loop result, verbatim: ```text Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage. ``` The final pass includes the memory correction, nested active-recovery regression, visible conflict naming and the interrupted-upload prefix checks. Earlier failed gates are recorded in the preceding comments. All final gates passed. No evidence or artifacts were committed; no issue was closed; no merge or deployment was performed.

kayg referenced this pull request

2026-10-28 22:23:29 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg referenced this pull request

2026-10-28 22:26:05 +00:00

Split storage: user files on HDD, app data on SSD; perf gates on that layout #642

kayg referenced this pull request

2026-11-04 13:04:31 +00:00

Recover interrupted renames and abandoned note save images #737

kayg referenced this pull request

2026-11-04 13:04:32 +00:00

Recover interrupted storage operations without discarding external staging bytes #740