kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Recover interrupted storage operations without discarding external staging bytes #740

Closed
kayg wants to merge 2 commits from fix/storage-staging-cleanup-race into dev
pull from: fix/storage-staging-cleanup-race
merge into: kayg:dev
Conversation 7 Commits 2 Files changed 20 +877 -6
kayg commented 2026-10-28 18:37:46 +00:00
Owner
Copy link

Related to #642. This replaces #737 on a new branch because this forge ignores pushes to an open PR branch. It carries forward that PR's interrupted rename recovery and staging cleanup.

A staging image could be the only remaining copy of a newer external edit. Recovery read matching canonical bytes, another program restored older canonical bytes during fsync, and cleanup then deleted the staging image. Even a final stable snapshot cannot protect against a replacement immediately after that check.

Recovery now requires independent durable journal evidence before deleting nonempty staging bytes. It no longer uses the mutable canonical pathname as disposal evidence. Unknown bytes remain at their staging path and are reported as a staging conflict copy, without importing a duplicate note ID. Exclusive staging claims, read leases, byte checks and directory flushes remain. The unsafe canonical comparison helper is removed. This changes startup recovery only; the save hot path is unchanged.

The permanent test reproduces the reviewer's FIFO/fsync race against the ordinary binary. It detects loss against the original head and passes against this head. Cases cover in-place rewrite and pathname replacement before recovery, the former canonical comparison boundary, and immediately after recovery. Since the unsafe check is removed, that boundary case races at completion when no canonical fsync occurs. There are no timing sleeps or product pause points. A second test checks equal unjournaled bytes and repeated recovery.

All final gates passed at e46ce1c306. The settled 1,000-stop loop completed 250 saves, 250 renames, 250 Trash operations and 250 restores. All 1,000 acknowledged Unicode markers survived exactly once after every restart, retry jobs completed, final canonical bytes matched notes.get, and no staging files remained after explicit save barriers in any round.

The initial workspace and canonical-race failures were a test-fixture mistake: notes.create lacked its required request key and returned HTTP 400 before recovery. The fixture was corrected and both gates passed on rerun. The original-head expected-loss gate means the regression successfully detected the original defect. Existing ignored tests remain ignored. These are process-death tests on local storage, not block-device power-loss tests.

Gate output, verbatim:

fmt exit=0
clippy exit=0
workspace-tests exit=101
staging-race exit=0
canonical-race exit=101
build-binary exit=0
durability-1000-settled exit=0
reviewer-repro-original-head expected-loss-observed exit=0
canonical-race-corrected exit=0
workspace-tests-corrected exit=0
fmt-final exit=0
clippy-final exit=0

All local heavy commands ran through one build slot at a time, with cargo -j3. Only product code and permanent tests are committed. Logs and gate evidence remain uncommitted under /home/agent/wt/storage-fix-1/.verification/. No merge or deployment.

Related to #642. This replaces #737 on a new branch because this forge ignores pushes to an open PR branch. It carries forward that PR's interrupted rename recovery and staging cleanup. A staging image could be the only remaining copy of a newer external edit. Recovery read matching canonical bytes, another program restored older canonical bytes during fsync, and cleanup then deleted the staging image. Even a final stable snapshot cannot protect against a replacement immediately after that check. Recovery now requires independent durable journal evidence before deleting nonempty staging bytes. It no longer uses the mutable canonical pathname as disposal evidence. Unknown bytes remain at their staging path and are reported as a staging conflict copy, without importing a duplicate note ID. Exclusive staging claims, read leases, byte checks and directory flushes remain. The unsafe canonical comparison helper is removed. This changes startup recovery only; the save hot path is unchanged. The permanent test reproduces the reviewer's FIFO/fsync race against the ordinary binary. It detects loss against the original head and passes against this head. Cases cover in-place rewrite and pathname replacement before recovery, the former canonical comparison boundary, and immediately after recovery. Since the unsafe check is removed, that boundary case races at completion when no canonical fsync occurs. There are no timing sleeps or product pause points. A second test checks equal unjournaled bytes and repeated recovery. All final gates passed at e46ce1c30652c051332123fbbb9db49abfc9c73b. The settled 1,000-stop loop completed 250 saves, 250 renames, 250 Trash operations and 250 restores. All 1,000 acknowledged Unicode markers survived exactly once after every restart, retry jobs completed, final canonical bytes matched notes.get, and no staging files remained after explicit save barriers in any round. The initial workspace and canonical-race failures were a test-fixture mistake: notes.create lacked its required request key and returned HTTP 400 before recovery. The fixture was corrected and both gates passed on rerun. The original-head expected-loss gate means the regression successfully detected the original defect. Existing ignored tests remain ignored. These are process-death tests on local storage, not block-device power-loss tests. Gate output, verbatim: ```text fmt exit=0 clippy exit=0 workspace-tests exit=101 staging-race exit=0 canonical-race exit=101 build-binary exit=0 durability-1000-settled exit=0 reviewer-repro-original-head expected-loss-observed exit=0 canonical-race-corrected exit=0 workspace-tests-corrected exit=0 fmt-final exit=0 clippy-final exit=0 ``` All local heavy commands ran through one build slot at a time, with cargo -j3. Only product code and permanent tests are committed. Logs and gate evidence remain uncommitted under /home/agent/wt/storage-fix-1/.verification/. No merge or deployment.
fix: recover interrupted note renames and staged saves
Some checks failed
CI / server (pull_request) Successful in 57s
Details
CI / rust (pull_request) Successful in 20m26s
Details
CI / web (pull_request) Failing after 2h59m10s
Details
66f27ce3cd
Preserve unjournaled staging bytes across canonical file races
Some checks failed
CI / server (pull_request) Successful in 1m18s
Details
CI / rust (pull_request) Successful in 15m56s
Details
CI / web (pull_request) Has been cancelled
Details
e46ce1c306
kayg commented 2026-10-28 19:05:56 +00:00
Author
Owner
Copy link

Started storage-fix-3b at 90d8834635 on fix/storage-exact-file-quarantine-642. Continuing the approved move-then-verify cleanup and replacing preload tests with deterministic pauses in Rust cleanup code. I will run fmt, clippy with warnings denied, workspace tests, replacement and earlier race tests, and a fresh settled 1,000-stop loop through build-slot.sh.

Branch choice: the job asks for an update to the saved branch, but its hard rule says pushes to an open PR branch are ignored and requires a new branch and PR. I will preserve the saved branch and publish fix/storage-exact-file-quarantine-642-3b into dev, explicitly replacing #740. No merge or deployment.

Started storage-fix-3b at 90d883463580bdd5b46dc8447a3317e8ea4c2fae on fix/storage-exact-file-quarantine-642. Continuing the approved move-then-verify cleanup and replacing preload tests with deterministic pauses in Rust cleanup code. I will run fmt, clippy with warnings denied, workspace tests, replacement and earlier race tests, and a fresh settled 1,000-stop loop through build-slot.sh. Branch choice: the job asks for an update to the saved branch, but its hard rule says pushes to an open PR branch are ignored and requires a new branch and PR. I will preserve the saved branch and publish fix/storage-exact-file-quarantine-642-3b into dev, explicitly replacing #740. No merge or deployment.
kayg commented 2026-10-28 19:07:26 +00:00
Author
Owner
Copy link

Finding: the saved claimed-path integration fixture could pass when it never reached its unlink pause: it replaced the file after recovery completed. It also required a preloaded C library, contrary to this job's test requirements. The canonical fixture had the same completion fallback.

Replaced these with cfg(test) Rust channel pauses inside the real recovery and quarantine claim/check paths. The cleanup worker sends the reached boundary and blocks; another thread atomically replaces the public file and signals cleanup to continue. A missing pause fails rather than falling back to a different window. Hooks are thread-local so parallel Homes cannot intercept one another. Production builds compile out every hook.

First targeted gate: cargo test --locked -p calternal-fsio -j3 staging_ -- --nocapture, through build-slot.sh, passed: 12 tests, 0 failed. The matrix exercises both original and old recovery basenames, equal and different lengths, before-check, before-move, after-move and after-check. It asserts replacement survival through a second recovery and acknowledged canonical bytes remain unchanged. Cancellation tests also verify rejection of a new inode between the earlier metadata check and move. Attachment sweep concurrency coverage is being added to the same descriptor-relative claim path. Both C fixtures and all preload use have been removed from tracked tests.

Finding: the saved claimed-path integration fixture could pass when it never reached its unlink pause: it replaced the file after recovery completed. It also required a preloaded C library, contrary to this job's test requirements. The canonical fixture had the same completion fallback. Replaced these with cfg(test) Rust channel pauses inside the real recovery and quarantine claim/check paths. The cleanup worker sends the reached boundary and blocks; another thread atomically replaces the public file and signals cleanup to continue. A missing pause fails rather than falling back to a different window. Hooks are thread-local so parallel Homes cannot intercept one another. Production builds compile out every hook. First targeted gate: cargo test --locked -p calternal-fsio -j3 staging_ -- --nocapture, through build-slot.sh, passed: 12 tests, 0 failed. The matrix exercises both original and old recovery basenames, equal and different lengths, before-check, before-move, after-move and after-check. It asserts replacement survival through a second recovery and acknowledged canonical bytes remain unchanged. Cancellation tests also verify rejection of a new inode between the earlier metadata check and move. Attachment sweep concurrency coverage is being added to the same descriptor-relative claim path. Both C fixtures and all preload use have been removed from tracked tests.
kayg commented 2026-10-28 19:08:15 +00:00
Author
Owner
Copy link

Gate progress on fix/storage-exact-file-quarantine-642-3b:

fmt-write exit=0
fmt exit=0
fsio exit=0
clippy exit=0

The fsio gate reports test result: ok. 49 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out. Workspace clippy ran cargo clippy --locked --workspace --all-targets -j3 -- -D warnings. All local heavy work is in one build-slot.sh job.

Change Test
Move public staging names into a private namespace before disposal 16 concurrent recovery cases: original/recovery basename × equal/different length × before-check/before-move/after-move/after-check; external bytes survive a second recovery
Compare the claimed inode with the previously checked inode 6 concurrent cancellation cases, including an equal-length replacement between metadata check and move
Use the pinned attachment parent for claims and removal 6 concurrent attachment sweep cases before move, after move and after proof
Keep both versions on failed proof and restore collision staging_failed_proof_and_restore_collision_preserve_both_versions
Recover crash-left quarantine entries and settle empty folders staging_crash_quarantine_preserves_destination_and_journal_proof; upload_marker_still_finds_an_image_after_a_crash_during_publication
Remove preload/interposition fixtures and completion fallbacks cfg(test) channel hook requires the selected cleanup boundary; ordinary-binary canonical before/after regression retained

The workspace test, earlier race and fresh 1,000-stop gates remain in progress.

Gate progress on fix/storage-exact-file-quarantine-642-3b: ``` fmt-write exit=0 fmt exit=0 fsio exit=0 clippy exit=0 ``` The fsio gate reports `test result: ok. 49 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out`. Workspace clippy ran `cargo clippy --locked --workspace --all-targets -j3 -- -D warnings`. All local heavy work is in one build-slot.sh job. | Change | Test | | --- | --- | | Move public staging names into a private namespace before disposal | 16 concurrent recovery cases: original/recovery basename × equal/different length × before-check/before-move/after-move/after-check; external bytes survive a second recovery | | Compare the claimed inode with the previously checked inode | 6 concurrent cancellation cases, including an equal-length replacement between metadata check and move | | Use the pinned attachment parent for claims and removal | 6 concurrent attachment sweep cases before move, after move and after proof | | Keep both versions on failed proof and restore collision | staging_failed_proof_and_restore_collision_preserve_both_versions | | Recover crash-left quarantine entries and settle empty folders | staging_crash_quarantine_preserves_destination_and_journal_proof; upload_marker_still_finds_an_image_after_a_crash_during_publication | | Remove preload/interposition fixtures and completion fallbacks | cfg(test) channel hook requires the selected cleanup boundary; ordinary-binary canonical before/after regression retained | The workspace test, earlier race and fresh 1,000-stop gates remain in progress.
kayg commented 2026-10-28 19:13:07 +00:00
Author
Owner
Copy link

The first full gate sequence has passed through the binary build:

fmt-write exit=0
fmt exit=0
fsio exit=0
clippy exit=0
workspace exit=0
earlier-races exit=0
canonical-races exit=0
binary exit=0

The fresh settled 1,000-stop loop is running with separate loop3b-home and loop3b-state directories. Its first 25 rounds passed acknowledged Unicode text, completed request-key retries, auxiliary path/title/body and no staging leftovers after save barriers.

During review I tightened the test-only recovery pause to sit immediately after the last identity/length/byte and lease checks, directly before removal. This exercises the reviewer's last-check/removal window. Final fmt, fsio, clippy and workspace checks will be repeated for that exact placement; the hook has no production-build effect. All remaining local heavy commands will remain sequential through build-slot.sh.

The first full gate sequence has passed through the binary build: ```text fmt-write exit=0 fmt exit=0 fsio exit=0 clippy exit=0 workspace exit=0 earlier-races exit=0 canonical-races exit=0 binary exit=0 ``` The fresh settled 1,000-stop loop is running with separate loop3b-home and loop3b-state directories. Its first 25 rounds passed acknowledged Unicode text, completed request-key retries, auxiliary path/title/body and no staging leftovers after save barriers. During review I tightened the test-only recovery pause to sit immediately after the last identity/length/byte and lease checks, directly before removal. This exercises the reviewer's last-check/removal window. Final fmt, fsio, clippy and workspace checks will be repeated for that exact placement; the hook has no production-build effect. All remaining local heavy commands will remain sequential through build-slot.sh.
kayg commented 2026-10-28 19:17:21 +00:00
Author
Owner
Copy link

Fresh settled durability gate passed. Output, verbatim:

fmt-write exit=0
fmt exit=0
fsio exit=0
clippy exit=0
workspace exit=0
earlier-races exit=0
canonical-races exit=0
binary exit=0
durability-1000 exit=0
final-fmt exit=0
final-fsio exit=0

Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage.

The final checks for the last-check/removal pause placement are now running.

Fresh settled durability gate passed. Output, verbatim: ```text fmt-write exit=0 fmt exit=0 fsio exit=0 clippy exit=0 workspace exit=0 earlier-races exit=0 canonical-races exit=0 binary exit=0 durability-1000 exit=0 final-fmt exit=0 final-fsio exit=0 ``` ```text Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage. ``` The final checks for the last-check/removal pause placement are now running.
kayg commented 2026-10-28 19:22:34 +00:00
Author
Owner
Copy link

Finished storage-fix-3b. Head ccc7a2b790; branch fix/storage-exact-file-quarantine-642-3b; saved starting head 90d8834635; recorded dev base d83e54dbe5. Replacement PR into dev: #742. It explicitly replaces #740. The saved branch is preserved; a fresh branch follows the job's hard rule for this forge.

Change Test
Move staging into a fresh private mode-0700 folder; remove through its pinned descriptor after proof 16 concurrent cases: original/recovery basename, equal/different length, before-check/before-move/after-move/after-check; replacement survives a second recovery
Reject a different inode between the earlier metadata check and the move 6 concurrent cancellation cases, including equal-length replacement
Claim attachment staging through its pinned parent before removal 6 concurrent sweep cases before move, after move and after proof
Restore exclusively or retain both versions on collision staging_failed_proof_and_restore_collision_preserve_both_versions
Recover a crash-left quarantine without staging leftovers staging_crash_quarantine_preserves_destination_and_journal_proof; upload_marker_still_finds_an_image_after_a_crash_during_publication
Keep independent journal proof; canonical equality cannot authorize deletion Notes staging and ordinary-binary canonical before/after tests
Keep displaced versions during guarded save and rollback displaced_replacement_and_rollback_replacement_keep_external_versions
Preserve acknowledged edits across process death and settle cleanup Fresh 1,000 stops: 250 saves, 250 renames, 250 Trash operations, 250 restores; all acknowledged Unicode markers checked exactly once after every restart; no staging leftovers after explicit save barriers

Gate output, verbatim:

fmt-write exit=0
fmt exit=0
fsio exit=0
clippy exit=0
workspace exit=0
earlier-races exit=0
canonical-races exit=0
binary exit=0
durability-1000 exit=0
final-fmt exit=0
final-fsio exit=0
final-clippy exit=0
final-workspace exit=0

Settled loop output, verbatim:

Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage.

The fsio suite passed 49 tests. The 28 concurrent replacement cases use cfg(test) product pauses and real atomic replacement from another thread. All selected boundaries must be reached. Unknown bytes survive repeated recovery, acknowledged canonical bytes are preserved, and collision tests preserve both versions. Both C fixtures and all preload use are removed from tracked tests. Existing ignored tests remain ignored. These are process-death checks on local storage, not block-device power-loss tests.

Only product code and permanent tests were committed. Evidence stays in these comments and the untracked .verification folder. No merge, deployment or issue closure.

Finished storage-fix-3b. Head ccc7a2b790890d661de8d16d6ca1f8632e93efde; branch fix/storage-exact-file-quarantine-642-3b; saved starting head 90d883463580bdd5b46dc8447a3317e8ea4c2fae; recorded dev base d83e54dbe571b625504001a8b6add90d44ca6161. Replacement PR into dev: https://git.kayg.org/kayg/calternal/pulls/742. It explicitly replaces #740. The saved branch is preserved; a fresh branch follows the job's hard rule for this forge. | Change | Test | | --- | --- | | Move staging into a fresh private mode-0700 folder; remove through its pinned descriptor after proof | 16 concurrent cases: original/recovery basename, equal/different length, before-check/before-move/after-move/after-check; replacement survives a second recovery | | Reject a different inode between the earlier metadata check and the move | 6 concurrent cancellation cases, including equal-length replacement | | Claim attachment staging through its pinned parent before removal | 6 concurrent sweep cases before move, after move and after proof | | Restore exclusively or retain both versions on collision | staging_failed_proof_and_restore_collision_preserve_both_versions | | Recover a crash-left quarantine without staging leftovers | staging_crash_quarantine_preserves_destination_and_journal_proof; upload_marker_still_finds_an_image_after_a_crash_during_publication | | Keep independent journal proof; canonical equality cannot authorize deletion | Notes staging and ordinary-binary canonical before/after tests | | Keep displaced versions during guarded save and rollback | displaced_replacement_and_rollback_replacement_keep_external_versions | | Preserve acknowledged edits across process death and settle cleanup | Fresh 1,000 stops: 250 saves, 250 renames, 250 Trash operations, 250 restores; all acknowledged Unicode markers checked exactly once after every restart; no staging leftovers after explicit save barriers | Gate output, verbatim: ```text fmt-write exit=0 fmt exit=0 fsio exit=0 clippy exit=0 workspace exit=0 earlier-races exit=0 canonical-races exit=0 binary exit=0 durability-1000 exit=0 final-fmt exit=0 final-fsio exit=0 final-clippy exit=0 final-workspace exit=0 ``` Settled loop output, verbatim: ```text Check 5 PASS: 1000/1000 abrupt stops and restarts completed: 250 saves, 250 renames, 250 Trash operations, 250 restores. All 1000 acknowledged text markers survived exactly once after every restart; final canonical bytes equal notes.get; no staging leftovers after restart and explicit save barriers in any round. Randomized SIGKILL timing does not prove a particular syscall was interrupted; deterministic boundary checks supply that coverage. ``` The fsio suite passed 49 tests. The 28 concurrent replacement cases use cfg(test) product pauses and real atomic replacement from another thread. All selected boundaries must be reached. Unknown bytes survive repeated recovery, acknowledged canonical bytes are preserved, and collision tests preserve both versions. Both C fixtures and all preload use are removed from tracked tests. Existing ignored tests remain ignored. These are process-death checks on local storage, not block-device power-loss tests. Only product code and permanent tests were committed. Evidence stays in these comments and the untracked .verification folder. No merge, deployment or issue closure.
kayg commented 2026-11-04 13:04:32 +00:00
Author
Owner
Copy link

Closing: superseded by #742, then #744, #749 and now #750 (named in the PR body or comments). Branch kept.

Closing: superseded by #742, then #744, #749 and now #750 (named in the PR body or comments). Branch kept.
kayg closed this pull request 2026-11-04 13:04:32 +00:00
Some checks failed
CI / server (pull_request) Successful in 1m18s
Details
CI / rust (pull_request) Successful in 15m56s
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!740
No description provided.