kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

storage: safe offline split and unsplit (#761 A; supersedes #784) #790

Closed
kayg wants to merge 7 commits from feat/tiering-761-a-migration into dev
pull from: feat/tiering-761-a-migration
merge into: kayg:dev
Conversation 9 Commits 7 Files changed 34 +3018 -73
kayg commented 2026-10-31 16:12:51 +00:00
Owner
Copy link

Plan — offline layout migration (#761), A of A → B → C → D

Separate production's offline migration from the reader, paged index and qualification tools in #784. A targets dev; later PRs stack on it. No merge or production operation is authorized here.

  • Keep legacy --data/--state spellings; add --userdata/--appdata. Omitted appdata resolves to the selected data root, preserving old CLI operations rather than writing a new cwd-relative ./appdata. First-time split setup supplies both roots explicitly.
  • Share one explicit calternal_core::layout classification with every state reader. Include contacts/raw, journal and SQLite side files. A real-server fixture exercises the list.
  • Preflight the full inventory and apparent byte requirement; allow filesystem-owned lost+found without traversing it. Unknown top-level entries require --unknown-to-appdata. Preserve symlinks as symlinks; list and refuse devices/FIFOs/sockets except a stopped admin socket.
  • Copy to two fresh sibling .split-partial trees; discard only matching owned staging trees on retry. Verify all copied bytes and metadata; rewalk and hash the source before publication. Never merge into a published destination. Refuse completed splits; a crash after publishing only re-marks an unchanged source, without copying.
  • Publish by exclusive renames, keeping incomplete pairs unstartable. Lock source directory identities and compatibility sentinels. Recheck the completion marker after obtaining startup locks.
  • Unsplit current roots to a fresh stage, refuse collisions, verify and publish once. Persist the input manifest before publication. A retry of owned published output must match both input snapshots exactly or refuse with a fresh-destination instruction. Source roots remain intact.

Cost: O(entries + bytes) for inventory/copy/verification; O(entries + retained xattr bytes) inventory/hash bookkeeping and a 1 MiB streaming buffer. Reserve total apparent bytes + 1% on each destination filesystem (conservative when roots use separate filesystems); source remains as rollback copy. Sparse zero blocks stay holes; hard links become independent logical files. No production-data read or cutover was performed.

Failure → effect → detection → recovery → test

Failure Effect / detection Recovery Test
Re-run completed split / deleted destination file Refused; no resurrection Use current roots; explicit documented cleanup before a fresh migration a_finished_split_never_runs_again_so_nothing_deleted_comes_back
Error or ENOSPC at a boundary No partial pair can serve Discard own staging, recopy current source a_stop_at_any_step_publishes_nothing_partial_and_the_rerun_starts_over
Contacts/state misclassified Originals stranded Single shared list; real-server inventory must all classify correctly isolation::layout::every_entry_the_server_writes_is_sorted_into_the_root_it_reads_it_from, realistic fixture
mtime/atime/mode/owner/xattr changes Copy fails verification; access times restored after verification Fix permissions/ownership, rerun a_realistic_root_splits_exactly_with_times_modes_links_and_attributes, verification_keeps_access_times_on_files_directories_and_symlinks, round-trip manifest
Source edit/add/remove/rename during copy Refuse before publishing, including equal-size rewrite/restored mtime Stop outside writers, rerun a_change_to_the_source_during_the_copy_publishes_nothing
Server races split Locks and source marker refuse startup Wait for split or choose new roots while_a_split_runs_no_server_can_start_on_the_old_root, root-lock replacement regression
Marker abandonment Clear refusal; no automatic source reuse Exact cleanup in docs/storage.md, only before new writes same startup test; old binary on marked source in round trip
Unsupported / unknown entries Complete preflight list, before copying Resolve special files; explicit unknown override dry_run_lists_everything_and_writes_nothing, lost_found_contents_are_left_to_the_filesystem
Dry run with no sentinel No lock or destination created Read-only preflight dry_run_on_a_root_without_a_lock_does_not_create_one
Unsplit interruption / roots changed before retry No published partial root or merge Fresh copy of current pair unsplit_stopped_at_any_step_starts_over_and_never_publishes_half, unsplit_after_new_writes_keeps_the_current_state_and_refuses_ambiguity

Validation

PASS at 5441212fd8b445fb80ff8ed640a4b33c17ae1d07 on dev-02: fmt, clippy with -D warnings, and the full workspace suite; 1178 passed, 0 failed, 20 ignored, job exit 0. All heavy commands ran in Nix through ~/bin/slot. Evidence: ~/tier-r2-final-a.log and ~/tier-r2-final-a.exit.

cargo fmt --all --check
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings
cargo test -j4 --workspace --locked --no-fail-fast

A's final end-to-end gate on dev-02 also passed (ALL PASS): 211 entries, 161 regular files, three users and three contact originals; 36 notes before the new write and 37 after. It verifies split, immediate unsplit, unchanged old-root refusal, new split-layout writes, unsplit against the current roots and old-binary startup.

cargo build -j4 -p calternal --locked
python3 ci/storage-split-e2e.py /home/cua/Developer/Worktrees/tier-r2-old/target/debug/calternal target/debug/calternal /home/cua/tier-r2-final-roundtrip

Old binary: d83e54db. Evidence: dev-02 ~/tier-r2-final-a.log, ~/tier-r2-final-a.exit = 0; fixture ~/tier-r2-final-roundtrip.

Round-2 findings

Finding Fix Failing-before / passing-after evidence
R2-A1 P1, stale published unsplit Persist a versioned digest of both input inventories, root identity/metadata, raw paths, xattrs and file hashes before publication. Recovery requires an exact match; missing/mismatched manifests refuse with instructions to preserve output and choose a fresh destination. Never recopy or merge into published output. r2_published_unsplit_refuses_changed_inputs: add/edit/delete/mode changes at published refuse and preserve output. Original A failed this regression.
R2-A2 P1, ignored durability errors Check writable-file fsync before close and after metadata; check final file/directory sync and closes. Remove ignored verification fsync and global sync/syncfs substitutes. Retract publication on parent flush/close failure; retract first root if publishing the second fails. r2_every_prepublication_fsync_close_and_rename_failure_aborts injects at every prepublication syscall hook for split and unsplit, requires no published roots/DONE and successful retry; r2_retraction_fsync_errors_are_reported_too checks repeated rollback errors. Independent LD_PRELOAD fsync #7 EIO: original A returned success, fixed A returns failure with no userdata/appdata/DONE.
R2-A3 P2, cleanup loses ownership Keep OWNER until all payload deletion completes and is flushed; then remove OWNER and the empty stage. r2_cleanup_keeps_ownership_until_payload_is_gone interrupts payload cleanup, checks OWNER survives, retries successfully. Original A failed.
R2-A4 P2, 64 KiB retained per tiny xattr Shrink attribute values before retaining inventory entries. Cost is O(entries + retained xattr bytes), plus the 1 MiB copy buffer. r2_tiny_xattrs_retain_only_their_bytes and r2_metadata_heavy_inventory_retains_actual_xattr_bytes cover capacity and 4,096 one-byte attributes; original A failed. Independent 4,096-file RSS gate: original 300,524 KiB, fixed 38,820 KiB, threshold 131,072 KiB.

The real-server layout test now compares every copied file, including SQLite WAL/SHM, against the immutable offline input snapshot. Live server descriptors can still mutate their own SHM after copying; comparing against those live bytes was a race.

Decisions and remaining risks

  • Legacy single-root serve/user/fix commands stay compatible until the operator explicitly splits. No automatic migration.
  • All outside writers must be stopped. Advisory locks cannot exclude an arbitrary editor retaining an fd. Final inventory/hash checks detect observed changes; this is an offline command.
  • I/O-boundary error injection is not SIGKILL or a power-cut durability proof. Privileged disk-full/mount/power-loss campaigns remain owed.
  • Linux cache eviction remains best effort; durability uses checked descriptor fsync/close. Unsplit SOURCE includes input content hashes; split SOURCE remains an inventory recovery digest. Preserve an independent backup manifest.
  • Hard-link sharing is lost; SELinux labels come from the target filesystem/container relabel. A Home entry at an exact reserved legacy state name is app state by the shared schema; arbitrary historical ambiguity needs operator inspection before migration.
  • Staging ownership permits cleanup of an empty/torn-marker stage. No online tier retirement, dedupe, FO1 or 15 ms latency certification is introduced.

Final-head macOS checks were not run: the shared lock is held by dav-recorder PR 789. Earlier targeted Mac results apply only to older heads. This change rejects symlinked ancestors, including aliases to the same physical root; configure physical paths. Real media errors, power cuts, 100k-note polling costs and latency qualification remain unrun.

Stack: A #790 → B #791 → C #792 → D #793. Ready for final round-3 review after these fixes; nothing merged.

## Plan — offline layout migration (#761), A of A → B → C → D Separate production's offline migration from the reader, paged index and qualification tools in #784. A targets `dev`; later PRs stack on it. No merge or production operation is authorized here. - Keep legacy `--data`/`--state` spellings; add `--userdata`/`--appdata`. Omitted appdata resolves to the selected data root, preserving old CLI operations rather than writing a new cwd-relative `./appdata`. First-time split setup supplies both roots explicitly. - Share one explicit `calternal_core::layout` classification with every state reader. Include contacts/raw, journal and SQLite side files. A real-server fixture exercises the list. - Preflight the full inventory and apparent byte requirement; allow filesystem-owned lost+found without traversing it. Unknown top-level entries require `--unknown-to-appdata`. Preserve symlinks as symlinks; list and refuse devices/FIFOs/sockets except a stopped admin socket. - Copy to two fresh sibling `.split-partial` trees; discard only matching owned staging trees on retry. Verify all copied bytes and metadata; rewalk and hash the source before publication. Never merge into a published destination. Refuse completed splits; a crash after publishing only re-marks an unchanged source, without copying. - Publish by exclusive renames, keeping incomplete pairs unstartable. Lock source directory identities and compatibility sentinels. Recheck the completion marker after obtaining startup locks. - Unsplit current roots to a fresh stage, refuse collisions, verify and publish once. Persist the input manifest before publication. A retry of owned published output must match both input snapshots exactly or refuse with a fresh-destination instruction. Source roots remain intact. Cost: O(entries + bytes) for inventory/copy/verification; O(entries + retained xattr bytes) inventory/hash bookkeeping and a 1 MiB streaming buffer. Reserve total apparent bytes + 1% on each destination filesystem (conservative when roots use separate filesystems); source remains as rollback copy. Sparse zero blocks stay holes; hard links become independent logical files. No production-data read or cutover was performed. ## Failure → effect → detection → recovery → test | Failure | Effect / detection | Recovery | Test | |---|---|---|---| | Re-run completed split / deleted destination file | Refused; no resurrection | Use current roots; explicit documented cleanup before a fresh migration | `a_finished_split_never_runs_again_so_nothing_deleted_comes_back` | | Error or ENOSPC at a boundary | No partial pair can serve | Discard own staging, recopy current source | `a_stop_at_any_step_publishes_nothing_partial_and_the_rerun_starts_over` | | Contacts/state misclassified | Originals stranded | Single shared list; real-server inventory must all classify correctly | `isolation::layout::every_entry_the_server_writes_is_sorted_into_the_root_it_reads_it_from`, realistic fixture | | mtime/atime/mode/owner/xattr changes | Copy fails verification; access times restored after verification | Fix permissions/ownership, rerun | `a_realistic_root_splits_exactly_with_times_modes_links_and_attributes`, `verification_keeps_access_times_on_files_directories_and_symlinks`, round-trip manifest | | Source edit/add/remove/rename during copy | Refuse before publishing, including equal-size rewrite/restored mtime | Stop outside writers, rerun | `a_change_to_the_source_during_the_copy_publishes_nothing` | | Server races split | Locks and source marker refuse startup | Wait for split or choose new roots | `while_a_split_runs_no_server_can_start_on_the_old_root`, root-lock replacement regression | | Marker abandonment | Clear refusal; no automatic source reuse | Exact cleanup in docs/storage.md, only before new writes | same startup test; old binary on marked source in round trip | | Unsupported / unknown entries | Complete preflight list, before copying | Resolve special files; explicit unknown override | `dry_run_lists_everything_and_writes_nothing`, `lost_found_contents_are_left_to_the_filesystem` | | Dry run with no sentinel | No lock or destination created | Read-only preflight | `dry_run_on_a_root_without_a_lock_does_not_create_one` | | Unsplit interruption / roots changed before retry | No published partial root or merge | Fresh copy of current pair | `unsplit_stopped_at_any_step_starts_over_and_never_publishes_half`, `unsplit_after_new_writes_keeps_the_current_state_and_refuses_ambiguity` | ## Validation **PASS at `5441212fd8b445fb80ff8ed640a4b33c17ae1d07`** on dev-02: fmt, clippy with `-D warnings`, and the full workspace suite; **1178 passed, 0 failed, 20 ignored**, job exit 0. All heavy commands ran in Nix through `~/bin/slot`. Evidence: `~/tier-r2-final-a.log` and `~/tier-r2-final-a.exit`. ```sh cargo fmt --all --check cargo clippy -j4 --workspace --all-targets --locked -- -D warnings cargo test -j4 --workspace --locked --no-fail-fast ``` A's final end-to-end gate on dev-02 also passed (`ALL PASS`): 211 entries, 161 regular files, three users and three contact originals; 36 notes before the new write and 37 after. It verifies split, immediate unsplit, unchanged old-root refusal, new split-layout writes, unsplit against the current roots and old-binary startup. ```sh cargo build -j4 -p calternal --locked python3 ci/storage-split-e2e.py /home/cua/Developer/Worktrees/tier-r2-old/target/debug/calternal target/debug/calternal /home/cua/tier-r2-final-roundtrip ``` Old binary: `d83e54db`. Evidence: dev-02 `~/tier-r2-final-a.log`, `~/tier-r2-final-a.exit` = 0; fixture `~/tier-r2-final-roundtrip`. ## Round-2 findings | Finding | Fix | Failing-before / passing-after evidence | |---|---|---| | R2-A1 P1, stale published unsplit | Persist a versioned digest of both input inventories, root identity/metadata, raw paths, xattrs and file hashes before publication. Recovery requires an exact match; missing/mismatched manifests refuse with instructions to preserve output and choose a fresh destination. Never recopy or merge into published output. | `r2_published_unsplit_refuses_changed_inputs`: add/edit/delete/mode changes at `published` refuse and preserve output. Original A failed this regression. | | R2-A2 P1, ignored durability errors | Check writable-file fsync before close and after metadata; check final file/directory sync and closes. Remove ignored verification fsync and global sync/syncfs substitutes. Retract publication on parent flush/close failure; retract first root if publishing the second fails. | `r2_every_prepublication_fsync_close_and_rename_failure_aborts` injects at every prepublication syscall hook for split and unsplit, requires no published roots/DONE and successful retry; `r2_retraction_fsync_errors_are_reported_too` checks repeated rollback errors. Independent LD_PRELOAD fsync #7 EIO: original A returned success, fixed A returns failure with no userdata/appdata/DONE. | | R2-A3 P2, cleanup loses ownership | Keep OWNER until all payload deletion completes and is flushed; then remove OWNER and the empty stage. | `r2_cleanup_keeps_ownership_until_payload_is_gone` interrupts payload cleanup, checks OWNER survives, retries successfully. Original A failed. | | R2-A4 P2, 64 KiB retained per tiny xattr | Shrink attribute values before retaining inventory entries. Cost is O(entries + retained xattr bytes), plus the 1 MiB copy buffer. | `r2_tiny_xattrs_retain_only_their_bytes` and `r2_metadata_heavy_inventory_retains_actual_xattr_bytes` cover capacity and 4,096 one-byte attributes; original A failed. Independent 4,096-file RSS gate: original 300,524 KiB, fixed 38,820 KiB, threshold 131,072 KiB. | The real-server layout test now compares every copied file, including SQLite WAL/SHM, against the immutable offline input snapshot. Live server descriptors can still mutate their own SHM after copying; comparing against those live bytes was a race. ## Decisions and remaining risks - Legacy single-root serve/user/fix commands stay compatible until the operator explicitly splits. No automatic migration. - All outside writers must be stopped. Advisory locks cannot exclude an arbitrary editor retaining an fd. Final inventory/hash checks detect observed changes; this is an offline command. - I/O-boundary error injection is not SIGKILL or a power-cut durability proof. Privileged disk-full/mount/power-loss campaigns remain owed. - Linux cache eviction remains best effort; durability uses checked descriptor fsync/close. Unsplit SOURCE includes input content hashes; split SOURCE remains an inventory recovery digest. Preserve an independent backup manifest. - Hard-link sharing is lost; SELinux labels come from the target filesystem/container relabel. A Home entry at an exact reserved legacy state name is app state by the shared schema; arbitrary historical ambiguity needs operator inspection before migration. - Staging ownership permits cleanup of an empty/torn-marker stage. No online tier retirement, dedupe, FO1 or 15 ms latency certification is introduced. Final-head macOS checks were not run: the shared lock is held by `dav-recorder PR 789`. Earlier targeted Mac results apply only to older heads. This change rejects symlinked ancestors, including aliases to the same physical root; configure physical paths. Real media errors, power cuts, 100k-note polling costs and latency qualification remain unrun. Stack: [A #790](https://git.kayg.org/kayg/calternal/pulls/790) → [B #791](https://git.kayg.org/kayg/calternal/pulls/791) → [C #792](https://git.kayg.org/kayg/calternal/pulls/792) → [D #793](https://git.kayg.org/kayg/calternal/pulls/793). Ready for final round-3 review after these fixes; nothing merged.
Preserve installation marker metadata and check full rollback manifests
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
3a1ed673ea
kayg commented 2026-10-31 17:33:43 +00:00
Author
Owner
Copy link

Independent review, round 2

Reviewed A at c5eaf6af7a258b51ba1608b72753da3f0cd8b571, against round 1 (#784, comment 70713) and the fix table (70847). Verdict: fix first. No new P0 found. No code pushed or production data accessed.

“Verified” below means the implementation and the named test actually cover the original scenario. I inspected existing gate/round-trip evidence and ran small independent reproductions on dev-01; I did not rerun the workspace suite. No builds/tests ran on the control host.

Round-1 item Round-2 status Does the test prove it?
P0-1 completed split/resume resurrects deletions Verified for split. DONE refuses completed runs; published-pair retry never copies; partial-pair retry recopies fresh staging. Yes: a_finished_split_never_runs_again_so_nothing_deleted_comes_back deletes a destination note, removes DONE, retries, and verifies it stays deleted. Every-boundary test changes source before retry. Unsplit has a new analogous gap, R2-A1 below.
P0-2 contacts originals misclassified Verified. Shared layout includes contacts/raw and SQLite side files. Yes: server-produced layout test creates real cards, waits for originals, then checks root placement and bytes. E2E checks originals for all three users. Exact reserved state names remain schema-reserved, as documented.
P1-1 times/xattrs/modes/owner Verified for successful Linux copies. Nanosecond times, uid/gid, modes and non-SELinux attrs are copied and checked. Realistic regression plus E2E checks file/directory mtimes, modes, owners, hashes and xattrs; separate verification_keeps_access_times_on_files_directories_and_symlinks checks destination atime. E2E itself does not compare atime or root-directory metadata. Flush errors remain R2-A2.
P1-2 offline/source change detection Verified within the stated offline contract. Root/sentinel locks, final inventory/ctime and byte rehash, startup marker recheck. Tests exercise server exclusion and edits/add/remove/rename during copy, including restored-mtime rewrites. This cannot exclude an arbitrary outside writer after the final check; the runbook requires stopping those writers.
P1-3 markers/abandonment Verified for split. Cleanup/rollback instructions and changed-old-root refusal exist. E2E starts the old binary on the marked source and consolidated rollback. Interrupted cleanup is not covered: R2-A3.
P1-4 unknown entries/dry-run Verified. Explicit override, lost+found skip, read-only dry-run without creating a lock. Tests cover unknown refusal/override, unreadable lost+found and absent sentinel.
P1-5 symlinks/special files Verified for migration. Links copied as links; unsupported objects listed/refused before staging. Realistic fixture checks links; special-file preflight tests exist. Notes link policy belongs to B.
P1-6 bad note/non-UTF-8 scan Not part of A. A leaves runtime discovery at its pre-reader behavior. B owns this regression and its tests. Migration preserves raw byte names.
P1-7 cwd-relative appdata Verified. Omitted appdata falls back to selected data; aliases/env precedence retained. Parser tests assert the resulting roots.
P1-8 macOS reader Not part of A. The new runtime reader is introduced in B. No claim of macOS migration durability qualification here.

New findings

R2-A1, P1: published unsplit retry silently accepts stale inputs. crates/calternal/src/storage_split/mod.rs:209-224 and :264-268. A crash after output rename/parent fsync but before OWNER removal leaves Mine::Yes. Restart the split-layout server, make a new note, stop it, then repeat unsplit: it removes OWNER and returns success without inventory/hash verification or copying. The advertised rollback root lacks the new note. This is the same ambiguity the split recovery digest explicitly guards against.

Independent dev-01 reproduction with A's existing binary synthesized exactly that post-publication marker state, added userdata/users/u/Notes/new.md, and retried: exit 0, “Already put back together; nothing copied”, input new file present, output new file absent. This is a boundary-state reproduction, not a SIGKILL test. Fix: bind the published rollback to both input snapshots and verify them before reporting successful recovery, otherwise refuse clearly and require a fresh destination. Never merge into that output, and account for an output that may already have been served. Add the changed-input case at the published hook; the current unsplit loop retries unchanged inputs only.

R2-A2, P1: observed flush failures can be ignored before successful publication. crates/calternal/src/storage_split/tree.rs:206-211, copy.rs:98-100 and :185-196. Copy closes its writable file without a checked fsync; verification explicitly ignores file.sync_all() errors. Later Linux syncfs opens a new root descriptor, so it is not a reliable substitute for reporting an already-observed writeback error (the superblock error cursor is sampled on open). Cache eviction is also best effort. A known EIO/ENOSPC must not be converted into “checked” success because subsequent reads happen to match cached bytes. Off Linux, the final sync() provides no error result at all.

Fix: propagate file durability errors, preferably sync each writable descriptor before closing it, and check the final metadata/directory flushes before publication; retain an error-observing filesystem descriptor from before writes if using batching. Add injected fsync/syncfs failures, including after final metadata, and assert no success/DONE. Current hooks inject returned errors around copy steps, not actual flush failures. This finding is source-verified; no media-error/power-cut reproduction was run.

R2-A3, P2: interrupted staging deletion loses the ownership proof. copy.rs:172-181, mod.rs:305-325. Recursive deletion may unlink OWNER before payload directories. Kill/error during the remaining removal leaves a nonempty stage without OWNER; the next retry rejects it as foreign. Synthesizing this state on dev-01 reproduced exit 1, “was not made by this split”; original data remained intact. Fix: retain a durable cleanup ownership record until payload removal completes (or rename into a separately recorded garbage namespace), and cover interruption during cleanup. The existing hooks never interrupt deletion. The current diagnostic permits manual recovery, but “run the same command again; it starts over” is not generally true.

R2-A4, P2: tiny xattrs retain 64 KiB each throughout manifests. tree.rs:162-165. Vec::truncate changes length, not capacity. One one-byte xattr on 100k files retains about 6.1 GiB in a single inventory, before the plan, rewalk and destination inventory coexist. A large tree can OOM even though its actual metadata is small. Source-verified, not scale-measured. Fix: allocate the returned length or shrink/box each value before retaining it; add a metadata-heavy RSS gate. The manifest is O(entries + retained attribute bytes), not just a small fixed entry record plus the 1 MiB copy buffer.

Evidence and limits

ci/storage-split-e2e.py really diffs SHA-256 content, kind/link target, modes, uid/gid, size, nanosecond mtime and xattrs, including directory mtimes. Immediate unsplit compares against the original with no installation-marker exception; post-write unsplit compares against the current pair. It excludes operational markers, lost+found, atime and root-directory metadata; xattr read failures are flattened to an empty list. That is meaningful successful-round-trip evidence, not disk-full/power-loss proof. The Python manifest also reads each whole file into memory, so the qualification script itself is not bounded-memory for huge attachments.

I checked the existing dev-01 strict E2E log, which ends ALL PASS. Existing Linux gates are author evidence, not independently rerun gates. Independent reproduction command: netbird ssh claude@calternal-dev-01-compute02.vpn.kayg.org 'python3 -' < /tmp/tier761-round2-repro.py; fixture /tmp/tier761-r2-3ks_g5ig on dev-01. Neither ENOSPC on a real filesystem, writeback EIO, SIGKILL nor power-cut ordering has been independently qualified. Exclusive renames plus parent fsync are present; a half pair at the configured final paths fails startup. Fix R2-A1/A2 before production use, and address or explicitly bound R2-A3/A4.

# Independent review, round 2 Reviewed A at `c5eaf6af7a258b51ba1608b72753da3f0cd8b571`, against round 1 (#784, comment 70713) and the fix table (70847). **Verdict: fix first.** No new P0 found. No code pushed or production data accessed. “Verified” below means the implementation and the named test actually cover the original scenario. I inspected existing gate/round-trip evidence and ran small independent reproductions on dev-01; I did not rerun the workspace suite. No builds/tests ran on the control host. | Round-1 item | Round-2 status | Does the test prove it? | |---|---|---| | P0-1 completed split/resume resurrects deletions | **Verified for split.** DONE refuses completed runs; published-pair retry never copies; partial-pair retry recopies fresh staging. | Yes: `a_finished_split_never_runs_again_so_nothing_deleted_comes_back` deletes a destination note, removes DONE, retries, and verifies it stays deleted. Every-boundary test changes source before retry. Unsplit has a new analogous gap, R2-A1 below. | | P0-2 contacts originals misclassified | **Verified.** Shared layout includes contacts/raw and SQLite side files. | Yes: server-produced layout test creates real cards, waits for originals, then checks root placement and bytes. E2E checks originals for all three users. Exact reserved state names remain schema-reserved, as documented. | | P1-1 times/xattrs/modes/owner | **Verified for successful Linux copies.** Nanosecond times, uid/gid, modes and non-SELinux attrs are copied and checked. | Realistic regression plus E2E checks file/directory mtimes, modes, owners, hashes and xattrs; separate `verification_keeps_access_times_on_files_directories_and_symlinks` checks destination atime. E2E itself does **not** compare atime or root-directory metadata. Flush errors remain R2-A2. | | P1-2 offline/source change detection | **Verified within the stated offline contract.** Root/sentinel locks, final inventory/ctime and byte rehash, startup marker recheck. | Tests exercise server exclusion and edits/add/remove/rename during copy, including restored-mtime rewrites. This cannot exclude an arbitrary outside writer after the final check; the runbook requires stopping those writers. | | P1-3 markers/abandonment | **Verified for split.** Cleanup/rollback instructions and changed-old-root refusal exist. | E2E starts the old binary on the marked source and consolidated rollback. Interrupted cleanup is not covered: R2-A3. | | P1-4 unknown entries/dry-run | **Verified.** Explicit override, lost+found skip, read-only dry-run without creating a lock. | Tests cover unknown refusal/override, unreadable lost+found and absent sentinel. | | P1-5 symlinks/special files | **Verified for migration.** Links copied as links; unsupported objects listed/refused before staging. | Realistic fixture checks links; special-file preflight tests exist. Notes link policy belongs to B. | | P1-6 bad note/non-UTF-8 scan | **Not part of A.** A leaves runtime discovery at its pre-reader behavior. | B owns this regression and its tests. Migration preserves raw byte names. | | P1-7 cwd-relative appdata | **Verified.** Omitted appdata falls back to selected data; aliases/env precedence retained. | Parser tests assert the resulting roots. | | P1-8 macOS reader | **Not part of A.** The new runtime reader is introduced in B. | No claim of macOS migration durability qualification here. | ## New findings **R2-A1, P1: published unsplit retry silently accepts stale inputs.** `crates/calternal/src/storage_split/mod.rs:209-224` and `:264-268`. A crash after output rename/parent fsync but before OWNER removal leaves `Mine::Yes`. Restart the split-layout server, make a new note, stop it, then repeat unsplit: it removes OWNER and returns success without inventory/hash verification or copying. The advertised rollback root lacks the new note. This is the same ambiguity the split recovery digest explicitly guards against. Independent dev-01 reproduction with A's existing binary synthesized exactly that post-publication marker state, added `userdata/users/u/Notes/new.md`, and retried: exit 0, “Already put back together; nothing copied”, input new file present, output new file absent. This is a boundary-state reproduction, not a SIGKILL test. Fix: bind the published rollback to both input snapshots and verify them before reporting successful recovery, otherwise refuse clearly and require a fresh destination. Never merge into that output, and account for an output that may already have been served. Add the changed-input case at the `published` hook; the current unsplit loop retries unchanged inputs only. **R2-A2, P1: observed flush failures can be ignored before successful publication.** `crates/calternal/src/storage_split/tree.rs:206-211`, `copy.rs:98-100` and `:185-196`. Copy closes its writable file without a checked fsync; verification explicitly ignores `file.sync_all()` errors. Later Linux `syncfs` opens a **new** root descriptor, so it is not a reliable substitute for reporting an already-observed writeback error (the superblock error cursor is sampled on open). Cache eviction is also best effort. A known EIO/ENOSPC must not be converted into “checked” success because subsequent reads happen to match cached bytes. Off Linux, the final `sync()` provides no error result at all. Fix: propagate file durability errors, preferably sync each writable descriptor before closing it, and check the final metadata/directory flushes before publication; retain an error-observing filesystem descriptor from before writes if using batching. Add injected fsync/syncfs failures, including after final metadata, and assert no success/DONE. Current hooks inject returned errors around copy steps, not actual flush failures. This finding is source-verified; no media-error/power-cut reproduction was run. **R2-A3, P2: interrupted staging deletion loses the ownership proof.** `copy.rs:172-181`, `mod.rs:305-325`. Recursive deletion may unlink OWNER before payload directories. Kill/error during the remaining removal leaves a nonempty stage without OWNER; the next retry rejects it as foreign. Synthesizing this state on dev-01 reproduced exit 1, “was not made by this split”; original data remained intact. Fix: retain a durable cleanup ownership record until payload removal completes (or rename into a separately recorded garbage namespace), and cover interruption during cleanup. The existing hooks never interrupt deletion. The current diagnostic permits manual recovery, but “run the same command again; it starts over” is not generally true. **R2-A4, P2: tiny xattrs retain 64 KiB each throughout manifests.** `tree.rs:162-165`. `Vec::truncate` changes length, not capacity. One one-byte xattr on 100k files retains about 6.1 GiB in a single inventory, before the plan, rewalk and destination inventory coexist. A large tree can OOM even though its actual metadata is small. Source-verified, not scale-measured. Fix: allocate the returned length or shrink/box each value before retaining it; add a metadata-heavy RSS gate. The manifest is O(entries + retained attribute bytes), not just a small fixed entry record plus the 1 MiB copy buffer. ## Evidence and limits `ci/storage-split-e2e.py` really diffs SHA-256 content, kind/link target, modes, uid/gid, size, nanosecond mtime and xattrs, including directory mtimes. Immediate unsplit compares against the original with **no installation-marker exception**; post-write unsplit compares against the current pair. It excludes operational markers, lost+found, atime and root-directory metadata; xattr read failures are flattened to an empty list. That is meaningful successful-round-trip evidence, not disk-full/power-loss proof. The Python manifest also reads each whole file into memory, so the qualification script itself is not bounded-memory for huge attachments. I checked the existing dev-01 strict E2E log, which ends ALL PASS. Existing Linux gates are author evidence, not independently rerun gates. Independent reproduction command: `netbird ssh claude@calternal-dev-01-compute02.vpn.kayg.org 'python3 -' < /tmp/tier761-round2-repro.py`; fixture `/tmp/tier761-r2-3ks_g5ig` on dev-01. Neither ENOSPC on a real filesystem, writeback EIO, SIGKILL nor power-cut ordering has been independently qualified. Exclusive renames plus parent fsync are present; a half pair at the configured final paths fails startup. Fix R2-A1/A2 before production use, and address or explicitly bound R2-A3/A4.
kayg commented 2026-10-31 17:38:36 +00:00
Author
Owner
Copy link

Round-2 implementation plan (last fix before final round 3): persist a versioned, content-and-metadata input manifest in the published unsplit ownership record; unchanged recovery only, otherwise refuse and require a fresh destination. Retain checked writable file descriptors through metadata fsync and close; replace syncfs/sync batching with checked file/directory fsync, with injection at every migration flush. Keep OWNER until all staged payload is removed. Store xattr values at their actual length. O(entries + actual xattr bytes) inventories, O(bytes) copy/hash and recovery verification, 1 MiB copy/hash buffer.

Failure Effect Detection Recovery Regression
Published unsplit inputs changed Refuse stale rollback Stored input manifest mismatch Choose absent destination, preserve existing output published-hook add/edit/delete/metadata
fsync/close/rename failure Abort; no successful publication Checked syscall result Retry fresh owned staging; retract failed rename when possible Every flush EIO injection, close/rename errors
Cleanup interrupted Ownership survives OWNER retained last Same command retries Fail after payload removal
Tiny xattrs Retain actual bytes Capacity/RSS checks No workaround needed Metadata-heavy fixture

Gates on exact heads: cargo fmt --all --check; cargo clippy -j4 --workspace --all-targets --locked -- -D warnings; cargo test -j4 --workspace --locked --no-fail-fast. Re-run ci/storage-split-e2e.py. Build/test through slot in Nix on dev-02. No merge.

Round-2 implementation plan (last fix before final round 3): persist a versioned, content-and-metadata input manifest in the published unsplit ownership record; unchanged recovery only, otherwise refuse and require a fresh destination. Retain checked writable file descriptors through metadata fsync and close; replace syncfs/sync batching with checked file/directory fsync, with injection at every migration flush. Keep OWNER until all staged payload is removed. Store xattr values at their actual length. O(entries + actual xattr bytes) inventories, O(bytes) copy/hash and recovery verification, 1 MiB copy/hash buffer. | Failure | Effect | Detection | Recovery | Regression | |---|---|---|---|---| | Published unsplit inputs changed | Refuse stale rollback | Stored input manifest mismatch | Choose absent destination, preserve existing output | published-hook add/edit/delete/metadata | | fsync/close/rename failure | Abort; no successful publication | Checked syscall result | Retry fresh owned staging; retract failed rename when possible | Every flush EIO injection, close/rename errors | | Cleanup interrupted | Ownership survives | OWNER retained last | Same command retries | Fail after payload removal | | Tiny xattrs | Retain actual bytes | Capacity/RSS checks | No workaround needed | Metadata-heavy fixture | Gates on exact heads: cargo fmt --all --check; cargo clippy -j4 --workspace --all-targets --locked -- -D warnings; cargo test -j4 --workspace --locked --no-fail-fast. Re-run ci/storage-split-e2e.py. Build/test through slot in Nix on dev-02. No merge.
kayg commented 2026-10-31 18:26:42 +00:00
Author
Owner
Copy link

Round-2 fixes ready for final round-3 review at 5441212fd8b445fb80ff8ed640a4b33c17ae1d07. Nothing merged.

Finding Fix Failing-before / passing-after evidence
R2-A1 P1, stale published unsplit Persist a versioned digest of both input inventories, root identity/metadata, raw paths, xattrs and file hashes before publication. Recovery requires an exact match; missing/mismatched manifests refuse with instructions to preserve output and choose a fresh destination. Never recopy or merge into published output. r2_published_unsplit_refuses_changed_inputs: add/edit/delete/mode changes at published refuse and preserve output. Original A failed this regression.
R2-A2 P1, ignored durability errors Check writable-file fsync before close and after metadata; check final file/directory sync and closes. Remove ignored verification fsync and global sync/syncfs substitutes. Retract publication on parent flush/close failure; retract first root if publishing the second fails. r2_every_prepublication_fsync_close_and_rename_failure_aborts injects at every prepublication syscall hook for split and unsplit, requires no published roots/DONE and successful retry; r2_retraction_fsync_errors_are_reported_too checks repeated rollback errors. Independent LD_PRELOAD fsync #7 EIO: original A returned success, fixed A returns failure with no userdata/appdata/DONE.
R2-A3 P2, cleanup loses ownership Keep OWNER until all payload deletion completes and is flushed; then remove OWNER and the empty stage. r2_cleanup_keeps_ownership_until_payload_is_gone interrupts payload cleanup, checks OWNER survives, retries successfully. Original A failed.
R2-A4 P2, 64 KiB retained per tiny xattr Shrink attribute values before retaining inventory entries. Cost is O(entries + retained xattr bytes), plus the 1 MiB copy buffer. r2_tiny_xattrs_retain_only_their_bytes and r2_metadata_heavy_inventory_retains_actual_xattr_bytes cover capacity and 4,096 one-byte attributes; original A failed. Independent 4,096-file RSS gate: original 300,524 KiB, fixed 38,820 KiB, threshold 131,072 KiB.

PASS at 5441212fd8b445fb80ff8ed640a4b33c17ae1d07 on dev-02: fmt, clippy with -D warnings, and the full workspace suite; 1178 passed, 0 failed, 20 ignored, job exit 0. All heavy commands ran in Nix through ~/bin/slot. Evidence: ~/tier-r2-final-a.log and ~/tier-r2-final-a.exit. Commands inside Nix through ~/bin/slot:

cargo fmt --all --check
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings
cargo test -j4 --workspace --locked --no-fail-fast
PR Final head Workspace tests fmt / strict clippy
#790 5441212fd8b445fb80ff8ed640a4b33c17ae1d07 1178 passed, 0 failed, 20 ignored PASS
#791 58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8 1187 passed, 0 failed, 20 ignored PASS
#792 9c8c287357240912daf895c4166bf00f2212ce4a 1205 passed, 0 failed, 20 ignored PASS
#793 659b0f4e420f6272e801ea78eba8fe10d66c1ca5 1208 passed, 0 failed, 20 ignored PASS

A's final end-to-end gate on dev-02 also passed (ALL PASS): 211 entries, 161 regular files, three users and three contact originals; 36 notes before the new write and 37 after. It verifies split, immediate unsplit, unchanged old-root refusal, new split-layout writes, unsplit against the current roots and old-binary startup.

cargo build -j4 -p calternal --locked
python3 ci/storage-split-e2e.py /home/cua/Developer/Worktrees/tier-r2-old/target/debug/calternal target/debug/calternal /home/cua/tier-r2-final-roundtrip

Old binary: d83e54db. Evidence: dev-02 ~/tier-r2-final-a.log, ~/tier-r2-final-a.exit = 0; fixture ~/tier-r2-final-roundtrip.

Before/after evidence on dev-02: ~/tier-r2-a-red.log (four expected failures), ~/tier-r2-b-red.log (two expected failures), ~/tier-r2-reader-ancestor-red.log (one expected failure), ~/tier-r2-fsync-probe.log and ~/tier-r2-xattr-rss.log (probe wrappers exit 0). External fsync/RSS probes used e688361c, whose runtime implementation is identical to final A; the later A change fixes only the layout test's live-SHM comparison race. The final workspace gates reran after that fix and the C/D needless-borrow fix.

Limits: this qualifies injected syscall failures, not real-media EIO/ENOSPC, SIGKILL or power-cut ordering. Persistent filesystem errors can prevent retraction; errors report this explicitly and retain ownership evidence for operator recovery. All outside writers must remain stopped. Final-head macOS validation was not run: shared /tmp/mac-bench.lock remains held by dav-recorder PR 789; no other job was disturbed. No production operation or merge.

C/D rows updated after C round-2 fixes. Final isolated C performance remains pending; see #792 status.

Round-2 fixes ready for final round-3 review at `5441212fd8b445fb80ff8ed640a4b33c17ae1d07`. Nothing merged. | Finding | Fix | Failing-before / passing-after evidence | |---|---|---| | R2-A1 P1, stale published unsplit | Persist a versioned digest of both input inventories, root identity/metadata, raw paths, xattrs and file hashes before publication. Recovery requires an exact match; missing/mismatched manifests refuse with instructions to preserve output and choose a fresh destination. Never recopy or merge into published output. | `r2_published_unsplit_refuses_changed_inputs`: add/edit/delete/mode changes at `published` refuse and preserve output. Original A failed this regression. | | R2-A2 P1, ignored durability errors | Check writable-file fsync before close and after metadata; check final file/directory sync and closes. Remove ignored verification fsync and global sync/syncfs substitutes. Retract publication on parent flush/close failure; retract first root if publishing the second fails. | `r2_every_prepublication_fsync_close_and_rename_failure_aborts` injects at every prepublication syscall hook for split and unsplit, requires no published roots/DONE and successful retry; `r2_retraction_fsync_errors_are_reported_too` checks repeated rollback errors. Independent LD_PRELOAD fsync #7 EIO: original A returned success, fixed A returns failure with no userdata/appdata/DONE. | | R2-A3 P2, cleanup loses ownership | Keep OWNER until all payload deletion completes and is flushed; then remove OWNER and the empty stage. | `r2_cleanup_keeps_ownership_until_payload_is_gone` interrupts payload cleanup, checks OWNER survives, retries successfully. Original A failed. | | R2-A4 P2, 64 KiB retained per tiny xattr | Shrink attribute values before retaining inventory entries. Cost is O(entries + retained xattr bytes), plus the 1 MiB copy buffer. | `r2_tiny_xattrs_retain_only_their_bytes` and `r2_metadata_heavy_inventory_retains_actual_xattr_bytes` cover capacity and 4,096 one-byte attributes; original A failed. Independent 4,096-file RSS gate: original 300,524 KiB, fixed 38,820 KiB, threshold 131,072 KiB. | **PASS at `5441212fd8b445fb80ff8ed640a4b33c17ae1d07`** on dev-02: fmt, clippy with `-D warnings`, and the full workspace suite; **1178 passed, 0 failed, 20 ignored**, job exit 0. All heavy commands ran in Nix through `~/bin/slot`. Evidence: `~/tier-r2-final-a.log` and `~/tier-r2-final-a.exit`. Commands inside Nix through `~/bin/slot`: ```sh cargo fmt --all --check cargo clippy -j4 --workspace --all-targets --locked -- -D warnings cargo test -j4 --workspace --locked --no-fail-fast ``` | PR | Final head | Workspace tests | fmt / strict clippy | |---|---|---|---| | #790 | `5441212fd8b445fb80ff8ed640a4b33c17ae1d07` | 1178 passed, 0 failed, 20 ignored | PASS | | #791 | `58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8` | 1187 passed, 0 failed, 20 ignored | PASS | | #792 | `9c8c287357240912daf895c4166bf00f2212ce4a` | 1205 passed, 0 failed, 20 ignored | PASS | | #793 | `659b0f4e420f6272e801ea78eba8fe10d66c1ca5` | 1208 passed, 0 failed, 20 ignored | PASS | A's final end-to-end gate on dev-02 also passed (`ALL PASS`): 211 entries, 161 regular files, three users and three contact originals; 36 notes before the new write and 37 after. It verifies split, immediate unsplit, unchanged old-root refusal, new split-layout writes, unsplit against the current roots and old-binary startup. ```sh cargo build -j4 -p calternal --locked python3 ci/storage-split-e2e.py /home/cua/Developer/Worktrees/tier-r2-old/target/debug/calternal target/debug/calternal /home/cua/tier-r2-final-roundtrip ``` Old binary: `d83e54db`. Evidence: dev-02 `~/tier-r2-final-a.log`, `~/tier-r2-final-a.exit` = 0; fixture `~/tier-r2-final-roundtrip`. Before/after evidence on dev-02: `~/tier-r2-a-red.log` (four expected failures), `~/tier-r2-b-red.log` (two expected failures), `~/tier-r2-reader-ancestor-red.log` (one expected failure), `~/tier-r2-fsync-probe.log` and `~/tier-r2-xattr-rss.log` (probe wrappers exit 0). External fsync/RSS probes used `e688361c`, whose runtime implementation is identical to final A; the later A change fixes only the layout test's live-SHM comparison race. The final workspace gates reran after that fix and the C/D needless-borrow fix. Limits: this qualifies injected syscall failures, not real-media EIO/ENOSPC, SIGKILL or power-cut ordering. Persistent filesystem errors can prevent retraction; errors report this explicitly and retain ownership evidence for operator recovery. All outside writers must remain stopped. Final-head macOS validation was not run: shared `/tmp/mac-bench.lock` remains held by `dav-recorder PR 789`; no other job was disturbed. No production operation or merge. C/D rows updated after C round-2 fixes. Final isolated C performance remains pending; see [#792 status](https://git.kayg.org/kayg/calternal/pulls/792#issuecomment-71522).
kayg commented 2026-10-31 21:32:32 +00:00
Author
Owner
Copy link

Independent review, round 3 (final)

Reviewed A at 5441212fd8b445fb80ff8ed640a4b33c17ae1d07, including round-1 comment 70713, round-2 comment 71053 and the final fix table. Verdict: ready within the documented offline migration contract.

Round-2 item Final status Does the test demonstrate it?
R2-A1 P1: stale published unsplit accepted on retry Resolved; no regression found. A versioned manifest binds both input trees, root identities, metadata, raw names, xattrs and file hashes. Published recovery refuses absent/mismatched manifests and preserves the output. Yes. r2_published_unsplit_refuses_changed_inputs stops at published, then tests add/edit/delete/mode changes, requires refusal, unchanged output and retained OWNER. The unchanged-input boundary tests cover successful recovery.
R2-A2 P1: ignored fsync errors before publication Resolved; no regression found. Writable descriptors are synced before close and after metadata; final file/directory sync and explicit closes propagate failures. Failed publication retracts the renamed root; second-root failure also retracts userdata. Retraction failure is reported. Yes for injected errors. r2_every_prepublication_fsync_close_and_rename_failure_aborts enumerates the hooked operations for split and unsplit, requires failure with no published roots/DONE, then successful retry. r2_retraction_fsync_errors_are_reported_too covers repeated flush errors. The recorded external fsync EIO probe also returns failure without userdata/appdata/DONE. This is not power-cut evidence.
R2-A3 P2: cleanup loses ownership Resolved; no regression found. Payload deletion and its directory flush precede OWNER removal. Yes. r2_cleanup_keeps_ownership_until_payload_is_gone interrupts payload cleanup, checks OWNER survives and reruns successfully.
R2-A4 P2: 64 KiB retained for each tiny xattr Resolved; no regression found. Values shrink before being retained. Yes. One-byte capacity and 4,096-file retained-capacity regressions pass. Recorded RSS probe falls from 300,524 to 38,820 KiB.

New findings, P0–P3: none. I also checked the new marker writes, publication/retraction ordering and cleanup paths for loss, duplication or stranded data. The fixes preserve the original roots and do not merge into published output. The prior completed-split and contacts placement protections remain present.

Independent verification on dev-01, at downstream D 659b0f4e containing these fixes, inside Nix through ~/bin/slot:

cargo test -j4 --workspace --locked r2_ -- --test-threads=2

22 passed, zero failed, including all six A regressions. Evidence: ~/tier-r3-independent.{log,exit}, exit 0. I inspected A's exact-head gate/E2E log on dev-02, ending ALL PASS, exit 0, and the recorded failing-before and external fsync/RSS probe logs. The full workspace suite and E2E were not independently rerun. No build/test ran on the control host; no code was pushed or production data accessed.

Open qualification limits remain explicit: final-head macOS checks, real-media EIO/ENOSPC, SIGKILL and power-cut ordering are unrun. The operator must stop all outside writers, keep an independent backup and follow the copy-of-data dry-run/cutover runbook. Readiness here is a code-review verdict, not a claim that those campaigns passed.

# Independent review, round 3 (final) Reviewed A at `5441212fd8b445fb80ff8ed640a4b33c17ae1d07`, including round-1 comment 70713, round-2 comment 71053 and the final fix table. **Verdict: ready** within the documented offline migration contract. | Round-2 item | Final status | Does the test demonstrate it? | |---|---|---| | R2-A1 P1: stale published unsplit accepted on retry | **Resolved; no regression found.** A versioned manifest binds both input trees, root identities, metadata, raw names, xattrs and file hashes. Published recovery refuses absent/mismatched manifests and preserves the output. | Yes. `r2_published_unsplit_refuses_changed_inputs` stops at `published`, then tests add/edit/delete/mode changes, requires refusal, unchanged output and retained OWNER. The unchanged-input boundary tests cover successful recovery. | | R2-A2 P1: ignored fsync errors before publication | **Resolved; no regression found.** Writable descriptors are synced before close and after metadata; final file/directory sync and explicit closes propagate failures. Failed publication retracts the renamed root; second-root failure also retracts userdata. Retraction failure is reported. | Yes for injected errors. `r2_every_prepublication_fsync_close_and_rename_failure_aborts` enumerates the hooked operations for split and unsplit, requires failure with no published roots/DONE, then successful retry. `r2_retraction_fsync_errors_are_reported_too` covers repeated flush errors. The recorded external fsync EIO probe also returns failure without userdata/appdata/DONE. This is not power-cut evidence. | | R2-A3 P2: cleanup loses ownership | **Resolved; no regression found.** Payload deletion and its directory flush precede OWNER removal. | Yes. `r2_cleanup_keeps_ownership_until_payload_is_gone` interrupts payload cleanup, checks OWNER survives and reruns successfully. | | R2-A4 P2: 64 KiB retained for each tiny xattr | **Resolved; no regression found.** Values shrink before being retained. | Yes. One-byte capacity and 4,096-file retained-capacity regressions pass. Recorded RSS probe falls from 300,524 to 38,820 KiB. | New findings, P0–P3: **none**. I also checked the new marker writes, publication/retraction ordering and cleanup paths for loss, duplication or stranded data. The fixes preserve the original roots and do not merge into published output. The prior completed-split and contacts placement protections remain present. Independent verification on dev-01, at downstream D `659b0f4e` containing these fixes, inside Nix through `~/bin/slot`: ```sh cargo test -j4 --workspace --locked r2_ -- --test-threads=2 ``` **22 passed, zero failed**, including all six A regressions. Evidence: `~/tier-r3-independent.{log,exit}`, exit 0. I inspected A's exact-head gate/E2E log on dev-02, ending `ALL PASS`, exit 0, and the recorded failing-before and external fsync/RSS probe logs. The full workspace suite and E2E were not independently rerun. No build/test ran on the control host; no code was pushed or production data accessed. Open qualification limits remain explicit: final-head macOS checks, real-media EIO/ENOSPC, SIGKILL and power-cut ordering are unrun. The operator must stop all outside writers, keep an independent backup and follow the copy-of-data dry-run/cutover runbook. Readiness here is a code-review verdict, not a claim that those campaigns passed.
kayg commented 2026-11-03 11:54:20 +00:00
Author
Owner
Copy link

A on dev 41f2efa3: resolutions and the prod-copy dry run

Head 91c034ebb36ca42a19530c50996244e892e4acbe, unchanged by this takeover (the earlier agent's rebase was already on 41f2efa3; its merge-base is 41f2efa3).

Resolutions (kept dev's side everywhere it overlaps)

  • Cache fast path and layout::NOTES state classification: dev's, kept. The split sorts with the same calternal_core::layout list the server opens state with, so a name the server treats as app state can never land in userdata.
  • Accepted-generation reads, executor recovery/deletion (#814), #828 restart stamp validation, search/peek source reads: A does not touch them. git diff origin/dev <D head> -- crates/notes/src/peek.rs crates/notes/src/meta/restart.rs crates/notes/src/write_executor/recovery.rs is empty for the whole stack.
  • fsio: both sides' coverage kept.

Migration dry run on a copy of prod (prod never written)

Source. The deploy backup backup-pre-41f2efa3-20261103T041034807656485Z.tgz holds app state only (docs/deploy.md): 712 note rows but no note files. So the copy has two parts:

  • that tarball, copied to dev-01 over NetBird (SHA-256 0f9f30d2… on both ends);
  • a read-only tar of the prod Home trees (nice/ionice, app-state names excluded), streamed to dev-01 (SHA-256 df415354… on both ends, 18.4 MB).

Combined: 1 person, 1,291 entries (15 folders, 1,276 files), 48.35 MB. Prod was only read.

Method. crates/calternal release binary at the D tree, run as the documented offline, copy-only steps (storage split --dry-run, --confirm-stopped, then storage unsplit). Before/after manifests hash every file and compare type, size, SHA-256, mode, mtime and link target. Servers ran on loopback with --dev-user, and comparisons go through the API: notes.list (all pages, both show modes), notes.get text hashes, notes.history, search.query for 40 title-derived terms, notes.rename + notes.undo.

Check Result
storage split --dry-run exit 0, 0.0 s, peak RSS 11 MB; wrote nothing
storage split --confirm-stopped exit 0, 98.9 s, peak RSS 12.9 MB; old root gets .calternal-split-done
Old root vs userdata+appdata (merged) pass: 1,291/1,291 entries, 1,276/1,276 files byte-identical; 0 missing, extra, mode or mtime differences
Old root unchanged apart from the marker pass (same comparison)
Sorting userdata 739 entries, appdata 558; 0 Home files in appdata; 0 history.db/meta.db/auth.db in userdata
storage unsplit round trip exit 0, 126.3 s, 12.9 MB; pass: identical to the old root, 1,276/1,276 files
Serve: dev on the old layout vs head on split roots, 3 starts each (+ head on the single root) at every matching start: 713/713 ids, paths, titles, kinds, tags; 713/713 texts; 40/40 identical search hit sets (658 hits); history 800 versions, 712/713 identical ³
Rename + undo on the split roots rename done, undo done, every path restored, same 713 ids; the renamed note's body byte-identical, only its last edited value changes. Same as dev on the single root.
Rollback after writes: storage unsplit, then the pre-split dev binary exit 0, 107.5 s, 13.4 MB; dev serves 713 notes, paths and titles 713/713; the renamed note carries its 2 new history versions (802 vs 800), as expected

³ The one history row that differs is today's daily note, whose version time is set at server start. It differs the same way between two dev copies started at different times.

Seen in dev as well (not tiering): on this data, the second start re-describes 273 list titles that came from the 04:10 meta.db. Dev 41f2efa3 does the same, and head matches dev at every start.

Prod migration (not run; needs an image that contains A):

# preflight (reads only)
podman run --rm --userns=keep-id:uid=10001,gid=10001 -v /srv/calternal-app:/srv/calternal-app:Z localhost/calternal-app:current \
  storage split --from /srv/calternal-app/data --userdata /srv/calternal-app/userdata --appdata /srv/calternal-app/appdata --dry-run
# stop calternal.service, then the same with --confirm-stopped, then switch the quadlet to two volumes
# (CALTERNAL_USERDATA=/userdata, CALTERNAL_APPDATA=/appdata) and start.

Expected stopped time: the split took 99 s for these 1,291 entries on dev-01 (one fsync per file), so plan ~2 min split + ~1 min stop/start ≈ 3 min. First start: healthy in 0.2 s, discovery done in 1.6 s.

Clean-up. Every restored tree is deleted (each script's EXIT trap). The two transfer files on dev-01 were deleted at the end. No note content was printed or posted: only counts, hashes of whole archives, timings and pass/fail.

## A on dev `41f2efa3`: resolutions and the prod-copy dry run Head `91c034ebb36ca42a19530c50996244e892e4acbe`, unchanged by this takeover (the earlier agent's rebase was already on `41f2efa3`; its merge-base is `41f2efa3`). ### Resolutions (kept dev's side everywhere it overlaps) - **Cache fast path and `layout::NOTES` state classification:** dev's, kept. The split sorts with the same `calternal_core::layout` list the server opens state with, so a name the server treats as app state can never land in userdata. - **Accepted-generation reads, executor recovery/deletion (#814), #828 restart stamp validation, search/peek source reads:** A does not touch them. `git diff origin/dev <D head> -- crates/notes/src/peek.rs crates/notes/src/meta/restart.rs crates/notes/src/write_executor/recovery.rs` is empty for the whole stack. - **fsio:** both sides' coverage kept. ### Migration dry run on a copy of prod (prod never written) **Source.** The deploy backup `backup-pre-41f2efa3-20261103T041034807656485Z.tgz` holds app state only (`docs/deploy.md`): 712 note rows but no note files. So the copy has two parts: - that tarball, copied to dev-01 over NetBird (SHA-256 `0f9f30d2…` on both ends); - a read-only `tar` of the prod Home trees (`nice`/`ionice`, app-state names excluded), streamed to dev-01 (SHA-256 `df415354…` on both ends, 18.4 MB). Combined: 1 person, 1,291 entries (15 folders, 1,276 files), 48.35 MB. Prod was only read. **Method.** `crates/calternal` release binary at the D tree, run as the documented offline, copy-only steps (`storage split --dry-run`, `--confirm-stopped`, then `storage unsplit`). Before/after manifests hash every file and compare type, size, SHA-256, mode, mtime and link target. Servers ran on loopback with `--dev-user`, and comparisons go through the API: `notes.list` (all pages, both `show` modes), `notes.get` text hashes, `notes.history`, `search.query` for 40 title-derived terms, `notes.rename` + `notes.undo`. | Check | Result | |---|---| | `storage split --dry-run` | exit 0, 0.0 s, peak RSS 11 MB; wrote nothing | | `storage split --confirm-stopped` | exit 0, **98.9 s, peak RSS 12.9 MB**; old root gets `.calternal-split-done` | | Old root vs userdata+appdata (merged) | **pass**: 1,291/1,291 entries, 1,276/1,276 files byte-identical; 0 missing, extra, mode or mtime differences | | Old root unchanged apart from the marker | **pass** (same comparison) | | Sorting | userdata 739 entries, appdata 558; 0 Home files in appdata; 0 `history.db`/`meta.db`/`auth.db` in userdata | | `storage unsplit` round trip | exit 0, 126.3 s, 12.9 MB; **pass**: identical to the old root, 1,276/1,276 files | | Serve: dev on the old layout vs head on split roots, 3 starts each (+ head on the single root) | at every matching start: **713/713** ids, paths, titles, kinds, tags; **713/713** texts; **40/40** identical search hit sets (658 hits); history 800 versions, 712/713 identical ³ | | Rename + undo on the split roots | rename `done`, undo `done`, every path restored, same 713 ids; the renamed note's body byte-identical, only its `last edited` value changes. **Same as dev on the single root.** | | Rollback after writes: `storage unsplit`, then the pre-split dev binary | exit 0, 107.5 s, 13.4 MB; dev serves 713 notes, paths and titles 713/713; the renamed note carries its 2 new history versions (802 vs 800), as expected | ³ The one history row that differs is today's daily note, whose version time is set at server start. It differs the same way between two dev copies started at different times. Seen in dev as well (not tiering): on this data, the **second** start re-describes **273** list titles that came from the 04:10 `meta.db`. Dev `41f2efa3` does the same, and head matches dev at every start. **Prod migration** (not run; needs an image that contains A): ```sh # preflight (reads only) podman run --rm --userns=keep-id:uid=10001,gid=10001 -v /srv/calternal-app:/srv/calternal-app:Z localhost/calternal-app:current \ storage split --from /srv/calternal-app/data --userdata /srv/calternal-app/userdata --appdata /srv/calternal-app/appdata --dry-run # stop calternal.service, then the same with --confirm-stopped, then switch the quadlet to two volumes # (CALTERNAL_USERDATA=/userdata, CALTERNAL_APPDATA=/appdata) and start. ``` Expected stopped time: the split took 99 s for these 1,291 entries on dev-01 (one fsync per file), so plan **~2 min split + ~1 min stop/start ≈ 3 min**. First start: healthy in 0.2 s, discovery done in 1.6 s. **Clean-up.** Every restored tree is deleted (each script's EXIT trap). The two transfer files on dev-01 were deleted at the end. No note content was printed or posted: only counts, hashes of whole archives, timings and pass/fail.
kayg commented 2026-11-03 11:54:21 +00:00
Author
Owner
Copy link

Gate table — stack on dev 41f2efa3 (takeover after the Codex usage limit)

Heads: #790 A 91c034eb → #791 B efcaaaf7 → #792 C 487e0589 → #793 D d422d0ca. Base: dev 41f2efa387b74175971a838f602535089d0d5fb4 (what prod runs). Nothing merged.

Gate (dev-01, Nix, ~/bin/slot) A 91c034eb B efcaaaf7 C 487e0589 D d422d0ca
cargo fmt --all --check pass pass pass pass
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings pass pass pass pass ¹
cargo test -j4 --workspace --locked -- --test-threads=2 1,495 passed, 0 failed, 22 ignored 1,504 / 0 / 22 1,537 / 0 / 22 ² 1,540 / 0 / 22 ¹

¹ Run by the earlier agent at 89989e62; git diff 89989e62 d422d0ca is empty (same tree).
² Run at 896babb4; 487e0589 differs only by the order of two #[cfg(test)] mod lines (cargo fmt).

Perf VMs (binaries built on dev-01 in release at the D tree, relayed through the control host, SHA-256 checked on both perf VMs: C 2006ab8e…, dev 724535bd…, harness 980045d9…):

Gate Head (C/D) dev 41f2efa3 Limit Result
cold.index, 3 cold pairs, perf-01, p95 = max of 3 25.0 s (22.6 / 25.0 / 24.3) 20.9 s (20.9 / 18.6 / 20.2) ≤ 1.25× dev and ≤ 60 s pass, 1.196× (was 1.248×)
cold.mem_peak (harness), 3 cold runs, max 227 MB 291 MB not worse than dev pass, 0.78× dev (was 175 vs 186 MiB = 0.94×)
server VmHWM, sampled every 200 ms, max over 4 samples 227.5 MiB 291.5 MiB — head lower
open.note@100000 p95, warm pair 12.4 ms 10.6 ms ≤ 15 ms pass
idle.rss@100000, warm pair 60.6 MB 71.8 MB ≤ 100 MB pass
200 sequential notes.append, 30 alternating pairs, perf-02, median total 1.578 s 1.595 s within 1% of dev pass, −1.09% (pair-ratio median 0.992)

Protocol unchanged from the fab01191 run: ~/bin/perf-lock ~/bin/slot run-perf.sh, per sample calternal-perf --sizes 100k --runs 1 --cold-runs {1,1,1,0} --seed 1 --idle-secs 60 --budgets {"open.note@100000":{"p95":15},"idle.rss@100000":{"p95":100}}, C then dev, same VM, no page-cache drop; all 8 reports have no scenario failures. No limit was changed.

Notes:

  • Absolute cold times rose for both sides against the fab01191 run (dev a55521f9 12.6 s → dev 41f2efa3 20.9 s on the same harness and VM), so only the ratio is comparable. The absolute memory numbers are from a different source (harness cold.mem_peak on perf-01 vs one dev-01 sample earlier); the ratio to dev improved.
  • The append gate is noisy: single pairs range 0.82–1.48×. A first 5-pair run read +1.15%; with 30 pairs the median is −1.09%. A 1% gate needs ≥ 30 pairs.
  • Reports, logs and scripts: control host ~/Developer/perf-results/pr792-41f2efa3/. Perf VM binaries and work dirs deleted.
## Gate table — stack on dev `41f2efa3` (takeover after the Codex usage limit) Heads: #790 A `91c034eb` → #791 B `efcaaaf7` → #792 C `487e0589` → #793 D `d422d0ca`. Base: dev `41f2efa387b74175971a838f602535089d0d5fb4` (what prod runs). Nothing merged. | Gate (dev-01, Nix, `~/bin/slot`) | A `91c034eb` | B `efcaaaf7` | C `487e0589` | D `d422d0ca` | |---|---|---|---|---| | `cargo fmt --all --check` | pass | pass | pass | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | pass | pass | pass ¹ | | `cargo test -j4 --workspace --locked -- --test-threads=2` | 1,495 passed, 0 failed, 22 ignored | 1,504 / 0 / 22 | 1,537 / 0 / 22 ² | 1,540 / 0 / 22 ¹ | ¹ Run by the earlier agent at `89989e62`; `git diff 89989e62 d422d0ca` is empty (same tree). ² Run at `896babb4`; `487e0589` differs only by the order of two `#[cfg(test)] mod` lines (`cargo fmt`). **Perf VMs** (binaries built on dev-01 in release at the D tree, relayed through the control host, SHA-256 checked on both perf VMs: C `2006ab8e…`, dev `724535bd…`, harness `980045d9…`): | Gate | Head (C/D) | dev `41f2efa3` | Limit | Result | |---|---:|---:|---|---| | cold.index, 3 cold pairs, perf-01, p95 = max of 3 | 25.0 s (22.6 / 25.0 / 24.3) | 20.9 s (20.9 / 18.6 / 20.2) | ≤ 1.25× dev and ≤ 60 s | **pass, 1.196×** (was 1.248×) | | cold.mem_peak (harness), 3 cold runs, max | 227 MB | 291 MB | not worse than dev | **pass**, 0.78× dev (was 175 vs 186 MiB = 0.94×) | | server VmHWM, sampled every 200 ms, max over 4 samples | 227.5 MiB | 291.5 MiB | — | head lower | | open.note@100000 p95, warm pair | 12.4 ms | 10.6 ms | ≤ 15 ms | pass | | idle.rss@100000, warm pair | 60.6 MB | 71.8 MB | ≤ 100 MB | pass | | 200 sequential `notes.append`, 30 alternating pairs, perf-02, median total | 1.578 s | 1.595 s | within 1% of dev | **pass, −1.09%** (pair-ratio median 0.992) | Protocol unchanged from the `fab01191` run: `~/bin/perf-lock ~/bin/slot run-perf.sh`, per sample `calternal-perf --sizes 100k --runs 1 --cold-runs {1,1,1,0} --seed 1 --idle-secs 60 --budgets {"open.note@100000":{"p95":15},"idle.rss@100000":{"p95":100}}`, C then dev, same VM, no page-cache drop; all 8 reports have no scenario failures. No limit was changed. Notes: - Absolute cold times rose for **both** sides against the `fab01191` run (dev a55521f9 12.6 s → dev 41f2efa3 20.9 s on the same harness and VM), so only the ratio is comparable. The absolute memory numbers are from a different source (harness `cold.mem_peak` on perf-01 vs one dev-01 sample earlier); the ratio to dev improved. - The append gate is noisy: single pairs range 0.82–1.48×. A first 5-pair run read +1.15%; with 30 pairs the median is −1.09%. A 1% gate needs ≥ 30 pairs. - Reports, logs and scripts: control host `~/Developer/perf-results/pr792-41f2efa3/`. Perf VM binaries and work dirs deleted.
kayg commented 2026-11-03 21:38:06 +00:00
Author
Owner
Copy link

Perf pass on the stack (branch perf/tiering-761-fix, head 3a65e27c): nothing in this PR's files changed. A/B tables, causes and the still-open cold-index/first-page gap are on #792 (comment above) and #793. The stack stays not mergeable under the no-regressions rule until the gap on #792 is closed or kayg decides.

Perf pass on the stack (branch `perf/tiering-761-fix`, head `3a65e27c`): nothing in this PR's files changed. A/B tables, causes and the still-open cold-index/first-page gap are on #792 (comment above) and #793. The stack stays **not mergeable** under the no-regressions rule until the gap on #792 is closed or kayg decides.
kayg commented 2026-11-07 19:10:07 +00:00
Author
Owner
Copy link

housekeeping #1009: needs-decision.

Old two-root split/unsplit stack conflicts with tiering v3 three-root architecture (#968/#970/#1008). Retain migration/race fixtures and offline-lock evidence.

original head: 91c034ebb36ca42a19530c50996244e892e4acbe. against dev cd7f84432febfc63b788fbb9b9f9950452982d74, git cherry reports 11 unmatched and 0 equivalent patches. the landing audit lists each candidate missing commit and stable patch match in pr-790.md; unmatched patches can include superseded behavior.

left open, source branch unchanged. no rebase or new test claim for this decision lane.

housekeeping #1009: needs-decision. Old two-root split/unsplit stack conflicts with tiering v3 three-root architecture (#968/#970/#1008). Retain migration/race fixtures and offline-lock evidence. original head: `91c034ebb36ca42a19530c50996244e892e4acbe`. against dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, git cherry reports 11 unmatched and 0 equivalent patches. the [landing audit](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5) lists each candidate missing commit and stable patch match in `pr-790.md`; unmatched patches can include superseded behavior. left open, source branch unchanged. no rebase or new test claim for this decision lane.
kayg commented 2026-11-07 21:21:15 +00:00
Author
Owner
Copy link

Superseded by tiering v3 (#968/#970/#1008). The branch and original tips are archived under archive/ (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.

Superseded by tiering v3 (#968/#970/#1008). The branch and original tips are archived under `archive/` (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.
kayg closed this pull request 2026-11-07 21:21:16 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!790
No description provided.