kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

feat: storage tiering slice 1 (#761): appdata/userdata split + offline migration #784

Closed
kayg wants to merge 8 commits from feat/tiering-761-s1 into dev
pull from: feat/tiering-761-s1
merge into: kayg:dev
Conversation 3 Commits 8 Files changed 60 +3380 -878
kayg commented 2026-10-31 05:32:49 +00:00
Owner
Copy link

Refs #761 (plan), #731. Slice 1 of storage tiering: the mandatory appdata/userdata split plus an offline migration command for single-root installs. No tier placement, no HDD moves, no dedupe, no production cutover.

Continues the paused Codex snapshot wip/6by32-01-tier-761-s1-pause-20261030 @ b343e0cb (60 files, gates never run). I checked it against the plan and decision; added what was missing (below).

Plan recap (what this slice is)

  • Layout A, decided: two non-nested roots. appdata = databases, indexes, auth, cache, admin socket, state lock (e.g. each user's index.db, so saves stop triggering quota recounts). userdata = users/<uuid>/... user files, the same logical tree an SSD/HDD pair will later join. One disk: /srv/calternal-app/appdata and /srv/calternal-app/userdata.
  • --userdata / --appdata (+ CALTERNAL_USERDATA / CALTERNAL_APPDATA) are canonical; --data / --state stay as aliases. Defaults are ./userdata and ./appdata; a shared or nested root is refused before anything is created. Startup never migrates a single root, even with --create.
  • Reader/root/index seam from the snapshot (pinned-root readers, paged identity index) is kept as is; see docs/plans/0761-slice-1-validation.md.
  • calternal storage split (snapshot) and new calternal storage unsplit (rollback).

Divergences found and fixed vs the plan

  • No rollback path existed beyond prose ("consolidate by hand"). Added storage unsplit: reads both roots, builds <to>.unsplit-partial, and publishes <to> by one atomic no-replace rename; resumable.
  • No disk-full or permission tests, and no operator guidance for either. Added explain() messages (nothing deleted, how to rerun) and tests.
  • A stale admin.sock left by a killed server inside system/ would have made the copy fail on a special file; it is now skipped (a process resource; the lock proves nobody owns it).
  • The validation doc claimed build-host evidence for gates that had never run; it now records the real run.
  • Fresh-install and CLI end-to-end coverage added (storage_startup.rs).

Failure table -> test map

All in crates/calternal/src/storage_split.rs (mod tests) unless noted.

Failure Required behaviour Test
Fresh install Separate roots from first start; matching pairing id; no DBs in userdata; nested/same roots refused with nothing created storage_startup.rs::fresh_install_is_split_from_the_first_start_and_never_nests
Single-root migration Every file byte-exact in the right root; WAL/SHM copied; modes preserved; source retained appdata_userdata_copy_is_exact_restartable_and_source_retained, storage_startup.rs::cli_split_serves_then_unsplit_rolls_back_to_the_exact_single_root (also starts the real server on the result)
Rerun / idempotent Same result and byte count; never overwrites newer destination state same first test (reruns, then edits appdata and expects refusal)
Crash at every step Fault injected at every write / sync / rename / mkdir boundary (first marker short write included), then rerun completes failure_at_every_copy_and_first_marker_boundary_recovers_repeatedly, crash_after_owned_directory_and_partial_payload_restarts
Disk full (ENOSPC) StorageFull injected at every boundary: source unchanged, completion marker only after every file, advice printed, rerun completes identically disk_full_at_every_step_keeps_the_source_and_resumes_after_space_is_freed
Permission error Unreadable source file; unwritable destination parent: PermissionDenied, no completion marker, nothing created, rerun after fixing succeeds (skips itself when run as root) unreadable_source_and_unwritable_destination_fail_closed_then_resume
Foreign/locked/unsafe input Locked source, foreign or nested destination, symlinks all refuse and preserve locks_foreign_destinations_symlinks_and_nested_layout_fail_closed
Rollback before new writes Source byte-identical after success and after interrupted runs, so the old binary just starts on it rollback_before_any_new_write_leaves_the_original_ready_for_the_old_binary
Rollback after new writes Current state wins (newer DB, new file, deleted file), inputs untouched, no partial root ever published rollback_after_new_writes_consolidates_current_state_into_one_root, rollback_survives_failure_at_every_step_and_never_publishes_a_partial_root
Rollback refusals Existing destination, mismatched pairing ids, Home entry shadowing app state rollback_refuses_foreign_pairs_collisions_and_existing_destinations
CLI safety --confirm-stopped required; split/unsplit flags validated args.rs tests

How to run the migration by hand

Stop Calternal and every outside writer (sync tools, editors), wait for lingering processes, take a coordinated backup of the whole root, run as the storage owner. Both destination parents must exist; both destination roots must be absent.

calternal storage split --from /srv/calternal-app/data \
  --userdata /srv/calternal-app/userdata \
  --appdata /srv/calternal-app/appdata --confirm-stopped
calternal serve --userdata /srv/calternal-app/userdata --appdata /srv/calternal-app/appdata ...

Failure: fix the cause (it says what) and rerun the identical command. Rollback before new writes: start the old binary on the untouched original. Rollback after new writes:

calternal storage unsplit --to /srv/calternal-app/data-rollback \
  --userdata /srv/calternal-app/userdata --appdata /srv/calternal-app/appdata --confirm-stopped

Full text: docs/storage.md.

What the prod step needs (not done here; by hand, approved separately)

  1. Deploy this binary but keep the service stopped. Confirm free space of at least the current data size on the target disk (the copy duplicates; the original is kept).
  2. Stop the service and verify no calternal process or lock holder; stop file-sync/outside writers; take and verify a backup of /srv/calternal-app/data.
  3. Run storage split as the data owner; record the printed verified byte count. Compare with du -sb of the source.
  4. Update the unit/env (--userdata, --appdata; CALTERNAL_DATA/CALTERNAL_STATE are no longer enough on their own since defaults changed), start, verify sign-in, notes, versions, attachments, quota page.
  5. Keep /srv/calternal-app/data and the backup until satisfied. After writes, rollback is unsplit, never "point the old binary at the old copy".

Gates

Run on the Linux dev VM inside the repo's Nix dev shell, -j4, through the slot helper. Gated commit 4f0fd265; head differs only by docs.

  • cargo fmt --all --check: pass
  • cargo clippy --workspace --all-targets --locked -- -D warnings: pass
  • cargo test --workspace --locked --no-fail-fast: 1180 passed, 0 failed, 20 ignored (stress/qualification gates that the plan reserves for the perf VMs)

Decisions and open risks

  • Decided without asking: unsplit instead of manual rollback; explain() text on ENOSPC/EACCES; stale admin.sock skipped; split writes three small marker files into the source root (.calternal-split-prepare, .calternal-split-id, .lock), which the old binary ignores. A source with hard links is copied as independent files (no mutation of the source; sharing is lost).
  • ENOSPC and EACCES are injected at the same boundaries as crashes, plus real permission bits; there is no real full-filesystem test (needs privileged tmpfs). Not a power-cut test: that campaign stays on the perf VMs.
  • The snapshot's ~2,700 lines outside the migration (paged index, pinned reader, writer queue, tiering-scale) were gated here but only lightly reviewed by me; docs/plans/0761-slice-1-validation.md lists the qualification still owed (history-heavy open, 15 ms open, FO1) and cutover stays withheld.
  • Rollback with unsplit copies the whole install again; needs free space equal to the data size.
  • Not merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN

Refs #761 (plan), #731. Slice 1 of storage tiering: the **mandatory appdata/userdata split** plus an **offline migration command** for single-root installs. No tier placement, no HDD moves, no dedupe, no production cutover. Continues the paused Codex snapshot `wip/6by32-01-tier-761-s1-pause-20261030` @ b343e0cb (60 files, gates never run). I checked it against the plan and decision; added what was missing (below). ## Plan recap (what this slice is) - Layout A, decided: two non-nested roots. `appdata` = databases, indexes, auth, cache, admin socket, state lock (e.g. each user's `index.db`, so saves stop triggering quota recounts). `userdata` = `users/<uuid>/...` user files, the same logical tree an SSD/HDD pair will later join. One disk: `/srv/calternal-app/appdata` and `/srv/calternal-app/userdata`. - `--userdata` / `--appdata` (+ `CALTERNAL_USERDATA` / `CALTERNAL_APPDATA`) are canonical; `--data` / `--state` stay as aliases. Defaults are `./userdata` and `./appdata`; a shared or nested root is refused before anything is created. Startup never migrates a single root, even with `--create`. - Reader/root/index seam from the snapshot (pinned-root readers, paged identity index) is kept as is; see `docs/plans/0761-slice-1-validation.md`. - `calternal storage split` (snapshot) and **new** `calternal storage unsplit` (rollback). ## Divergences found and fixed vs the plan - No rollback path existed beyond prose ("consolidate by hand"). Added `storage unsplit`: reads both roots, builds `<to>.unsplit-partial`, and publishes `<to>` by one atomic no-replace rename; resumable. - No disk-full or permission tests, and no operator guidance for either. Added `explain()` messages (nothing deleted, how to rerun) and tests. - A stale `admin.sock` left by a killed server inside `system/` would have made the copy fail on a special file; it is now skipped (a process resource; the lock proves nobody owns it). - The validation doc claimed build-host evidence for gates that had never run; it now records the real run. - Fresh-install and CLI end-to-end coverage added (`storage_startup.rs`). ## Failure table -> test map All in `crates/calternal/src/storage_split.rs` (`mod tests`) unless noted. | Failure | Required behaviour | Test | |---|---|---| | Fresh install | Separate roots from first start; matching pairing id; no DBs in userdata; nested/same roots refused with nothing created | `storage_startup.rs::fresh_install_is_split_from_the_first_start_and_never_nests` | | Single-root migration | Every file byte-exact in the right root; WAL/SHM copied; modes preserved; source retained | `appdata_userdata_copy_is_exact_restartable_and_source_retained`, `storage_startup.rs::cli_split_serves_then_unsplit_rolls_back_to_the_exact_single_root` (also starts the real server on the result) | | Rerun / idempotent | Same result and byte count; never overwrites newer destination state | same first test (reruns, then edits appdata and expects refusal) | | Crash at every step | Fault injected at every write / sync / rename / mkdir boundary (first marker short write included), then rerun completes | `failure_at_every_copy_and_first_marker_boundary_recovers_repeatedly`, `crash_after_owned_directory_and_partial_payload_restarts` | | Disk full (ENOSPC) | `StorageFull` injected at every boundary: source unchanged, completion marker only after every file, advice printed, rerun completes identically | `disk_full_at_every_step_keeps_the_source_and_resumes_after_space_is_freed` | | Permission error | Unreadable source file; unwritable destination parent: `PermissionDenied`, no completion marker, nothing created, rerun after fixing succeeds (skips itself when run as root) | `unreadable_source_and_unwritable_destination_fail_closed_then_resume` | | Foreign/locked/unsafe input | Locked source, foreign or nested destination, symlinks all refuse and preserve | `locks_foreign_destinations_symlinks_and_nested_layout_fail_closed` | | Rollback before new writes | Source byte-identical after success and after interrupted runs, so the old binary just starts on it | `rollback_before_any_new_write_leaves_the_original_ready_for_the_old_binary` | | Rollback after new writes | Current state wins (newer DB, new file, deleted file), inputs untouched, no partial root ever published | `rollback_after_new_writes_consolidates_current_state_into_one_root`, `rollback_survives_failure_at_every_step_and_never_publishes_a_partial_root` | | Rollback refusals | Existing destination, mismatched pairing ids, Home entry shadowing app state | `rollback_refuses_foreign_pairs_collisions_and_existing_destinations` | | CLI safety | `--confirm-stopped` required; split/unsplit flags validated | `args.rs` tests | ## How to run the migration by hand Stop Calternal and every outside writer (sync tools, editors), wait for lingering processes, take a coordinated backup of the whole root, run as the storage owner. Both destination parents must exist; both destination roots must be absent. ```sh calternal storage split --from /srv/calternal-app/data \ --userdata /srv/calternal-app/userdata \ --appdata /srv/calternal-app/appdata --confirm-stopped calternal serve --userdata /srv/calternal-app/userdata --appdata /srv/calternal-app/appdata ... ``` Failure: fix the cause (it says what) and rerun the identical command. Rollback before new writes: start the old binary on the untouched original. Rollback after new writes: ```sh calternal storage unsplit --to /srv/calternal-app/data-rollback \ --userdata /srv/calternal-app/userdata --appdata /srv/calternal-app/appdata --confirm-stopped ``` Full text: `docs/storage.md`. ## What the prod step needs (not done here; by hand, approved separately) 1. Deploy this binary but keep the service stopped. Confirm free space of at least the current data size on the target disk (the copy duplicates; the original is kept). 2. Stop the service and verify no `calternal` process or lock holder; stop file-sync/outside writers; take and verify a backup of `/srv/calternal-app/data`. 3. Run `storage split` as the data owner; record the printed verified byte count. Compare with `du -sb` of the source. 4. Update the unit/env (`--userdata`, `--appdata`; `CALTERNAL_DATA`/`CALTERNAL_STATE` are no longer enough on their own since defaults changed), start, verify sign-in, notes, versions, attachments, quota page. 5. Keep `/srv/calternal-app/data` and the backup until satisfied. After writes, rollback is `unsplit`, never "point the old binary at the old copy". ## Gates Run on the Linux dev VM inside the repo's Nix dev shell, `-j4`, through the slot helper. Gated commit `4f0fd265`; head differs only by docs. - `cargo fmt --all --check`: pass - `cargo clippy --workspace --all-targets --locked -- -D warnings`: pass - `cargo test --workspace --locked --no-fail-fast`: 1180 passed, 0 failed, 20 ignored (stress/qualification gates that the plan reserves for the perf VMs) ## Decisions and open risks - Decided without asking: unsplit instead of manual rollback; `explain()` text on ENOSPC/EACCES; stale `admin.sock` skipped; split writes three small marker files into the source root (`.calternal-split-prepare`, `.calternal-split-id`, `.lock`), which the old binary ignores. A source with hard links is copied as independent files (no mutation of the source; sharing is lost). - ENOSPC and EACCES are injected at the same boundaries as crashes, plus real permission bits; there is no real full-filesystem test (needs privileged tmpfs). Not a power-cut test: that campaign stays on the perf VMs. - The snapshot's ~2,700 lines outside the migration (paged index, pinned reader, writer queue, tiering-scale) were gated here but only lightly reviewed by me; `docs/plans/0761-slice-1-validation.md` lists the qualification still owed (history-heavy open, 15 ms open, FO1) and cutover stays withheld. - Rollback with `unsplit` copies the whole install again; needs free space equal to the data size. - Not merged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN 
Codex job tier-761-s1 was paused mid-work; 60 changed files, gates not run. Uses appdata/userdata naming per kayg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: record real gate evidence for slice 1
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
effb44dc93
kayg commented 2026-10-31 05:36:52 +00:00
Author
Owner
Copy link

Independent review, round 1

Reviewed head effb44dc against origin/dev. Read the PR body, #761 and docs/plans/0761-slice-1-validation.md. This was a read-only source review: nothing was built or run (control host). Items marked verify are read from the code and have not been reproduced. Line numbers are in the PR head.

Verdict: split the PR, and fix the P0/P1 migration items before anyone runs it on prod. Prod runs storage split by hand, so it is a different risk class from the roughly 2,700 lines of runtime change (paged index, pinned reader, scan/watch semantics) that every user gets. Details at the end.

P0: can lose, strand or duplicate user data

P0-1. A completed split can be re-run, and re-running resurrects deleted data.
crates/calternal/src/storage_split.rs:27-45, 136, 454. Once split succeeds, the source intent still matches (Ok(old) if old == binding), so the same command proceeds. copy() only checks "target exists and is equal, else differs-refuse". A target that is absent is copied again. Scenario: split, start the new server, user deletes a note or purges Trash, operator re-runs "the identical command" (the PR tells them to on any failure). The deleted file comes back into live userdata, and a file created or moved in the new layout is not touched. Nothing consults .calternal-split-complete. Same for any deletion between a failed first attempt and a retry. Also, a destination is never swept for entries that are not in the source, so stale extras survive a resume.
Fix: if .calternal-split-complete exists, refuse (or run verify-only, never write). Record source file inventory (path, size, hash) in the intent at first run and refuse on any change at resume. Test: split, delete one file in destination, re-run, expect refusal and file still absent.

P0-2. users/<uuid>/contacts/ is classified as a Home file, but the code reads it from appdata. verify
storage_split.rs:262-277 (is_state_item allowlist has notes, search, attachments, tasks, index.db*), vs crates/contacts-actions/src/cards.rs:33 (Cards::new(user_state).root = user_state/contacts/raw) and write.rs:156 (self.state.join("users")...). After the split, the original-card copies (unknown fields, original inline photos; the fixture in this very file calls them out) land in userdata/users/<u>/contacts/raw and the server looks in appdata/.... They are stranded, not deleted, but contacts will treat the originals as missing and may regenerate lossy copies. Tests miss it because the fixture uses Notes/card.vcf, not contacts/raw.
Fix: derive the state list from the code (each plugin's app_data_dir / state joins: tasks, contacts, search, attachments, notes, plus any I missed), and add a test that fails if a directory the server creates in appdata is classified as userdata. Better: invert to a startup-written manifest or have a fixture produced by a real server run on a legacy root and then split.
Related: a user Home folder literally named notes, search, tasks, attachments or contacts (case-sensitive fs) is silently moved into appdata, and unsplit refuses it as ambiguous (:200-215). Classify by exact known path plus refusal on collision at split time too.

P1: must fix before prod

P1-1. mtimes are not preserved (files or directories). fsio/src/reader.rs copy_attributes copies mode and xattrs only; no futimens. Effects: the notes scan state is (size, mtime) per file (notes/src/scan.rs:408-443), so the first start after split sees every note as changed (full cold scan, re-identify of every head, 100k-note cost, risk of spurious "changed outside" events); outside sync tools (Syncthing etc.) will see every file as modified; file managers show wrong dates. The PR body says "modes preserved" only. Fix: copy atime/mtime (nanosecond) on files and, after children, on directories; test it; keep scan state valid.

P1-2. "Offline" is only a .lock flock plus a flag. args.rs (--confirm-stopped is just a bool). Outside writers are not excluded; the only defense is a re-compare of each file right after its own copy (:482-500). A file edited after it was copied, a file created after read_dir of its directory, or a file renamed mid-run is not noticed on the first run: source.validate() (:101) checks root identity only, and no final verification pass runs before the completion marker. Fix: after all copies, re-walk the source and compare inventory (path, size, mtime, ino) to the one taken at the start; refuse completion on any difference. Also add a test that mutates the source mid-run (hook in boundary()).

P1-3. Marker files in the source root are an operational trap. .calternal-split-prepare, .calternal-split-id stay in the source forever. A later split to different paths hits "different or torn split intent" (:29-34) with no documented way out; abandoning the split and running the old binary for days, then re-splitting, mixes the P0-1 resume with a changed source. Also: the PR claims the old binary "ignores" them, but the new split itself rejects unknown root names, and I could not find a test starting the old binary on a marked source. Fix: document exact cleanup; make the intent carry a source inventory hash so a changed source refuses with a clear message; test old-binary start on a marked source (or at least check_legacy).

P1-4. Unknown-entry refusal has no override. :96, :228 refuse any unknown root entry. If /srv/calternal-app/data is a mount point, lost+found is there, and the operator cannot classify it, only delete or move it. Also cache/ is copied in full though regenerable. Add a preflight storage split --dry-run that lists classification (counts and bytes per root, every unknown entry, every symlink, hard link, sparse file, special file, non-UTF-8 name) without touching anything, and have the runbook require it on a copy of prod data first.

P1-5. Symlinks and special files in user data abort the whole split. open_inventory uses NO_SYMLINKS and demands regular files (reader.rs). Safe, but any single symlink in Documents or Photos strands the migration, and note that the same change makes the Notes watcher stop following symlinked folders (notes/src/watch.rs:562-590 now uses DirEntry::file_type, which does not follow). verify: a symlinked folder inside Notes/ worked on dev and silently disappears from scans after this PR. Decide and document; add tests.

P1-6. One bad file poisons the whole Home scan. watch.rs:588 root.open_file(&rel, ...)? inside walk(): an unreadable note (mode 000, ACL), a file that vanishes between listing and open, or a non-regular entry returns Err for the entire walk, so scan::run fails and the Home stays "incomplete". On dev one such file was skipped. Also non-UTF-8 names are now silently skipped instead of listed (:571-575). Skip-and-report per file instead. Also one extra openat2 per file per poll (snapshot) at 100k notes: measure.

P1-7. Operator CLI defaults can write to the wrong place. args.rs:184, 302, 311, 356: --appdata now defaults to ./appdata (cwd-relative) for user create, user recover, fix. An operator who runs calternal user create bob --data /srv/calternal-app/data (as in every old runbook) creates auth.db under the current directory. require_separate passes (they are distinct). Fix: require both flags explicitly for non-serve commands, or refuse when only one is given.

P1-8. macOS (and any non-Linux) is broken. reader.rs: Root::open, validate, open_file, open_inventory return Unsupported off Linux, and Home now opens a Root unconditionally (notes/src/home.rs:248). Notes cannot open at all on the Mac VM or a Mac dev machine. If macOS is a supported dev target, gate it; if not, say so in the PR. No macOS gate was run.

P2

  • Durability of "verified". copy() re-reads the destination after sync_all and rename, but from page cache, so media errors are not caught; there is no checksum recorded for later audit. A recorded BLAKE3 per file in the intent would make the printed byte count meaningful ("compare with du -sb" is weak: it ignores holes and hard links).
  • Sparse files are expanded (read/write loop, :487-497), so the destination can be larger than the source; the free-space estimate in the runbook ("at least the current data size") is then wrong. Hard links are copied independent (acknowledged); this also inflates space and the quota recount, so say so in the runbook.
  • Directory fsync coverage is good for files (sync then move_exclusive then parent sync_dir), but private_dir makes ancestors up to the root with sync_dir per level, which is O(depth) per file and fine, while copy_attributes on a directory runs after its children and is followed by only sync_dir(target), not the parent. A crash there is repaired on resume (fine), but there is no test that observes the order.
  • "Crash at every step" is error injection, not a crash. boundary() returns an Err in-process; no kill -9, no dropped page cache, no reordering. The tests show resumability after clean error returns, not fsync ordering. The PR states this for power cuts; it should say it for crashes too. The tests also skip the permission case when run as root (CI?), and run on a one-user, 8-file fixture: no multi-user, no file above the 1 MiB buffer, no contacts, no hard link, no sparse, no symlink, no mid-run source change, no mtime check.
  • Unsplit resume is brittle. A rerun with a partial <to>.unsplit-partial fails with "destination differs" if either root changed in between (e.g. server started once), and there is no cleanup instruction, so the rollback path fails when it is needed most. Offer --discard-partial or tell the operator to delete the staging dir (it is owned by the marker). Also unsplit does not collision-check appdata items against userdata items outside the four state names; equal files merge silently (:195-215).
  • Index (index.rs): (a) the single Mutex<Connection> is held by the writer during compaction while it deletes and re-inserts every row (:535-560), so all lookups for that Home block for the whole rebuild (seconds at 100k), against the 15 ms goal; (b) log.send on a bounded sync_channel(64) blocks the calling async thread when the writer is behind an fsync; (c) when the writer thread exits on error, later put_soon/remove_soon calls are lost silently (let _ = log.send) and the Home never recovers until restart; (d) lines is restored as len/70 (project), a guess that skews compaction triggers; (e) 128-lease cap refuses a new Home when full: availability regression for a deployment with unload/reload churn, no metric.
  • Reader pinning: identity includes mnt_id, which changes when the filesystem is remounted; every open then returns PermissionDenied until restart. Intentional per the doc, but needs an operator message and a test of remount.
  • NFC removed from rel_of, free_name and Cmd::Moved (watch.rs, rename.rs): paths are now raw. Existing NFD-named files and existing NFC entries in ids.jsonl rely on fold matching; verify with an NFD fixture on migrated data (Mac-origin files are the likely case).
  • After split, the id projection identity (dev:ino of ids.jsonl) differs, so every Home rebuilds its projection at first open (O(n) per Home). Expected, but budget it in the runbook.

P3

  • owned_root treats any directory with only .calternal-split-owner.tmp as ours; a stray empty directory created by someone else before the first marker is claimed (:404-415). Low risk because the source intent requires the destination to be absent at first run.
  • fs::read_dir order and user.path() use non-canonical paths while Root is pinned on from; fine offline, note it.
  • tree_bytes after a completed unsplit reports apparent size, unlike copied.
  • Message text: "Offline rollback complete ... Start the pre-split binary with --data" omits --state, and the OLD binary does not know .calternal-install-id-pairing semantics; confirm it starts on the result in a test (the CLI test starts the new server).

Failure table: does each test prove its row?

Row Verdict
Fresh install OK for what it says.
Single-root migration Partly: byte-exact yes; "right root" only for the four names the author chose (P0-2), modes yes, mtimes/xattrs untested.
Rerun / idempotent No: only the "differs" case is tested; absent-target resurrection (P0-1) is not.
Crash at every step Proves resume after injected Err, not durability ordering.
Disk full Same injection; no real ENOSPC; a half-written temp is reused, that is good.
Permission Skips as root; no test for unreadable directory.
Foreign/locked/unsafe Reasonable; add hard-link and special-file cases.
Rollback before new writes True by construction (source read-only), but never starts the old binary.
Rollback after new writes Good; add contacts and multi-user.
Rollback refusals Good.
CLI safety Flag only.

Should anything leave slice 1?

Yes. Suggested split, in order:

  1. PR A (small, high care): --userdata/--appdata flags and defaults (fix P1-7), require_separate, storage split/unsplit, DataLock directory lock, storage_startup.rs, docs. No notes or index changes. This is what prod will run; keep it reviewable and run it against a copy of real prod data with a dry-run.
  2. PR B: pinned Root reader and the notes watch/scan changes (symlink, NFC, per-file failure semantics, P1-5/P1-6/P1-8, Linux-only gate).
  3. PR C: paged identity index (SQLite projection, writer queue, cache lease) with its own scale and compaction-lock work.
  4. PR D: tiering-scale and tiering_model.rs (no runtime effect; reviewable independently).
    The Root types are needed by the split (A), so either keep a minimal open_inventory subset in A, or land B first.

Gates run by the author are green on Linux, but they do not exercise P0-1 or P0-2, which is why they are green.

# Independent review, round 1 Reviewed head `effb44dc` against `origin/dev`. Read the PR body, #761 and `docs/plans/0761-slice-1-validation.md`. This was a read-only source review: nothing was built or run (control host). Items marked *verify* are read from the code and have not been reproduced. Line numbers are in the PR head. **Verdict: split the PR, and fix the P0/P1 migration items before anyone runs it on prod.** Prod runs `storage split` by hand, so it is a different risk class from the roughly 2,700 lines of runtime change (paged index, pinned reader, scan/watch semantics) that every user gets. Details at the end. ## P0: can lose, strand or duplicate user data **P0-1. A completed split can be re-run, and re-running resurrects deleted data.** `crates/calternal/src/storage_split.rs:27-45, 136, 454`. Once split succeeds, the source intent still matches (`Ok(old) if old == binding`), so the same command proceeds. `copy()` only checks "target exists and is equal, else differs-refuse". A target that is **absent** is copied again. Scenario: split, start the new server, user deletes a note or purges Trash, operator re-runs "the identical command" (the PR tells them to on any failure). The deleted file comes back into live userdata, and a file created or moved in the new layout is not touched. Nothing consults `.calternal-split-complete`. Same for any deletion between a failed first attempt and a retry. Also, a destination is never swept for entries that are not in the source, so stale extras survive a resume. Fix: if `.calternal-split-complete` exists, refuse (or run verify-only, never write). Record source file inventory (path, size, hash) in the intent at first run and refuse on any change at resume. Test: split, delete one file in destination, re-run, expect refusal and file still absent. **P0-2. `users/<uuid>/contacts/` is classified as a Home file, but the code reads it from appdata.** *verify* `storage_split.rs:262-277` (`is_state_item` allowlist has notes, search, attachments, tasks, index.db*), vs `crates/contacts-actions/src/cards.rs:33` (`Cards::new(user_state).root = user_state/contacts/raw`) and `write.rs:156` (`self.state.join("users")...`). After the split, the original-card copies (unknown fields, original inline photos; the fixture in this very file calls them out) land in `userdata/users/<u>/contacts/raw` and the server looks in `appdata/...`. They are stranded, not deleted, but contacts will treat the originals as missing and may regenerate lossy copies. Tests miss it because the fixture uses `Notes/card.vcf`, not `contacts/raw`. Fix: derive the state list from the code (each plugin's `app_data_dir` / state joins: tasks, contacts, search, attachments, notes, plus any I missed), and add a test that fails if a directory the server creates in appdata is classified as userdata. Better: invert to a startup-written manifest or have a fixture produced by a real server run on a legacy root and then split. Related: a user Home folder literally named `notes`, `search`, `tasks`, `attachments` or `contacts` (case-sensitive fs) is silently moved into appdata, and unsplit refuses it as ambiguous (`:200-215`). Classify by exact known path plus refusal on collision at split time too. ## P1: must fix before prod **P1-1. mtimes are not preserved (files or directories).** `fsio/src/reader.rs` `copy_attributes` copies mode and xattrs only; no `futimens`. Effects: the notes scan state is (size, mtime) per file (`notes/src/scan.rs:408-443`), so the first start after split sees every note as changed (full cold scan, re-identify of every head, 100k-note cost, risk of spurious "changed outside" events); outside sync tools (Syncthing etc.) will see every file as modified; file managers show wrong dates. The PR body says "modes preserved" only. Fix: copy atime/mtime (nanosecond) on files and, after children, on directories; test it; keep scan state valid. **P1-2. "Offline" is only a `.lock` flock plus a flag.** `args.rs` (`--confirm-stopped` is just a bool). Outside writers are not excluded; the only defense is a re-compare of each file right after its own copy (`:482-500`). A file edited after it was copied, a file created after `read_dir` of its directory, or a file renamed mid-run is not noticed on the first run: `source.validate()` (`:101`) checks root identity only, and no final verification pass runs before the completion marker. Fix: after all copies, re-walk the source and compare inventory (path, size, mtime, ino) to the one taken at the start; refuse completion on any difference. Also add a test that mutates the source mid-run (hook in `boundary()`). **P1-3. Marker files in the source root are an operational trap.** `.calternal-split-prepare`, `.calternal-split-id` stay in the source forever. A later split to different paths hits "different or torn split intent" (`:29-34`) with no documented way out; abandoning the split and running the old binary for days, then re-splitting, mixes the P0-1 resume with a changed source. Also: the PR claims the old binary "ignores" them, but the new `split` itself rejects unknown root names, and I could not find a test starting the old binary on a marked source. Fix: document exact cleanup; make the intent carry a source inventory hash so a changed source refuses with a clear message; test old-binary start on a marked source (or at least `check_legacy`). **P1-4. Unknown-entry refusal has no override.** `:96, :228` refuse any unknown root entry. If `/srv/calternal-app/data` is a mount point, `lost+found` is there, and the operator cannot classify it, only delete or move it. Also `cache/` is copied in full though regenerable. Add a preflight `storage split --dry-run` that lists classification (counts and bytes per root, every unknown entry, every symlink, hard link, sparse file, special file, non-UTF-8 name) without touching anything, and have the runbook require it on a copy of prod data first. **P1-5. Symlinks and special files in user data abort the whole split.** `open_inventory` uses `NO_SYMLINKS` and demands regular files (`reader.rs`). Safe, but any single symlink in Documents or Photos strands the migration, and note that the same change makes the Notes watcher stop following symlinked folders (`notes/src/watch.rs:562-590` now uses `DirEntry::file_type`, which does not follow). *verify*: a symlinked folder inside `Notes/` worked on `dev` and silently disappears from scans after this PR. Decide and document; add tests. **P1-6. One bad file poisons the whole Home scan.** `watch.rs:588` `root.open_file(&rel, ...)?` inside `walk()`: an unreadable note (mode 000, ACL), a file that vanishes between listing and open, or a non-regular entry returns `Err` for the entire walk, so `scan::run` fails and the Home stays "incomplete". On `dev` one such file was skipped. Also non-UTF-8 names are now silently skipped instead of listed (`:571-575`). Skip-and-report per file instead. Also one extra `openat2` per file per poll (`snapshot`) at 100k notes: measure. **P1-7. Operator CLI defaults can write to the wrong place.** `args.rs:184, 302, 311, 356`: `--appdata` now defaults to `./appdata` (cwd-relative) for `user create`, `user recover`, `fix`. An operator who runs `calternal user create bob --data /srv/calternal-app/data` (as in every old runbook) creates `auth.db` under the current directory. `require_separate` passes (they are distinct). Fix: require both flags explicitly for non-serve commands, or refuse when only one is given. **P1-8. macOS (and any non-Linux) is broken.** `reader.rs`: `Root::open`, `validate`, `open_file`, `open_inventory` return `Unsupported` off Linux, and `Home` now opens a `Root` unconditionally (`notes/src/home.rs:248`). Notes cannot open at all on the Mac VM or a Mac dev machine. If macOS is a supported dev target, gate it; if not, say so in the PR. No macOS gate was run. ## P2 - **Durability of "verified".** `copy()` re-reads the destination after `sync_all` and rename, but from page cache, so media errors are not caught; there is no checksum recorded for later audit. A recorded BLAKE3 per file in the intent would make the printed byte count meaningful ("compare with `du -sb`" is weak: it ignores holes and hard links). - **Sparse files** are expanded (read/write loop, `:487-497`), so the destination can be larger than the source; the free-space estimate in the runbook ("at least the current data size") is then wrong. **Hard links** are copied independent (acknowledged); this also inflates space and the quota recount, so say so in the runbook. - **Directory fsync coverage is good** for files (sync then `move_exclusive` then parent `sync_dir`), but `private_dir` makes ancestors up to the root with `sync_dir` per level, which is O(depth) per file and fine, while `copy_attributes` on a directory runs **after** its children and is followed by only `sync_dir(target)`, not the parent. A crash there is repaired on resume (fine), but there is no test that observes the order. - **"Crash at every step" is error injection, not a crash.** `boundary()` returns an `Err` in-process; no kill -9, no dropped page cache, no reordering. The tests show resumability after clean error returns, not fsync ordering. The PR states this for power cuts; it should say it for crashes too. The tests also skip the permission case when run as root (CI?), and run on a one-user, 8-file fixture: no multi-user, no file above the 1 MiB buffer, no contacts, no hard link, no sparse, no symlink, no mid-run source change, no mtime check. - **Unsplit resume is brittle.** A rerun with a partial `<to>.unsplit-partial` fails with "destination differs" if either root changed in between (e.g. server started once), and there is no cleanup instruction, so the rollback path fails when it is needed most. Offer `--discard-partial` or tell the operator to delete the staging dir (it is owned by the marker). Also unsplit does not collision-check appdata items against userdata items outside the four state names; equal files merge silently (`:195-215`). - **Index (index.rs)**: (a) the single `Mutex<Connection>` is held by the writer during compaction while it deletes and re-inserts every row (`:535-560`), so all lookups for that Home block for the whole rebuild (seconds at 100k), against the 15 ms goal; (b) `log.send` on a bounded `sync_channel(64)` blocks the calling async thread when the writer is behind an fsync; (c) when the writer thread exits on error, later `put_soon`/`remove_soon` calls are lost silently (`let _ = log.send`) and the Home never recovers until restart; (d) `lines` is restored as `len/70` (`project`), a guess that skews compaction triggers; (e) 128-lease cap refuses a new Home when full: availability regression for a deployment with unload/reload churn, no metric. - **Reader pinning**: identity includes `mnt_id`, which changes when the filesystem is remounted; every open then returns PermissionDenied until restart. Intentional per the doc, but needs an operator message and a test of remount. - **NFC removed** from `rel_of`, `free_name` and `Cmd::Moved` (`watch.rs`, `rename.rs`): paths are now raw. Existing NFD-named files and existing NFC entries in `ids.jsonl` rely on `fold` matching; verify with an NFD fixture on migrated data (Mac-origin files are the likely case). - After split, the id projection identity (dev:ino of `ids.jsonl`) differs, so every Home rebuilds its projection at first open (O(n) per Home). Expected, but budget it in the runbook. ## P3 - `owned_root` treats any directory with only `.calternal-split-owner.tmp` as ours; a stray empty directory created by someone else before the first marker is claimed (`:404-415`). Low risk because the source intent requires the destination to be absent at first run. - `fs::read_dir` order and `user.path()` use non-canonical paths while `Root` is pinned on `from`; fine offline, note it. - `tree_bytes` after a completed unsplit reports apparent size, unlike `copied`. - Message text: "Offline rollback complete ... Start the pre-split binary with --data" omits `--state`, and the OLD binary does not know `.calternal-install-id`-pairing semantics; confirm it starts on the result in a test (the CLI test starts the new server). ## Failure table: does each test prove its row? | Row | Verdict | |---|---| | Fresh install | OK for what it says. | | Single-root migration | Partly: byte-exact yes; "right root" only for the four names the author chose (P0-2), modes yes, mtimes/xattrs untested. | | Rerun / idempotent | **No**: only the "differs" case is tested; absent-target resurrection (P0-1) is not. | | Crash at every step | Proves resume after injected `Err`, not durability ordering. | | Disk full | Same injection; no real ENOSPC; a half-written temp is reused, that is good. | | Permission | Skips as root; no test for unreadable directory. | | Foreign/locked/unsafe | Reasonable; add hard-link and special-file cases. | | Rollback before new writes | True by construction (source read-only), but never starts the old binary. | | Rollback after new writes | Good; add contacts and multi-user. | | Rollback refusals | Good. | | CLI safety | Flag only. | ## Should anything leave slice 1? Yes. Suggested split, in order: 1. **PR A (small, high care):** `--userdata/--appdata` flags and defaults (fix P1-7), `require_separate`, `storage split`/`unsplit`, `DataLock` directory lock, `storage_startup.rs`, docs. No notes or index changes. This is what prod will run; keep it reviewable and run it against a copy of real prod data with a dry-run. 2. **PR B:** pinned `Root` reader and the notes `watch`/`scan` changes (symlink, NFC, per-file failure semantics, P1-5/P1-6/P1-8, Linux-only gate). 3. **PR C:** paged identity index (SQLite projection, writer queue, cache lease) with its own scale and compaction-lock work. 4. **PR D:** `tiering-scale` and `tiering_model.rs` (no runtime effect; reviewable independently). The `Root` types are needed by the split (A), so either keep a minimal `open_inventory` subset in A, or land B first. Gates run by the author are green on Linux, but they do not exercise P0-1 or P0-2, which is why they are green.
kayg commented 2026-10-31 16:24:21 +00:00
Author
Owner
Copy link

Superseded for implementation/review by four stacked PRs, ultimately into dev:

  1. A #790 — offline CLI, split/unsplit, shared state layout and directory locks.
  2. B #791 — portable pinned reader and resilient Notes scan/watch behavior.
  3. C #792 — paged identity projection.
  4. D #793 — qualification binary, independent models and validation ledger.

This PR remains open. Nothing was merged or closed, and no production operation was performed. Each replacement PR begins with its plan, sizes/complexity, failure → effect → detection → recovery → test map, exact gate commands, decisions and remaining risks.

Round-1 finding (comment 70713) Status / evidence
P0-1 completed split can resurrect deleted data A refuses completed sources. Retry recopies into fresh owned staging; it never merges into live destinations. A post-publication retry only marks an unchanged source, without copying. Completed/deleted-file and every-boundary retry regressions pass.
P0-2 contacts/raw and state list A uses one explicit calternal_core::layout list in migration and state readers. Real-server state-layout fixture passes; round trip puts all 3 original cards in appdata.
P1-1 mtime/atime/perms/xattrs A preserves times to nanoseconds, ownership, permissions, attributes/ACLs and installation-marker metadata. Verification access times are restored afterward. File/directory/link regression and hash+metadata round trip pass.
P1-2 final source manifest / mid-split startup A rewalks inventory and rehashes copied files, detecting changed sizes, mtimes, inode/ctime and content; mutations fail before publication. Source directory/sentinel locks and startup marker recheck refuse concurrent startup. Tests pass.
P1-3 marker cleanup Exact cleanup/rollback instructions in docs/storage.md; tests remove the completion marker explicitly. Round trip starts the old binary on both marked source and consolidated rollback.
P1-4 dry-run / lost+found / override Read-only preflight, no newly created lock sentinel; filesystem-owned lost+found is reported without traversing unreadable contents. Unknown root entries require --unknown-to-appdata (unsplit: --unknown-to-root). Tests pass.
P1-5 links / special files A copies links as links and lists unsupported devices/FIFOs/sockets before copying. B deliberately reports/skips Notes links instead of following them out of a Home; policy and tests are explicit.
P1-6 bad note poisons scan B skips/reports per-entry failures, non-UTF-8 names and unreadable subdirectories; vanished-head test preserves other notes. Notes-root listing/replacement still fails closed. Targeted tests and full Linux workspace gates pass.
P1-7 cwd-relative appdata A retains legacy data/state fallback: missing appdata resolves to the selected data root, never an unrelated cwd ./appdata. Both aliases and env precedence are tested. Explicit split setup supplies both roots.
P1-8 macOS reader unsupported B implements component-wise O_NOFOLLOW/device checks on macOS. Reader and changed scan tests pass on the Mac VM. Final legacy-NFC/NFD refresh rerun is queued under the shared Mac lock. Broader unrelated Mac failures are listed in B, not claimed green.
P2 sparse/hard-link expansion Sparse zero ranges remain holes; space preflight budgets apparent bytes + 1%. Hard-link names become independent files and the runbook explains size/quota effects.
P2 unsplit resume and collision ambiguity A always starts over in owned staging; no stale-file merging. It refuses duplicate destinations and Home/state ambiguity. Tests pass.
P2 id-index record-count estimate / silent writer loss C records the actual record count (5 long-path records previously reopened as 31; red/green regression). Writer disconnects now emit errors; durable failures cannot acknowledge.
P2 reader remount / Unicode spelling / first projection rebuild B reports restart advice for root/remount replacement. Raw spelling and legacy folded mappings have a regression (also found and fixed a baseline-scan mapping bug). Runbook budgets one projection rebuild per Home. Privileged remount qualification remains owed.
P2 directory fsync ordering / cache copy cost Changed directories are synced, but no independent syscall-order or power-cut campaign was run. Cache contents are still copied conservatively in full. Both remain documented qualification/cost limits.
P3 offline path spelling / staging ownership Directory roots are pinned; ancestor/name changes still require stopped writers and offline coordination. Empty/torn-marker staging ownership remains an explicit low-risk cleanup exception, not a claim of ownership proof.
P3 rollback message and size reporting CLI says old binary with --data and no --state; old-binary startup is tested. Reports use planned apparent bytes consistently.

A validation: dev-01 fmt, clippy and cargo test -j4 --workspace --locked --no-fail-fast: 1,172 passed, 0 failed, 20 ignored. ci/storage-split-e2e.py ends ALL PASS: 3 users, 211 entries, 160 regular files, 3 contact originals, old times, xattrs, large attachments, symlinks and lost+found. Original → split → immediate unsplit is hash+metadata identical. After server startup/new writes, unsplit is exactly the current roots; original Home files are untouched, and the old binary sees 37 notes (36 + the new note).

Linux final gates: all PASS. B: 1,178 passed; C: 1,183 passed; D: 1,186 passed. Each has 0 failed, 20 ignored, fmt and clippy green. Nix only, dev-01 slot, -j4. Logs: ~/tier761-exact-{a,b,c,d}.log, exit files all 0. Exact commands in each PR: cargo fmt --all --check; cargo clippy -j4 --workspace --all-targets --locked -- -D warnings; cargo test -j4 --workspace --locked --no-fail-fast.

Final heads: A c5eaf6af, B 977d26c8, C 5594f8f5, D 5b96787b. Final heads differ from gate-run heads only in one stricter inherited Python assertion; all Rust/workspace sources are identical. A's stricter full round trip also passed (~/tier761-strict-e2e.log, exit 0, fixture ~/tier761-roundtrip-strict-6): even installation-marker equality has no exception. Independent review of the replacement stack remains pending. Latest Mac reader/scan checks are still queued under the active shared benchmark lock.

Still owed / intentionally not folded into this migration fix: per-file checksum audit persistence and real media/power-loss proof; privileged ENOSPC/inode/mount campaigns; independent SIGKILL/fsync-order testing; id-index compaction mutex latency, async queue backpressure, automatic writer recovery and fair cache-lease eviction/metrics; full 100k/aggregate/100k-edit-history successful-open, 15 ms, RSS/FD/watch and FO1 qualification. Empty/torn-marker staging ownership and offline ancestor/path coordination remain explicit limits. Models in D are independent specifications, not production certification. General Mac suite failures include Linux /proc-dependent shutdown inspection, delayed SQLite cleanup, and an unchanged case-only rename collision fixture. Production preflight/round trip on a stopped backup copy of real data is still required by the runbook; no real production data was accessed here.

Final Mac rerun remains externally blocked after over an hour queued: /tmp/mac-bench.lock belongs to BM1-mac-measurement (PID 16512, started 2026-10-31T16:02:50Z). B waiter PID 18599 remains queued; evidence will appear in ~/tier761-b-exact-mac.log and ~/tier761-b-exact-mac.exit. No foreign lock/process was removed. The latest-head Mac rerun and independent review are still pending.

Superseded for implementation/review by four stacked PRs, ultimately into `dev`: 1. **A #790** — offline CLI, split/unsplit, shared state layout and directory locks. 2. **B #791** — portable pinned reader and resilient Notes scan/watch behavior. 3. **C #792** — paged identity projection. 4. **D #793** — qualification binary, independent models and validation ledger. This PR remains open. Nothing was merged or closed, and no production operation was performed. Each replacement PR begins with its plan, sizes/complexity, failure → effect → detection → recovery → test map, exact gate commands, decisions and remaining risks. | Round-1 finding (comment 70713) | Status / evidence | |---|---| | P0-1 completed split can resurrect deleted data | A refuses completed sources. Retry recopies into fresh owned staging; it never merges into live destinations. A post-publication retry only marks an unchanged source, without copying. Completed/deleted-file and every-boundary retry regressions pass. | | P0-2 contacts/raw and state list | A uses one explicit `calternal_core::layout` list in migration and state readers. Real-server state-layout fixture passes; round trip puts all 3 original cards in appdata. | | P1-1 mtime/atime/perms/xattrs | A preserves times to nanoseconds, ownership, permissions, attributes/ACLs and installation-marker metadata. Verification access times are restored afterward. File/directory/link regression and hash+metadata round trip pass. | | P1-2 final source manifest / mid-split startup | A rewalks inventory and rehashes copied files, detecting changed sizes, mtimes, inode/ctime and content; mutations fail before publication. Source directory/sentinel locks and startup marker recheck refuse concurrent startup. Tests pass. | | P1-3 marker cleanup | Exact cleanup/rollback instructions in docs/storage.md; tests remove the completion marker explicitly. Round trip starts the old binary on both marked source and consolidated rollback. | | P1-4 dry-run / lost+found / override | Read-only preflight, no newly created lock sentinel; filesystem-owned lost+found is reported without traversing unreadable contents. Unknown root entries require `--unknown-to-appdata` (unsplit: `--unknown-to-root`). Tests pass. | | P1-5 links / special files | A copies links as links and lists unsupported devices/FIFOs/sockets before copying. B deliberately reports/skips Notes links instead of following them out of a Home; policy and tests are explicit. | | P1-6 bad note poisons scan | B skips/reports per-entry failures, non-UTF-8 names and unreadable subdirectories; vanished-head test preserves other notes. Notes-root listing/replacement still fails closed. Targeted tests and full Linux workspace gates pass. | | P1-7 cwd-relative appdata | A retains legacy data/state fallback: missing appdata resolves to the selected data root, never an unrelated cwd `./appdata`. Both aliases and env precedence are tested. Explicit split setup supplies both roots. | | P1-8 macOS reader unsupported | B implements component-wise O_NOFOLLOW/device checks on macOS. Reader and changed scan tests pass on the Mac VM. Final legacy-NFC/NFD refresh rerun is queued under the shared Mac lock. Broader unrelated Mac failures are listed in B, not claimed green. | | P2 sparse/hard-link expansion | Sparse zero ranges remain holes; space preflight budgets apparent bytes + 1%. Hard-link names become independent files and the runbook explains size/quota effects. | | P2 unsplit resume and collision ambiguity | A always starts over in owned staging; no stale-file merging. It refuses duplicate destinations and Home/state ambiguity. Tests pass. | | P2 id-index record-count estimate / silent writer loss | C records the actual record count (5 long-path records previously reopened as 31; red/green regression). Writer disconnects now emit errors; durable failures cannot acknowledge. | | P2 reader remount / Unicode spelling / first projection rebuild | B reports restart advice for root/remount replacement. Raw spelling and legacy folded mappings have a regression (also found and fixed a baseline-scan mapping bug). Runbook budgets one projection rebuild per Home. Privileged remount qualification remains owed. | | P2 directory fsync ordering / cache copy cost | Changed directories are synced, but no independent syscall-order or power-cut campaign was run. Cache contents are still copied conservatively in full. Both remain documented qualification/cost limits. | | P3 offline path spelling / staging ownership | Directory roots are pinned; ancestor/name changes still require stopped writers and offline coordination. Empty/torn-marker staging ownership remains an explicit low-risk cleanup exception, not a claim of ownership proof. | | P3 rollback message and size reporting | CLI says old binary with --data and no --state; old-binary startup is tested. Reports use planned apparent bytes consistently. | **A validation:** dev-01 fmt, clippy and `cargo test -j4 --workspace --locked --no-fail-fast`: 1,172 passed, 0 failed, 20 ignored. `ci/storage-split-e2e.py` ends ALL PASS: 3 users, 211 entries, 160 regular files, 3 contact originals, old times, xattrs, large attachments, symlinks and lost+found. Original → split → immediate unsplit is hash+metadata identical. After server startup/new writes, unsplit is exactly the current roots; original Home files are untouched, and the old binary sees 37 notes (36 + the new note). **Linux final gates: all PASS.** B: 1,178 passed; C: 1,183 passed; D: 1,186 passed. Each has 0 failed, 20 ignored, fmt and clippy green. Nix only, dev-01 slot, -j4. Logs: `~/tier761-exact-{a,b,c,d}.log`, exit files all 0. Exact commands in each PR: `cargo fmt --all --check`; `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`; `cargo test -j4 --workspace --locked --no-fail-fast`. Final heads: A `c5eaf6af`, B `977d26c8`, C `5594f8f5`, D `5b96787b`. Final heads differ from gate-run heads only in one stricter inherited Python assertion; all Rust/workspace sources are identical. A's stricter full round trip also passed (`~/tier761-strict-e2e.log`, exit 0, fixture `~/tier761-roundtrip-strict-6`): even installation-marker equality has no exception. Independent review of the replacement stack remains pending. Latest Mac reader/scan checks are still queued under the active shared benchmark lock. **Still owed / intentionally not folded into this migration fix:** per-file checksum audit persistence and real media/power-loss proof; privileged ENOSPC/inode/mount campaigns; independent SIGKILL/fsync-order testing; id-index compaction mutex latency, async queue backpressure, automatic writer recovery and fair cache-lease eviction/metrics; full 100k/aggregate/100k-edit-history successful-open, 15 ms, RSS/FD/watch and FO1 qualification. Empty/torn-marker staging ownership and offline ancestor/path coordination remain explicit limits. Models in D are independent specifications, not production certification. General Mac suite failures include Linux /proc-dependent shutdown inspection, delayed SQLite cleanup, and an unchanged case-only rename collision fixture. Production preflight/round trip on a stopped backup copy of real data is still required by the runbook; no real production data was accessed here. Final Mac rerun remains externally blocked after over an hour queued: `/tmp/mac-bench.lock` belongs to `BM1-mac-measurement` (PID 16512, started 2026-10-31T16:02:50Z). B waiter PID 18599 remains queued; evidence will appear in `~/tier761-b-exact-mac.log` and `~/tier761-b-exact-mac.exit`. No foreign lock/process was removed. The latest-head Mac rerun and independent review are still pending.
kayg commented 2026-11-04 13:04:41 +00:00
Author
Owner
Copy link

Closing: superseded by #790 (named in the PR body or comments). Branch kept.

Closing: superseded by #790 (named in the PR body or comments). Branch kept.
kayg closed this pull request 2026-11-04 13:04:41 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!784
No description provided.