Plan: storage tiering (#731) #761

Closed
kayg wants to merge 1 commit from docs/tiering-731-plan into dev
Owner

Plan the joined SSD/HDD Home for #731 before implementation. Covers operation costs and budgets, crash and outside-writer races, safe limits, the shared reader and quota contracts, tests required in the first code PR, delivery slices, and five likely failures. Follows ADRs 0062–0064 and the accepted T-A1–T-A3 and FO1 decisions.

Only docs/plans/0731-tiering.md changes: 2,321 words, four A4 pages. The usage-limit checkpoint and its evidence files are excluded from this branch history; artifacts remain outside the repo. No product code, merge or deploy. Refs #731.

Base: d83e54dbe5
Head: 2cf7e3fbd8

The first PDF attempt failed during Chromium Crashpad setup. The retry with isolated job-owned config and cache directories exited 0. All local rendering ran through the shared build-slot wrapper. Product implementation, crash, isolation and performance gates are requirements for later code PRs, not results claimed here.

Gate output verbatim:

PASS Markdown: Forgejo GFM HTTP 200; seven required sections, two tables, five second-pass risks.
PASS Links: five relative links resolve locally and render to the correct branch paths.
PASS Length: 2,321 words; 4 A4 pages at 10.5 pt body / 9.5 pt tables, 14 mm margins.
PASS Cost review: seven operations include complexity, syscalls, disks and budgets; existing gates unchanged.
PASS Failure review: eleven rows cover journal boundaries, outside races, open inodes, disk faults, two processes and state loss.
PASS Plan review: first-PR scale/model/crash gates, FO1, joined-reader/quota contracts, slices and five failure checks covered.
PASS git diff --check: exit 0 against origin/dev.
PASS Scope: only docs/plans/0731-tiering.md changed in every proposed commit; worktree clean; no product code or evidence artifacts.
PASS Remote: docs/tiering-731-plan points to 2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80.
NOTE Docs only: no product build, crash campaign, isolation audit or performance measurement was run.
Plan the joined SSD/HDD Home for #731 before implementation. Covers operation costs and budgets, crash and outside-writer races, safe limits, the shared reader and quota contracts, tests required in the first code PR, delivery slices, and five likely failures. Follows ADRs 0062–0064 and the accepted T-A1–T-A3 and FO1 decisions. Only docs/plans/0731-tiering.md changes: 2,321 words, four A4 pages. The usage-limit checkpoint and its evidence files are excluded from this branch history; artifacts remain outside the repo. No product code, merge or deploy. Refs #731. Base: d83e54dbe571b625504001a8b6add90d44ca6161 Head: 2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80 The first PDF attempt failed during Chromium Crashpad setup. The retry with isolated job-owned config and cache directories exited 0. All local rendering ran through the shared build-slot wrapper. Product implementation, crash, isolation and performance gates are requirements for later code PRs, not results claimed here. Gate output verbatim: ```text PASS Markdown: Forgejo GFM HTTP 200; seven required sections, two tables, five second-pass risks. PASS Links: five relative links resolve locally and render to the correct branch paths. PASS Length: 2,321 words; 4 A4 pages at 10.5 pt body / 9.5 pt tables, 14 mm margins. PASS Cost review: seven operations include complexity, syscalls, disks and budgets; existing gates unchanged. PASS Failure review: eleven rows cover journal boundaries, outside races, open inodes, disk faults, two processes and state loss. PASS Plan review: first-PR scale/model/crash gates, FO1, joined-reader/quota contracts, slices and five failure checks covered. PASS git diff --check: exit 0 against origin/dev. PASS Scope: only docs/plans/0731-tiering.md changed in every proposed commit; worktree clean; no product code or evidence artifacts. PASS Remote: docs/tiering-731-plan points to 2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80. NOTE Docs only: no product build, crash campaign, isolation audit or performance measurement was run. ```
docs: plan storage tiering before implementation (#731)
All checks were successful
CI / server (pull_request) Successful in 1m37s
CI / rust (pull_request) Successful in 15m36s
CI / web (pull_request) Successful in 2h40m55s
2cf7e3fbd8
Author
Owner

Independent review of PR #761: revise

Reviewed docs/plans/0731-tiering.md at 2cf7e3fbd8, based on d83e54dbe571b625504001a8b6add90d44ca6161. Read issue #731 and its discussion, the referenced FO1 decision (#642 comment 67053), ADRs 0062–0064, and the current storage, notes, attachments, contacts and quota paths. Also inspected origin/perf/quota-simple at bb36381dd1; its behavior is materially different from quota on the PR base. A scoped filename search found neither CLAUDE.md nor AGENTS.md in the repository/worktrees, so no instructions from those absent files were assumed.

Verdict: revise before approving this as the implementation plan. The decisions are mostly carried forward correctly, but several safety properties are statements of desired outcomes without the protocol needed to achieve them. The most consequential gaps concern ordinary saves, Home-only recovery, quota consistency, bounded opens and dedupe privacy.

The plan does improve on the recent failures: it explicitly requires 100k-note, 10k-line/repeated-block, 100k-edit-history, 50k-sibling, 1M-HDD-file and multi-person tests in the first code PR (lines 130–155, 171–173). It includes open writable handles, repeated recovery crashes and a power-loss persistence model. It honestly permits indefinitely pending tier retirement without verified exclusion and describes outside cross-tree renames as delete/create rather than promising an atomic external view. Those are strengths, not missing requirements.

This was a design and source review. No product code was written, built or tested; no performance or crash campaign was run. Three small deterministic kernel probes ran only in the review job folder. They confirm syscall hazards, not that a future implementation necessarily has those bugs. Paths and line numbers below refer to the reviewed commit unless the quota branch is explicitly named. All findings describe a schedule that the plan must resolve, rather than claiming an unimplemented feature already fails.

Blocking gaps

B1. Ordinary saves need the same old-inode protection as tier retirement

Plan: lines 33, 73, 76–77, 94–96, 177–186. Code: fsio/swap.rs, particularly after_swap at lines 191–219; notes/src/room/save.rs:153–199.

Scenario: an outside program holds a writable descriptor to a note. Calternal checks the old bytes and exchanges its freshly synced snapshot into the public name. The outside program writes and fsyncs through its old descriptor. The current writer sees the same device/inode at the displaced temporary name and unlinks it. Closing the outside descriptor destroys that durable outside version. This does not require a tier move or dedupe.

A deterministic syscall probe reproduced this sequence. The current guard compares device/inode, not a guarantee that the inode cannot change. Adding a final hash/version check merely moves the last race boundary: the outside write can follow that check. The plan explicitly requires exclusion for retirement/dedupe but still describes ordinary saves as reusable “guarded placement.” Its outside-write row therefore assumes a stronger primitive than the existing one provides.

Required revision: cover every destructive replacement/cleanup, including saves, companion updates, undo and restore. Define either verified exclusion or retention of displaced writable versions with a safe reclamation rule. State what happens to FO1's one-second requirement when such protection cannot be obtained. Put this exact old-descriptor schedule in the materialization gate, before tier moves are enabled. “Pending move” alone does not solve a save that already acknowledged typing.

B2. Home-only recovery lacks a byte-complete transaction and a restartable bootstrap

Plan: lines 50–59, 70–80, 177–179. Decision: ADR 0062:404–458; ADR 0063:64–85.

Scenario 1: updating a contact replaces its note, then crashes before replacing the original-card companion. State is lost at the same time. The specified intent fields are paths, IDs, expected versions/hashes, directory identities and progress. None reconstructs the missing new card, including an unknown field or original inline photo, or necessarily restores the overwritten old note. A record saying “finish the pair” is insufficient if its only complete payload was in state. The permitted pre-materialization loss window does not license leaving an incoherent materialized pair with neither a complete old nor complete new version available.

Scenario 2: empty-state reconstruction creates some databases and crashes. Next startup sees nonempty, incomplete state. The plan says to preserve/refuse corrupt nonempty state, but also promises repeated recovery is idempotent. It does not identify a durable rebuild generation/marker that distinguishes its own interrupted bootstrap from foreign corruption. Refusing forever after the first interrupted rebuild violates the recovery requirement.

Scenario 3: a revocation reaches the Home log before the state transaction, or the state transaction commits before its final Home proof. Crash between them, then lose state. “Current auth” and “Home proofs precede acknowledgements” do not identify the commit point or resolve which authorization generation is authoritative. Falling back to an older complete snapshot can resurrect a revoked credential; refusing all partial snapshots can make an otherwise recoverable Home unusable.

Required revision: give a small state machine with its durable commit point, immutable byte payload/staging ownership, authorization sequence, paired-file recovery rule, snapshot/log cutoff and retirement rule. Specify creation and fsync of the recovery directory and all changed parents on their own filesystems. Show how a new complete snapshot plus its log suffix replaces the old generation without a gap, how torn tails differ from corrupt committed records, and how recovery itself resumes after each boundary. This is a design requirement, not a request to write the implementation in this PR.

B3. Quota is not reconciled with the actual whole-Home owner or a consistent two-tree scan

Plan: lines 117–126; operation table at 30–38. Code on quota branch: usage.rs, lines 1–26, 188–255; usage/scan.rs:52–110; auth/src/quota.rs:74–108.

perf/quota-simple counts the whole physical Home, charges every regular-file name including hard-link aliases, fences directory/inode events, excludes foreign mounts, and refuses admission on incomplete coverage. Admission joins or runs a fresh recount with an approximately two-second deadline. It does not already implement the plan's stable logical-ID event ledger or journal-staging exclusions. Simply routing it through the paged reader changes its correctness mechanism.

Scenario: count SSD before an outside cross-tree move/copy-delete and HDD afterwards. The file can be counted twice, or neither time in the reverse schedule. If watcher delivery is delayed or lost, a logical commit generation need not change at all. Even an app tier move changes its physical copies without changing logical usage. A cursor that crosses commit generations or skips a “pending” prefix cannot certify a fresh whole-Home total. A separate contact-history provider can change between the file walk and provider read as well.

Scenario: a large dirty Home requires a cold recount before an upload. One million entries in two seconds requires at least 500,000 entries/s before watch installation, inode coverage and fence costs. The plan budgets 120 seconds for reconciliation but omits quota admission from the cost table. It neither promises successful admission for that Home nor explicitly chooses retryable unavailability. Replacing a cold walk with an index lookup without proving freshness is unsafe.

Required revision: agree an owner contract that covers both roots under one validated observation/fence, membership and move receipts, contact-history/provider generations, failed entries, raw names, native overflow, mounts and reservations. Define whether pending scans retry, restart or retain a pinned generation; bound retries and make unknown usage fail closed. List create/upload, overwrite, Trash/restore/purge, cancellation, outside changes and state-loss reservation recovery explicitly. Add recount/admission cost and availability budgets at 100k/1M. Preserve the branch's known failure vectors instead of reducing the problem to “no double-release.”

B4. The 15 ms open is not established by construction

Plan: lines 32, 45–46, 100–108, 171–173. Code: notes/src/lib.rs:253–285; home.rs:237–241,291–295; index.rs:144; scan.rs:144–160; room/load.rs:44; history.rs:803–823.

The proposed joined lookup removes enumeration from one layer, but the current caller still has these costs:

  • First use loads the entire ids.jsonl; reopening an unloaded index does the same under its mutex: O(N) bytes/entries.
  • An unknown non-daily ID can wait for the startup scan, up to 60 seconds. “Dirty or unindexed names wait” does not provide an open latency bound.
  • A room collects all history updates into a vector, then applies every update. H is unbounded; the plan notices this but chooses no checkpoint/replay bound. Update-application cost and retained CRDT state also need measurement; counting serialized history bytes alone is not a time proof.
  • Every direct open synchronously probes HDD metadata. At depth d, cold pathname lookup can require several storage reads even for an SSD note. O(d) and O(log N) do not bound rotational seek/queue latency below 15 ms under concurrent scan, quota and dedupe traffic.

Scenario: open one cold note after index unloading, with 100k siblings, 100k historical edits and HDD metadata displaced by a recount. The reader can do zero directory enumeration and still miss the gate badly. Returning “pending” in 1 ms is not a successful 15 ms note open.

Required revision: name the replacement of the full index reload, the ID lookup and incomplete-index behavior, bounded checkpoint plus tail replay without pruning history, and the cold-HDD strategy/latency allocation. Keep exact-path conflict probes if required, but acknowledge that they make 15 ms a measured feasibility gate, not a consequence of the big-O expression. Separate successful completion latency from pending/error response latency. Current perf opens only 24 sampled editor notes (crates/perf/src/scenarios/open.rs:12–26); ensure the gate covers cold first-person use and history-heavy notes rather than a namespace microbenchmark.

B5. Full-upload-first does not make physical-capacity errors invisible across users

Plan: lines 183–192. Decision: issue #731 comment 67049, T-A2.

Scenario: B uploads and receives a full-file acknowledgement. Background work can reclaim that independent allocation by cloning A's matching file. Near physical exhaustion, B's next upload succeeds in the matching case and fails ENOSPC in the unmatched case. Logical quota and a logical free-space display can be identical in both worlds. A generic error string still leaks through success versus failure. The same issue affects copy-on-write headroom, snapshot/companion placement and FO1 materialization. Deduplication being delayed until after the first acknowledgement does not erase later observations.

The plan says errors reveal no match, but supplies no physical capacity reservation/admission policy that makes this true. Its finite timing gate also examines only p50/p95 within ±5%. A repeatable difference under that margin, in tails, or in post-dedupe read/write latency can still be an existence signal. This is not resolved by hiding XFS usage; the issue's quota probe already warns against sharing-dependent physical accounting.

Required revision: explicitly gate dedupe on a capacity/headroom and resource-isolation policy, with a safe disabled outcome if the strict T-A2 promise cannot be established. Test controlled near-full disks, delayed dedupe, peer purge, identical local edits and repeated adaptive observations, including binary success/error outcomes and later operations. State the measurable threat/observation model honestly; two quantiles and an isolation audit do not prove “no timing path.” Do not silently weaken T-A2 or claim the current design meets it.

Should-fix gaps

S1. Descriptor safety needs an ownership/ancestor and mount policy, not just beneath

Plan: lines 61–65, 76, 79, 100–108. Code: fsio/src/beneath.rs:52–82,223–250; calternal/src/storage.rs:99–118.

Scenario: after the server opens A/Work, an outside tool renames that directory elsewhere, then the server places a file relative to its still-open directory descriptor. Placement can succeed in the new location. The descriptor's inode identity is unchanged. A deterministic probe confirmed this. Post-placement detection cannot undo the period during which another namespace could see the bytes, and rollback itself can race a replacement.

Separately, Linux beneath uses BENEATH | NO_MAGICLINKS, not NO_XDEV; it permits a mount below the Home. Root role validation does not constrain such descendants. A same-filesystem bind mount can retain st_dev while introducing another ownership or quota domain. Mount disappearance beneath an already-open descriptor can also leave operations using a detached tree while the pathname now resolves elsewhere.

Revision: state the trusted outside-writer privileges and supported namespace model. Define whether renaming ancestors is excluded during placement, what generation a pinned reader authorizes, and which mount identities are admitted/rechecked. Safest initial policy: reject/report nested mounts and unsupported topology consistently in reader, writer, migration and quota; do not follow them simply because path lookup remains beneath a root. Add rename-out, bind-mount and detached-root schedules. The existing two-process/root-lock row is useful, but should also test root/lock replacement and aliases with different state directories while a blocking writer outlives shutdown.

S2. Raw names, aliases and case-only rename need explicit rows

Plan: lines 75–77, 87–93, 103–108. Code: fsio/src/beneath.rs:104–115; notes/src/rename.rs:210–224; fsio/src/lib.rs:109–146; quota branch usage/scan.rs:83–101.

Scenario: two names with raw bytes ff 2e 6d 64 and fe 2e 6d 64 both become the same replacement-character string under lossy UTF-8 conversion. Current Folder::names and disk_names use that conversion. The kernel probe confirmed the collision. A Unicode-folded index cannot use that lossy string as a safe write/delete identity. Silently dropping them from the joined scan can also undercount quota.

Scenario: outside hard links alias a file within a Home or into another accessible tree; a write through an alias is missed by path-only watching. Symlinks can point into the other tier or form cycles. “No app-created hardlinks” does not define handling of outside aliases. Dedupe's independent-inode assumption and per-name quota rule need an explicit compatibility policy. True hard links across distinct filesystems are impossible, but SSD Home/state or two SSD Homes may share a filesystem; symlinks and bind aliases can span tiers.

Scenario: a case-only rename on a supported case-insensitive filesystem needs source -> temp -> destination, as current rename_safely shows. Crash after the intermediate rename, or an outside create of either spelling, leaves more states than “two directory renames.” Also test NFC/NFD collisions and case changes of ancestors on both trees.

Revision: preserve raw filesystem identity separately from display/portable keys; choose lossless representation or explicit preserved unsupported-entry status. Define symlink/hard-link behavior and per-name charges, link-count changes during operations, no-follow behavior and alias event coverage. Model every extra rename/fsync boundary. The eventual-observation limit for newly created outside case collisions is acceptable if stated; exact-path probes must not be presented as detecting all folded collisions immediately.

S3. Sizes specify counts but not a capacity or aggregate workload

Plan: lines 15–23, 150–159.

Scenario: 200k photos averaging 8 MiB occupy about 1.53 TiB. Hashing all of them once in 120 seconds would require about 13 GiB/s, before metadata, reads for validation or other people. The plan does not necessarily require hashing them all in that interval, but “reconciliation” and “changed bytes” do not say whether content validation, indexing or only metadata discovery is complete. A first deployment, plugin update or fingerprint-index loss changes that answer dramatically.

Likewise 100 people each with 100k notes means 10M SSD notes; ten active large Homes means 1M active notes. The specified “one large Home, others sharing the remaining files” distribution does not test that case or establish a supported aggregate maximum. A 1 MiB/10k-line note is a useful regression fixture, but claimed text and original-card photos have no tiering size cutoff. H=100k edits gives no byte/CRDT-state budget without edit sizes. Queue/disk headroom and path depth are unspecified.

Revision: keep all existing counts, but add byte-size distributions and totals per tier, simultaneous versus individual maxima, aggregate active-Homes limits, note/history/photo-companion extremes, directory depth and empty-directory counts, RAM/cache/watch/FD budgets, and required temporary/recovery space. Define metadata discovery, full validation and dedupe completion as separate costs. Do not invent a placement-by-size rule to solve this.

S4. “Every operation” omits important paths and understates some current work

Plan: lines 25–38, 110–126. Code: fsio/src/lib.rs:95–160,318–358; fsio/src/usage.rs:287–341; notes/src/rename.rs:210–248; notes/src/room/save.rs:145–180; notes/src/history.rs:803–823.

Scenario: reusing rename_safely for a 10k-descendant folder invokes usage::moved, which scans the registered counter's entire file map while holding its state mutex. That is O(all counted files), not the advertised parent-ID switch. Existing note rename also enumerates all sibling disk names. The plan proposes stable parent IDs, but does not explicitly retire these old hooks and flat-path indexes; a reader seam alone leaves them intact. Per-entry B-tree mutations can add log N factors to K/R completion work, and rewriting links entails loading, editing, journaling and materializing the linking notes, not just changing an index key.

Save also serializes current text, hashes it and encodes CRDT state. Tier-copy verification needs target rereads and source verification: say which bytes are read on each disk and when, not only O(B). Bulk fsio currently uses syncfs on Linux; reusing it can flush unrelated writes on the same filesystem, which matters for latency isolation.

Revision: add upload/admission, classify/reclassify, dedupe, conflict retention, Trash/restore/purge, export/backup, auth snapshotting, Home-only rebuild, quota recount and rollback. Charge buffers, full content/CRDT passes, indexes and flushes to the actual owner and disk, and distinguish admission from successful durable completion. Define which old helpers/indexes are replaced in each slice. This need not be a huge table: operations sharing one protocol can refer to it with their differing byte/entry costs.

S5. The one-second maximum needs a deadline protocol and an explicit fault boundary

Plan: lines 33, 66, 78, 163, 177–179. Code: notes/src/home.rs:58–69 currently uses 750 ms quiet and 5 seconds maximum; room/save.rs:83–123,145–199 accepts live changes while a frozen save runs.

Scenario: continuous typing is acknowledged just after the materializer freezes its snapshot. That acknowledged edit needs the next snapshot. An already-running copy, fsync, pair placement or merge occupies the necessary resource for another second. Deferring admission of new moves does not preempt that I/O. Existing scheduling cannot meet the new bound simply by changing placement disks. With an outside old-inode writer, B1's safe deferral can itself outlast the deadline.

Scenario: a Home becomes full or stalls immediately after an SSD ack. Reporting degraded status afterwards is correct, but cannot make that already-acknowledged edit satisfy a universal one-second guarantee. The ADR treats this as a visible failed gate, not permission to erase the sample.

Revision: define oldest-unmaterialized-ack tracking, coalescing/coverage semantics, maximum batching interval with execution headroom, scheduling/resource reservations, and when new acknowledgements are backpressured. Define coverage for an acknowledged edit subsequently undone/deleted, and deletion tombstones if applicable. Require deterministic clocks/barriers, independent durable pair reads, and report all misses even under faults. Say plainly that a stalled/unavailable disk can violate the target; do not promise an unconditional physical deadline.

S6. Missed events plus preserved timestamps are missing from reconciliation

Plan: lines 55, 76, 87–89, 133–145. Code: notes/src/scan.rs:7–10,397–410 caches size and mtime; fsio/src/lib.rs:641–651 uses wall time in temporary names.

Scenario: an outside program rewrites a same-length file and restores its mtime, while events are lost; the clock can also move backwards. Reusing the current size/mtime scan treats it as unchanged. Classification, search and fingerprints can remain stale even though the journal never inferred a phase from timestamps. Avoiding timestamp-based replay is good but does not establish discovery freshness.

Revision: add equal-size/preserved-mtime replacement, inode reuse, backward/forward wall-clock jumps and missed events to the model and real tests. Use monotonic time for deadlines and durable sequence/unique IDs for protocol identity; specify how a full validation finds metadata-identical changes and charge that byte work separately. If only eventual content auditing is feasible, expose that exact limit and prohibit provisional metadata from certifying fresh quota or a dedupe candidate.

S7. The test outline needs protocol-specific adversaries and anti-vacuity checks

Plan: lines 130–164, 183–192.

The first-code-PR timing is already explicit and correct. The problem is what those harnesses model. Maps of file versions plus syscall boundaries do not automatically model shared inodes, raw names, moved directory descriptors, mount IDs, sparse logical sizes, mmap writers, permission/ACL inheritance or partial overwrite of a journal record. A model with a magical safe replacement primitive will approve B1. A persistence model that always persists a whole intent at once will approve B2.

Scenario: all contended operations return pending forever. No half-action is acknowledged, unauthorized bytes stay hidden, and the invariants can pass, while useful successful opens/moves never finish. A reader-only first slice can also ship a generic crash harness without exercising the later byte-bearing pair/recovery protocol. “Interleave outside actions at every boundary” does not by itself check outcomes before an eventually quiescent replay.

Revision: require the concrete counterexamples above as named, shrunk vectors; short/partial writes and failed fsyncs on intent creation, phase append, checkpoint and cleanup; independent disk persistence orders; request cancellation after publication; duplicate requests during recovery; revocation with in-flight readers; cross-user alias/dedupe/move/quota interactions; and byte-exact Home-only bootstrap interrupted repeatedly. Test temporary/recovery ENOSPC and inode exhaustion separately from a full final destination. Make every slice activate its actual protocol hooks, and measure successful completion counts, pending duration, bounded queues/watch counts and quiescent progress under stated availability assumptions. The statistical gate must include B5's discrete outcomes, not only quantiles. Run tests on filesystems/topologies matching the claimed support; local syscall probes on the review host are not substitutes.

Failure-table coverage assessment

The eleven broad rows are not eleven complete protocols. The following distinguishes already acknowledged limits from missing decisions; missing cases should be linked to exact state-machine transitions and permanent vectors.

Requested surface Assessment and concrete consequence
Outside writes/truncation/replacement at every stage Named, but ordinary-save old-inode retirement is not resolved; B1.
Outside cross-tree rename Delete/create and ambiguous-ID limits are honestly stated. Add races with a paged recount and contact-pair association; B3/S2.
Partial journal writes Torn tails are named. Byte-complete payloads, corrupt committed records, append-after-tail repair and snapshot cutoff are unspecified; B2/S7.
Fsync ordering on both disks Folder parent flushes are required. Recovery/staging directory creation, pair payload durability and checkpoint/proof retirement ordering remain unspecified; B2.
Missing/full/error disk mid-step T-A1 fail-closed is carried through correctly. Add detached/replaced mount paths, temporary/inode exhaustion and post-ack deadline failures; S1/S5/S7.
Clock changes Not a standalone case; phase timestamps are excluded, but preserved-mtime discovery and timeout identity still need rules; S6.
Two processes Root roles, locks and lock lifetime are included. Prove protection from root/lock replacement and alias roots, including lingering blocking work; S1.
Crash during recovery Named for intents, but interrupted empty-state bootstrap conflicts with corrupt-nonempty refusal; B2.
Hard links and symlinks across trees Outside alias policy, raw inode watch coverage, per-name charge and mount/symlink traversal are not defined; S1/S2/B3.
Case-only/Unicode rename Collisions are named, intermediate temporary-name phases and ancestor spelling changes are not; S2.
Non-UTF-8 names Missing; lossy current helpers merge distinct identities; S2.
Mounts inside a Home Missing as a namespace policy; current reader and quota branch use different traversal rules; S1/B3.
T-A2 cross-user dedupe invisibility Independent files, full upload first and audit are included; later capacity/CoW outcomes and adaptive timing remain unproved; B5.
Quota, moves and whole-Home recount Desired arithmetic is stated, but consistent cross-root freshness, provider generation and bounded admission are not designed; B3.

No separate nit findings: the unresolved items affect correctness, feasibility or acceptance, rather than wording alone.

Simpler design within the accepted decisions

Keep real SSD/HDD trees, format-owned placement/companions, the shared reader, existing action executor/bulk scheduler, and the chosen offline fallback. Do not replace them with stubs, hardlinks or a content-addressed originals store.

Simplify the authority model: use one authoritative protected Home recovery log on SSD Home, plus immutable staged payloads where needed; make state-side job/index records projections of that log rather than a second independently authoritative phase journal. A generation-linked checkpoint stores current auth/account recovery and a committed log cutoff. State loss then rebuilds from that same authority. This keeps recovery metadata on a valid Home as FO1 requires and reuses existing owners; it needs a precise replay and payload-retention proof, not a second generic transaction system.

For a folder operation, a small monotone protocol can be: durable intent; first tree applied and parent directories synced; second tree applied and parent directories synced; required links/companions and recovery data durable; one logical commit/event; receipt-based cleanup. Record enough to recognize syscalls that happened before a progress append. Keep the existing intersecting-prefix serialization and allow pending rather than adding an online exclusion backend that has not been proved. Contact saves use a distinct byte-bearing pair recipe within the same owner protocol; cross-disk copies additionally stage, verify and retire only under exclusion.

This reduces disagreement between state and Home journals; it does not manufacture cross-disk atomicity, safe arbitrary-writer cleanup or strict dedupe privacy. Dedupe should remain disabled unless B5's strict release gate is feasible. It is valid to complete the tiered layout and offline tidy-up first while retaining a no-go decision for online dedupe; that must be stated as incomplete T-A2 delivery, not treated as fulfilling it.

Review evidence and limits

Read-only spot checks confirmed the costs and API differences cited above. Local probes reproduced these exact hazards:

REPRODUCED ordinary-save schedule: same inode check allows unlink of a durable outside write
REPRODUCED ancestor schedule: directory-relative placement succeeds after outside rename into another Home
REPRODUCED name encoding: two distinct non-UTF-8 names collapse under lossy UTF-8 conversion
SYSCALL COUNTEREXAMPLES: PASS (expected hazards observed; no product or performance result claimed)

The probes are syscall counterexamples corresponding to inspected source, not a compiled execution of the Rust helpers. Throughput numbers in S3 and B3 are lower-bound arithmetic, not measurements. No product test or perf result is asserted. Review files and evidence remain outside the repository. No branch, commit, push, merge, deployment or issue closure was performed.

# Independent review of PR #761: revise Reviewed `docs/plans/0731-tiering.md` at **2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80**, based on `d83e54dbe571b625504001a8b6add90d44ca6161`. Read issue #731 and its discussion, the referenced FO1 decision (#642 comment 67053), ADRs 0062–0064, and the current storage, notes, attachments, contacts and quota paths. Also inspected `origin/perf/quota-simple` at **bb36381dd14878c3bdafba1794f4ef89eb0f07a8**; its behavior is materially different from quota on the PR base. A scoped filename search found neither CLAUDE.md nor AGENTS.md in the repository/worktrees, so no instructions from those absent files were assumed. **Verdict: revise before approving this as the implementation plan.** The decisions are mostly carried forward correctly, but several safety properties are statements of desired outcomes without the protocol needed to achieve them. The most consequential gaps concern ordinary saves, Home-only recovery, quota consistency, bounded opens and dedupe privacy. The plan does improve on the recent failures: it explicitly requires 100k-note, 10k-line/repeated-block, 100k-edit-history, 50k-sibling, 1M-HDD-file and multi-person tests in the **first code PR** (lines 130–155, 171–173). It includes open writable handles, repeated recovery crashes and a power-loss persistence model. It honestly permits indefinitely pending tier retirement without verified exclusion and describes outside cross-tree renames as delete/create rather than promising an atomic external view. Those are strengths, not missing requirements. This was a design and source review. No product code was written, built or tested; no performance or crash campaign was run. Three small deterministic kernel probes ran only in the review job folder. They confirm syscall hazards, not that a future implementation necessarily has those bugs. Paths and line numbers below refer to the reviewed commit unless the quota branch is explicitly named. All findings describe a schedule that the plan must resolve, rather than claiming an unimplemented feature already fails. ## Blocking gaps ### B1. Ordinary saves need the same old-inode protection as tier retirement **Plan:** lines 33, 73, 76–77, 94–96, 177–186. **Code:** [fsio/swap.rs](https://git.kayg.org/kayg/calternal/src/commit/2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80/crates/fsio/src/swap.rs#L152), particularly `after_swap` at lines 191–219; `notes/src/room/save.rs:153–199`. **Scenario:** an outside program holds a writable descriptor to a note. Calternal checks the old bytes and exchanges its freshly synced snapshot into the public name. The outside program writes and fsyncs through its old descriptor. The current writer sees the same device/inode at the displaced temporary name and unlinks it. Closing the outside descriptor destroys that durable outside version. This does not require a tier move or dedupe. A deterministic syscall probe reproduced this sequence. The current guard compares device/inode, not a guarantee that the inode cannot change. Adding a final hash/version check merely moves the last race boundary: the outside write can follow that check. The plan explicitly requires exclusion for retirement/dedupe but still describes ordinary saves as reusable “guarded placement.” Its outside-write row therefore assumes a stronger primitive than the existing one provides. **Required revision:** cover every destructive replacement/cleanup, including saves, companion updates, undo and restore. Define either verified exclusion or retention of displaced writable versions with a safe reclamation rule. State what happens to FO1's one-second requirement when such protection cannot be obtained. Put this exact old-descriptor schedule in the materialization gate, before tier moves are enabled. “Pending move” alone does not solve a save that already acknowledged typing. ### B2. Home-only recovery lacks a byte-complete transaction and a restartable bootstrap **Plan:** lines 50–59, 70–80, 177–179. **Decision:** ADR 0062:404–458; ADR 0063:64–85. **Scenario 1:** updating a contact replaces its note, then crashes before replacing the original-card companion. State is lost at the same time. The specified intent fields are paths, IDs, expected versions/hashes, directory identities and progress. None reconstructs the missing new card, including an unknown field or original inline photo, or necessarily restores the overwritten old note. A record saying “finish the pair” is insufficient if its only complete payload was in state. The permitted pre-materialization loss window does not license leaving an incoherent materialized pair with neither a complete old nor complete new version available. **Scenario 2:** empty-state reconstruction creates some databases and crashes. Next startup sees nonempty, incomplete state. The plan says to preserve/refuse corrupt nonempty state, but also promises repeated recovery is idempotent. It does not identify a durable rebuild generation/marker that distinguishes its own interrupted bootstrap from foreign corruption. Refusing forever after the first interrupted rebuild violates the recovery requirement. **Scenario 3:** a revocation reaches the Home log before the state transaction, or the state transaction commits before its final Home proof. Crash between them, then lose state. “Current auth” and “Home proofs precede acknowledgements” do not identify the commit point or resolve which authorization generation is authoritative. Falling back to an older complete snapshot can resurrect a revoked credential; refusing all partial snapshots can make an otherwise recoverable Home unusable. **Required revision:** give a small state machine with its durable commit point, immutable byte payload/staging ownership, authorization sequence, paired-file recovery rule, snapshot/log cutoff and retirement rule. Specify creation and fsync of the recovery directory and all changed parents on their own filesystems. Show how a new complete snapshot plus its log suffix replaces the old generation without a gap, how torn tails differ from corrupt committed records, and how recovery itself resumes after each boundary. This is a design requirement, not a request to write the implementation in this PR. ### B3. Quota is not reconciled with the actual whole-Home owner or a consistent two-tree scan **Plan:** lines 117–126; operation table at 30–38. **Code on quota branch:** [usage.rs](https://git.kayg.org/kayg/calternal/src/commit/bb36381dd14878c3bdafba1794f4ef89eb0f07a8/crates/fsio/src/usage.rs#L1), lines 1–26, 188–255; `usage/scan.rs:52–110`; `auth/src/quota.rs:74–108`. `perf/quota-simple` counts the **whole physical Home**, charges every regular-file name including hard-link aliases, fences directory/inode events, excludes foreign mounts, and refuses admission on incomplete coverage. Admission joins or runs a fresh recount with an approximately two-second deadline. It does not already implement the plan's stable logical-ID event ledger or journal-staging exclusions. Simply routing it through the paged reader changes its correctness mechanism. **Scenario:** count SSD before an outside cross-tree move/copy-delete and HDD afterwards. The file can be counted twice, or neither time in the reverse schedule. If watcher delivery is delayed or lost, a logical commit generation need not change at all. Even an app tier move changes its physical copies without changing logical usage. A cursor that crosses commit generations or skips a “pending” prefix cannot certify a fresh whole-Home total. A separate contact-history provider can change between the file walk and provider read as well. **Scenario:** a large dirty Home requires a cold recount before an upload. One million entries in two seconds requires at least 500,000 entries/s before watch installation, inode coverage and fence costs. The plan budgets 120 seconds for reconciliation but omits quota admission from the cost table. It neither promises successful admission for that Home nor explicitly chooses retryable unavailability. Replacing a cold walk with an index lookup without proving freshness is unsafe. **Required revision:** agree an owner contract that covers both roots under one validated observation/fence, membership and move receipts, contact-history/provider generations, failed entries, raw names, native overflow, mounts and reservations. Define whether pending scans retry, restart or retain a pinned generation; bound retries and make unknown usage fail closed. List create/upload, overwrite, Trash/restore/purge, cancellation, outside changes and state-loss reservation recovery explicitly. Add recount/admission cost and availability budgets at 100k/1M. Preserve the branch's known failure vectors instead of reducing the problem to “no double-release.” ### B4. The 15 ms open is not established by construction **Plan:** lines 32, 45–46, 100–108, 171–173. **Code:** `notes/src/lib.rs:253–285`; `home.rs:237–241,291–295`; [index.rs:144](https://git.kayg.org/kayg/calternal/src/commit/2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80/crates/notes/src/index.rs#L144); `scan.rs:144–160`; [room/load.rs:44](https://git.kayg.org/kayg/calternal/src/commit/2cf7e3fbd8ca3d3180ce50fbe2cafb8ecc179c80/crates/notes/src/room/load.rs#L44); `history.rs:803–823`. The proposed joined lookup removes enumeration from one layer, but the current caller still has these costs: - First use loads the entire `ids.jsonl`; reopening an unloaded index does the same under its mutex: O(N) bytes/entries. - An unknown non-daily ID can wait for the startup scan, up to 60 seconds. “Dirty or unindexed names wait” does not provide an open latency bound. - A room collects **all** history updates into a vector, then applies every update. H is unbounded; the plan notices this but chooses no checkpoint/replay bound. Update-application cost and retained CRDT state also need measurement; counting serialized history bytes alone is not a time proof. - Every direct open synchronously probes HDD metadata. At depth d, cold pathname lookup can require several storage reads even for an SSD note. O(d) and O(log N) do not bound rotational seek/queue latency below 15 ms under concurrent scan, quota and dedupe traffic. **Scenario:** open one cold note after index unloading, with 100k siblings, 100k historical edits and HDD metadata displaced by a recount. The reader can do zero directory enumeration and still miss the gate badly. Returning “pending” in 1 ms is not a successful 15 ms note open. **Required revision:** name the replacement of the full index reload, the ID lookup and incomplete-index behavior, bounded checkpoint plus tail replay without pruning history, and the cold-HDD strategy/latency allocation. Keep exact-path conflict probes if required, but acknowledge that they make 15 ms a measured feasibility gate, not a consequence of the big-O expression. Separate successful completion latency from pending/error response latency. Current perf opens only 24 sampled editor notes (`crates/perf/src/scenarios/open.rs:12–26`); ensure the gate covers cold first-person use and history-heavy notes rather than a namespace microbenchmark. ### B5. Full-upload-first does not make physical-capacity errors invisible across users **Plan:** lines 183–192. **Decision:** issue #731 comment 67049, T-A2. **Scenario:** B uploads and receives a full-file acknowledgement. Background work can reclaim that independent allocation by cloning A's matching file. Near physical exhaustion, B's next upload succeeds in the matching case and fails ENOSPC in the unmatched case. Logical quota and a logical free-space display can be identical in both worlds. A generic error string still leaks through success versus failure. The same issue affects copy-on-write headroom, snapshot/companion placement and FO1 materialization. Deduplication being delayed until after the first acknowledgement does not erase later observations. The plan says errors reveal no match, but supplies no physical capacity reservation/admission policy that makes this true. Its finite timing gate also examines only p50/p95 within ±5%. A repeatable difference under that margin, in tails, or in post-dedupe read/write latency can still be an existence signal. This is not resolved by hiding XFS usage; the issue's quota probe already warns against sharing-dependent physical accounting. **Required revision:** explicitly gate dedupe on a capacity/headroom and resource-isolation policy, with a safe disabled outcome if the strict T-A2 promise cannot be established. Test controlled near-full disks, delayed dedupe, peer purge, identical local edits and repeated adaptive observations, including binary success/error outcomes and later operations. State the measurable threat/observation model honestly; two quantiles and an isolation audit do not prove “no timing path.” Do not silently weaken T-A2 or claim the current design meets it. ## Should-fix gaps ### S1. Descriptor safety needs an ownership/ancestor and mount policy, not just `beneath` **Plan:** lines 61–65, 76, 79, 100–108. **Code:** `fsio/src/beneath.rs:52–82,223–250`; `calternal/src/storage.rs:99–118`. **Scenario:** after the server opens `A/Work`, an outside tool renames that directory elsewhere, then the server places a file relative to its still-open directory descriptor. Placement can succeed in the new location. The descriptor's inode identity is unchanged. A deterministic probe confirmed this. Post-placement detection cannot undo the period during which another namespace could see the bytes, and rollback itself can race a replacement. Separately, Linux `beneath` uses `BENEATH | NO_MAGICLINKS`, **not** `NO_XDEV`; it permits a mount below the Home. Root role validation does not constrain such descendants. A same-filesystem bind mount can retain st_dev while introducing another ownership or quota domain. Mount disappearance beneath an already-open descriptor can also leave operations using a detached tree while the pathname now resolves elsewhere. **Revision:** state the trusted outside-writer privileges and supported namespace model. Define whether renaming ancestors is excluded during placement, what generation a pinned reader authorizes, and which mount identities are admitted/rechecked. Safest initial policy: reject/report nested mounts and unsupported topology consistently in reader, writer, migration and quota; do not follow them simply because path lookup remains beneath a root. Add rename-out, bind-mount and detached-root schedules. The existing two-process/root-lock row is useful, but should also test root/lock replacement and aliases with different state directories while a blocking writer outlives shutdown. ### S2. Raw names, aliases and case-only rename need explicit rows **Plan:** lines 75–77, 87–93, 103–108. **Code:** `fsio/src/beneath.rs:104–115`; `notes/src/rename.rs:210–224`; `fsio/src/lib.rs:109–146`; quota branch `usage/scan.rs:83–101`. **Scenario:** two names with raw bytes `ff 2e 6d 64` and `fe 2e 6d 64` both become the same replacement-character string under lossy UTF-8 conversion. Current `Folder::names` and `disk_names` use that conversion. The kernel probe confirmed the collision. A Unicode-folded index cannot use that lossy string as a safe write/delete identity. Silently dropping them from the joined scan can also undercount quota. **Scenario:** outside hard links alias a file within a Home or into another accessible tree; a write through an alias is missed by path-only watching. Symlinks can point into the other tier or form cycles. “No app-created hardlinks” does not define handling of outside aliases. Dedupe's independent-inode assumption and per-name quota rule need an explicit compatibility policy. True hard links across distinct filesystems are impossible, but SSD Home/state or two SSD Homes may share a filesystem; symlinks and bind aliases can span tiers. **Scenario:** a case-only rename on a supported case-insensitive filesystem needs `source -> temp -> destination`, as current `rename_safely` shows. Crash after the intermediate rename, or an outside create of either spelling, leaves more states than “two directory renames.” Also test NFC/NFD collisions and case changes of ancestors on both trees. **Revision:** preserve raw filesystem identity separately from display/portable keys; choose lossless representation or explicit preserved unsupported-entry status. Define symlink/hard-link behavior and per-name charges, link-count changes during operations, no-follow behavior and alias event coverage. Model every extra rename/fsync boundary. The eventual-observation limit for newly created outside case collisions is acceptable if stated; exact-path probes must not be presented as detecting all folded collisions immediately. ### S3. Sizes specify counts but not a capacity or aggregate workload **Plan:** lines 15–23, 150–159. **Scenario:** 200k photos averaging 8 MiB occupy about **1.53 TiB**. Hashing all of them once in 120 seconds would require about **13 GiB/s**, before metadata, reads for validation or other people. The plan does not necessarily require hashing them all in that interval, but “reconciliation” and “changed bytes” do not say whether content validation, indexing or only metadata discovery is complete. A first deployment, plugin update or fingerprint-index loss changes that answer dramatically. Likewise 100 people each with 100k notes means 10M SSD notes; ten active large Homes means 1M active notes. The specified “one large Home, others sharing the remaining files” distribution does not test that case or establish a supported aggregate maximum. A 1 MiB/10k-line note is a useful regression fixture, but claimed text and original-card photos have no tiering size cutoff. H=100k edits gives no byte/CRDT-state budget without edit sizes. Queue/disk headroom and path depth are unspecified. **Revision:** keep all existing counts, but add byte-size distributions and totals per tier, simultaneous versus individual maxima, aggregate active-Homes limits, note/history/photo-companion extremes, directory depth and empty-directory counts, RAM/cache/watch/FD budgets, and required temporary/recovery space. Define metadata discovery, full validation and dedupe completion as separate costs. Do not invent a placement-by-size rule to solve this. ### S4. “Every operation” omits important paths and understates some current work **Plan:** lines 25–38, 110–126. **Code:** `fsio/src/lib.rs:95–160,318–358`; `fsio/src/usage.rs:287–341`; `notes/src/rename.rs:210–248`; `notes/src/room/save.rs:145–180`; `notes/src/history.rs:803–823`. **Scenario:** reusing `rename_safely` for a 10k-descendant folder invokes `usage::moved`, which scans the registered counter's entire file map while holding its state mutex. That is O(all counted files), not the advertised parent-ID switch. Existing note rename also enumerates all sibling disk names. The plan proposes stable parent IDs, but does not explicitly retire these old hooks and flat-path indexes; a reader seam alone leaves them intact. Per-entry B-tree mutations can add log N factors to K/R completion work, and rewriting links entails loading, editing, journaling and materializing the linking notes, not just changing an index key. Save also serializes current text, hashes it and encodes CRDT state. Tier-copy verification needs target rereads and source verification: say which bytes are read on each disk and when, not only O(B). Bulk fsio currently uses `syncfs` on Linux; reusing it can flush unrelated writes on the same filesystem, which matters for latency isolation. **Revision:** add upload/admission, classify/reclassify, dedupe, conflict retention, Trash/restore/purge, export/backup, auth snapshotting, Home-only rebuild, quota recount and rollback. Charge buffers, full content/CRDT passes, indexes and flushes to the actual owner and disk, and distinguish admission from successful durable completion. Define which old helpers/indexes are replaced in each slice. This need not be a huge table: operations sharing one protocol can refer to it with their differing byte/entry costs. ### S5. The one-second maximum needs a deadline protocol and an explicit fault boundary **Plan:** lines 33, 66, 78, 163, 177–179. **Code:** `notes/src/home.rs:58–69` currently uses 750 ms quiet and **5 seconds** maximum; `room/save.rs:83–123,145–199` accepts live changes while a frozen save runs. **Scenario:** continuous typing is acknowledged just after the materializer freezes its snapshot. That acknowledged edit needs the next snapshot. An already-running copy, fsync, pair placement or merge occupies the necessary resource for another second. Deferring admission of new moves does not preempt that I/O. Existing scheduling cannot meet the new bound simply by changing placement disks. With an outside old-inode writer, B1's safe deferral can itself outlast the deadline. **Scenario:** a Home becomes full or stalls immediately after an SSD ack. Reporting degraded status afterwards is correct, but cannot make that already-acknowledged edit satisfy a universal one-second guarantee. The ADR treats this as a visible failed gate, not permission to erase the sample. **Revision:** define oldest-unmaterialized-ack tracking, coalescing/coverage semantics, maximum batching interval with execution headroom, scheduling/resource reservations, and when new acknowledgements are backpressured. Define coverage for an acknowledged edit subsequently undone/deleted, and deletion tombstones if applicable. Require deterministic clocks/barriers, independent durable pair reads, and report all misses even under faults. Say plainly that a stalled/unavailable disk can violate the target; do not promise an unconditional physical deadline. ### S6. Missed events plus preserved timestamps are missing from reconciliation **Plan:** lines 55, 76, 87–89, 133–145. **Code:** `notes/src/scan.rs:7–10,397–410` caches size and mtime; `fsio/src/lib.rs:641–651` uses wall time in temporary names. **Scenario:** an outside program rewrites a same-length file and restores its mtime, while events are lost; the clock can also move backwards. Reusing the current size/mtime scan treats it as unchanged. Classification, search and fingerprints can remain stale even though the journal never inferred a phase from timestamps. Avoiding timestamp-based replay is good but does not establish discovery freshness. **Revision:** add equal-size/preserved-mtime replacement, inode reuse, backward/forward wall-clock jumps and missed events to the model and real tests. Use monotonic time for deadlines and durable sequence/unique IDs for protocol identity; specify how a full validation finds metadata-identical changes and charge that byte work separately. If only eventual content auditing is feasible, expose that exact limit and prohibit provisional metadata from certifying fresh quota or a dedupe candidate. ### S7. The test outline needs protocol-specific adversaries and anti-vacuity checks **Plan:** lines 130–164, 183–192. The first-code-PR timing is already explicit and correct. The problem is what those harnesses model. Maps of file versions plus syscall boundaries do not automatically model shared inodes, raw names, moved directory descriptors, mount IDs, sparse logical sizes, mmap writers, permission/ACL inheritance or partial overwrite of a journal record. A model with a magical safe replacement primitive will approve B1. A persistence model that always persists a whole intent at once will approve B2. **Scenario:** all contended operations return pending forever. No half-action is acknowledged, unauthorized bytes stay hidden, and the invariants can pass, while useful successful opens/moves never finish. A reader-only first slice can also ship a generic crash harness without exercising the later byte-bearing pair/recovery protocol. “Interleave outside actions at every boundary” does not by itself check outcomes before an eventually quiescent replay. **Revision:** require the concrete counterexamples above as named, shrunk vectors; short/partial writes and failed fsyncs on intent creation, phase append, checkpoint and cleanup; independent disk persistence orders; request cancellation after publication; duplicate requests during recovery; revocation with in-flight readers; cross-user alias/dedupe/move/quota interactions; and byte-exact Home-only bootstrap interrupted repeatedly. Test temporary/recovery ENOSPC and inode exhaustion separately from a full final destination. Make every slice activate its actual protocol hooks, and measure successful completion counts, pending duration, bounded queues/watch counts and quiescent progress under stated availability assumptions. The statistical gate must include B5's discrete outcomes, not only quantiles. Run tests on filesystems/topologies matching the claimed support; local syscall probes on the review host are not substitutes. ## Failure-table coverage assessment The eleven broad rows are not eleven complete protocols. The following distinguishes already acknowledged limits from missing decisions; missing cases should be linked to exact state-machine transitions and permanent vectors. | Requested surface | Assessment and concrete consequence | |---|---| | Outside writes/truncation/replacement at every stage | Named, but ordinary-save old-inode retirement is not resolved; B1. | | Outside cross-tree rename | Delete/create and ambiguous-ID limits are honestly stated. Add races with a paged recount and contact-pair association; B3/S2. | | Partial journal writes | Torn tails are named. Byte-complete payloads, corrupt committed records, append-after-tail repair and snapshot cutoff are unspecified; B2/S7. | | Fsync ordering on both disks | Folder parent flushes are required. Recovery/staging directory creation, pair payload durability and checkpoint/proof retirement ordering remain unspecified; B2. | | Missing/full/error disk mid-step | T-A1 fail-closed is carried through correctly. Add detached/replaced mount paths, temporary/inode exhaustion and post-ack deadline failures; S1/S5/S7. | | Clock changes | Not a standalone case; phase timestamps are excluded, but preserved-mtime discovery and timeout identity still need rules; S6. | | Two processes | Root roles, locks and lock lifetime are included. Prove protection from root/lock replacement and alias roots, including lingering blocking work; S1. | | Crash during recovery | Named for intents, but interrupted empty-state bootstrap conflicts with corrupt-nonempty refusal; B2. | | Hard links and symlinks across trees | Outside alias policy, raw inode watch coverage, per-name charge and mount/symlink traversal are not defined; S1/S2/B3. | | Case-only/Unicode rename | Collisions are named, intermediate temporary-name phases and ancestor spelling changes are not; S2. | | Non-UTF-8 names | Missing; lossy current helpers merge distinct identities; S2. | | Mounts inside a Home | Missing as a namespace policy; current reader and quota branch use different traversal rules; S1/B3. | | T-A2 cross-user dedupe invisibility | Independent files, full upload first and audit are included; later capacity/CoW outcomes and adaptive timing remain unproved; B5. | | Quota, moves and whole-Home recount | Desired arithmetic is stated, but consistent cross-root freshness, provider generation and bounded admission are not designed; B3. | No separate nit findings: the unresolved items affect correctness, feasibility or acceptance, rather than wording alone. ## Simpler design within the accepted decisions Keep real SSD/HDD trees, format-owned placement/companions, the shared reader, existing action executor/bulk scheduler, and the chosen offline fallback. Do not replace them with stubs, hardlinks or a content-addressed originals store. Simplify the authority model: use **one authoritative protected Home recovery log** on SSD Home, plus immutable staged payloads where needed; make state-side job/index records projections of that log rather than a second independently authoritative phase journal. A generation-linked checkpoint stores current auth/account recovery and a committed log cutoff. State loss then rebuilds from that same authority. This keeps recovery metadata on a valid Home as FO1 requires and reuses existing owners; it needs a precise replay and payload-retention proof, not a second generic transaction system. For a folder operation, a small monotone protocol can be: durable intent; first tree applied and parent directories synced; second tree applied and parent directories synced; required links/companions and recovery data durable; one logical commit/event; receipt-based cleanup. Record enough to recognize syscalls that happened before a progress append. Keep the existing intersecting-prefix serialization and allow pending rather than adding an online exclusion backend that has not been proved. Contact saves use a distinct byte-bearing pair recipe within the same owner protocol; cross-disk copies additionally stage, verify and retire only under exclusion. This reduces disagreement between state and Home journals; it does not manufacture cross-disk atomicity, safe arbitrary-writer cleanup or strict dedupe privacy. Dedupe should remain disabled unless B5's strict release gate is feasible. It is valid to complete the tiered layout and offline tidy-up first while retaining a no-go decision for online dedupe; that must be stated as incomplete T-A2 delivery, not treated as fulfilling it. ## Review evidence and limits Read-only spot checks confirmed the costs and API differences cited above. Local probes reproduced these exact hazards: ```text REPRODUCED ordinary-save schedule: same inode check allows unlink of a durable outside write REPRODUCED ancestor schedule: directory-relative placement succeeds after outside rename into another Home REPRODUCED name encoding: two distinct non-UTF-8 names collapse under lossy UTF-8 conversion SYSCALL COUNTEREXAMPLES: PASS (expected hazards observed; no product or performance result claimed) ``` The probes are syscall counterexamples corresponding to inspected source, not a compiled execution of the Rust helpers. Throughput numbers in S3 and B3 are lower-bound arithmetic, not measurements. No product test or perf result is asserted. Review files and evidence remain outside the repository. No branch, commit, push, merge, deployment or issue closure was performed.
Author
Owner

Round 2 design revision pushed to docs/tiering-731-plan: b1e3ad02. Replaces the plan reviewed in comment 69394. Maximum three review rounds per change; no additional review round was started here.

All blocking and should-fix findings are addressed by a mechanism or an explicit limit:

  • B1 — ordinary saves/old writable descriptors: every replacement, including saves, companions, undo/restore and cleanup, retains displaced inodes. Hashes, quiet and inode checks never authorize deletion. Reclamation initially requires an offline exclusion pass covering old descriptors, mmap, aliases and new opens; indefinite retention/space backpressure and FO1 failures are explicit. The exact late-write-after-exchange vector gates materialization before moves.
  • B2 — Home-only recovery: one protected authoritative Home log; immutable byte-complete note/card payloads; PREPARE/APPLY/COMMIT/CLEAN with a synced HEAD commit point. Added auth/revocation ordering, checkpoint cutoff plus linked suffix, committed-corruption refusal versus torn-tail repair, payload retirement, per-filesystem parent syncs and a durable bootstrap-generation marker that survives repeated rebuild crashes. State is a projection. Protected recovery bytes are excluded from ordinary reads/lists/exports.
  • B3 — quota: explicitly owned by #764, perf/quota-simple at d0e6d6f6 (three independent rounds passed). Requires one two-root observation/invalidation generation, native directory/inode fence, stable provider generations and existing fail-closed coverage rules. Every regular-file name counts full logical size, including deduped files and temporary/retained copies; no invented staging exclusions or second accounting ledger. Admission is ≤2 s or quietly retried; cold 100k/1M successful admission is not promised. Create/overwrite/Trash/restore/purge/cancellation/state-loss behavior is stated.
  • B4 — 15 ms opens: replace full ids.jsonl reload and flat paths with paged ID/child indexes; replace all-history replay with checkpoints and a ≤256-update/256 KiB tail without pruning history. Distinguish unavailable/pending from successful opens. Both-tree probes remain, with explicit latency allocation and cold-HDD/large-CRDT feasibility risk; failure blocks cutover. Cold first-person, index-reopen and history-heavy editor opens are required.
  • B5 — dedupe privacy: split ships with dedupe OFF. Users see quota-full warnings only; physical-capacity warnings are admin-only. Later dedupe needs sharing-independent headroom/capacity and resource isolation, including binary success/error outcomes after dedupe, peer purge and CoW/materialization. Adaptive near-full and later-operation distributions/tails are required; finite tests are not claimed to prove no signal. T-A2 dedupe delivery remains explicitly incomplete.
  • S1 — topology/descriptor safety: online outside writers may edit files; ancestor moves, cross-person aliases, root/lock replacement and mount changes require offline coordination. Reject nested/bind/detached mounts consistently, validate mount/root identities, and hold identity locks through lingering writer exit across different state aliases. Post-checks cannot repair rename-out exposure.
  • S2 — names/aliases: preserve raw names separately from display/folded keys; non-UTF-8 entries remain counted/preserved but unsupported for mutation. Symlinks are no-follow; hard links charge per name, require inode coverage and are unsupported for mutation/dedupe. Journal every case-only intermediate rename/fsync; NFC/NFD and ancestor case collisions have explicit eventual-observation limits.
  • S3 — scale/resources: added byte distributions and tier totals, large text/cards/history, depth/empty directories, 10M-note aggregate/1M-active fixture and 1M-HDD aggregate cap. Added cache/buffer/job/FD/watch/kernel-memory and temporary/recovery space budgets. The 2M-watch ceiling explicitly does not promise simultaneous quota readiness for all 10M notes. Metadata discovery, full byte audit and dedupe completion are distinct costs.
  • S4 — operation costs/owners: expanded the cost table to upload/admission, classification, conflict retention, Trash/restore/purge/undo, export/backup/auth, rebuild, dedupe and rollback. Charge serialization/hash/CRDT passes, target rereads/source verification, B-tree factors and link materialization. Slices explicitly replace old ID reload/sibling scan, whole-map move hook, destructive after_swap, debounce and broad syncfs paths.
  • S5 — FO1 deadline: track oldest uncovered ack/sequence on a monotonic clock; ≤100 ms batching, post-freeze snapshots, causal successor/tombstone coverage, reserved space/I/O and pre-ack backpressure. Every >1 s sample, including a post-ack disk fault, fails; no unconditional physical deadline is claimed.
  • S6 — freshness/clocks: full byte audits detect metadata-identical rewrites after missed events; classify/search freshness remains eventual until audit completion. Fresh quota uses #764 fences, dedupe uses verified bytes/exclusion. Added preserved-mtime/inode-reuse/clock-jump vectors and sequence/random protocol identities.
  • S7 — adversarial tests/progress: first-code-PR harnesses model real shared inodes/mmap/raw names/mounts/ACLs/sparse sizes, short/partial writes, failed fsync and independent disk persistence. Every slice activates its actual hooks. Added repeated byte-exact bootstrap, cancellation/duplicate publication, in-flight revocation, cross-user schedules, temporary/recovery ENOSPC versus inode exhaustion, and successful-completion/pending/resource metrics so all-pending cannot pass under supported availability/exclusion assumptions.

The failure table and second-pass five likely failures were rewritten. Final performance measurements now run only on x86_64 perf-test VMs with uncached 4K disks, using build-host binaries; never on 10by32-* because Mac RAM caches their reads.

Validation: docs-only diff; git diff --check against both previous HEAD and PR base; relative links resolve; all 12 findings covered; four-page A4 print layout (10 pt body / 9 pt tables). Local print proof: /home/agent/codex-jobs/revise-761-round2/plan-print-check.pdf. No product build/tests, performance/crash runs, deploy, merge or production cutover.

Round 2 design revision pushed to `docs/tiering-731-plan`: [b1e3ad02](https://git.kayg.org/kayg/calternal/commit/b1e3ad02c6236bb59c2ee5fc348a8241bc870167). Replaces the plan reviewed in [comment 69394](https://git.kayg.org/kayg/calternal/pulls/761#issuecomment-69394). Maximum three review rounds per change; no additional review round was started here. All blocking and should-fix findings are addressed by a mechanism or an explicit limit: - **B1 — ordinary saves/old writable descriptors:** every replacement, including saves, companions, undo/restore and cleanup, retains displaced inodes. Hashes, quiet and inode checks never authorize deletion. Reclamation initially requires an offline exclusion pass covering old descriptors, mmap, aliases and new opens; indefinite retention/space backpressure and FO1 failures are explicit. The exact late-write-after-exchange vector gates materialization before moves. - **B2 — Home-only recovery:** one protected authoritative Home log; immutable byte-complete note/card payloads; PREPARE/APPLY/COMMIT/CLEAN with a synced HEAD commit point. Added auth/revocation ordering, checkpoint cutoff plus linked suffix, committed-corruption refusal versus torn-tail repair, payload retirement, per-filesystem parent syncs and a durable bootstrap-generation marker that survives repeated rebuild crashes. State is a projection. Protected recovery bytes are excluded from ordinary reads/lists/exports. - **B3 — quota:** explicitly owned by #764, `perf/quota-simple` at `d0e6d6f6` (three independent rounds passed). Requires one two-root observation/invalidation generation, native directory/inode fence, stable provider generations and existing fail-closed coverage rules. Every regular-file name counts full logical size, including deduped files and temporary/retained copies; no invented staging exclusions or second accounting ledger. Admission is ≤2 s or quietly retried; cold 100k/1M successful admission is not promised. Create/overwrite/Trash/restore/purge/cancellation/state-loss behavior is stated. - **B4 — 15 ms opens:** replace full `ids.jsonl` reload and flat paths with paged ID/child indexes; replace all-history replay with checkpoints and a ≤256-update/256 KiB tail without pruning history. Distinguish unavailable/pending from successful opens. Both-tree probes remain, with explicit latency allocation and cold-HDD/large-CRDT feasibility risk; failure blocks cutover. Cold first-person, index-reopen and history-heavy editor opens are required. - **B5 — dedupe privacy:** split ships with dedupe OFF. Users see quota-full warnings only; physical-capacity warnings are admin-only. Later dedupe needs sharing-independent headroom/capacity and resource isolation, including binary success/error outcomes after dedupe, peer purge and CoW/materialization. Adaptive near-full and later-operation distributions/tails are required; finite tests are not claimed to prove no signal. T-A2 dedupe delivery remains explicitly incomplete. - **S1 — topology/descriptor safety:** online outside writers may edit files; ancestor moves, cross-person aliases, root/lock replacement and mount changes require offline coordination. Reject nested/bind/detached mounts consistently, validate mount/root identities, and hold identity locks through lingering writer exit across different state aliases. Post-checks cannot repair rename-out exposure. - **S2 — names/aliases:** preserve raw names separately from display/folded keys; non-UTF-8 entries remain counted/preserved but unsupported for mutation. Symlinks are no-follow; hard links charge per name, require inode coverage and are unsupported for mutation/dedupe. Journal every case-only intermediate rename/fsync; NFC/NFD and ancestor case collisions have explicit eventual-observation limits. - **S3 — scale/resources:** added byte distributions and tier totals, large text/cards/history, depth/empty directories, 10M-note aggregate/1M-active fixture and 1M-HDD aggregate cap. Added cache/buffer/job/FD/watch/kernel-memory and temporary/recovery space budgets. The 2M-watch ceiling explicitly does not promise simultaneous quota readiness for all 10M notes. Metadata discovery, full byte audit and dedupe completion are distinct costs. - **S4 — operation costs/owners:** expanded the cost table to upload/admission, classification, conflict retention, Trash/restore/purge/undo, export/backup/auth, rebuild, dedupe and rollback. Charge serialization/hash/CRDT passes, target rereads/source verification, B-tree factors and link materialization. Slices explicitly replace old ID reload/sibling scan, whole-map move hook, destructive `after_swap`, debounce and broad `syncfs` paths. - **S5 — FO1 deadline:** track oldest uncovered ack/sequence on a monotonic clock; ≤100 ms batching, post-freeze snapshots, causal successor/tombstone coverage, reserved space/I/O and pre-ack backpressure. Every >1 s sample, including a post-ack disk fault, fails; no unconditional physical deadline is claimed. - **S6 — freshness/clocks:** full byte audits detect metadata-identical rewrites after missed events; classify/search freshness remains eventual until audit completion. Fresh quota uses #764 fences, dedupe uses verified bytes/exclusion. Added preserved-mtime/inode-reuse/clock-jump vectors and sequence/random protocol identities. - **S7 — adversarial tests/progress:** first-code-PR harnesses model real shared inodes/mmap/raw names/mounts/ACLs/sparse sizes, short/partial writes, failed fsync and independent disk persistence. Every slice activates its actual hooks. Added repeated byte-exact bootstrap, cancellation/duplicate publication, in-flight revocation, cross-user schedules, temporary/recovery ENOSPC versus inode exhaustion, and successful-completion/pending/resource metrics so all-pending cannot pass under supported availability/exclusion assumptions. The failure table and second-pass five likely failures were rewritten. Final performance measurements now run only on **x86_64 perf-test VMs with uncached 4K disks**, using build-host binaries; never on `10by32-*` because Mac RAM caches their reads. Validation: docs-only diff; `git diff --check` against both previous HEAD and PR base; relative links resolve; all 12 findings covered; four-page A4 print layout (10 pt body / 9 pt tables). Local print proof: `/home/agent/codex-jobs/revise-761-round2/plan-print-check.pdf`. No product build/tests, performance/crash runs, deploy, merge or production cutover.
Author
Owner

Independent design review, round 2: approve with required edits

PR #761, docs/tiering-731-plan, reviewed at b1e3ad02c6. Base: d83e54dbe571b625504001a8b6add90d44ca6161. Review worktree: /home/agent/wt/review-761b, on 10by32-01-std.

Verdict: approve with required edits. Exactly two small plan edits are required below. The five round-1 blockers no longer call for a broad redesign. The revised plan supplies mechanisms for the central safety problems and explicitly declines claims it cannot yet establish. The remaining edits close a bootstrap interruption edge and make one quota accounting boundary unambiguous. This approves the design subject to those edits; it does not approve implementation, cutover, performance, or dedupe enablement.

I read the round-1 review, the itemized response, issue #731's decisions/discussion including the 29 October quota findings, ADRs 0062–0064, and the save/placement code. I also read PR #764's review/fix discussion through its final ship verdict and inspected its counter, scanner, event fence and Auth admission at d0e6d6f60f. Its three review rounds concern that quota implementation; they do not establish a two-root extension that has yet to be written. User instructions supplied the later warning/dedupe/perf-host decisions. References to plan lines below use the reviewed head.

Required edits — complete list

E1. Make the first bootstrap marker creation restartable (lines 54–57, 187, 211–214)

The durable marker resolves an interrupted rebuild after the marker is durable. Its own interrupted creation is still unspecified.

Concrete schedule: valid Home, empty state; bootstrap creates its marker, writes only a prefix, then loses power before marker sync. The next startup sees nonempty state without a valid marker. “Resume/recreate only marked staging” plus “preserve/refuse unmarked corrupt nonempty state” refuses the installation's own first rebuild indefinitely. A crash after creating the generation directory but before its marker has the same problem if that directory lies inside the guarded state location.

Small edit: add an explicit ownership rule covering the interval before the marker is durable. One sufficient mechanism is a synced bootstrap PREPARE in the existing Home recovery log, recording the unique staging generation and target state location before any state creation. Replay can then recognize and rebuild that exact owned generation even if its local marker is absent/torn; it still preserves/refuses unrelated nonempty state. Add one named vector: crash at directory creation and every short-write/sync boundary of the first marker, then repeatedly recover with state lost again. No second journal or new recovery owner is needed.

E2. Specify the current-card/history quota split (lines 85–89)

Charging every Home regular-file name at full logical size is clear and follows the quota decision. “Providers add only their existing non-Home charges” does not resolve the overlap with current contact cards.

The current provider (crates/contacts-actions/src/write.rs:135–141, also present at #764's pinned head) returns history::retained_bytes. That includes the current card version, not just additional historical versions (crates/notes/src/history/companions.rs:205–244,765–780). ADR 0064's Import section explicitly says the mandatory Home companion is the same current logical card and is not a second quota charge.

Concrete schedule: one newly imported, unchunked card of V bytes has one retained current version and a Home companion of V bytes. Ignoring other charges, a literal reuse of the provider adds V to the Home walk's V: 2V for the current card. The shared fence can certify that number perfectly; it cannot fix the accounting overlap. Large inline photos make the difference material.

Small edit: state that the contacts owner must adapt its provider to report the additional retained-history charge beyond the current card already charged by the Home walk, preserving ADR 0064's unchanged-photo sharing within retained history. Tie that split to the same stable provider/pair generation already required here. Add the one-current-version example: its card contribution is V, plus separately counted note/recovery/holding files, not V+V. This does not exempt any Home file, discount deduped files, or require a second quota ledger.

Disposition of every round-1 finding

“Resolved” below means resolved as a design contract, with implementation gates still outstanding.

Finding Disposition Evidence / remaining boundary
B1 ordinary saves / old descriptors Resolved by retention; reclamation limit honest. Lines 25–41 cover every replacement, including saves, companions, undo/restore and cleanup. Displaced public inodes stay linked; only verified exclusion permits reclaim. Indefinite charges/backpressure and deadline misses are explicit. Slice 3 removes destructive after_swap cleanup before moves enable.
B2 Home-only recovery / bootstrap Core resolved; E1 remains open. Complete immutable pair payloads, one authoritative Home log, durable HEAD commit, auth ordering, snapshot+suffix retention and marked rebuild replace the previous missing protocol. The first marker's own interruption needs E1's ownership proof.
B3 two-tree quota Observation/admission mechanism resolved; E2 remains open. Lines 75–101 retain #764's whole-Home owner and require one root-set generation, complete native coverage, shared event/mount fence, stable providers, serialized admission and no stale/unknown admission. E2 specifies which provider bytes belong in the sum.
B4 15 ms open Honest measured gate, with cutover withheld on failure. Lines 103–115 replace ID reload/all-history replay, distinguish pending from successful opens and acknowledge cold HDD/CRDT costs. Lines 171–177 and 234–235 keep the old layout until gates pass. This is no claim of feasibility by construction.
B5 dedupe capacity/privacy Honest deferred limit. Dedupe ships OFF and T-A2 delivery remains incomplete (9,177,194–207). Later enablement needs sharing-independent capacity/CoW reserves and isolation, including binary outcomes and later operations. Generic errors alone are expressly insufficient.
S1 topology/descriptor authority Resolved through an explicit supported namespace limit. Lines 43–45,60–69 require current auth checks, root/mount identity validation and locks through writer exit; ancestor moves, root replacement, cross-person aliases and mount changes require offline coordination. This is a cooperative outside-writer model, not a sandbox against a process able to move ancestors arbitrarily. Post-checks cannot prevent that exposure.
S2 raw names, links, case Resolved, with explicit unsupported operations. Raw names remain identities/countable; no-follow symlinks and hard-link mutation/dedupe restrictions are explicit. Case-only intermediate renames and each parent sync are journaled. Newly introduced folded collisions remain eventually observed (27–29,66–71,185).
S3 capacity/aggregate scale Resolved as a qualification envelope, with limits honest. Lines 134–151 add byte distributions, 10M total/1M active notes, aggregate HDD cap, large cards/history, depth/empty directories and resource/headroom budgets. Full simultaneous quota readiness for 10M notes is expressly unclaimed.
S4 operation costs/old paths Resolved. Lines 153–177 charge serialization, CRDT, verification reads, links and B-tree costs; cover admission, recovery, backup and rollback; and name the old helpers/indexes to remove. Quota stays with #764. E2 is the remaining provider accounting clarification.
S5 FO1 deadline Concrete scheduling mechanism plus honest fault limit. Lines 117–125 track oldest uncovered ack/sequence, bound batching, cover post-freeze edits and causal successors, reserve resources and backpressure new acks. Every >1 s sample fails, including disk faults; earlier acks cannot be rescued by later backpressure.
S6 preserved timestamps / lost events Resolved by eventual byte auditing and separate freshness proofs. Lines 127–130 read metadata-identical files, keep classify/search freshness eventual, and prohibit metadata discovery from certifying quota or dedupe. Monotonic deadlines and unique durable identities replace wall-clock decisions.
S7 adversaries / vacuous success Resolved as a test contract, with E1's precise vector to add. Lines 209–230 model actual inode/mmap/name/mount/persistence hazards, activate each slice's real hooks, require successful completions/quiescent progress, and put final performance on perf-test VMs. All-pending is not a passing result.

Attempts to break the revised failure table

B1: old fd survives an ordinary save

  1. Outside writer opens inode A for writing.
  2. Calternal prepares immutable new bytes B, exchanges B into the public name, and keeps A linked in protected holding.
  3. Outside writer writes and fsyncs A after exchange, after the final hash check, or after Calternal's commit.
  4. Writer closes; Calternal crashes and loses state.

A survives because it was never unlinked, and Home recovery owns its holding reference. A hash/inode check, quiet interval or advisory lock cannot authorize deletion. Its late bytes can be audited into a conflict; reclaim still waits for verified exclusion. This defeats the round-1 destructive-cleanup schedule.

The source code still has the hazard: fsio/src/swap.rs:191–209 unlinks the displaced file on device/inode equality, and notes/src/room/save.rs:375–388 calls that placement path. The plan correctly requires replacing it, rather than treating it as an already-safe primitive. Recovery from subsequent outside truncation/overwriting of the same held inode is not promised; the plan explicitly cannot recover versions destroyed before observation. Retention consumes logical quota and may block further saves/admission until offline cleanup. Those are stated limits, not hidden completion guarantees.

B2: state disappears halfway through pair recovery

Prepare complete note N and original card C, including opaque fields/photos; sync payloads and the Home intent; place N; crash before C and erase all state. Recovery can finish C from Home bytes and recognize the earlier N placement even if its progress append never landed. Crash again during that replay: ownership/identities and retained payloads allow the same reconstruction. A valid prepared record survives as work to replay; a torn uncommitted suffix can be repaired, while corruption of committed evidence fails closed. Payload retirement waits for references/receipts and durable HEAD.

For revocation, crashing before durable HEAD keeps the prior auth generation and produces no successful security ack; crashing after HEAD must restore the new generation even without state. Gating/draining readers and checking current auth before returning bytes closes the previously unspecified state/Home ordering.

The remaining counterexample is E1: the first bootstrap marker itself can tear before there is a valid local ownership marker. Repeating an already-marked bootstrap is covered; starting one needs the same crash completeness.

B3: copy/delete races the two-tree recount

Count SSD before a file is copied/deleted across tiers, then HDD afterwards; reverse the direction to try to omit it instead. A single two-root generation plus coverage on both roots and one shared native fence invalidates either mixed observation. Overflow, watch loss, mount/root-set change or provider mutation prevents acceptance. When quiescent, both physical names count while both exist; only durable deletion followed by a fresh count releases the source charge. Retained old-writer copies therefore remain charged.

Two independent #764 counters fenced separately, or a joined-index cursor substituted for its complete walk, would fail this schedule. The revised plan explicitly excludes those shortcuts. The extension must also implement the plan's changed inclusion policy: #764 currently skips TEMP_PREFIX names/subtrees and corresponding events (usage/scan.rs:166–169, usage/watch/events.rs:109–112); those exclusions cannot silently hide the new charged staging/holding payloads. This is already required by lines 85–87, not an additional review request.

The shared fence is an observation boundary, not a lock against arbitrary future external writes. External changes after it dirty the next generation; app admissions stay serialized through publication/invalidation. An unavailable required disk means unknown rather than #764's otherwise valid deleted-Home zero. A slow/continuously invalidated recount returns retryably within the total two-second admission budget, with bounded retries. That sacrifices admission availability rather than correctness. At 1M files, successful cold admission in two seconds is expressly not promised. E2 is arithmetic/provider ownership, separate from this consistency mechanism.

Remaining limits tested on paper

An ancestor moved after a directory fd is pinned can still expose placement elsewhere; it is outside the supported online namespace model and requires coordination. No claim that a post-check fixes it remains. A disk stalling just after an SSD-state ack can still violate FO1; it must be recorded as a failed sample, with degraded status/backpressure. No safety or privacy pass may be obtained by returning pending to every request. Dedupe capacity and adaptive timing attacks remain unsolved here, but dedupe is disabled, not silently accepted.

Open latency and simplicity

The successful 15 ms open is not feasible by construction from this document. It is clearly a measured go/no-go gate. Paged ID lookup and bounded tails remove known unbounded work, but expanded CRDT state and cold HDD metadata probes can still exceed the allocation. Unknown-ID pending responses do not count as successful opens. The fallback is to withhold cutover/retain the existing layout; no lowered gate or partial-success mode is authorized. Final measurements belong on the specified perf-test VMs, not this shared 10by32 host.

The design is simpler in authority and accounting than round 1. One Home log replaces competing state/Home authority; #764 remains the quota owner instead of a new logical ledger; deferred dedupe removes a premature cross-user capacity mechanism. Paged indexes are justified by the stated 50k-sibling/10M-entry envelope. A separate generic transaction framework, quota journal or online exclusion service would add unnecessary machinery; none is needed for these edits.

The expensive part is deliberately conservative retention of every displaced public inode until verified exclusion. It can grow even with ordinary repeated saves, not just exceptional conflicts. The plan states the resulting quota/space/availability cost and requires backpressure and offline reclamation. That is a material operating limit to measure, not evidence that the byte-safety mechanism is unnecessary. Watch/RSS/readiness limits also mean fixture qualification must not be advertised as unrestricted support for all maxima simultaneously.

Evidence and scope

This was a reading/writing design review and paper failure analysis. No product code, product tests, syscall probes, builds, performance runs, deploys, merges or pushes were performed. No claims from #764's independent runs are presented as tests run in this review. The review is saved outside the repository. The two required edits are the complete edit list; implementation gates already specified by the plan remain implementation work, not demands for another design expansion.

# Independent design review, round 2: approve with required edits PR [#761](https://git.kayg.org/kayg/calternal/pulls/761), `docs/tiering-731-plan`, reviewed at **b1e3ad02c6236bb59c2ee5fc348a8241bc870167**. Base: `d83e54dbe571b625504001a8b6add90d44ca6161`. Review worktree: `/home/agent/wt/review-761b`, on `10by32-01-std`. **Verdict: approve with required edits. Exactly two small plan edits are required below.** The five round-1 blockers no longer call for a broad redesign. The revised plan supplies mechanisms for the central safety problems and explicitly declines claims it cannot yet establish. The remaining edits close a bootstrap interruption edge and make one quota accounting boundary unambiguous. This approves the design subject to those edits; it does not approve implementation, cutover, performance, or dedupe enablement. I read the [round-1 review](https://git.kayg.org/kayg/calternal/pulls/761#issuecomment-69394), the [itemized response](https://git.kayg.org/kayg/calternal/pulls/761#issuecomment-69728), issue #731's decisions/discussion including the 29 October quota findings, ADRs 0062–0064, and the save/placement code. I also read PR #764's review/fix discussion through its final ship verdict and inspected its counter, scanner, event fence and Auth admission at **d0e6d6f60f688a8e1f71abbc54f28a558d471ab1**. Its three review rounds concern that quota implementation; they do not establish a two-root extension that has yet to be written. User instructions supplied the later warning/dedupe/perf-host decisions. References to plan lines below use the reviewed head. ## Required edits — complete list ### E1. Make the first bootstrap marker creation restartable (lines 54–57, 187, 211–214) The durable marker resolves an interrupted rebuild **after the marker is durable**. Its own interrupted creation is still unspecified. Concrete schedule: valid Home, empty state; bootstrap creates its marker, writes only a prefix, then loses power before marker sync. The next startup sees nonempty state without a valid marker. “Resume/recreate only marked staging” plus “preserve/refuse unmarked corrupt nonempty state” refuses the installation's own first rebuild indefinitely. A crash after creating the generation directory but before its marker has the same problem if that directory lies inside the guarded state location. **Small edit:** add an explicit ownership rule covering the interval before the marker is durable. One sufficient mechanism is a synced bootstrap PREPARE in the existing Home recovery log, recording the unique staging generation and target state location before any state creation. Replay can then recognize and rebuild that exact owned generation even if its local marker is absent/torn; it still preserves/refuses unrelated nonempty state. Add one named vector: crash at directory creation and every short-write/sync boundary of the first marker, then repeatedly recover with state lost again. No second journal or new recovery owner is needed. ### E2. Specify the current-card/history quota split (lines 85–89) Charging every Home regular-file name at full logical size is clear and follows the quota decision. “Providers add only their existing non-Home charges” does not resolve the overlap with current contact cards. The current provider (`crates/contacts-actions/src/write.rs:135–141`, also present at #764's pinned head) returns `history::retained_bytes`. That includes the current card version, not just additional historical versions (`crates/notes/src/history/companions.rs:205–244,765–780`). ADR 0064's Import section explicitly says the mandatory Home companion is the same current logical card and is not a second quota charge. Concrete schedule: one newly imported, unchunked card of V bytes has one retained current version and a Home companion of V bytes. Ignoring other charges, a literal reuse of the provider adds V to the Home walk's V: 2V for the current card. The shared fence can certify that number perfectly; it cannot fix the accounting overlap. Large inline photos make the difference material. **Small edit:** state that the contacts owner must adapt its provider to report the additional retained-history charge beyond the current card already charged by the Home walk, preserving ADR 0064's unchanged-photo sharing within retained history. Tie that split to the same stable provider/pair generation already required here. Add the one-current-version example: its card contribution is V, plus separately counted note/recovery/holding files, not V+V. This does not exempt any Home file, discount deduped files, or require a second quota ledger. ## Disposition of every round-1 finding “Resolved” below means resolved as a design contract, with implementation gates still outstanding. | Finding | Disposition | Evidence / remaining boundary | |---|---|---| | **B1 ordinary saves / old descriptors** | **Resolved by retention; reclamation limit honest.** | Lines 25–41 cover every replacement, including saves, companions, undo/restore and cleanup. Displaced public inodes stay linked; only verified exclusion permits reclaim. Indefinite charges/backpressure and deadline misses are explicit. Slice 3 removes destructive `after_swap` cleanup before moves enable. | | **B2 Home-only recovery / bootstrap** | **Core resolved; E1 remains open.** | Complete immutable pair payloads, one authoritative Home log, durable HEAD commit, auth ordering, snapshot+suffix retention and marked rebuild replace the previous missing protocol. The first marker's own interruption needs E1's ownership proof. | | **B3 two-tree quota** | **Observation/admission mechanism resolved; E2 remains open.** | Lines 75–101 retain #764's whole-Home owner and require one root-set generation, complete native coverage, shared event/mount fence, stable providers, serialized admission and no stale/unknown admission. E2 specifies which provider bytes belong in the sum. | | **B4 15 ms open** | **Honest measured gate, with cutover withheld on failure.** | Lines 103–115 replace ID reload/all-history replay, distinguish pending from successful opens and acknowledge cold HDD/CRDT costs. Lines 171–177 and 234–235 keep the old layout until gates pass. This is no claim of feasibility by construction. | | **B5 dedupe capacity/privacy** | **Honest deferred limit.** | Dedupe ships OFF and T-A2 delivery remains incomplete (9,177,194–207). Later enablement needs sharing-independent capacity/CoW reserves and isolation, including binary outcomes and later operations. Generic errors alone are expressly insufficient. | | **S1 topology/descriptor authority** | **Resolved through an explicit supported namespace limit.** | Lines 43–45,60–69 require current auth checks, root/mount identity validation and locks through writer exit; ancestor moves, root replacement, cross-person aliases and mount changes require offline coordination. This is a cooperative outside-writer model, not a sandbox against a process able to move ancestors arbitrarily. Post-checks cannot prevent that exposure. | | **S2 raw names, links, case** | **Resolved, with explicit unsupported operations.** | Raw names remain identities/countable; no-follow symlinks and hard-link mutation/dedupe restrictions are explicit. Case-only intermediate renames and each parent sync are journaled. Newly introduced folded collisions remain eventually observed (27–29,66–71,185). | | **S3 capacity/aggregate scale** | **Resolved as a qualification envelope, with limits honest.** | Lines 134–151 add byte distributions, 10M total/1M active notes, aggregate HDD cap, large cards/history, depth/empty directories and resource/headroom budgets. Full simultaneous quota readiness for 10M notes is expressly unclaimed. | | **S4 operation costs/old paths** | **Resolved.** | Lines 153–177 charge serialization, CRDT, verification reads, links and B-tree costs; cover admission, recovery, backup and rollback; and name the old helpers/indexes to remove. Quota stays with #764. E2 is the remaining provider accounting clarification. | | **S5 FO1 deadline** | **Concrete scheduling mechanism plus honest fault limit.** | Lines 117–125 track oldest uncovered ack/sequence, bound batching, cover post-freeze edits and causal successors, reserve resources and backpressure new acks. Every >1 s sample fails, including disk faults; earlier acks cannot be rescued by later backpressure. | | **S6 preserved timestamps / lost events** | **Resolved by eventual byte auditing and separate freshness proofs.** | Lines 127–130 read metadata-identical files, keep classify/search freshness eventual, and prohibit metadata discovery from certifying quota or dedupe. Monotonic deadlines and unique durable identities replace wall-clock decisions. | | **S7 adversaries / vacuous success** | **Resolved as a test contract, with E1's precise vector to add.** | Lines 209–230 model actual inode/mmap/name/mount/persistence hazards, activate each slice's real hooks, require successful completions/quiescent progress, and put final performance on perf-test VMs. All-pending is not a passing result. | ## Attempts to break the revised failure table ### B1: old fd survives an ordinary save 1. Outside writer opens inode A for writing. 2. Calternal prepares immutable new bytes B, exchanges B into the public name, and keeps A linked in protected holding. 3. Outside writer writes and fsyncs A after exchange, after the final hash check, or after Calternal's commit. 4. Writer closes; Calternal crashes and loses state. A survives because it was never unlinked, and Home recovery owns its holding reference. A hash/inode check, quiet interval or advisory lock cannot authorize deletion. Its late bytes can be audited into a conflict; reclaim still waits for verified exclusion. This defeats the round-1 destructive-cleanup schedule. The source code still has the hazard: `fsio/src/swap.rs:191–209` unlinks the displaced file on device/inode equality, and `notes/src/room/save.rs:375–388` calls that placement path. The plan correctly requires replacing it, rather than treating it as an already-safe primitive. Recovery from subsequent outside truncation/overwriting of the same held inode is not promised; the plan explicitly cannot recover versions destroyed before observation. Retention consumes logical quota and may block further saves/admission until offline cleanup. Those are stated limits, not hidden completion guarantees. ### B2: state disappears halfway through pair recovery Prepare complete note N and original card C, including opaque fields/photos; sync payloads and the Home intent; place N; crash before C and erase all state. Recovery can finish C from Home bytes and recognize the earlier N placement even if its progress append never landed. Crash again during that replay: ownership/identities and retained payloads allow the same reconstruction. A valid prepared record survives as work to replay; a torn uncommitted suffix can be repaired, while corruption of committed evidence fails closed. Payload retirement waits for references/receipts and durable HEAD. For revocation, crashing before durable HEAD keeps the prior auth generation and produces no successful security ack; crashing after HEAD must restore the new generation even without state. Gating/draining readers and checking current auth before returning bytes closes the previously unspecified state/Home ordering. The remaining counterexample is E1: the *first bootstrap marker itself* can tear before there is a valid local ownership marker. Repeating an already-marked bootstrap is covered; starting one needs the same crash completeness. ### B3: copy/delete races the two-tree recount Count SSD before a file is copied/deleted across tiers, then HDD afterwards; reverse the direction to try to omit it instead. A single two-root generation plus coverage on both roots and one shared native fence invalidates either mixed observation. Overflow, watch loss, mount/root-set change or provider mutation prevents acceptance. When quiescent, both physical names count while both exist; only durable deletion followed by a fresh count releases the source charge. Retained old-writer copies therefore remain charged. Two independent #764 counters fenced separately, or a joined-index cursor substituted for its complete walk, would fail this schedule. The revised plan explicitly excludes those shortcuts. The extension must also implement the plan's changed inclusion policy: #764 currently skips `TEMP_PREFIX` names/subtrees and corresponding events (`usage/scan.rs:166–169`, `usage/watch/events.rs:109–112`); those exclusions cannot silently hide the new charged staging/holding payloads. This is already required by lines 85–87, not an additional review request. The shared fence is an observation boundary, not a lock against arbitrary future external writes. External changes after it dirty the next generation; app admissions stay serialized through publication/invalidation. An unavailable required disk means unknown rather than #764's otherwise valid deleted-Home zero. A slow/continuously invalidated recount returns retryably within the total two-second admission budget, with bounded retries. That sacrifices admission availability rather than correctness. At 1M files, successful cold admission in two seconds is expressly not promised. E2 is arithmetic/provider ownership, separate from this consistency mechanism. ### Remaining limits tested on paper An ancestor moved after a directory fd is pinned can still expose placement elsewhere; it is outside the supported online namespace model and requires coordination. No claim that a post-check fixes it remains. A disk stalling just after an SSD-state ack can still violate FO1; it must be recorded as a failed sample, with degraded status/backpressure. No safety or privacy pass may be obtained by returning pending to every request. Dedupe capacity and adaptive timing attacks remain unsolved here, but dedupe is disabled, not silently accepted. ## Open latency and simplicity **The successful 15 ms open is not feasible by construction from this document. It is clearly a measured go/no-go gate.** Paged ID lookup and bounded tails remove known unbounded work, but expanded CRDT state and cold HDD metadata probes can still exceed the allocation. Unknown-ID pending responses do not count as successful opens. The fallback is to withhold cutover/retain the existing layout; no lowered gate or partial-success mode is authorized. Final measurements belong on the specified perf-test VMs, not this shared 10by32 host. **The design is simpler in authority and accounting than round 1.** One Home log replaces competing state/Home authority; #764 remains the quota owner instead of a new logical ledger; deferred dedupe removes a premature cross-user capacity mechanism. Paged indexes are justified by the stated 50k-sibling/10M-entry envelope. A separate generic transaction framework, quota journal or online exclusion service would add unnecessary machinery; none is needed for these edits. The expensive part is deliberately conservative retention of every displaced public inode until verified exclusion. It can grow even with ordinary repeated saves, not just exceptional conflicts. The plan states the resulting quota/space/availability cost and requires backpressure and offline reclamation. That is a material operating limit to measure, not evidence that the byte-safety mechanism is unnecessary. Watch/RSS/readiness limits also mean fixture qualification must not be advertised as unrestricted support for all maxima simultaneously. ## Evidence and scope This was a reading/writing design review and paper failure analysis. No product code, product tests, syscall probes, builds, performance runs, deploys, merges or pushes were performed. No claims from #764's independent runs are presented as tests run in this review. The review is saved outside the repository. The two required edits are the complete edit list; implementation gates already specified by the plan remain implementation work, not demands for another design expansion.
Author
Owner

Commit: 8b534793335ae98757a9c1f6766977c447751caf.

Applied the two required edits from round-2 review 69748, with matching failure-table vectors. Only docs/plans/0731-tiering.md changes.

Restartable bootstrap — changed section:

Restartable bootstrap: before creating state, append/fsync a checksummed PREPARE in the
existing Home SSD recovery log (fsync its parent if newly created): Home UUID/HEAD/schema,
unique generation, pinned state parent/root identities, reserved private staging path and verified
absent/empty baseline. Retain unfinished intent across checkpoints. Under the state lock,
exclusively create the generation; fsync it/changed parents on state SSD. There create-exclusive
marker.tmp, write the PREPARE-bound checksummed marker, fsync file, rename to marker,
fsync generation directory. Home/state syncs are independent if on different filesystems;
no DB creation before durable marker.
Restart before durable PREPARE has no owned state; after directory creation, absent/empty staging;
after temp creation/short write, absent/torn/complete temp; after file fsync, absent/complete temp;
after rename before directory fsync, temp/marker/neither; after directory fsync, valid marker.
Matching durable PREPARE plus pinned identities/private reservation authorizes recreating exactly
that owned staging/marker, then rebuilding/resuming DBs, even after repeated crashes/state loss.
Unrelated entries, mismatched markers or corrupt committed evidence are preserved/refused;
absent marker/torn owned temp alone is not foreign state. Revalidate HEAD/replay suffix, verify
byte-exact inventory/auth, atomically publish complete-generation pointer and fsync state parent
before serving/recording Home completion. Completed bootstrap with lost state starts a new intent.
Home-only recovery loses history and FO1-lag edits; prepared pairs remain byte-complete.

Current card versus retained history — changed section:

Contacts current card versus retained history: #764's whole-Home walk charges the person's
current .vcf at full logical st_size (kayg's rule), regardless of sparse/reflink/dedupe savings.
Contacts must adapt history::retained_bytes, which includes the current version: report only
additional retained-history bytes beyond the current card already charged in Home. Keep ADR 0064
sharing: retained parts count at full logical size once; exclude current-card parts already
covered by Home, charge older text/photos beyond those. Bind this split to the same stable
provider/pair generation through #764's joint recount/admission fence; unstable means unknown.
One newly imported unchunked V-byte card contributes V, not V+V, plus separately counted
note/recovery/holding files. All Home copies still count per name in full; state current-card
caches add no charge. #764 owns the recount; contacts supplies only the history supplement,
no new ledger. Other providers add only non-Home charges with stable generations.

Validation: git diff --check passes; 3,033 words; four A4 pages at 10 pt body / 9 pt tables, 12 mm margins. Failure-table rows cover first-marker creation boundaries with repeated state loss, plus current/history/photo accounting and generation races. Documentation only; no product tests, builds, deployment, merge or performance claims.

Commit: `8b534793335ae98757a9c1f6766977c447751caf`. Applied the two required edits from round-2 review [69748](https://git.kayg.org/kayg/calternal/pulls/761#issuecomment-69748), with matching failure-table vectors. Only `docs/plans/0731-tiering.md` changes. **Restartable bootstrap — changed section:** > **Restartable bootstrap:** before creating state, append/fsync a checksummed PREPARE in the > existing **Home SSD** recovery log (fsync its parent if newly created): Home UUID/HEAD/schema, > unique generation, pinned state parent/root identities, reserved private staging path and verified > absent/empty baseline. Retain unfinished intent across checkpoints. Under the state lock, > exclusively create the generation; fsync it/changed parents on **state SSD**. There create-exclusive > `marker.tmp`, write the PREPARE-bound checksummed marker, fsync file, rename to `marker`, > fsync generation directory. Home/state syncs are independent if on different filesystems; > no DB creation before durable marker. > Restart before durable PREPARE has no owned state; after directory creation, absent/empty staging; > after temp creation/short write, absent/torn/complete temp; after file fsync, absent/complete temp; > after rename before directory fsync, temp/marker/neither; after directory fsync, valid marker. > Matching durable PREPARE plus pinned identities/private reservation authorizes recreating exactly > that owned staging/marker, then rebuilding/resuming DBs, even after repeated crashes/state loss. > Unrelated entries, mismatched markers or corrupt committed evidence are preserved/refused; > absent marker/torn owned temp alone is not foreign state. Revalidate HEAD/replay suffix, verify > byte-exact inventory/auth, atomically publish complete-generation pointer and fsync state parent > before serving/recording Home completion. Completed bootstrap with lost state starts a new intent. > Home-only recovery loses history and FO1-lag edits; prepared pairs remain byte-complete. **Current card versus retained history — changed section:** > **Contacts current card versus retained history:** #764's whole-Home walk charges the person's > current `.vcf` at full logical `st_size` (kayg's rule), regardless of sparse/reflink/dedupe savings. > Contacts must adapt `history::retained_bytes`, which includes the current version: report only > additional retained-history bytes beyond the current card already charged in Home. Keep ADR 0064 > sharing: retained parts count at full logical size once; exclude current-card parts already > covered by Home, charge older text/photos beyond those. Bind this split to the same stable > provider/pair generation through #764's joint recount/admission fence; unstable means unknown. > One newly imported unchunked V-byte card contributes **V, not V+V**, plus separately counted > note/recovery/holding files. All Home copies still count per name in full; state current-card > caches add no charge. #764 owns the recount; contacts supplies only the history supplement, > no new ledger. Other providers add only non-Home charges with stable generations. Validation: `git diff --check` passes; 3,033 words; four A4 pages at 10 pt body / 9 pt tables, 12 mm margins. Failure-table rows cover first-marker creation boundaries with repeated state loss, plus current/history/photo accounting and generation races. Documentation only; no product tests, builds, deployment, merge or performance claims.
Author
Owner

housekeeping #1009: needs-decision.

Older two-root tiering plan predates accepted three-root tiering v3 (#968, dev 789fdaa9 and active #970/#1008). Retain its accounting amendments and failure cases for the v3 plan.

original head: 8b534793335ae98757a9c1f6766977c447751caf. against dev cd7f84432febfc63b788fbb9b9f9950452982d74, git cherry reports 3 unmatched and 0 equivalent patches. the landing audit lists each candidate missing commit and stable patch match in pr-761.md; unmatched patches can include superseded behavior.

left open, source branch unchanged. no rebase or new test claim for this decision lane.

housekeeping #1009: needs-decision. Older two-root tiering plan predates accepted three-root tiering v3 (#968, dev 789fdaa9 and active #970/#1008). Retain its accounting amendments and failure cases for the v3 plan. original head: `8b534793335ae98757a9c1f6766977c447751caf`. against dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, git cherry reports 3 unmatched and 0 equivalent patches. the [landing audit](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5) lists each candidate missing commit and stable patch match in `pr-761.md`; unmatched patches can include superseded behavior. left open, source branch unchanged. no rebase or new test claim for this decision lane.
Author
Owner

Superseded by tiering v3 plan #968 (on dev). The branch and original tips are archived under archive/ (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.

Superseded by tiering v3 plan #968 (on dev). The branch and original tips are archived under `archive/` (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.
kayg closed this pull request 2026-11-07 21:21:17 +00:00
All checks were successful
CI / server (pull_request) Successful in 1m37s
CI / rust (pull_request) Successful in 15m36s
CI / web (pull_request) Successful in 2h40m55s

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!761
No description provided.