kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

feat: persist Apple Notes IMAP UID authority (#766, #775) #824

Closed
kayg wants to merge 0 commits from feat/imap-766-uid-authority into dev
pull from: feat/imap-766-uid-authority
merge into: kayg:dev
Conversation 3 Commits - Files changed -
kayg commented 2026-11-01 08:11:49 +00:00
Owner
Copy link

Plan

Size: Small. One backend model crate, one short I1 ADR, and persistence tests.

Big-O: UID allocation keeps O(1) in-memory state and does one durable authority reservation per 4096 UIDs; each SQLite insert is O(log N) plus payload bytes. Exact lookup and delete are O(log N). Restore and re-index are O(N); the in-memory restore snapshot is O(N).

Failure Effect Detection Recovery Test
Crash before authority commit No UID has been exposed; the range can be retried. The reservation did not reach the durable record. Retry allocation. Before-authority checkpoint.
Crash after authority commit, before freshness witness Authority is ahead and writes stop. Generation/floor mismatch on open. Admin supplies an independently verified floor. After-authority checkpoint and older authority copy.
Crash after witness or content commit The committed block stays burned; restart begins after its high-water. Durable authority and witness on reopen. Reserve a new block. After-witness and after-content checkpoints.
Restore an older content backup Restored projection cannot lower the host floor. The next UID starts after max(restored, authority) + 4096. Restore identity and witness checks. Install the restored content, then reserve before issue. restore_older_backup and migration rollback.
New directory entry is not durable A newly created root or account path may be lost after power loss. Create each path component and sync its parent before mailbox files are written. A sync error aborts before mailbox persistence. durable_directory_creation_syncs_each_new_parent_in_order.
Restore gap exceeds UID space Restore returns UidExhausted before writing a pending witness. Check the restored floor and next authority generation before the marker. Existing authority and content remain reopenable; UID allocation stays exhausted. restore_exhaustion_keeps_the_live_and_reopened_writer_state_consistent.
Missing, stale or inconsistent authority/witness Account writes remain disabled. Authority, witness and content identity checks. Admin recovery from an independent floor. Missing/stale witness, older authority, and identity mismatch cases.
Wrong or absent delete tuple, or ineligible intent No message is deleted. Exact composite-key match and explicit intent. No action required. Stale UID 20002 fixture and tuple tests.
Second writer or exhausted UID space Second writer is fenced; allocation stops before wrap. Exclusive host lock and checked 32-bit boundary. Wait for the current writer or require a later rollover design. Writer-fence and exhaustion tests.

Summary

  • Adds a stable Notes root with fixed UIDVALIDITY and a server-owned per-account UID counter.
  • Keeps authority and restore freshness records in the host-owned root, separate from restorable content. Each reservation is committed before issue; restore burns the explicit 4096-value gap.
  • Stores opaque payloads under the exact (mailbox_id, UIDVALIDITY, UID) key. Deletes require eligible intent.
  • Adds the I1 ADR and focused persistence tests, including randomized allocation/restore/restart coverage.

This PR adds no IMAP listener, TLS, or native writes. It references #766 and #775.

Gates on dev-01

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -j4 -- -D warnings
  • cargo test -p calternal-fsio -j4
  • cargo test -p calternal-imap-uid -j4
  • cargo test --workspace -j4

Each command ran in nix develop through ~/bin/slot.

## Plan **Size:** Small. One backend model crate, one short I1 ADR, and persistence tests. **Big-O:** UID allocation keeps O(1) in-memory state and does one durable authority reservation per 4096 UIDs; each SQLite insert is O(log N) plus payload bytes. Exact lookup and delete are O(log N). Restore and re-index are O(N); the in-memory restore snapshot is O(N). | Failure | Effect | Detection | Recovery | Test | | --- | --- | --- | --- | --- | | Crash before authority commit | No UID has been exposed; the range can be retried. | The reservation did not reach the durable record. | Retry allocation. | Before-authority checkpoint. | | Crash after authority commit, before freshness witness | Authority is ahead and writes stop. | Generation/floor mismatch on open. | Admin supplies an independently verified floor. | After-authority checkpoint and older authority copy. | | Crash after witness or content commit | The committed block stays burned; restart begins after its high-water. | Durable authority and witness on reopen. | Reserve a new block. | After-witness and after-content checkpoints. | | Restore an older content backup | Restored projection cannot lower the host floor. The next UID starts after `max(restored, authority) + 4096`. | Restore identity and witness checks. | Install the restored content, then reserve before issue. | `restore_older_backup` and migration rollback. | | New directory entry is not durable | A newly created root or account path may be lost after power loss. | Create each path component and sync its parent before mailbox files are written. | A sync error aborts before mailbox persistence. | `durable_directory_creation_syncs_each_new_parent_in_order`. | | Restore gap exceeds UID space | Restore returns `UidExhausted` before writing a pending witness. | Check the restored floor and next authority generation before the marker. | Existing authority and content remain reopenable; UID allocation stays exhausted. | `restore_exhaustion_keeps_the_live_and_reopened_writer_state_consistent`. | | Missing, stale or inconsistent authority/witness | Account writes remain disabled. | Authority, witness and content identity checks. | Admin recovery from an independent floor. | Missing/stale witness, older authority, and identity mismatch cases. | | Wrong or absent delete tuple, or ineligible intent | No message is deleted. | Exact composite-key match and explicit intent. | No action required. | Stale UID 20002 fixture and tuple tests. | | Second writer or exhausted UID space | Second writer is fenced; allocation stops before wrap. | Exclusive host lock and checked 32-bit boundary. | Wait for the current writer or require a later rollover design. | Writer-fence and exhaustion tests. | ## Summary - Adds a stable `Notes` root with fixed UIDVALIDITY and a server-owned per-account UID counter. - Keeps authority and restore freshness records in the host-owned root, separate from restorable content. Each reservation is committed before issue; restore burns the explicit 4096-value gap. - Stores opaque payloads under the exact `(mailbox_id, UIDVALIDITY, UID)` key. Deletes require eligible intent. - Adds the I1 ADR and focused persistence tests, including randomized allocation/restore/restart coverage. This PR adds no IMAP listener, TLS, or native writes. It references #766 and #775. ## Gates on dev-01 - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -j4 -- -D warnings` - `cargo test -p calternal-fsio -j4` - `cargo test -p calternal-imap-uid -j4` - `cargo test --workspace -j4` Each command ran in `nix develop` through `~/bin/slot`.
test: write authority fixtures atomically
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
b130c95f60
kayg commented 2026-11-01 08:20:20 +00:00
Author
Owner
Copy link

Independent review, round 1

Reviewed b130c95f60, crates/imap-uid and ADR 0069.

Verdict: request changes. One P1 and one P2; no P0 or additional P3 findings.

P1: persist newly created directory entries before acknowledging mailbox writes

crates/imap-uid/src/lib.rs:238–239, 845–846: create / prepare_roots use create_dir_all for roots and account directories without syncing their containing directories. write_json correctly syncs the temporary file and the immediate account directory, and SQLite syncs its database/journal and immediate directory. Neither makes the account directory's entry in its parent durably committed.

Failure scenario: create a new account, successfully append/acknowledge a message, then lose power before the newly created parent entries reach durable storage. The content account directory, authority account directory, or a newly created root can disappear on a filesystem that requires explicit parent-directory sync. Losing content loses acknowledged messages; losing authority stops writes and requires independent recovery. The successful append has therefore not established the promised durable state. This concerns initial creation, even when the authority/content roots already exist.

Evidence: an LD_PRELOAD trace of both fsync and fdatasync during notes_identity_and_uidvalidity_survive_restart_and_reindex recorded syncs of authority temporary files, authority account directory, SQLite files and content account directory, but none of host-authority, restorable-content, or the parent containing those newly created roots. This is a missing durability barrier established by code and syscall inspection, not a claim that a physical power-cut test was run.

Fix: durably publish every newly created directory from an already durable ancestor before any UID/message is acknowledged. Add a gate that checks the directory-publication ordering.

P2: restore exhaustion leaves a pending witness with a writable live store

crates/imap-uid/src/lib.rs:513–526: restore_content commits pending_restore = true before checking whether the 4096-value restore gap fits. On UidExhausted, it returns through ? without setting disabled_reason or updating the in-memory witness/reservation.

Reproduction: recover to u32::MAX - 2, append once, then restore an older valid snapshot. Restore returns UidExhausted; the durable witness says pending restore, but writes_enabled() remains true and another append succeeds from the existing reservation. Reopening the same store disables writes as InterruptedWrite. Thus live and reopened writers disagree about whether the authority is usable, and a rejected restore creates an unnecessary recovery requirement.

Fix: perform exhaustion/generation checks before persisting the pending marker, or disable the live writer on every error after that marker is committed. Gate both the current process and reopen behavior.

Probes and limits

Uncommitted review worktree on dev-01: /home/claude/Developer/Worktrees/review824-r1. No commits or pushes.

Commands ran in that worktree with the Nix PATH configured:

  • nix develop -c bash -c '~/bin/slot cargo test -p calternal-imap-uid -j4 -- --skip review_': 12 original tests passed.
  • nix develop -c bash -c '~/bin/slot cargo test -p calternal-imap-uid -j4 -- --test-threads=1': 19 tests passed, including seven independent probe/helper tests. The two finding probes deliberately assert the observed problematic behavior.
  • Compiled the small fsync/fdatasync tracing library through nix develop and ~/bin/slot cc, then ran the exact identity/restart test through ~/bin/slot with that library.

Coverage:

  • Abrupt child-process exits before authority persistence, after authority persistence, after witness persistence, and after content commit. Before reservation can retry UID 1; authority-only commit disables writes; completed reservation burns through 4096 and resumes at 4097.
  • A separate process cannot acquire the held writer fence. Allocation and restore use that same open-time fence; restore after release advances the floor.
  • Missing, empty, truncated and corrupt authority, plus witness-floor mismatch, all block append, eligible delete, restore and reindex without fallback.
  • Older authority plus older content is rejected while the current freshness witness survives.
  • As a boundary check, rolling back authority, witness and content together reproduces UID reuse. ADR 0069 explicitly excludes both host records from every backup/restore set; this experiment is outside that stated trust boundary, not a separate finding.
  • u32::MAX is issued without wrapping; subsequent allocation stops. Stable Notes identity, the restore gap, exact-tuple deletes and eligible intent pass the focused tests.

The first combined run used parallel test threads and encountered transient WriterFenced failures while subprocess probes were spawning. The complete serial run passed; the original tests also passed separately with normal parallelism. Logs: ~/review824-r1-serial.log, ~/review824-original.log, ~/review824-fsync-full.log.

# Independent review, round 1 Reviewed **b130c95f6090aa8799178a9fd608c9c3d338cc3a**, `crates/imap-uid` and ADR 0069. **Verdict: request changes.** One P1 and one P2; no P0 or additional P3 findings. ### P1: persist newly created directory entries before acknowledging mailbox writes `crates/imap-uid/src/lib.rs:238–239, 845–846`: `create` / `prepare_roots` use `create_dir_all` for roots and account directories without syncing their containing directories. `write_json` correctly syncs the temporary file and the immediate account directory, and SQLite syncs its database/journal and immediate directory. Neither makes the account directory's entry in its parent durably committed. Failure scenario: create a new account, successfully append/acknowledge a message, then lose power before the newly created parent entries reach durable storage. The content account directory, authority account directory, or a newly created root can disappear on a filesystem that requires explicit parent-directory sync. Losing content loses acknowledged messages; losing authority stops writes and requires independent recovery. The successful append has therefore not established the promised durable state. This concerns initial creation, even when the authority/content roots already exist. Evidence: an `LD_PRELOAD` trace of both `fsync` and `fdatasync` during `notes_identity_and_uidvalidity_survive_restart_and_reindex` recorded syncs of authority temporary files, authority account directory, SQLite files and content account directory, but none of `host-authority`, `restorable-content`, or the parent containing those newly created roots. This is a missing durability barrier established by code and syscall inspection, not a claim that a physical power-cut test was run. Fix: durably publish every newly created directory from an already durable ancestor before any UID/message is acknowledged. Add a gate that checks the directory-publication ordering. ### P2: restore exhaustion leaves a pending witness with a writable live store `crates/imap-uid/src/lib.rs:513–526`: `restore_content` commits `pending_restore = true` before checking whether the 4096-value restore gap fits. On `UidExhausted`, it returns through `?` without setting `disabled_reason` or updating the in-memory witness/reservation. Reproduction: recover to `u32::MAX - 2`, append once, then restore an older valid snapshot. Restore returns `UidExhausted`; the durable witness says pending restore, but `writes_enabled()` remains true and another append succeeds from the existing reservation. Reopening the same store disables writes as `InterruptedWrite`. Thus live and reopened writers disagree about whether the authority is usable, and a rejected restore creates an unnecessary recovery requirement. Fix: perform exhaustion/generation checks before persisting the pending marker, or disable the live writer on every error after that marker is committed. Gate both the current process and reopen behavior. ### Probes and limits Uncommitted review worktree on dev-01: `/home/claude/Developer/Worktrees/review824-r1`. No commits or pushes. Commands ran in that worktree with the Nix PATH configured: - `nix develop -c bash -c '~/bin/slot cargo test -p calternal-imap-uid -j4 -- --skip review_'`: **12 original tests passed**. - `nix develop -c bash -c '~/bin/slot cargo test -p calternal-imap-uid -j4 -- --test-threads=1'`: **19 tests passed**, including seven independent probe/helper tests. The two finding probes deliberately assert the observed problematic behavior. - Compiled the small fsync/fdatasync tracing library through `nix develop` and `~/bin/slot cc`, then ran the exact identity/restart test through `~/bin/slot` with that library. Coverage: - Abrupt child-process exits before authority persistence, after authority persistence, after witness persistence, and after content commit. Before reservation can retry UID 1; authority-only commit disables writes; completed reservation burns through 4096 and resumes at 4097. - A separate process cannot acquire the held writer fence. Allocation and restore use that same open-time fence; restore after release advances the floor. - Missing, empty, truncated and corrupt authority, plus witness-floor mismatch, all block append, eligible delete, restore and reindex without fallback. - Older authority plus older content is rejected while the current freshness witness survives. - As a boundary check, rolling back authority, witness **and** content together reproduces UID reuse. ADR 0069 explicitly excludes both host records from every backup/restore set; this experiment is outside that stated trust boundary, not a separate finding. - `u32::MAX` is issued without wrapping; subsequent allocation stops. Stable Notes identity, the restore gap, exact-tuple deletes and eligible intent pass the focused tests. The first combined run used parallel test threads and encountered transient `WriterFenced` failures while subprocess probes were spawning. The complete serial run passed; the original tests also passed separately with normal parallelism. Logs: `~/review824-r1-serial.log`, `~/review824-original.log`, `~/review824-fsync-full.log`.
kayg commented 2026-11-01 08:35:13 +00:00
Author
Owner
Copy link

Round-one review status (finding → fix → test)

  • P1, directory durability: added calternal_fsio::create_dir_all_durable, reusing fsio::sync_dir. Every directory-creation path in imap-uid now creates missing components from the existing ancestor and syncs each parent before continuing. durable_directory_creation_syncs_each_new_parent_in_order records and checks the parent-sync sequence.
  • P2, restore exhaustion: restore now checks the 4096-value floor and generation increment before writing pending_restore. restore_exhaustion_keeps_the_live_and_reopened_writer_state_consistent verifies the failed restore leaves the live and reopened authority state consistent, while allocations still return UidExhausted.

Dev-01 gates passed in the Nix devshell through ~/bin/slot:

  • cargo fmt --all -- --check
  • cargo test -p calternal-fsio -j4 (41 passed)
  • cargo test -p calternal-imap-uid -j4 (13 passed)
  • cargo clippy --workspace --all-targets -j4 -- -D warnings
  • cargo test --workspace -j4 (passed; 0 failures)

Head: f1c1aa961e169ef99e132063a97ccbd9a89a5def.

Round-one review status (finding → fix → test) - **P1, directory durability:** added `calternal_fsio::create_dir_all_durable`, reusing `fsio::sync_dir`. Every directory-creation path in `imap-uid` now creates missing components from the existing ancestor and syncs each parent before continuing. `durable_directory_creation_syncs_each_new_parent_in_order` records and checks the parent-sync sequence. - **P2, restore exhaustion:** restore now checks the 4096-value floor and generation increment before writing `pending_restore`. `restore_exhaustion_keeps_the_live_and_reopened_writer_state_consistent` verifies the failed restore leaves the live and reopened authority state consistent, while allocations still return `UidExhausted`. Dev-01 gates passed in the Nix devshell through `~/bin/slot`: - `cargo fmt --all -- --check` - `cargo test -p calternal-fsio -j4` (41 passed) - `cargo test -p calternal-imap-uid -j4` (13 passed) - `cargo clippy --workspace --all-targets -j4 -- -D warnings` - `cargo test --workspace -j4` (passed; 0 failures) Head: `f1c1aa961e169ef99e132063a97ccbd9a89a5def`.
kayg commented 2026-11-04 13:04:50 +00:00
Author
Owner
Copy link

Closing: head f1c1aa96 is already in dev (reached via 4b9cebe1); nothing left to merge. Branch kept.

Closing: head f1c1aa96 is already in dev (reached via 4b9cebe1); nothing left to merge. Branch kept.
kayg closed this pull request 2026-11-04 13:04:50 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!824
No description provided.