kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

docs: revise IMAP Notes plan after slice 0 (#766) #775

Open
kayg wants to merge 8 commits from docs/imap-766-plan-v2 into docs/imap-notes-plan
pull from: docs/imap-766-plan-v2
merge into: kayg:docs/imap-notes-plan
Conversation 9 Commits 8 Files changed 72 +13745
kayg commented 2026-10-31 05:05:40 +00:00
Owner
Copy link

Revises the Apple Notes over IMAP plan (#766) after the 1 November native evidence and independent review 72369. Documentation only; no implementation code.

Base branch

This PR targets spike/imap-766-s0 because the plan and slice 0 evidence exist there. Merging it will not put the plan on dev.

Plan changes

  • Keep one stable Notes mailbox name per account and fixed UIDVALIDITY for the mailbox life. Allocate strictly increasing UIDs only from the server-owned counter; routine state changes preserve the name, UIDVALIDITY and counter.
  • Keep the per-account (UIDVALIDITY, next_uid) authority and restore freshness witness outside every restorable VM/Home/content set. On every content restore, use max(restored_next_uid, authority_next_uid) + 4096; missing, stale or inconsistent authority disables writes until an admin recovers a trustworthy floor.
  • Resolve deletes by exact (mailbox_id, UIDVALIDITY, UID) plus explicit eligible intent. A missing or mismatched tuple is a no-op; never look up a UID number alone.
  • Reset only for disaster recovery. Export pending text with macOS Automation before account removal, re-add the account, reconcile, then verify the exact pending text is visible before writes resume.
  • Use two concurrently active macOS users on the one Mac VM for E2-6. The physical iPhone has a separate short K1 checklist for kayg before native writes. iOS Simulator equivalence remains unproven.

First implementation PR

Small backend model and persistence slice: stable mailbox identity, external UID authority and writer fencing, a minimal exact-tuple UID map and eligible-delete resolver, a short I1 ADR, and targeted reserve/crash/restore/exhaustion tests. Keep opaque fixtures and explicit delete intent; defer inferred cleanup classification, parsing, reconciliation and other later subsystems. This PR does not add an IMAP listener or enable native writes.

Gate status

Scoped native passes: account setup/initial sync/one edit; same-name higher-UID online refresh after relaunch and a session kick; and stale UID 20002 leaving the new UID range and bystanders unchanged. Fresh-root discovery failed. Offline same-name reset uploaded pending A while Notes displayed the base, so visible pending-edit recovery and E2-6 re-pair remain open. K1 and the two-user macOS gate are required before native writes, not before the model PR. E2-5 and G1–G3 remain later integration gates.

Validation: git diff --check passed. No product tests, builds or native experiments were run for this documentation revision. Do not merge until reviewed.

Revises the Apple Notes over IMAP plan (#766) after the 1 November native evidence and independent review 72369. Documentation only; no implementation code. ## Base branch This PR targets `spike/imap-766-s0` because the plan and slice 0 evidence exist there. Merging it will not put the plan on `dev`. ## Plan changes - Keep one stable `Notes` mailbox name per account and fixed UIDVALIDITY for the mailbox life. Allocate strictly increasing UIDs only from the server-owned counter; routine state changes preserve the name, UIDVALIDITY and counter. - Keep the per-account `(UIDVALIDITY, next_uid)` authority and restore freshness witness outside every restorable VM/Home/content set. On every content restore, use `max(restored_next_uid, authority_next_uid) + 4096`; missing, stale or inconsistent authority disables writes until an admin recovers a trustworthy floor. - Resolve deletes by exact `(mailbox_id, UIDVALIDITY, UID)` plus explicit eligible intent. A missing or mismatched tuple is a no-op; never look up a UID number alone. - Reset only for disaster recovery. Export pending text with macOS Automation before account removal, re-add the account, reconcile, then verify the exact pending text is visible before writes resume. - Use two concurrently active macOS users on the one Mac VM for E2-6. The physical iPhone has a separate short K1 checklist for kayg before native writes. iOS Simulator equivalence remains unproven. ## First implementation PR Small backend model and persistence slice: stable mailbox identity, external UID authority and writer fencing, a minimal exact-tuple UID map and eligible-delete resolver, a short I1 ADR, and targeted reserve/crash/restore/exhaustion tests. Keep opaque fixtures and explicit delete intent; defer inferred cleanup classification, parsing, reconciliation and other later subsystems. This PR does not add an IMAP listener or enable native writes. ## Gate status Scoped native passes: account setup/initial sync/one edit; same-name higher-UID online refresh after relaunch and a session kick; and stale UID 20002 leaving the new UID range and bystanders unchanged. Fresh-root discovery failed. Offline same-name reset uploaded pending A while Notes displayed the base, so visible pending-edit recovery and E2-6 re-pair remain open. K1 and the two-user macOS gate are required before native writes, not before the model PR. E2-5 and G1–G3 remain later integration gates. Validation: `git diff --check` passed. No product tests, builds or native experiments were run for this documentation revision. Do not merge until reviewed.
pr775-original-native-evidence.zip
3.4 MiB
docs: revise IMAP Notes plan after slice 0 (#766)
Some checks failed
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
d239d4f1e6 
Apply the 13 plan changes from the slice 0 findings (section 6). Add the
never-reuse-UID invariant (I1) with tests, UID map and mailbox state sizes
and big-O, a five-column failure table, scaling tests, and slice 1 scope and
exit criteria. Add the open slice 0 experiments with scripted imaplib steps.

Documentation only. No product code.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN
kayg commented 2026-10-31 21:40:04 +00:00
Author
Owner
Copy link

Updated and pushed 6f24ecb2215e3a448a86581dd3dcd7e235d904ef on docs/imap-766-plan-v2 (base unchanged: spike/imap-766-s0). Documentation only, based on continuation 71773.

Area Change Evidence / gate
T2 conflict rule New plan §2a: Calternal is canonical; APPEND is a proposal against a retained base. Detect concurrency by base revision, not timing. Define UTF-16/grapheme-safe disjoint edits, identical-edit deduplication, insertion/boundary overlap and guarded union verification. T2 remains failed until reconciliation is implemented and rerun; two ordinary divergent versions are not acceptance.
Overlap / unknown base Preserve canonical text and exact private proposal; create a linked unresolved note with a fresh UUID, titled <visible title> (conflicted copy, <Installation>), safely visible in both Calternal and Notes. Receipt-proven retries reuse its identity; cleanup cannot remove another version. Unsafe display keeps the native path gated. F13 revised; F29/F30 and two_session_merge, two_session_conflict, conflict_copy_replay specified, not executed.
Existing note behavior Checked crates/notes/src/room/edit.rs, room/merge.rs and diff.rs against ADRs 0020/0045. Use the existing version-based patch/room merge for accepted deltas; IMAP preflight diverts uncertain overlap. Existing CRDT typing and structured-owner rules remain intact. Conflict-copy presentation adopts ADR 0045's recommendation for overlapping text, which is still an open question in that ADR.
E2-4b Recorded passed: new epoch UIDs 400/401, stale cleanup 379 harmless, pending edit 402 reconciled to 403, both notes retained. Supports never reusing numeric UIDs across epochs and canonical supersession; active-client reset remains open. I1 unchanged.
T4 Recorded passed for Mac + scripted imaplib B: delete, republish, second delete, then five minutes absent with no further republish. Supports the observed no-loop path only; true two-Apple quiescence and repeated-cycle coverage remain open.
E2-5 Explicit open slice 1 exit gate X11, without blocking pure-model entry. Updated exact procedure in 766-slice0-open-experiments.md: 70 mapped edits, actual base comments/carriers, relaunch, split, clear and retained-token reconnect; driver/raw/canonical CSV and MIME; ≥69 valid comments and all 70 ordinary edits resolved without proposals. Prior unmapped driver/R73–R76 missing comments and ten-edit stripped reconnect do not close it. Missing token requires complete bounded evidence; cleanup UID is only a hint. If token-only fails, revise fallback design before exit.
Slice 1 Add pure reconciliation/conflict-publication planner and corresponding failure vectors to scope/X10. Native publication and two-Apple acceptance stay later integration gates. No sockets, product code, builds or native experiments added/run here.

Validation: git diff --check HEAD^ HEAD passed; git status --short is empty; git ls-remote origin refs/heads/docs/imap-766-plan-v2 matches the full SHA above. Only the plan and its experiment document changed. New test names are planned gates, not passing results. Review remains pending.

Updated and pushed `6f24ecb2215e3a448a86581dd3dcd7e235d904ef` on `docs/imap-766-plan-v2` (base unchanged: `spike/imap-766-s0`). Documentation only, based on [continuation 71773](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-71773). | Area | Change | Evidence / gate | | --- | --- | --- | | T2 conflict rule | New plan §2a: Calternal is canonical; APPEND is a proposal against a retained base. Detect concurrency by base revision, not timing. Define UTF-16/grapheme-safe disjoint edits, identical-edit deduplication, insertion/boundary overlap and guarded union verification. | T2 remains failed until reconciliation is implemented and rerun; two ordinary divergent versions are not acceptance. | | Overlap / unknown base | Preserve canonical text and exact private proposal; create a linked unresolved note with a fresh UUID, titled `<visible title> (conflicted copy, <Installation>)`, safely visible in both Calternal and Notes. Receipt-proven retries reuse its identity; cleanup cannot remove another version. Unsafe display keeps the native path gated. | F13 revised; F29/F30 and `two_session_merge`, `two_session_conflict`, `conflict_copy_replay` specified, not executed. | | Existing note behavior | Checked `crates/notes/src/room/edit.rs`, `room/merge.rs` and `diff.rs` against ADRs 0020/0045. Use the existing version-based patch/room merge for accepted deltas; IMAP preflight diverts uncertain overlap. | Existing CRDT typing and structured-owner rules remain intact. Conflict-copy presentation adopts ADR 0045's recommendation for overlapping text, which is still an open question in that ADR. | | E2-4b | Recorded passed: new epoch UIDs 400/401, stale cleanup 379 harmless, pending edit 402 reconciled to 403, both notes retained. | Supports never reusing numeric UIDs across epochs and canonical supersession; active-client reset remains open. I1 unchanged. | | T4 | Recorded passed for Mac + scripted imaplib B: delete, republish, second delete, then five minutes absent with no further republish. | Supports the observed no-loop path only; true two-Apple quiescence and repeated-cycle coverage remain open. | | E2-5 | Explicit open slice 1 exit gate X11, without blocking pure-model entry. Updated exact procedure in `766-slice0-open-experiments.md`: 70 mapped edits, actual base comments/carriers, relaunch, split, clear and retained-token reconnect; driver/raw/canonical CSV and MIME; ≥69 valid comments and all 70 ordinary edits resolved without proposals. | Prior unmapped driver/R73–R76 missing comments and ten-edit stripped reconnect do not close it. Missing token requires complete bounded evidence; cleanup UID is only a hint. If token-only fails, revise fallback design before exit. | | Slice 1 | Add pure reconciliation/conflict-publication planner and corresponding failure vectors to scope/X10. Native publication and two-Apple acceptance stay later integration gates. | No sockets, product code, builds or native experiments added/run here. | Validation: `git diff --check HEAD^ HEAD` passed; `git status --short` is empty; `git ls-remote origin refs/heads/docs/imap-766-plan-v2` matches the full SHA above. Only the plan and its experiment document changed. New test names are planned gates, not passing results. Review remains pending.
kayg commented 2026-10-31 21:43:22 +00:00
Author
Owner
Copy link

Plan review

Reviewed docs/imap-766-plan-v2 at 6f24ecb2215e3a448a86581dd3dcd7e235d904ef, against spike/imap-766-s0, continuation 71773, ADRs 0020/0045, the existing room Patch path, and the reported contracts of #809/#814. Plan review only; no builds or native experiments run.

Verdict: rework. The canonical/proposal/conflict-copy policy is consistent with ADR 0045 and conservatively preserves uncertain edits. The remaining gaps concern evidence, the transaction boundary and testable guarantees.

Blocking

  1. I1 is stronger than the restore algorithm can guarantee (§4 F3, §5, X1–X3). An unknown backup age plus 2^24 is not an upper bound on UIDs issued since that backup. At the allowed 200 UIDs/s, that jump covers less than 24 hours, before accounting for reset allocations and reservation skips. A restore after a longer interval can reuse a previously issued UID. For known ages, specify rounding and include every allocation source, repeated crash/reservation loss, and resets in the bound. For unknown or unbounded loss, require a surviving monotonic authority or refuse allocation pending explicit account re-pairing; a guessed floor cannot satisfy I1. Add a deterministic restore test whose lost issuance exceeds the proposed jump, plus repeated restore/crash cases. S4 must compare against an independent complete issued-UID oracle, not only the restored high-water.

  2. Specify the shared executor handoff and prerequisites (§2a, §4, §8). The plan currently submits through the existing version-based room Patch action and describes an A–E publication marker, but does not connect either to #809/#814. #809 is a byte transaction boundary; #814 adds leases, authored history and publication for stable existing-note replacements, while explicitly refusing create/delete/rename and unsupported private participants. A linked conflict copy is a creation plus private relation/receipt and mailbox publication, which those slices do not yet support. Define the IMAP planner → guarded executor input, including exact current pre/post images, authored base/delta, source/owner/visibility dependencies, operation identity, conflict identity and publication participants. Compute and verify the expected union before durable mutation; the current room API can already write history before returning, so a post-call comparison cannot safely divert to conflict. Gate IMAP writes on the required creation/private-state/outbox adapters, and specify how a stale executor input is re-planned without reusing an aborted operation or returning success for partial publication. Add integration exits for conflict creation, ordinary-save races, cancellation and crashes between COMMIT/history/index/mailbox publication. Pure F30 tests alone cannot prove this boundary.

  3. The base-evidence gate misses ordinary successive replacements (§2 token placement, F11/F13, X10/X11). Notes does not supply a reliable revision in its UUID or rewritten headers. A retained signed body token proves the exported ancestor, but does not prove that a later full-body save was authored directly from that ancestor: Notes can edit its previous raw save before fetching the new canonical projection. Example: B=one, first P=one A, next P=one AB, both retaining B's token. B→C and B→P overlap under §2a, so normal continued typing takes the conflict path. This is conservative preservation, but not successful ordinary two-way editing. X11's spaced, mostly append-only edits do not establish this case. Require successive replacement/deletion/backspace tests before canonical FETCH, offline coalescing, and a competing client edit between the saves. Define which retained proposal/receipt evidence can establish the intermediate base; when it cannot, explicitly require unchanged C plus the visible conflict copy. Do not promote a cleanup UID, last FETCH or matching UUID into proof. Make the ordinary-edit acceptance gate fail if these normal cases continually create conflicts; token survival alone cannot close it.

Should change

  • Make native acceptance concrete for two Macs plus iPhone. Existing T2 uses scripted B, and the new gates name true two-Apple acceptance without a three-device schedule. Add both admission orders, one device offline through multiple canonical revisions, overlap and disjoint edits, delayed cleanup, reconnect, and a subsequent genuine edit/delete of both original and conflict copy. Record every connection and assert both apps expose all authorized text, one canonical projection per UUID, and quiescence. Also distinguish a proven transport retry from an identical intentional edit; if Notes supplies no distinguishing evidence, document the unresolved fallback and measure copy growth rather than claiming deduplication.
  • Complete collapse/retention states and bounds. §2a correctly requires reconciliation before collapse; encode that precondition in the planner and S7. F6/F7 also need separate tests for an unreconciled raw proposal and the sole canonical message flagged Deleted, with no follow-up EXPUNGE. Specify the resulting canonical republication/deletion decision without inferring user intent. Test the republish loop breaker for continued access to canonical and pending text. §6's T<10% of N needs an admission/backpressure rule, especially for small mailboxes; include unresolved conflict copies, pinned raw/base history, receipts and publication state in storage accounting and churn tests.

Nits

  • F20 references S12 for base-budget scaling; that is S11.
  • §5 still calls a server that rejects unsupported extension commands an outstanding experiment; continuation 71773 reports the explicit BAD responses. Update the scoped evidence statement.
  • The PR description says 28 failure rows; the reviewed plan has F1–F30.

Base-proof answer: it cannot always be established from what Notes sends. Missing, expired, ambiguous or over-budget evidence must leave canonical text unchanged and preserve a safely visible conflict copy, or keep the native path gated if safe display is impossible. That fallback and the ADR alignment are sound; the blocking items above must be resolved before the plan's stronger guarantees and exit criteria are reviewable as complete.

# Plan review Reviewed `docs/imap-766-plan-v2` at `6f24ecb2215e3a448a86581dd3dcd7e235d904ef`, against `spike/imap-766-s0`, continuation [71773](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-71773), ADRs 0020/0045, the existing room Patch path, and the reported contracts of #809/#814. Plan review only; no builds or native experiments run. **Verdict: rework.** The canonical/proposal/conflict-copy policy is consistent with ADR 0045 and conservatively preserves uncertain edits. The remaining gaps concern evidence, the transaction boundary and testable guarantees. ## Blocking 1. **I1 is stronger than the restore algorithm can guarantee (§4 F3, §5, X1–X3).** An unknown backup age plus `2^24` is not an upper bound on UIDs issued since that backup. At the allowed 200 UIDs/s, that jump covers less than 24 hours, before accounting for reset allocations and reservation skips. A restore after a longer interval can reuse a previously issued UID. For known ages, specify rounding and include every allocation source, repeated crash/reservation loss, and resets in the bound. For unknown or unbounded loss, require a surviving monotonic authority or refuse allocation pending explicit account re-pairing; a guessed floor cannot satisfy I1. Add a deterministic restore test whose lost issuance exceeds the proposed jump, plus repeated restore/crash cases. S4 must compare against an independent complete issued-UID oracle, not only the restored high-water. 2. **Specify the shared executor handoff and prerequisites (§2a, §4, §8).** The plan currently submits through the existing version-based room Patch action and describes an A–E publication marker, but does not connect either to #809/#814. #809 is a byte transaction boundary; #814 adds leases, authored history and publication for stable existing-note replacements, while explicitly refusing create/delete/rename and unsupported private participants. A linked conflict copy is a creation plus private relation/receipt and mailbox publication, which those slices do not yet support. Define the IMAP planner → guarded executor input, including exact current pre/post images, authored base/delta, source/owner/visibility dependencies, operation identity, conflict identity and publication participants. Compute and verify the expected union before durable mutation; the current room API can already write history before returning, so a post-call comparison cannot safely divert to conflict. Gate IMAP writes on the required creation/private-state/outbox adapters, and specify how a stale executor input is re-planned without reusing an aborted operation or returning success for partial publication. Add integration exits for conflict creation, ordinary-save races, cancellation and crashes between COMMIT/history/index/mailbox publication. Pure F30 tests alone cannot prove this boundary. 3. **The base-evidence gate misses ordinary successive replacements (§2 token placement, F11/F13, X10/X11).** Notes does not supply a reliable revision in its UUID or rewritten headers. A retained signed body token proves the exported ancestor, but does not prove that a later full-body save was authored directly from that ancestor: Notes can edit its previous raw save before fetching the new canonical projection. Example: B=`one`, first P=`one A`, next P=`one AB`, both retaining B's token. B→C and B→P overlap under §2a, so normal continued typing takes the conflict path. This is conservative preservation, but not successful ordinary two-way editing. X11's spaced, mostly append-only edits do not establish this case. Require successive replacement/deletion/backspace tests before canonical FETCH, offline coalescing, and a competing client edit between the saves. Define which retained proposal/receipt evidence can establish the intermediate base; when it cannot, explicitly require unchanged C plus the visible conflict copy. Do not promote a cleanup UID, last FETCH or matching UUID into proof. Make the ordinary-edit acceptance gate fail if these normal cases continually create conflicts; token survival alone cannot close it. ## Should change - **Make native acceptance concrete for two Macs plus iPhone.** Existing T2 uses scripted B, and the new gates name true two-Apple acceptance without a three-device schedule. Add both admission orders, one device offline through multiple canonical revisions, overlap and disjoint edits, delayed cleanup, reconnect, and a subsequent genuine edit/delete of both original and conflict copy. Record every connection and assert both apps expose all authorized text, one canonical projection per UUID, and quiescence. Also distinguish a proven transport retry from an identical intentional edit; if Notes supplies no distinguishing evidence, document the unresolved fallback and measure copy growth rather than claiming deduplication. - **Complete collapse/retention states and bounds.** §2a correctly requires reconciliation before collapse; encode that precondition in the planner and S7. F6/F7 also need separate tests for an unreconciled raw proposal and the sole canonical message flagged Deleted, with no follow-up EXPUNGE. Specify the resulting canonical republication/deletion decision without inferring user intent. Test the republish loop breaker for continued access to canonical and pending text. §6's T<10% of N needs an admission/backpressure rule, especially for small mailboxes; include unresolved conflict copies, pinned raw/base history, receipts and publication state in storage accounting and churn tests. ## Nits - F20 references S12 for base-budget scaling; that is S11. - §5 still calls a server that rejects unsupported extension commands an outstanding experiment; continuation 71773 reports the explicit BAD responses. Update the scoped evidence statement. - The PR description says 28 failure rows; the reviewed plan has F1–F30. **Base-proof answer:** it cannot always be established from what Notes sends. Missing, expired, ambiguous or over-budget evidence must leave canonical text unchanged and preserve a safely visible conflict copy, or keep the native path gated if safe display is impossible. That fallback and the ADR alignment are sound; the blocking items above must be resolved before the plan's stronger guarantees and exit criteria are reviewable as complete.
kayg commented 2026-10-31 21:49:58 +00:00
Author
Owner
Copy link

Updated and pushed e3e53ae43cf03cdc52fd18725b7d6c6671833251 on docs/imap-766-plan-v2, addressing plan review 71798. Documentation only; base unchanged. Each step was committed and pushed: af82fd98, 4ecac173, 959427ee, e3e53ae4.

Review item Change Required evidence / remaining gate
Blocking 1: restore / UID non-reuse I1, F3, §5, S4 and X3/X9 replace guessed restore jumps with permanent old-name retirement on any incomplete UID history. Compare epoch + external high-water against fresh-name recovery; choose a fresh name + UIDVALIDITY given E2-4. Complete authority covers every reservation; lost namespace authority refuses writes pending explicit account re-pairing. No alias/redirect of retired names. Independent never-restored oracle includes issuance beyond 2^24, unknown ages, repeated restores/crashes and corrupt/lost authority. Active/offline native migration and stale cleanup remain release gates; E2-4b remains a scoped pass only.
Blocking 2: executor / publication §4a specifies planner input, exact pre/post images, isolated authored delta and verified union before mutation, authority guards, action/conflict identities, private participants and outbox. Every IMAP mutation is one shared executor action: PREPARE journal → files → COMMIT → complete publication → response. Stale plans abort and use a new action ID; COMMIT replay resumes publication only. G1 requires real Home integration for conflict creation, ordinary-save races, cancellation, every publication boundary and lost response. #809/#814 do not yet support the required creation/private/mailbox adapters: all IMAP mutations remain gated; pure model tests cannot enable them.
Blocking 3: rapid successive replacements New §2b/I7 defines per-note, per-connection chains and proven own-version fast-forward, including one → one A → one AB → backspace without canonical FETCH. An ancestor token is not immediate-parent proof. Foreign intervening edits merge against the resolved parent; incomplete/ambiguous evidence preserves C plus a visible conflict. X12/G2 and the extended E2-1b procedure require rapid insertion/replacement/deletion/backspace, offline coalescing, both competing admission orders and every offset including 12 s/1 s. Ordinary typing producing repeated conflicts fails acceptance. If Apple supplies only an ancestor token and no complete parent proof, the native path stays disabled; no unsupported proof claim.
Native acceptance / retry ambiguity G3 gives a two-Mac-plus-iPhone schedule, offline multi-revision edits, overlap/disjoint cases, delayed cleanup, reconnect, subsequent genuine edit/delete of original and copy, and five-minute quiescence. Capture all connections and UI versions; measure copy growth when retry cannot be distinguished from identical intentional editing. Scripted-B results do not close G3.
Collapse and retention Reconcile and durably pin every raw proposal before collapse. Cover unreconciled raw and sole canonical Deleted without EXPUNGE; pause republish loops while retaining canonical/pending access. §6/S7 add slot admission max(8, ceil(N/10)), small-mailbox cases and accounting for conflicts, pinned chains/history/receipts/publication state. Explicit durable-abort/backpressure at limits; never discard required evidence to meet a cap. New tests are specified, not run.
Review nits F20 now references S11. Capability section records continuation 71773's explicit BAD responses for unsupported extension forms, scoped to that experiment. PR description failure count corrected where applicable. Broader capability/reselect acceptance stays open.

Validation: git diff --check 6f24ecb2..HEAD passed; git status --short empty; git ls-remote origin refs/heads/docs/imap-766-plan-v2 matches the full SHA above. Documentation checks confirmed unique F1–F30/S1–S12/X1–X12 rows, resolving local Markdown links and removal of the guessed restore jump. Only docs/plans/0543-imap-notes.md and docs/plans/766-slice0-open-experiments.md changed. No builds, product tests or native experiments ran on the control host. Review remains pending; no merge.

Updated and pushed `e3e53ae43cf03cdc52fd18725b7d6c6671833251` on `docs/imap-766-plan-v2`, addressing plan review [71798](https://git.kayg.org/kayg/calternal/pulls/775#issuecomment-71798). Documentation only; base unchanged. Each step was committed and pushed: `af82fd98`, `4ecac173`, `959427ee`, `e3e53ae4`. | Review item | Change | Required evidence / remaining gate | | --- | --- | --- | | Blocking 1: restore / UID non-reuse | I1, F3, §5, S4 and X3/X9 replace guessed restore jumps with permanent old-name retirement on any incomplete UID history. Compare epoch + external high-water against fresh-name recovery; choose a fresh name + UIDVALIDITY given E2-4. Complete authority covers every reservation; lost namespace authority refuses writes pending explicit account re-pairing. No alias/redirect of retired names. | Independent never-restored oracle includes issuance beyond 2^24, unknown ages, repeated restores/crashes and corrupt/lost authority. Active/offline native migration and stale cleanup remain release gates; E2-4b remains a scoped pass only. | | Blocking 2: executor / publication | §4a specifies planner input, exact pre/post images, isolated authored delta and verified union before mutation, authority guards, action/conflict identities, private participants and outbox. Every IMAP mutation is one shared executor action: PREPARE journal → files → COMMIT → complete publication → response. Stale plans abort and use a new action ID; COMMIT replay resumes publication only. | G1 requires real Home integration for conflict creation, ordinary-save races, cancellation, every publication boundary and lost response. #809/#814 do not yet support the required creation/private/mailbox adapters: all IMAP mutations remain gated; pure model tests cannot enable them. | | Blocking 3: rapid successive replacements | New §2b/I7 defines per-note, per-connection chains and proven own-version fast-forward, including `one` → `one A` → `one AB` → backspace without canonical FETCH. An ancestor token is not immediate-parent proof. Foreign intervening edits merge against the resolved parent; incomplete/ambiguous evidence preserves C plus a visible conflict. | X12/G2 and the extended E2-1b procedure require rapid insertion/replacement/deletion/backspace, offline coalescing, both competing admission orders and every offset including 12 s/1 s. Ordinary typing producing repeated conflicts fails acceptance. If Apple supplies only an ancestor token and no complete parent proof, the native path stays disabled; no unsupported proof claim. | | Native acceptance / retry ambiguity | G3 gives a two-Mac-plus-iPhone schedule, offline multi-revision edits, overlap/disjoint cases, delayed cleanup, reconnect, subsequent genuine edit/delete of original and copy, and five-minute quiescence. | Capture all connections and UI versions; measure copy growth when retry cannot be distinguished from identical intentional editing. Scripted-B results do not close G3. | | Collapse and retention | Reconcile and durably pin every raw proposal before collapse. Cover unreconciled raw and sole canonical Deleted without EXPUNGE; pause republish loops while retaining canonical/pending access. §6/S7 add slot admission `max(8, ceil(N/10))`, small-mailbox cases and accounting for conflicts, pinned chains/history/receipts/publication state. | Explicit durable-abort/backpressure at limits; never discard required evidence to meet a cap. New tests are specified, not run. | | Review nits | F20 now references S11. Capability section records continuation 71773's explicit BAD responses for unsupported extension forms, scoped to that experiment. PR description failure count corrected where applicable. | Broader capability/reselect acceptance stays open. | Validation: `git diff --check 6f24ecb2..HEAD` passed; `git status --short` empty; `git ls-remote origin refs/heads/docs/imap-766-plan-v2` matches the full SHA above. Documentation checks confirmed unique F1–F30/S1–S12/X1–X12 rows, resolving local Markdown links and removal of the guessed restore jump. Only `docs/plans/0543-imap-notes.md` and `docs/plans/766-slice0-open-experiments.md` changed. No builds, product tests or native experiments ran on the control host. Review remains pending; no merge.
kayg commented 2026-10-31 21:53:02 +00:00
Author
Owner
Copy link

Plan review, round 2

Reviewed docs/imap-766-plan-v2 at e3e53ae43cf03cdc52fd18725b7d6c6671833251, including both revised documents, review 71798, response 71824, ADRs 0020/0045 and the #809/#814 boundaries. Plan review only; no builds or native experiments run. git diff --check 6f24ecb2 e3e53ae4 passed.

Verdict: rework. The executor and successive-save changes resolve the corresponding round 1 specification gaps. Fresh-name recovery fixes the allocator argument, but leaves the native recovery experience insufficiently specified and untested.

Round 1 blockers

  1. UID non-reuse: allocation guarantee addressed; recovery contract still blocking below. I1/F3/§5/S4 now retire an exposed name whenever allocation history is incomplete, use independent namespace authority, and refuse writes if that authority is lost. The independent oracle and repeated-loss cases replace the invalid guessed floor. This is a sound server-side direction, conditional on proving authority continuity.
  2. Shared executor: addressed at plan level. §4a supplies guarded exact images, authored deltas verified before mutation, conflict identity/private participants, publication ordering, terminal abort and COMMIT-only publication replay. Unsupported creation/private/mailbox adapters explicitly prevent IMAP mutation enablement. G1 requires real Home integration and boundary failures; #809/#814 alone are correctly insufficient. These are required future tests, not existing passes.
  3. Rapid replacements: addressed at plan level, feasibility remains open. §2b/X12/G2 distinguish ancestor evidence from immediate-parent proof, retain raw chains, cover continued typing/backspace/interleaving/offline cases, and fail ordinary-edit acceptance on repeated conflicts. Synthetic parent evidence cannot close G2. The revision correctly leaves native writes disabled if Apple cannot supply sufficient evidence; it does not establish that a working parent mechanism already exists.

Blocking: define and verify fresh-mailbox recovery in Notes.app

B1. Specify the user-visible transition and pending-edit handoff (§5/F3). Refusing the old name and rebuilding a fresh mailbox does not define what Notes displays. State the expected folder names and counts, whether the old cached folder remains beside the new one, how Notes discovers/subscribes to the new mailbox, what is re-downloaded, and when the old folder disappears. Define safe old-folder removal separately from canonical note deletion; old names must remain permanently retired. No alias or server rename may reconnect stale UID commands to the rebuilt mailbox.

The removal/re-addition fallback also needs an ordered recovery procedure. Unsynced edits may exist only on an offline Apple device: private server retention cannot protect text never received. Require recovery/export of those edits before any account removal that could discard the local cache, and explain what happens when that recovery cannot be established. Reconcile recovered edits under fresh evidence before enabling writes. Specify the interruption/retry behavior and the plain user instructions. A fresh folder plus re-download can be an acceptable explicit recovery cost; unexplained duplicate folders, inaccessible pending text or unproven cache removal are not an accepted contract.

B2. Add a concrete native migration gate and experiment, tied to write enablement. §5 currently names a release gate, but the experiment document contains no fresh-name migration procedure; E2-4b changes UIDVALIDITY/numbers under the same mailbox name. Neither it nor G3 demonstrates this recovery path. Add a named gate covering active and offline clients on two Macs plus iPhone, local-only pending edits, old folder cached/selected, discovery of the fresh folder, and the documented account re-add fallback. Include repeated recovery and interruption during the transition.

Capture every connection and before/during/after Notes UI: LIST/LSUB/subscription behavior, SELECT name/UIDVALIDITY, fetched bytes/re-download, folder/note counts and exact pending text. Exercise delayed old STORE/EXPUNGE/CLOSE, including cleanup carried onto a newly selected mailbox, old-folder removal and a subsequent genuine edit/delete. Assert bystanders and pending text survive, one usable canonical projection per UUID in the final view, no persistent unexplained duplicate folders, and quiescence. If intent remains indistinguishable, keep destructive operations disabled. Native evidence must close this gate before recovery enables writes; model namespace tests cannot substitute.

Recovery answer today: the plan specifies server retirement and rebuilding, not an observed Notes.app migration. Duplicate-folder behavior, re-download and safe old-folder removal remain unknown. No native fresh-mailbox recovery pass is reported. Pure-model work can proceed within the stated disabled-write scope, but the recovery contract and measurable gate need revision before plan approval.

# Plan review, round 2 Reviewed `docs/imap-766-plan-v2` at `e3e53ae43cf03cdc52fd18725b7d6c6671833251`, including both revised documents, review [71798](https://git.kayg.org/kayg/calternal/pulls/775#issuecomment-71798), response [71824](https://git.kayg.org/kayg/calternal/pulls/775#issuecomment-71824), ADRs 0020/0045 and the #809/#814 boundaries. Plan review only; no builds or native experiments run. `git diff --check 6f24ecb2 e3e53ae4` passed. **Verdict: rework.** The executor and successive-save changes resolve the corresponding round 1 specification gaps. Fresh-name recovery fixes the allocator argument, but leaves the native recovery experience insufficiently specified and untested. ## Round 1 blockers 1. **UID non-reuse: allocation guarantee addressed; recovery contract still blocking below.** I1/F3/§5/S4 now retire an exposed name whenever allocation history is incomplete, use independent namespace authority, and refuse writes if that authority is lost. The independent oracle and repeated-loss cases replace the invalid guessed floor. This is a sound server-side direction, conditional on proving authority continuity. 2. **Shared executor: addressed at plan level.** §4a supplies guarded exact images, authored deltas verified before mutation, conflict identity/private participants, publication ordering, terminal abort and COMMIT-only publication replay. Unsupported creation/private/mailbox adapters explicitly prevent IMAP mutation enablement. G1 requires real Home integration and boundary failures; #809/#814 alone are correctly insufficient. These are required future tests, not existing passes. 3. **Rapid replacements: addressed at plan level, feasibility remains open.** §2b/X12/G2 distinguish ancestor evidence from immediate-parent proof, retain raw chains, cover continued typing/backspace/interleaving/offline cases, and fail ordinary-edit acceptance on repeated conflicts. Synthetic parent evidence cannot close G2. The revision correctly leaves native writes disabled if Apple cannot supply sufficient evidence; it does not establish that a working parent mechanism already exists. ## Blocking: define and verify fresh-mailbox recovery in Notes.app **B1. Specify the user-visible transition and pending-edit handoff (§5/F3).** Refusing the old name and rebuilding a fresh mailbox does not define what Notes displays. State the expected folder names and counts, whether the old cached folder remains beside the new one, how Notes discovers/subscribes to the new mailbox, what is re-downloaded, and when the old folder disappears. Define safe old-folder removal separately from canonical note deletion; old names must remain permanently retired. No alias or server rename may reconnect stale UID commands to the rebuilt mailbox. The removal/re-addition fallback also needs an ordered recovery procedure. Unsynced edits may exist only on an offline Apple device: private server retention cannot protect text never received. Require recovery/export of those edits before any account removal that could discard the local cache, and explain what happens when that recovery cannot be established. Reconcile recovered edits under fresh evidence before enabling writes. Specify the interruption/retry behavior and the plain user instructions. A fresh folder plus re-download can be an acceptable explicit recovery cost; unexplained duplicate folders, inaccessible pending text or unproven cache removal are not an accepted contract. **B2. Add a concrete native migration gate and experiment, tied to write enablement.** §5 currently names a release gate, but the experiment document contains no fresh-name migration procedure; E2-4b changes UIDVALIDITY/numbers under the same mailbox name. Neither it nor G3 demonstrates this recovery path. Add a named gate covering active and offline clients on two Macs plus iPhone, local-only pending edits, old folder cached/selected, discovery of the fresh folder, and the documented account re-add fallback. Include repeated recovery and interruption during the transition. Capture every connection and before/during/after Notes UI: LIST/LSUB/subscription behavior, SELECT name/UIDVALIDITY, fetched bytes/re-download, folder/note counts and exact pending text. Exercise delayed old STORE/EXPUNGE/CLOSE, including cleanup carried onto a newly selected mailbox, old-folder removal and a subsequent genuine edit/delete. Assert bystanders and pending text survive, one usable canonical projection per UUID in the final view, no persistent unexplained duplicate folders, and quiescence. If intent remains indistinguishable, keep destructive operations disabled. Native evidence must close this gate before recovery enables writes; model namespace tests cannot substitute. **Recovery answer today:** the plan specifies server retirement and rebuilding, not an observed Notes.app migration. Duplicate-folder behavior, re-download and safe old-folder removal remain unknown. No native fresh-mailbox recovery pass is reported. Pure-model work can proceed within the stated disabled-write scope, but the recovery contract and measurable gate need revision before plan approval.
kayg commented 2026-10-31 22:25:36 +00:00
Author
Owner
Copy link

Fresh-mailbox recovery evidence is now in new #766 comment and the findings note. Scripted IMAP showed duplicate folders/UUIDs when the fresh name coexisted with the old one, and a stale UID could remove a different message after selecting the fresh mailbox. Notes.app setup stopped at Keychain trust before account creation; offline-edit handoff and safe cache removal remain unverified, so the native gate stays open.

Fresh-mailbox recovery evidence is now in [new #766 comment](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-71907) and the [findings note](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0-cases/docs/plans/766-slice0-evidence/fresh-mailbox/findings.md). Scripted IMAP showed duplicate folders/UUIDs when the fresh name coexisted with the old one, and a stale UID could remove a different message after selecting the fresh mailbox. Notes.app setup stopped at Keychain trust before account creation; offline-edit handoff and safe cache removal remain unverified, so the native gate stays open.
kayg commented 2026-11-01 07:13:49 +00:00
Author
Owner
Copy link

Plan update, round 3

Updated docs/plans/0543-imap-notes.md and the native gate procedure in docs/plans/766-slice0-open-experiments.md on docs/imap-766-plan-v2 at 610aa9d7.

The new native report and #766 comment 72340 show that Notes does not discover a fresh root name and continues selecting the cached Notes name. A same-name higher-UID refresh worked after relaunch and an explicit session kick. During an offline same-name reset, Notes uploaded pending A but displayed the old base while the server retained duplicate A; its old cleanup UID was a no-op because the new range was higher. Offline fresh-root migration left the local edit cached and did not upload it.

The plan now requires one stable Notes root, fixed UIDVALIDITY during the mailbox life, and server-owned strictly increasing UID allocation across restarts, re-index, restore, migration, tiering, failover and compaction. It lists the continuity guard for each operation. Cleanup must resolve the exact (mailbox_id, selected UIDVALIDITY, UID) tuple; an absent or stale tuple cannot be looked up by number alone. Duplicate A stays retained until reconciliation makes the pending text visible.

Disaster recovery is the only planned reset path: export pending text with the working macOS Automation before account removal, re-add against the same Notes root, reconcile, then verify the exact pending text is visible. New gate E2-6 requires two Macs and an iPhone, interruption coverage, stale cleanup, bystander preservation and a subsequent edit/delete.

Scoped passes: initial account setup/sync/edit, online same-name high-UID refresh after a kick, and the measured stale-cleanup no-op. Transparent fresh-root discovery and offline pending-text visibility failed. Re-pair recovery and the routine identity-preservation matrix remain open, as do E2-5 and G1–G3. The first implementation slice is large: one model-only PR with 11 deliverables and no listener, TLS or Apple UI.

Documentation only. git diff --check passed; no product tests, builds or native experiments were run for this revision. The native release gates remain open; no merge requested.

### Plan update, round 3 Updated `docs/plans/0543-imap-notes.md` and the native gate procedure in `docs/plans/766-slice0-open-experiments.md` on `docs/imap-766-plan-v2` at `610aa9d7`. The new [native report](https://git.kayg.org/kayg/calternal/src/commit/610aa9d7/docs/plans/766-slice0-evidence/native-fresh-mailbox/README.md) and [#766 comment 72340](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-72340) show that Notes does not discover a fresh root name and continues selecting the cached `Notes` name. A same-name higher-UID refresh worked after relaunch and an explicit session kick. During an offline same-name reset, Notes uploaded pending A but displayed the old base while the server retained duplicate A; its old cleanup UID was a no-op because the new range was higher. Offline fresh-root migration left the local edit cached and did not upload it. The plan now requires one stable `Notes` root, fixed UIDVALIDITY during the mailbox life, and server-owned strictly increasing UID allocation across restarts, re-index, restore, migration, tiering, failover and compaction. It lists the continuity guard for each operation. Cleanup must resolve the exact `(mailbox_id, selected UIDVALIDITY, UID)` tuple; an absent or stale tuple cannot be looked up by number alone. Duplicate A stays retained until reconciliation makes the pending text visible. Disaster recovery is the only planned reset path: export pending text with the working macOS Automation before account removal, re-add against the same `Notes` root, reconcile, then verify the exact pending text is visible. New gate E2-6 requires two Macs and an iPhone, interruption coverage, stale cleanup, bystander preservation and a subsequent edit/delete. Scoped passes: initial account setup/sync/edit, online same-name high-UID refresh after a kick, and the measured stale-cleanup no-op. Transparent fresh-root discovery and offline pending-text visibility failed. Re-pair recovery and the routine identity-preservation matrix remain open, as do E2-5 and G1–G3. The first implementation slice is large: one model-only PR with 11 deliverables and no listener, TLS or Apple UI. Documentation only. `git diff --check` passed; no product tests, builds or native experiments were run for this revision. The native release gates remain open; no merge requested.
kayg commented 2026-11-01 07:18:48 +00:00
Author
Owner
Copy link

Independent plan review, round 3

Reviewed 610aa9d70e, both plan documents, the native-fresh-mailbox README, native evidence 72340, and planner reply 72355. Review only; no builds, native experiments, repository edits or pushes.

Verdict: revise. The core rule and interpretation of the evidence are now sound. Narrow the first PR and specify the surviving UID authority and feasible native gate. This is round 3; the smaller design below is the proposed next step rather than another expansion of the plan.

Findings, P0–P3

P0: none identified in the revised contract. Missing authority stops writes; re-pairing never permits UID reuse. That is the right safety boundary.

P1: “non-rewinding authority” still needs a concrete durability and freshness boundary (§5, F3, S4/S13). The older-Home-backup case is explicitly guarded now: restore content separately, retain the current reservation authority, and refuse allocation when continuity is unknown. But “outside any Home-only restore set” does not establish what happens after an older whole-VM/volume restore or promotion of a self-consistent older authority copy. Comparing two restored records cannot reveal reservations lost from both. This is an implementation specification gap, not evidence that the revised algorithm intentionally rewinds.

Specify where the authority survives, which restore operations cannot rewind it, the durable reservation commit point before issuance, and how startup/promotion proves freshness and fences the previous writer. A local lease alone is insufficient after restoring both lease and counter. If the surviving authority is unavailable, require an explicit write-disabled recovery state; never infer completeness from an intact old log. One acceptable first design is one writer with a reservation authority outside the restored VM/content volume, no writable standby promotion, and manual recovery on authority loss. Other choices are possible, but select one before implementing durability claims.

Required deterministic vector: take backup B, reserve and issue more blocks, burn a block through a crash, restore B, and require the next UID to exceed the independent oracle's highest reserved UID. Repeat with the authority itself missing and with an intact older authority copy: both must refuse writes unless a surviving freshness witness establishes the current floor. Include migration rollback and stale-writer promotion. Never use backup age or a guessed jump.

Operation audit of §5:

Operation Guard present in the plan Remaining proof
Restart Recover reservation log before writes; fenced allocator Durable commit/restart ordering
Re-index Rebuild derived indexes from immutable assignments; no allocation Preserve registry/map independently of index
Older backup restore Separate content restore from current authority; fail closed Concrete surviving authority and stale-copy detection, as above
Migration Copy identity/map/log/tombstones; verify atomic cutover; retain maximum on rollback Cutover/rollback crash vectors
Tiering Preserve identity/authority; verify payload copy; no allocation on cold read Transition vectors
Failover Recover current log; fence allocator; reject older standby Freshness and writer-fencing mechanism, as above
Compaction Retain assignments/tombstones/authority; move payload only Compaction interruption vectors

These are meaningful plan-level preconditions, not passing implementation results. The counter is monotonic across an older content-only restore conditional on the surviving authority; all-state rollback is not yet concretely covered.

P2: E2-6's physical-device count blocks a feasible equivalent (§5; experiment E2-6). Require independent native client state and the specified observations, rather than two physical Macs. Proposed gate below. Also scope recovery honestly: the working macOS export does not recover text that exists only on an offline iPhone. Either test a separate iPhone export/re-pair procedure or explicitly keep that recovery case unsupported and preserve its configured account/cache.

P2: slice 1 is still several subsystems in one PR (§8, deliverables 1–11). MIME/attachments, visibility, parent-chain evidence, reconciliation/conflict publication, collapse and broad scale work are not prerequisites for proving stable identity and exact cleanup addressing. X11 also couples a server identity PR to an unrelated 70-edit native token experiment.

P3: update the PR description. It still advertises backup-age UID jumps and an unknown-age 2^24 fallback, which the reviewed files correctly reject. The reviewable summary should describe the final design.

Evidence assessment

The revision follows the native evidence. Notes used cleanup UID 20002 after receiving the new UIDVALIDITY; safety came from the higher numeric range, not the epoch alone. Exact (mailbox_id, selected UIDVALIDITY, UID) resolution and lifetime numeric non-reuse are both necessary. Pending A at 30004 coexisted with base A at 30002, while Notes displayed the base. Keeping raw MIME is therefore not a visibility pass. F2, X4 and E2-6 correctly retain both versions and require visible reconciliation before collapse. Fresh Notes-* discovery failed; the stable Notes root and export-before-account-removal procedure follow that evidence. Re-pairing remains untested, as stated.

Concrete smaller first PR

Keep deliverable 1, the minimal exact-tuple UID map from 2, and only the target-resolution/no-op part of 3. Use opaque fixture payloads and an explicit eligible-delete intent; defer inferred cleanup classification. Add the short I1 ADR and targeted crash/restore/exhaustion tests.

Cut 4–8 and 11 entirely from this PR: collapse, token codec, MIME/attachments, visibility, parent-chain resolution and conflict-publication planning. Cut the full wire parser/replay harness 9; retain a small structured fixture of the observed stale 20002 command. Replace broad deliverable 10 with the identity-relevant parts of S2/S4/S13 and X1–X3. Keep an independent issued/reserved-range oracle. No E2-5 dependency for this PR's exit; E2-5 and G1–G3 still block their later integration/write-enablement scopes.

Exit: one Notes identity, fixed routine UIDVALIDITY, durable reserve-before-use, zero reuse across every listed transition, unavailable/stale authority refusing allocation, exhaustion stopping, and absent/wrong-mailbox/wrong-epoch cleanup changing nothing. Model tests establish the transition rules; actual durable-store crash tests establish persistence. Neither enables native writes.

Feasible native gate

Use two separate macOS users concurrently logged into the existing VM, plus a physical iPhone. Give each macOS user an independent Internet Account/Notes database and distinguishable connections. Apple documents that switched-out sessions continue running and communicating (Fast User Switching); this supports the proposed equivalence, but Notes concurrency must be confirmed in the preflight. This covers independent caches/client sessions, not different macOS versions or physical-machine failures.

Keep user A online; isolate only user B's test connections at the server proxy, not the VM's whole network. Create B's local-only edit, change A concurrently, export B, perform the same-name disaster replacement above the proven reserved high-water, and re-pair B. Run E2-6's interruption points, old-selection replay, old numeric UID under the new selection, bystander checks, exact visible pending text, subsequent edit/delete and five-minute quiet period. Swap roles and record all connections and both users' native bodies/UI, plus the iPhone observations. Preserve all other G3 schedules. If simultaneous Notes activity cannot be demonstrated, this substitute has not passed.

Do not substitute the iOS Simulator. A usable native Notes-with-IMAP simulator configuration was not established by web research. Apple's Simulator guide explicitly describes a limited installed-app environment; it does not promise the device's Notes/account stack. Standard simulator Notes availability cannot be assumed, and I did not inspect the installed runtime. Treat it as unavailable for this acceptance gate unless someone first demonstrates Apple's actual Notes app, Other/IMAP account setup, offline edits and server sync on that exact runtime. A recreated Notes app is not equivalent. Apple documents other-account Notes setup on a physical iPhone.

If no physical iPhone is available, run the two-user macOS gate now and label it macOS-only; leave iOS acceptance open. Hardware availability need not block the smaller server-identity PR.

## Independent plan review, round 3 Reviewed **610aa9d70edbb5f477c68fd70c5c962753512da8**, both plan documents, the native-fresh-mailbox README, [native evidence 72340](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-72340), and [planner reply 72355](https://git.kayg.org/kayg/calternal/pulls/775#issuecomment-72355). Review only; no builds, native experiments, repository edits or pushes. **Verdict: revise.** The core rule and interpretation of the evidence are now sound. Narrow the first PR and specify the surviving UID authority and feasible native gate. This is round 3; the smaller design below is the proposed next step rather than another expansion of the plan. ### Findings, P0–P3 **P0: none identified in the revised contract.** Missing authority stops writes; re-pairing never permits UID reuse. That is the right safety boundary. **P1: “non-rewinding authority” still needs a concrete durability and freshness boundary (§5, F3, S4/S13).** The older-Home-backup case is explicitly guarded now: restore content separately, retain the current reservation authority, and refuse allocation when continuity is unknown. But “outside any Home-only restore set” does not establish what happens after an older whole-VM/volume restore or promotion of a self-consistent older authority copy. Comparing two restored records cannot reveal reservations lost from both. This is an implementation specification gap, not evidence that the revised algorithm intentionally rewinds. Specify where the authority survives, which restore operations cannot rewind it, the durable reservation commit point before issuance, and how startup/promotion proves freshness and fences the previous writer. A local lease alone is insufficient after restoring both lease and counter. If the surviving authority is unavailable, require an explicit write-disabled recovery state; never infer completeness from an intact old log. One acceptable first design is one writer with a reservation authority outside the restored VM/content volume, no writable standby promotion, and manual recovery on authority loss. Other choices are possible, but select one before implementing durability claims. Required deterministic vector: take backup B, reserve and issue more blocks, burn a block through a crash, restore B, and require the next UID to exceed the independent oracle's **highest reserved** UID. Repeat with the authority itself missing and with an intact older authority copy: both must refuse writes unless a surviving freshness witness establishes the current floor. Include migration rollback and stale-writer promotion. Never use backup age or a guessed jump. Operation audit of §5: | Operation | Guard present in the plan | Remaining proof | | --- | --- | --- | | Restart | Recover reservation log before writes; fenced allocator | Durable commit/restart ordering | | Re-index | Rebuild derived indexes from immutable assignments; no allocation | Preserve registry/map independently of index | | Older backup restore | Separate content restore from current authority; fail closed | Concrete surviving authority and stale-copy detection, as above | | Migration | Copy identity/map/log/tombstones; verify atomic cutover; retain maximum on rollback | Cutover/rollback crash vectors | | Tiering | Preserve identity/authority; verify payload copy; no allocation on cold read | Transition vectors | | Failover | Recover current log; fence allocator; reject older standby | Freshness and writer-fencing mechanism, as above | | Compaction | Retain assignments/tombstones/authority; move payload only | Compaction interruption vectors | These are meaningful plan-level preconditions, not passing implementation results. The counter is monotonic across an older **content-only** restore conditional on the surviving authority; all-state rollback is not yet concretely covered. **P2: E2-6's physical-device count blocks a feasible equivalent (§5; experiment E2-6).** Require independent native client state and the specified observations, rather than two physical Macs. Proposed gate below. Also scope recovery honestly: the working macOS export does not recover text that exists only on an offline iPhone. Either test a separate iPhone export/re-pair procedure or explicitly keep that recovery case unsupported and preserve its configured account/cache. **P2: slice 1 is still several subsystems in one PR (§8, deliverables 1–11).** MIME/attachments, visibility, parent-chain evidence, reconciliation/conflict publication, collapse and broad scale work are not prerequisites for proving stable identity and exact cleanup addressing. X11 also couples a server identity PR to an unrelated 70-edit native token experiment. **P3: update the PR description.** It still advertises backup-age UID jumps and an unknown-age `2^24` fallback, which the reviewed files correctly reject. The reviewable summary should describe the final design. ### Evidence assessment The revision follows the native evidence. Notes used cleanup UID **20002 after receiving the new UIDVALIDITY**; safety came from the higher numeric range, not the epoch alone. Exact `(mailbox_id, selected UIDVALIDITY, UID)` resolution and lifetime numeric non-reuse are both necessary. Pending A at **30004** coexisted with base A at **30002**, while Notes displayed the base. Keeping raw MIME is therefore not a visibility pass. F2, X4 and E2-6 correctly retain both versions and require visible reconciliation before collapse. Fresh `Notes-*` discovery failed; the stable `Notes` root and export-before-account-removal procedure follow that evidence. Re-pairing remains untested, as stated. ### Concrete smaller first PR Keep deliverable **1**, the minimal exact-tuple UID map from **2**, and only the target-resolution/no-op part of **3**. Use opaque fixture payloads and an explicit eligible-delete intent; defer inferred cleanup classification. Add the short I1 ADR and targeted crash/restore/exhaustion tests. Cut **4–8 and 11** entirely from this PR: collapse, token codec, MIME/attachments, visibility, parent-chain resolution and conflict-publication planning. Cut the full wire parser/replay harness **9**; retain a small structured fixture of the observed stale 20002 command. Replace broad deliverable **10** with the identity-relevant parts of S2/S4/S13 and X1–X3. Keep an independent issued/reserved-range oracle. No E2-5 dependency for this PR's exit; E2-5 and G1–G3 still block their later integration/write-enablement scopes. Exit: one `Notes` identity, fixed routine UIDVALIDITY, durable reserve-before-use, zero reuse across every listed transition, unavailable/stale authority refusing allocation, exhaustion stopping, and absent/wrong-mailbox/wrong-epoch cleanup changing nothing. Model tests establish the transition rules; actual durable-store crash tests establish persistence. Neither enables native writes. ### Feasible native gate **Use two separate macOS users concurrently logged into the existing VM, plus a physical iPhone.** Give each macOS user an independent Internet Account/Notes database and distinguishable connections. Apple documents that switched-out sessions continue running and communicating ([Fast User Switching](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPMultipleUsers/Concepts/FastUserSwitching.html)); this supports the proposed equivalence, but Notes concurrency must be confirmed in the preflight. This covers independent caches/client sessions, not different macOS versions or physical-machine failures. Keep user A online; isolate only user B's test connections at the server proxy, not the VM's whole network. Create B's local-only edit, change A concurrently, export B, perform the same-name disaster replacement above the proven reserved high-water, and re-pair B. Run E2-6's interruption points, old-selection replay, old numeric UID under the new selection, bystander checks, exact visible pending text, subsequent edit/delete and five-minute quiet period. Swap roles and record all connections and both users' native bodies/UI, plus the iPhone observations. Preserve all other G3 schedules. If simultaneous Notes activity cannot be demonstrated, this substitute has not passed. **Do not substitute the iOS Simulator.** A usable native Notes-with-IMAP simulator configuration was not established by web research. Apple's [Simulator guide](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/GettingStartedwithiOSSimulator/GettingStartedwithiOSSimulator.html) explicitly describes a limited installed-app environment; it does not promise the device's Notes/account stack. Standard simulator Notes availability cannot be assumed, and I did not inspect the installed runtime. Treat it as unavailable for this acceptance gate unless someone first demonstrates Apple's actual Notes app, Other/IMAP account setup, offline edits and server sync on that exact runtime. A recreated Notes app is not equivalent. Apple documents other-account Notes setup on a [physical iPhone](https://support.apple.com/guide/iphone/add-or-remove-accounts-iph7262fd4fe/ios). If no physical iPhone is available, run the two-user macOS gate now and label it **macOS-only**; leave iOS acceptance open. Hardware availability need not block the smaller server-identity PR.
kayg commented 2026-11-01 07:30:25 +00:00
Author
Owner
Copy link

Revision after independent review 72369

Updated both plan documents on docs/imap-766-plan-v2 at 39d0553d5f896c8e5593272dfc0b6133cf25d855; replaced the PR description to match the revised design.

  • Specified a host-owned per-account (UIDVALIDITY, next_uid) authority and separate restore freshness witness outside the restorable VM/Home/content set. Every content restore advances to max(restored_next_uid, authority_next_uid) + 4096; missing, stale or inconsistent authority leaves writes disabled pending admin recovery. Added restore_older_backup and missing/stale authority, migration rollback, stale-writer and exhaustion checks.
  • Narrowed the first PR to stable mailbox identity, the server-owned allocator and writer fence, a minimal exact-tuple UID map/delete resolver, short I1 ADR, and targeted persistence tests. Deletes require explicit eligible intent and exact tuple matches; UID number alone is never sufficient. Added a structured fixture for the observed stale UID 20002. Reconciliation, MIME, visibility, parsing, collapse and broad scale work move to later slices.
  • Changed E2-6/G3 to two concurrently active macOS users with independent Notes state on the one Mac VM. Added K1 as a separate short physical-iPhone checklist for kayg, required before native writes but not before the model PR. Simulator equivalence is explicitly unproven.
  • Kept the scoped native passes explicit; fresh-root discovery failed, while re-pair recovery and visible pending-edit recovery remain open.

git diff --check passed. No code, product tests, builds or native experiments were run for this revision.

### Revision after independent review 72369 Updated both plan documents on `docs/imap-766-plan-v2` at `39d0553d5f896c8e5593272dfc0b6133cf25d855`; replaced the PR description to match the revised design. - Specified a host-owned per-account `(UIDVALIDITY, next_uid)` authority and separate restore freshness witness outside the restorable VM/Home/content set. Every content restore advances to `max(restored_next_uid, authority_next_uid) + 4096`; missing, stale or inconsistent authority leaves writes disabled pending admin recovery. Added `restore_older_backup` and missing/stale authority, migration rollback, stale-writer and exhaustion checks. - Narrowed the first PR to stable mailbox identity, the server-owned allocator and writer fence, a minimal exact-tuple UID map/delete resolver, short I1 ADR, and targeted persistence tests. Deletes require explicit eligible intent and exact tuple matches; UID number alone is never sufficient. Added a structured fixture for the observed stale UID 20002. Reconciliation, MIME, visibility, parsing, collapse and broad scale work move to later slices. - Changed E2-6/G3 to two concurrently active macOS users with independent Notes state on the one Mac VM. Added K1 as a separate short physical-iPhone checklist for kayg, required before native writes but not before the model PR. Simulator equivalence is explicitly unproven. - Kept the scoped native passes explicit; fresh-root discovery failed, while re-pair recovery and visible pending-edit recovery remain open. `git diff --check` passed. No code, product tests, builds or native experiments were run for this revision.
kayg changed target branch from spike/imap-766-s0 to docs/imap-notes-plan 2026-11-07 19:32:56 +00:00
kayg commented 2026-11-07 19:33:41 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+green, left open for review.

rebased IMAP plan follow-up on #766. preserved original native evidence as Forgejo attachment a0d5563c-586f-41e0-b0c0-f125b7abeb4a and removed evidence files from the rebased repository content.

head: aac806602f0b50879f9d4f2d23e6e3f9d83dd372. rebased onto dev cd7f84432febfc63b788fbb9b9f9950452982d74 and pushed with the original exact force-with-lease.

validation on the Mac VM, through Nix and the shared Mac lock/slot: root bun install --frozen-lockfile; in apps/web, bun run check, bun run lint, bun run build, bun run test; PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. exact commands, logs, and dev bundle control.

original-head missing-commit/patch inventory, file pr-775.md.

housekeeping #1009: rebased+green, left open for review. rebased IMAP plan follow-up on #766. preserved original native evidence as Forgejo attachment a0d5563c-586f-41e0-b0c0-f125b7abeb4a and removed evidence files from the rebased repository content. head: `aac806602f0b50879f9d4f2d23e6e3f9d83dd372`. rebased onto dev `cd7f84432febfc63b788fbb9b9f9950452982d74` and pushed with the original exact force-with-lease. validation on the Mac VM, through Nix and the shared Mac lock/slot: root `bun install --frozen-lockfile`; in `apps/web`, `bun run check`, `bun run lint`, `bun run build`, `bun run test`; `PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate`. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. [exact commands, logs, and dev bundle control](https://git.kayg.org/attachments/64244177-fd81-4992-860e-4d5d22ca01cd). [original-head missing-commit/patch inventory](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5), file `pr-775.md`.
Some checks failed
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
This pull request is broken due to missing fork information.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin docs/imap-766-plan-v2:docs/imap-766-plan-v2
git switch docs/imap-766-plan-v2

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch docs/imap-notes-plan
git merge --no-ff docs/imap-766-plan-v2
git switch docs/imap-766-plan-v2
git rebase docs/imap-notes-plan
git switch docs/imap-notes-plan
git merge --ff-only docs/imap-766-plan-v2
git switch docs/imap-766-plan-v2
git rebase docs/imap-notes-plan
git switch docs/imap-notes-plan
git merge --no-ff docs/imap-766-plan-v2
git switch docs/imap-notes-plan
git merge --squash docs/imap-766-plan-v2
git switch docs/imap-notes-plan
git merge --ff-only docs/imap-766-plan-v2
git switch docs/imap-notes-plan
git merge docs/imap-766-plan-v2
git push origin docs/imap-notes-plan
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!775
No description provided.