kayg/calternal

Watch

Fork 0

Plan: Apple Notes over IMAP (#543) #766

Open

kayg wants to merge 1 commit from docs/imap-notes-plan into dev

kayg commented

2026-10-29 17:14:57 +00:00

Owner

Adds docs/plans/0543-imap-notes.md for two-way Apple Notes in calternaldav before any product code. The design retains exact source and sent bases, preserves pending Apple edits as versions, prevents replacement expunges from deleting newer work, and excludes hidden content from every IMAP read path.

Covers 100k notes per account, 10k-line notes, attachment-heavy notes, multiple devices and accounts; operation costs and two-second online visibility gates; crash/offline/conflict recovery; per-device app passwords; mapping properties, real-library IMAP conformance, native macOS VM acceptance, and scale tests in the first code PR. Rendering-only Apple features remain read-only where edits cannot round-trip. Actual MIME transfers and Apple save timing are included as explicit latency constraints.

Validation:

Read #543 and all 20 comments, including the macOS 27 feature report and real-iPhone Siri check; ADRs 0056/0063/0064/0066–0068; current primitives, the unmerged CardDAV read bridge, earlier IMAP/DAV source, and CLAUDE guidance.
Checked all eight required sections, local source links, and documentation-only diff.
git diff --cached --check passed.
Temporary A4 rendering: four pages at 10.5-point text.
Product builds/tests not run: one documentation file added. No product code or deployments.

Related to #543. Design proposal for review, not an implementation or measured performance claim.

Adds `docs/plans/0543-imap-notes.md` for two-way Apple Notes in calternaldav before any product code. The design retains exact source and sent bases, preserves pending Apple edits as versions, prevents replacement expunges from deleting newer work, and excludes hidden content from every IMAP read path. Covers 100k notes per account, 10k-line notes, attachment-heavy notes, multiple devices and accounts; operation costs and two-second online visibility gates; crash/offline/conflict recovery; per-device app passwords; mapping properties, real-library IMAP conformance, native macOS VM acceptance, and scale tests in the first code PR. Rendering-only Apple features remain read-only where edits cannot round-trip. Actual MIME transfers and Apple save timing are included as explicit latency constraints. Validation: - Read #543 and all 20 comments, including the macOS 27 feature report and real-iPhone Siri check; ADRs 0056/0063/0064/0066–0068; current primitives, the unmerged CardDAV read bridge, earlier IMAP/DAV source, and CLAUDE guidance. - Checked all eight required sections, local source links, and documentation-only diff. - `git diff --cached --check` passed. - Temporary A4 rendering: four pages at 10.5-point text. - Product builds/tests not run: one documentation file added. No product code or deployments. Related to #543. Design proposal for review, not an implementation or measured performance claim.

kayg added 1 commit

2026-10-29 17:14:57 +00:00

docs: plan Apple Notes over IMAP (#543 )

CI / server (pull_request) Successful in 1m31s

Details

CI / rust (pull_request) Successful in 15m23s

Details

CI / web (pull_request) Has been cancelled

Details

a4230326f5

kayg referenced this pull request

2026-10-29 17:15:26 +00:00

calternaldav: CalDAV, WebDAV, reminders, notes, mail and calendar sync (epic, grill first) #543

kayg commented

2026-10-29 17:25:27 +00:00

Author

Owner

Independent design review — Apple Notes over IMAP

Verdict: revise. Round 1 of at most 3.

Reviewed PR #766, branch docs/imap-notes-plan, head a4230326f54f1131bc8eb172df779b09c3d10e40. All plan line references below refer to that head's docs/plans/0543-imap-notes.md.

The direction matches kayg's decisions: two-way notes, server authority and history, no hidden export, read-only checklist state, base64 attachments, and no blind pending-edit overwrite. Retaining source and sent projections is appropriate. The first-code-PR scale requirement is explicit and should stay. The blockers are the operational rules between these invariants: several distinct Apple operations still map to the same unspecified receipt/guard machinery, and a required visibility primitive does not exist at this head.

Ranked blocking gaps and exact required edits

R1 — P1: make visibility an explicit prerequisite and define its publication boundary

Plan lines 19, 43–49, 66, 84, 89–91. “Use search's authoritative visibility policy” currently names a requirement, not an available implementation. At this head, crates/notes/src/search.rs:151–179 extracts title/tags and cleans every body line without filtering hidden:. The supported hidden value in ADR 0068 is a block-ID list; this plan also requires whole-note hiding but does not specify its accepted representation or how it composes with block hiding. Do not inherit whatever a permissive YAML getter happens to return.

Concrete failure: export a visible note with a hidden paragraph and hidden attachment, then edit only that paragraph. If the source revision drives Message-ID, Date, a signed base header, UID or MODSEQ, an IMAP observer can detect the hidden edit despite unchanged visible content. More directly, a slow BODY[] or partial attachment FETCH can already hold an old readable generation when Hide commits. Invalidating the cache and checking authorization at command start does not settle what can still leave the socket afterward.

Required plan edits:

Name the shared visibility/marker module as a dependency, define valid whole-note/block forms and malformed/duplicate-key behavior, and require the same vectors for Calternal search and IMAP. Distinguish privacy hiding from task notes being excluded from ordinary Notes lists.
Build the entire wire projection and SEARCH index from the authorized view, including Subject, link labels, image alt text, filenames, CIDs, previews, reference definitions and MIME parts. Opaque/source fallbacks must pass that same filter. No attachment referenced only by omitted content is emitted.
Publish a new wire revision only for an authorized-view or relevant mailbox-state change. Hidden-only source edits must not advance externally visible revision headers/counters or emit IDLE hints. Preserve their private source revisions separately. Test hidden notes that have never been exported separately from visible-to-hidden removal, whose disappearance is necessarily observable.
Define a visibility-generation fence shared by Hide/Unhide, FETCH/partial FETCH, SEARCH and queued IDLE delivery. State how an in-flight literal is aborted/connection closed and when bytes count as already delivered; do not promise recall of previously delivered bytes. Expunge the old immutable message and publish a sanitized replacement under a new UID for partial hides; never mutate a UID's message bytes.
Add hide/unhide races with stale Apple APPEND, replay, restore and overlapping FETCH, plus tests that two Homes with identical exported content but different never-exported hidden content produce identical ordinary mailbox counts/search/size/revision behavior. Existing per-user/account keying is a good isolation requirement; also keep hidden and foreign-ID errors indistinguishable from unavailable objects.

R2 — P1: specify Apple's actual move/delete/APPEND state machine, not only abstract MOVE

Plan lines 15–17, 45–49, 75–85. The older code is not the rewrite architecture, but its captured client behavior is relevant evidence. calternal-v0/docs/research/apple-notes-imap.md:150–188 records a Notes folder move as APPEND in the destination, then STORE Deleted/UID EXPUNGE in the source; it explicitly saw no UID MOVE. The new #543 probe verifies folder creation/rename and replacement saves, not an updated complete move/copy/delete matrix. The plan specifies identity-preserving MOVE, identity-changing COPY and same-note replacement receipts, but not the observed cross-mailbox APPEND sequence.

Concrete failure: a device moves note X from A to B by APPEND, disconnects, then deletes X's source UID after a concurrent edit or another move. Depending on how the unspecified receipt is matched, the server can trash X, restore A, duplicate X, or treat a real copy as a move. A durable receipt alone does not define which result is correct.

There is also a protocol/semantic boundary to settle. “Repeat identical input returns its original result” can mean valid action deduplication, but APPEND has no standardized idempotency key. It cannot generally mean returning an old or expunged APPENDUID for every later identical APPEND, especially across a UIDVALIDITY reset. RFC 3501 §6.3.11 and RFC 9051 §6.3.12 append a new message; RFC 4315 §3 describes the UID assigned in the destination mailbox. Likewise, an IMAP COPY copies a message, while a Calternal copy needs a fresh identity graph. Specify any subsequent canonical reconciliation rather than silently rewriting the copied message under its assigned UID.

Required plan edits: add one compact transition table covering same-folder replacement, cross-folder APPEND move, explicit MOVE, COPY, standalone delete, cancelled deletion (-FLAGS Deleted), failed APPEND followed by cleanup, and disconnect at each command boundary. For each transition give:

Durable key: account, Installation, mailbox ID, UIDVALIDITY, source UID/revision and operation/base evidence; exact scope of any pre-APPEND UID set.
Raw-message UIDs versus canonical note identity, flags, canonical folder membership, and retained proposals.
Exact OK/NO response and APPENDUID/COPYUID, resulting EXISTS/EXPUNGE/VANISHED, and what another selected session can FETCH.
Receipt completion/retention rules. An unfinished receipt must neither authorize deleting a future revision nor mask a later genuine delete. When intent cannot be distinguished, retain a proposal rather than guess.
Recovery and a subsequent legitimate operation after recovery, not only retry of the interrupted operation.

Separate message-level APPEND behavior from deduplicating the canonical action. Distinguish an echo/retry from a deliberate edit that returns the note to an earlier visible value. Do not claim exactly-once client intent from content hashes alone. Measure the native move/copy/delete sequences before committing to folder semantics; a one-folder first slice avoids much of this ambiguity.

R3 — P1: prove base provenance early and bound all candidate-base work

Plan lines 19, 35, 47, 61, 78, 95–105. The signed-base fallback is correctly conditional, but its successful path is unverified. The #543 imported/edited pair contains the ordinary UUID/creation-date headers; it did not test an added signed base header or HTML binding attributes. The raw conflict samples also share X-Mail-Created-Date across the base, server replacement and Apple edit. The older research/code explicitly warns that a newer FETCH is not the edit base and that creation dates survive multiple saves. These are reasons to keep the conservative fallback, not evidence that the proposed automatic path works.

Concrete failure: an Installation has fetched hundreds of revisions, including repeated visible paragraphs whose private block identities or metadata differ, then reconnects with an edit made weeks ago and no retained custom header. Comparing all “plausible” bases is not the advertised O(S+A+B) operation. If a deadline expires, a conflict is safe, but ordinary editing after a long history can become permanently conflict-only. Truncating the candidate set and calling the surviving candidates unanimous is unsafe.

Required plan edits:

Put a native compatibility/provenance experiment before broad mapping/protocol implementation. Test signed-header and binding retention through ordinary saves, successive saves without reselect, attachments, restart, reconnect, offline edits and the measured pending-edit race. Record actual retained fields; do not assume custom headers survive.
Define how the complete plausible-base set is constructed. Define “same safe patch” by target identities, semantic effects and protected-source preservation, not just matching visible text or offsets.
Add history depth/candidate count H to the cost model. Bound candidate discovery, loading, parse, matching, merge and re-render verification together. If complete evidence cannot be established within the budget, preserve the incoming proposal; never sample away possible bases.
Require repeated successful automatic ordinary-text edits after long histories and offline reconnect, alongside deliberate conflicts for truly ambiguous cases. This strengthens the existing “not every edit to conflicts” gate rather than replacing it.

R4 — P1: define quota/full-disk outcomes across the durable stages

Plan lines 37, 45, 61–65, 74–85, 89. Quotas and streaming are named, but quota-full/ENOSPC is absent from the failure table. A crash-replay recipe does not by itself handle a permanently unavailable write target. “Keep raw incoming MIME,” “retain both byte sets,” and “no success before C/D” need a stated limit when there is no space to retain the incoming version.

Concrete failure: two concurrent 100 MiB attachment APPENDs each pass a preflight check, then jointly exhaust quota. Or B commits a receipt and raw version, C changes the room/history, and D fails because the mailbox database/WAL is full. An unqualified replay-before-reads barrier can then make the account unavailable indefinitely. Retrying from Apple can also amplify retained raw/conflict data.

Required plan edits:

Add quota reservation under the same authority as admission, including new decoded blobs, exact MIME/layout records, retained proposals/companions and bounded staging. Define release on failure and replay. Use existing attachment type/path guards; do not assume the current photo-specific companion deduper handles MIME (crates/notes/src/history/blobs.rs:12–18,113–128 recognizes vCard PHOTO lines).
Set separate raw-literal, decoded-total, individual attachment, part/header/DOM and staging/concurrency limits. They must admit the declared fixtures, account for base64 expansion, and reject larger transfers explicitly without truncation or altering the prior note.
Specify pre-B rejection, post-B retryable failure, post-C/D failure and recovery behavior, including which reads remain safe while a generation is pending. Distinguish user quota from physical disk/WAL/Home failures; reconcile this with ADR 0062's degraded Home and one-second materialization rules.
Test two competing admissions, full staging, full history, full projection/WAL, failed Home placement and failure during recovery. Native acceptance must confirm a rejected APPEND keeps the Apple edit recoverable and that subsequent cleanup cannot delete the canonical note. If retaining rejected bytes is impossible, say so and return failure; do not promise retention that did not happen.

R5 — P2: define protected semantics and format-owner composition

Plan lines 19–35, 47, 95–105. A source companion makes byte preservation possible, and conservative rejection is valid. It does not establish that all accepted text/list/link operations preserve Calternal meaning. “Unchanged protected structure” needs to include the semantic graph, not only an HTML/Markdown tree.

Concrete failures: an apparently ordinary text edit changes a relationship link into/out of a standalone child; reordering a task/event list changes ownership; deleting a visible parent would also consume a hidden child; editing a contact list value races a CardDAV update while each adapter updates only its own companion. A generic room merge is not automatically a semantic compare-and-swap. At this head crates/notes/src/room/edit.rs:84–118 forks/merges a known-base patch and has a format-owner rebase hook for a matching staged companion. The history store supports companions per owner, but an individual API edit carries one optional companion; cross-format atomicity needs an explicit integration rule.

Required plan edits: define the protected semantic inventory (IDs/retirement, hidden membership, relationship/owner bindings, task/event facts, frontmatter, table-width associations and reference-link definitions). Have the owning format modules validate the candidate against the current generation inside the guarded action; replan or retain a proposal if that generation changes. Cover notes already claimed by contacts/tasks/calendar and simultaneous changes through those protocols. State which such fields are editable through IMAP and how their companions are updated coherently; unsupported structured edits can be read-only in the first slice. Do not allocate public IDs merely to make the mapper easier.

Mapping audit: what “lossless” can actually mean

There is no implementation in this PR to execute. These are design counterexamples and required vectors, checked against the existing native MIME evidence, not claims of new Apple feature tests. Byte-exact preservation of unchanged source is achievable with a trusted matching companion; full editable Apple feature parity is not. Lossless source retention and an editable projection must remain separate acceptance checks.

Source / attachment kind	Required note → HTML → Apple edit → source result
Paragraphs, blank lines, soft/hard breaks	Test bare body text, div/p/br rewrites, empty paragraphs, trailing spaces, tabs and NBSP. Preserve untouched blank lines and CRLF/LF style; a collapsed soft break cannot become an invented paragraph.
H1–H6 and title	Test editing the first line, Subject/body disagreement, duplicate headings and duplicate titles. One rename/link action must own the title change; native heading authoring is not established.
Bold/italic/strike/underline/inline code	Normalize b/strong, i/em, span styles and nested marks. Escape literal Markdown punctuation; preserve backtick runs. Neither HTML spelling nor a text-only comparison proves a mark edit.
Ordered/unordered/nested lists	Test multiple real items, repeated identical items, start numbers, loose items, indent/outdent, reorder/delete and list-to-paragraph conversion. #543's authored bullet fixture was one item with a literal backslash-n, not a nested/multi-item proof. Ambiguity retains the original plus incoming proposal.
Tasks, all checkbox states, journal/events	A text edit must not tick, skip, change clocks/recurrence/owner or create an outcome. Test `[/]`, `[-]`, unknown boxes and hidden children. Disabled HTML controls still need a native save test; observed checkbox clicks are not durable state.
Quotes/callouts	Test nested quote prefixes, lists/code inside quotes and removal of the visible callout label. Preserve kind and children or retain a proposal.
Fenced/indented code and dividers	Test tabs, leading/trailing spaces, CRLF, blank lines, literal HTML, fence delimiters/languages and long repeated code. WebKit whitespace normalization must not silently rewrite code or turn it into task/event syntax.
GFM tables	Test real cell edits, pipes/escapes, empty/repeated cells, nested formatting, tbody insertion and flattening. The #543 edit was after the table. Preserve alignments and the right table's width metadata; ambiguous boundaries remain unresolved.
Links and reference links	Test label-only/target edits, shared definitions, renamed/moved/reused paths, block links, fragments/percent-encoding and Unicode. Keep UUID authority and relative source paths; verify safe HTTPS navigation on the device. `applenotes:` rendering did not prove navigation.
Images	Test img → Apple object → img normalization, same bytes under changed CID/name/order, duplicate/missing CIDs and image removal. Accept only proven attachment changes; do not confuse a preview with the original.
PDF and generic files	Send base64, decode/hash-verify, then test Apple save and return, not only server serialization. Include the 599-byte ASCII PDF, binary PDF, zero-byte, text, ZIP and arbitrary binary data. Incoming 7bit corruption cannot be repaired or made lossless by changing the outgoing encoding later.
Audio/video and drawings	No native round-trip proof in #543. Define downloadable original versus preview and preserve both separately. Test MIME type, filenames, bytes and whether the device exposes the promised download. Keep native drawing/ink semantics read-only.
Footnotes, math, diagrams, raw HTML, plugin/unknown blocks	Display-only/opaque preservation is acceptable. Test an adjacent ordinary edit, attempted edit/removal of the representation, hidden content inside the source, script/remote-resource sanitization and immutable raw-MIME retention. Never silently accept an edit to a read-only representation as a no-op.
Frontmatter, comments and markers	Preserve BOM, unknown keys, comments, aliases, duplicate/invalid YAML and exact untouched bytes. Source markers never reach UI/accessibility/search. Malformed visibility fails closed. Whole-note copies follow identity rules without copying authority/history.
Character sets / MIME transport	Outgoing UTF-8 and base64; incoming us-ascii/UTF-8 plus an explicit policy for other declared charsets. Test quoted-printable soft breaks, base64 wrapping, header folding/encoded Subject, NFC/NFD, emoji/combining characters and invalid sequences. Decode once; preserve opaque bytes rather than use lossy replacement characters.
Very long/repeated notes	10k lines and 1 MiB text with repeated blocks, Unicode and pathological alternating edits must bound projection, matching and verification, not just the final diff call. No per-block whole-document scan or public-ID allocation.

I independently decoded the supplied #543 samples: both recorded PNG paths retain the original 209 bytes and SHA-256; the recorded PDF changes from 599 to 631 bytes and exactly equals LF→CRLF conversion. The provided evidence does not establish outgoing-base64 PDF survival through a later Apple save, arbitrary attachments, binding/header retention, nested-list/cell editing, or every failure ordering.

Protocol requirements to add to the acceptance checklist

Normative references: RFC 3501, RFC 9051, RFC 7162, RFC 2177, RFC 4315. Native evidence: #543 report, raw MIME, and earlier Apple trace.

The UID-per-immutable-message model is correct. Complete the protocol contract around it:

Choose rev1 plus extensions explicitly. RFC 9051 is useful guidance, but do not advertise IMAP4rev2 without its additional requirements. Rev1 international mailbox names use modified UTF-7 unless the relevant UTF-8 extension is negotiated; MIME body charset is a separate concern. Verify Unicode folders and the actual Apple capability-dependent grammar, including non-synchronizing literals and CHANGEDSINCE 0 seen in the older trace.
CONDSTORE is metadata concurrency, not an APPEND body precondition. RFC 7162 §3.1 requires positive unsigned 63-bit monotonic sequences, persistent behavior, new-message sequences and correct flag updates. Include UNCHANGEDSINCE partial success/MODIFIED UID sets, .SILENT MODSEQ responses, no-op flag changes, BODY FETCH setting Seen and BODY.PEEK not doing so. Do not infer Apple's use of a conditional STORE from the server advertising CONDSTORE.
QRESYNC requires ENABLE. RFC 7162 §3.2.3 requires implementing/advertising ENABLE and enabling QRESYNC per connection. Cover rejection before enable, HIGHESTMODSEQ including expunges, VANISHED versus VANISHED (EARLIER), UID/sequence-number accounting and the ordering that prevents losing delayed expunges after reconnect. Test non-QRESYNC and CONDSTORE-only clients too.
Epoch reset applies to active sessions. Force all old selected views to reselect or close before accepting subsequent UID commands; clearing persisted mappings alone is insufficient. Test a crash/restore that reuses numeric UIDs while an old session is alive. New UIDVALIDITY must obey rev1 monotonicity/no-reuse rules for the mailbox name, including delete/recreate and rename/recreate. Verify stale APPEND receipts do not return obsolete destination-epoch acknowledgements.
Define tombstone expiry. For an older MODSEQ than retained detail, use a correct complete missing-UID reconciliation or safe epoch reset; do not silently return only recent tombstones. RFC 7162 §5 discusses bounded retention. This must coexist with offline devices and bounded memory/disk.
IDLE is a per-connection command for the selected mailbox. RFC 2177 requires the continuation/DONE/tagged completion flow and correct queued responses. Cover the SELECT→IDLE subscription race, DONE→next command, reconnect, 29-minute refresh behavior, slow-reader backpressure, cancellation/revocation and sequence-number-safe unsolicited responses. In particular, test an edit to a folder that the device has not selected. One active IDLE does not promise all-folder two-second delivery.
Use more than one ordinary client library as the correctness oracle. async-imap socket tests are useful; add scripted raw-wire cases for invalid literals, pipelining, partial FETCH/HEADER sections, empty UID sets, CLOSE, failed writes, capability negotiation and reconnect. Captured Apple replay plus RFC invariants is stronger than merely accepting one library's successful requests.

Security and isolation assessment

The per-Installation Notes-only app-password design, TLS, live revocation, and user/account-scoped lookup keys are appropriate. Make the two-user/two-account test cover every command and code path: authentication, LIST/LSUB/STATUS, SEARCH predicates and counts, FETCH headers/sections/attachments, APPEND UUID aliases, COPY/MOVE, flags/expunge, partial replies, receipts, reconnect, recovery and submission compatibility. A Notes credential must not gain web/Mail/Contacts access or another account through an alias, shared hash or cache hit. Revocation must fence in-flight reads and mutations as well as wake an idle connection. Invalid foreign identities must not expose titles, paths, blob availability or another user’s deduplication result. The signed base, if used, must be verified under the authenticated account; an Apple UUID is a claim, not authorization. R1 is the remaining concrete hidden-content blocker.

Failure-table additions / required outcomes

The existing A–E crash rows and preserved-conflict policy are useful. Extend them with these concrete sequences and assertions:

Scenario	Required assertion
Weeks offline, deltas expired, then edit and delete	Full reconciliation does not establish a newer edit base. Retain unknown-base writing and deletion proposals; never resurrect a retired/hidden note. Bound candidate-base work.
Two devices edit; one immediately deletes/moves	Disjoint merge is applied once; overlapping writing survives privately; delayed cleanup affects only its proven message/membership. Follow with a genuine new delete to catch overbroad receipts.
UIDVALIDITY reset during active selection or lost APPEND reply	Old session UIDs cannot address new messages. Retries preserve the canonical action without inventing a valid old-epoch APPENDUID.
Crash between APPEND, STORE and EXPUNGE, including across folders	Exact flags, membership and canonical identity agree after repeated recovery. Failure to publish/ack must not permit later cleanup to trash surviving work.
Hide/unhide while offline or while a literal streams	Current policy controls every served generation; no post-fence protected output, stale resurrection or hidden-only change signal. Previously downloaded bytes remain outside recall guarantees.
Device mass-deletes while edits continue	Guard each target against its expected revision; partial command/batch completion is represented correctly. Grouped restore retains UUIDs, folder metadata, all companions and attachments, including moved notes.
Quota full, physical disk/WAL full, or Home unavailable	Admission reserves capacity; no acknowledged half-generation or endless global replay lock. Failed input is not falsely described as retained. Prior canonical content remains recoverable.
Huge/slow attachment, disconnect and retry	Bound raw/decoded/staging/queue memory and deadlines; remove abandoned staging safely; retain verified referenced originals. Another user and ordinary note writes remain responsive.
Broken HTML/MIME, duplicate identity/CID, charset failure	Distinguish safe but unrepresentable edits from unsafe/over-limit input. Explicit response, no partial canonical mutation; no script/remote fetch/path traversal.
State loss / Home-only rebuild, renderer upgrade, or companion outside edit	Preserve current source and opaque data, aliases/membership and authorization; rotate the UID epoch when the ledger cannot be proved. Old sent bases remain usable only when actually retained; missing history cannot be guessed from current files.

Cost, scale and simpler delivery

The budgets are correctly called gates, not results. The plan does require scale tests in the first code PR (lines 101 and 105), which directly addresses the earlier small-fixture failure. Keep that requirement and make the harness non-vacuous:

Add H (retained candidate bases/history) and C (connections/selected mailboxes) to N/S/B/A/K/D/Δ. Fan-out and memory depend on actual sessions/subscriptions, not physical device count alone. Bound per-Installation, per-user and global connections, queue bytes and slow-client work; coalesce wake hints and recover via persistent cursors.
First-PR tests must cover 100k notes/account, two concurrently active accounts, four devices each, 10k-line/1 MiB notes with repeated content, 100 attachments and 100 MiB decoded attachment totals, long histories, and 52 text edits with changed boundaries/CIDs/base64 wrapping. Assert raw-MIME, layout, companion, database/WAL and blob amplification, not just the original attachment directory size. Existing vCard PHOTO chunking is not a general MIME solution.
The first slice can measure mapping/storage work without a listener. Require its scale measurements before merge; add server/conformance and native measurements in the slices where those exist. Do not label the pure format harness an end-to-end 100k-note Apple acceptance pass.
Treat 1 TiB as an account-storage/streaming/initial-transfer test with an explicit fixture policy; do not allocate or materialize it in memory. State whether memory ceilings include database cache, DOM/source trees, TLS buffers, active room and page-cache effects; record incremental bridge and total-process memory.
The 60-second gate is for metadata, not full content delivery. Publish initial-indexing and first-usable-note criteria separately. MIME base64 expands large payloads before line wrapping; a 100 MiB decoded edit is roughly 137 MiB on the wire. Meeting the reserved half-second network budget for even one full such transfer requires roughly 2.3 Gbit/s of payload throughput. Record bandwidth and concurrent-device traffic rather than silently excluding heavy fixtures. Real device rendering is an independent gate.
Avoid parsing all notes into per-account or per-connection trees. Persist compact metadata/offset indexes and load bounded active notes; test cancellation and concurrent foreground writes during rebuild/SEARCH. A diff timeout alone does not bound parser, allocation, history lookup or verification time.

A simpler first vertical slice is: native provenance experiment → one Notes mailbox → safe text/style edits on known bases, source-preserving display of unsupported structures, base64 verified attachments, durable conflicts and hidden enforcement → two-device/crash/scale acceptance. Keep true two-way editing as the milestone; defer nested-folder mutation, identity-changing COPY, broad structural edits and unproved media affordances until their evidence exists. QRESYNC can land after a correct persistent UID/CONDSTORE/IDLE path if not required by the captured client; advertise it only when complete. This reduces moving parts without changing the owner's final scope or reintroducing public IDs for every block.

What must be verified on macos-01-std first

Before committing to the mapper/receipt architecture, capture redacted wire plus original/returned MIME for:

Custom signed base headers and binding attributes across successive edits, reselect, restart, reconnect and offline save; establish whether ordinary automatic merges remain possible without them.
Native edit/replacement, move, copy, delete, undo/cancel-delete and failed APPEND cleanup, including multiple connections and the UID 118/119/120 pending-edit ordering from #543. Confirm flags and UIDVALIDITY/MODSEQ behavior for the advertised capability sets; do not assume Apple enables QRESYNC.
Base64 ASCII/binary PDF and image byte retention after a text edit, non-image downloadable files, CID rewriting and part omission/reordering. Add audio/video/drawing only if that support is promised.
Multi-item/nested lists, actual table-cell/code/quote/heading/link edits, repeated content and 10k-line notes. Record which source forms are editable and which preserve only an inert view.
IDLE pickup in selected and non-selected folders, account/profile setup on implicit TLS 993, whether authenticated submission validation is needed, and revocation during quiet IDLE and FETCH.
Hide removal from Notes and Spotlight UI, then Siri on kayg's real iPhone. The #543 real-iPhone Siri positive result is valid; the VM's negative mdfind result establishes neither UI search nor removal. Keep the existing cleanup/reservation rules.

These are required experiments to put into the plan, not tests run by this writing job.

Review evidence and validation

Read issue #543 and all 21 currently returned comments, including its plan announcement; downloaded/read the macOS 27 report and examined the supplied raw MIME archive. Read ADRs 0028, 0056, 0062–0064 and the relevant 0066–0068 format/sync contracts; examined current notes/search/history/companion/attachment/action code, the unmerged origin/issue/carddav-read projection, and earlier calternal-v0 IMAP/DAV source/research at ebaa86165145f871441efe5ae950c7cfc94cc953. Read the available global and v0 CLAUDE.md contracts; there is no tracked CLAUDE.md in the reviewed rewrite tree. Checked RFC 3501/9051 message/command/session rules, RFC 7162 CONDSTORE/QRESYNC, RFC 2177 IDLE, and RFC 4315 UIDPLUS.

Worktree: /home/agent/wt/review-766, freshly created detached at origin/docs/imap-notes-plan, verified head a4230326f54f1131bc8eb172df779b09c3d10e40. Review artifact: /home/agent/codex-jobs/review-766/review.md. Evidence-check output: /home/agent/codex-jobs/review-766/evidence-check.txt.

No product code changed. No build, heavy run, native feature run, deploy, push or merge was performed. The native checks above remain unverified requirements. This review requests concrete plan edits; it does not claim measured implementation failures for code that does not yet exist.

# Independent design review — Apple Notes over IMAP **Verdict: revise. Round 1 of at most 3.** Reviewed PR [#766](https://git.kayg.org/kayg/calternal/pulls/766), branch `docs/imap-notes-plan`, head `a4230326f54f1131bc8eb172df779b09c3d10e40`. All plan line references below refer to that head's `docs/plans/0543-imap-notes.md`. The direction matches kayg's decisions: two-way notes, server authority and history, no hidden export, read-only checklist state, base64 attachments, and no blind pending-edit overwrite. Retaining source and sent projections is appropriate. The first-code-PR scale requirement is explicit and should stay. The blockers are the operational rules between these invariants: several distinct Apple operations still map to the same unspecified receipt/guard machinery, and a required visibility primitive does not exist at this head. ## Ranked blocking gaps and exact required edits ### R1 — P1: make visibility an explicit prerequisite and define its publication boundary **Plan lines 19, 43–49, 66, 84, 89–91.** “Use search's authoritative visibility policy” currently names a requirement, not an available implementation. At this head, `crates/notes/src/search.rs:151–179` extracts title/tags and cleans every body line without filtering `hidden:`. The supported `hidden` value in ADR 0068 is a block-ID list; this plan also requires whole-note hiding but does not specify its accepted representation or how it composes with block hiding. Do not inherit whatever a permissive YAML getter happens to return. **Concrete failure:** export a visible note with a hidden paragraph and hidden attachment, then edit only that paragraph. If the source revision drives Message-ID, Date, a signed base header, UID or MODSEQ, an IMAP observer can detect the hidden edit despite unchanged visible content. More directly, a slow `BODY[]` or partial attachment FETCH can already hold an old readable generation when Hide commits. Invalidating the cache and checking authorization at command start does not settle what can still leave the socket afterward. **Required plan edits:** - Name the shared visibility/marker module as a dependency, define valid whole-note/block forms and malformed/duplicate-key behavior, and require the same vectors for Calternal search and IMAP. Distinguish privacy hiding from task notes being excluded from ordinary Notes lists. - Build the entire wire projection and SEARCH index from the authorized view, including Subject, link labels, image alt text, filenames, CIDs, previews, reference definitions and MIME parts. Opaque/source fallbacks must pass that same filter. No attachment referenced only by omitted content is emitted. - Publish a new wire revision only for an authorized-view or relevant mailbox-state change. Hidden-only source edits must not advance externally visible revision headers/counters or emit IDLE hints. Preserve their private source revisions separately. Test hidden notes that have never been exported separately from visible-to-hidden removal, whose disappearance is necessarily observable. - Define a visibility-generation fence shared by Hide/Unhide, FETCH/partial FETCH, SEARCH and queued IDLE delivery. State how an in-flight literal is aborted/connection closed and when bytes count as already delivered; do not promise recall of previously delivered bytes. Expunge the old immutable message and publish a sanitized replacement under a new UID for partial hides; never mutate a UID's message bytes. - Add hide/unhide races with stale Apple APPEND, replay, restore and overlapping FETCH, plus tests that two Homes with identical exported content but different never-exported hidden content produce identical ordinary mailbox counts/search/size/revision behavior. Existing per-user/account keying is a good isolation requirement; also keep hidden and foreign-ID errors indistinguishable from unavailable objects. ### R2 — P1: specify Apple's actual move/delete/APPEND state machine, not only abstract MOVE **Plan lines 15–17, 45–49, 75–85.** The older code is not the rewrite architecture, but its captured client behavior is relevant evidence. `calternal-v0/docs/research/apple-notes-imap.md:150–188` records a Notes folder move as **APPEND in the destination, then STORE Deleted/UID EXPUNGE in the source**; it explicitly saw no UID MOVE. The new #543 probe verifies folder creation/rename and replacement saves, not an updated complete move/copy/delete matrix. The plan specifies identity-preserving MOVE, identity-changing COPY and same-note replacement receipts, but not the observed cross-mailbox APPEND sequence. **Concrete failure:** a device moves note X from A to B by APPEND, disconnects, then deletes X's source UID after a concurrent edit or another move. Depending on how the unspecified receipt is matched, the server can trash X, restore A, duplicate X, or treat a real copy as a move. A durable receipt alone does not define which result is correct. There is also a protocol/semantic boundary to settle. “Repeat identical input returns its original result” can mean valid action deduplication, but APPEND has no standardized idempotency key. It cannot generally mean returning an old or expunged APPENDUID for every later identical APPEND, especially across a UIDVALIDITY reset. RFC 3501 §6.3.11 and RFC 9051 §6.3.12 append a new message; RFC 4315 §3 describes the UID assigned in the destination mailbox. Likewise, an IMAP COPY copies a message, while a Calternal copy needs a fresh identity graph. Specify any subsequent canonical reconciliation rather than silently rewriting the copied message under its assigned UID. **Required plan edits:** add one compact transition table covering same-folder replacement, cross-folder APPEND move, explicit MOVE, COPY, standalone delete, cancelled deletion (`-FLAGS Deleted`), failed APPEND followed by cleanup, and disconnect at each command boundary. For each transition give: 1. Durable key: account, Installation, mailbox ID, UIDVALIDITY, source UID/revision and operation/base evidence; exact scope of any pre-APPEND UID set. 2. Raw-message UIDs versus canonical note identity, flags, canonical folder membership, and retained proposals. 3. Exact OK/NO response and APPENDUID/COPYUID, resulting EXISTS/EXPUNGE/VANISHED, and what another selected session can FETCH. 4. Receipt completion/retention rules. An unfinished receipt must neither authorize deleting a future revision nor mask a later genuine delete. When intent cannot be distinguished, retain a proposal rather than guess. 5. Recovery and a subsequent legitimate operation after recovery, not only retry of the interrupted operation. Separate message-level APPEND behavior from deduplicating the canonical action. Distinguish an echo/retry from a deliberate edit that returns the note to an earlier visible value. Do not claim exactly-once client intent from content hashes alone. Measure the native move/copy/delete sequences before committing to folder semantics; a one-folder first slice avoids much of this ambiguity. ### R3 — P1: prove base provenance early and bound all candidate-base work **Plan lines 19, 35, 47, 61, 78, 95–105.** The signed-base fallback is correctly conditional, but its successful path is unverified. The #543 imported/edited pair contains the ordinary UUID/creation-date headers; it did not test an added signed base header or HTML binding attributes. The raw conflict samples also share `X-Mail-Created-Date` across the base, server replacement and Apple edit. The older research/code explicitly warns that a newer FETCH is not the edit base and that creation dates survive multiple saves. These are reasons to keep the conservative fallback, not evidence that the proposed automatic path works. **Concrete failure:** an Installation has fetched hundreds of revisions, including repeated visible paragraphs whose private block identities or metadata differ, then reconnects with an edit made weeks ago and no retained custom header. Comparing all “plausible” bases is not the advertised O(S+A+B) operation. If a deadline expires, a conflict is safe, but ordinary editing after a long history can become permanently conflict-only. Truncating the candidate set and calling the surviving candidates unanimous is unsafe. **Required plan edits:** - Put a native compatibility/provenance experiment **before** broad mapping/protocol implementation. Test signed-header and binding retention through ordinary saves, successive saves without reselect, attachments, restart, reconnect, offline edits and the measured pending-edit race. Record actual retained fields; do not assume custom headers survive. - Define how the complete plausible-base set is constructed. Define “same safe patch” by target identities, semantic effects and protected-source preservation, not just matching visible text or offsets. - Add history depth/candidate count H to the cost model. Bound candidate discovery, loading, parse, matching, merge and re-render verification together. If complete evidence cannot be established within the budget, preserve the incoming proposal; never sample away possible bases. - Require repeated successful automatic ordinary-text edits after long histories and offline reconnect, alongside deliberate conflicts for truly ambiguous cases. This strengthens the existing “not every edit to conflicts” gate rather than replacing it. ### R4 — P1: define quota/full-disk outcomes across the durable stages **Plan lines 37, 45, 61–65, 74–85, 89.** Quotas and streaming are named, but quota-full/ENOSPC is absent from the failure table. A crash-replay recipe does not by itself handle a permanently unavailable write target. “Keep raw incoming MIME,” “retain both byte sets,” and “no success before C/D” need a stated limit when there is no space to retain the incoming version. **Concrete failure:** two concurrent 100 MiB attachment APPENDs each pass a preflight check, then jointly exhaust quota. Or B commits a receipt and raw version, C changes the room/history, and D fails because the mailbox database/WAL is full. An unqualified replay-before-reads barrier can then make the account unavailable indefinitely. Retrying from Apple can also amplify retained raw/conflict data. **Required plan edits:** - Add quota reservation under the same authority as admission, including new decoded blobs, exact MIME/layout records, retained proposals/companions and bounded staging. Define release on failure and replay. Use existing attachment type/path guards; do not assume the current photo-specific companion deduper handles MIME (`crates/notes/src/history/blobs.rs:12–18,113–128` recognizes vCard PHOTO lines). - Set separate raw-literal, decoded-total, individual attachment, part/header/DOM and staging/concurrency limits. They must admit the declared fixtures, account for base64 expansion, and reject larger transfers explicitly without truncation or altering the prior note. - Specify pre-B rejection, post-B retryable failure, post-C/D failure and recovery behavior, including which reads remain safe while a generation is pending. Distinguish user quota from physical disk/WAL/Home failures; reconcile this with ADR 0062's degraded Home and one-second materialization rules. - Test two competing admissions, full staging, full history, full projection/WAL, failed Home placement and failure during recovery. Native acceptance must confirm a rejected APPEND keeps the Apple edit recoverable and that subsequent cleanup cannot delete the canonical note. If retaining rejected bytes is impossible, say so and return failure; do not promise retention that did not happen. ### R5 — P2: define protected semantics and format-owner composition **Plan lines 19–35, 47, 95–105.** A source companion makes byte preservation possible, and conservative rejection is valid. It does not establish that all accepted text/list/link operations preserve Calternal meaning. “Unchanged protected structure” needs to include the semantic graph, not only an HTML/Markdown tree. **Concrete failures:** an apparently ordinary text edit changes a relationship link into/out of a standalone child; reordering a task/event list changes ownership; deleting a visible parent would also consume a hidden child; editing a contact list value races a CardDAV update while each adapter updates only its own companion. A generic room merge is not automatically a semantic compare-and-swap. At this head `crates/notes/src/room/edit.rs:84–118` forks/merges a known-base patch and has a format-owner rebase hook for a matching staged companion. The history store supports companions per owner, but an individual API edit carries one optional companion; cross-format atomicity needs an explicit integration rule. **Required plan edits:** define the protected semantic inventory (IDs/retirement, hidden membership, relationship/owner bindings, task/event facts, frontmatter, table-width associations and reference-link definitions). Have the owning format modules validate the candidate against the current generation **inside** the guarded action; replan or retain a proposal if that generation changes. Cover notes already claimed by contacts/tasks/calendar and simultaneous changes through those protocols. State which such fields are editable through IMAP and how their companions are updated coherently; unsupported structured edits can be read-only in the first slice. Do not allocate public IDs merely to make the mapper easier. ## Mapping audit: what “lossless” can actually mean There is no implementation in this PR to execute. These are design counterexamples and required vectors, checked against the existing native MIME evidence, not claims of new Apple feature tests. Byte-exact preservation of unchanged source is achievable with a trusted matching companion; full editable Apple feature parity is not. Lossless source retention and an editable projection must remain separate acceptance checks. | Source / attachment kind | Required note → HTML → Apple edit → source result | | --- | --- | | Paragraphs, blank lines, soft/hard breaks | Test bare body text, div/p/br rewrites, empty paragraphs, trailing spaces, tabs and NBSP. Preserve untouched blank lines and CRLF/LF style; a collapsed soft break cannot become an invented paragraph. | | H1–H6 and title | Test editing the first line, Subject/body disagreement, duplicate headings and duplicate titles. One rename/link action must own the title change; native heading authoring is not established. | | Bold/italic/strike/underline/inline code | Normalize b/strong, i/em, span styles and nested marks. Escape literal Markdown punctuation; preserve backtick runs. Neither HTML spelling nor a text-only comparison proves a mark edit. | | Ordered/unordered/nested lists | Test multiple real items, repeated identical items, start numbers, loose items, indent/outdent, reorder/delete and list-to-paragraph conversion. #543's authored bullet fixture was one item with a literal backslash-n, not a nested/multi-item proof. Ambiguity retains the original plus incoming proposal. | | Tasks, all checkbox states, journal/events | A text edit must not tick, skip, change clocks/recurrence/owner or create an outcome. Test `[/]`, `[-]`, unknown boxes and hidden children. Disabled HTML controls still need a native save test; observed checkbox clicks are not durable state. | | Quotes/callouts | Test nested quote prefixes, lists/code inside quotes and removal of the visible callout label. Preserve kind and children or retain a proposal. | | Fenced/indented code and dividers | Test tabs, leading/trailing spaces, CRLF, blank lines, literal HTML, fence delimiters/languages and long repeated code. WebKit whitespace normalization must not silently rewrite code or turn it into task/event syntax. | | GFM tables | Test real cell edits, pipes/escapes, empty/repeated cells, nested formatting, tbody insertion and flattening. The #543 edit was after the table. Preserve alignments and the right table's width metadata; ambiguous boundaries remain unresolved. | | Links and reference links | Test label-only/target edits, shared definitions, renamed/moved/reused paths, block links, fragments/percent-encoding and Unicode. Keep UUID authority and relative source paths; verify safe HTTPS navigation on the device. `applenotes:` rendering did not prove navigation. | | Images | Test img → Apple object → img normalization, same bytes under changed CID/name/order, duplicate/missing CIDs and image removal. Accept only proven attachment changes; do not confuse a preview with the original. | | PDF and generic files | Send base64, decode/hash-verify, then test **Apple save and return**, not only server serialization. Include the 599-byte ASCII PDF, binary PDF, zero-byte, text, ZIP and arbitrary binary data. Incoming 7bit corruption cannot be repaired or made lossless by changing the outgoing encoding later. | | Audio/video and drawings | No native round-trip proof in #543. Define downloadable original versus preview and preserve both separately. Test MIME type, filenames, bytes and whether the device exposes the promised download. Keep native drawing/ink semantics read-only. | | Footnotes, math, diagrams, raw HTML, plugin/unknown blocks | Display-only/opaque preservation is acceptable. Test an adjacent ordinary edit, attempted edit/removal of the representation, hidden content inside the source, script/remote-resource sanitization and immutable raw-MIME retention. Never silently accept an edit to a read-only representation as a no-op. | | Frontmatter, comments and markers | Preserve BOM, unknown keys, comments, aliases, duplicate/invalid YAML and exact untouched bytes. Source markers never reach UI/accessibility/search. Malformed visibility fails closed. Whole-note copies follow identity rules without copying authority/history. | | Character sets / MIME transport | Outgoing UTF-8 and base64; incoming us-ascii/UTF-8 plus an explicit policy for other declared charsets. Test quoted-printable soft breaks, base64 wrapping, header folding/encoded Subject, NFC/NFD, emoji/combining characters and invalid sequences. Decode once; preserve opaque bytes rather than use lossy replacement characters. | | Very long/repeated notes | 10k lines and 1 MiB text with repeated blocks, Unicode and pathological alternating edits must bound projection, matching and verification, not just the final diff call. No per-block whole-document scan or public-ID allocation. | I independently decoded the supplied #543 samples: both recorded PNG paths retain the original 209 bytes and SHA-256; the recorded PDF changes from 599 to 631 bytes and exactly equals LF→CRLF conversion. The provided evidence does **not** establish outgoing-base64 PDF survival through a later Apple save, arbitrary attachments, binding/header retention, nested-list/cell editing, or every failure ordering. ## Protocol requirements to add to the acceptance checklist Normative references: [RFC 3501](https://www.rfc-editor.org/rfc/rfc3501), [RFC 9051](https://www.rfc-editor.org/rfc/rfc9051), [RFC 7162](https://www.rfc-editor.org/rfc/rfc7162), [RFC 2177](https://www.rfc-editor.org/rfc/rfc2177), [RFC 4315](https://www.rfc-editor.org/rfc/rfc4315). Native evidence: [#543 report](https://git.kayg.org/attachments/31813979-6271-4813-87cc-662e60a36210), [raw MIME](https://git.kayg.org/attachments/e474f897-b52a-484b-8e73-40a82a099cbc), and [earlier Apple trace](https://git.kayg.org/kayg/calternal-v0/src/commit/ebaa86165145f871441efe5ae950c7cfc94cc953/docs/research/apple-notes-imap.md#L150). The UID-per-immutable-message model is correct. Complete the protocol contract around it: - **Choose rev1 plus extensions explicitly.** RFC 9051 is useful guidance, but do not advertise IMAP4rev2 without its additional requirements. Rev1 international mailbox names use modified UTF-7 unless the relevant UTF-8 extension is negotiated; MIME body charset is a separate concern. Verify Unicode folders and the actual Apple capability-dependent grammar, including non-synchronizing literals and `CHANGEDSINCE 0` seen in the older trace. - **CONDSTORE is metadata concurrency, not an APPEND body precondition.** RFC 7162 §3.1 requires positive unsigned 63-bit monotonic sequences, persistent behavior, new-message sequences and correct flag updates. Include UNCHANGEDSINCE partial success/MODIFIED UID sets, `.SILENT` MODSEQ responses, no-op flag changes, BODY FETCH setting Seen and BODY.PEEK not doing so. Do not infer Apple's use of a conditional STORE from the server advertising CONDSTORE. - **QRESYNC requires ENABLE.** RFC 7162 §3.2.3 requires implementing/advertising ENABLE and enabling QRESYNC per connection. Cover rejection before enable, HIGHESTMODSEQ including expunges, VANISHED versus VANISHED (EARLIER), UID/sequence-number accounting and the ordering that prevents losing delayed expunges after reconnect. Test non-QRESYNC and CONDSTORE-only clients too. - **Epoch reset applies to active sessions.** Force all old selected views to reselect or close before accepting subsequent UID commands; clearing persisted mappings alone is insufficient. Test a crash/restore that reuses numeric UIDs while an old session is alive. New UIDVALIDITY must obey rev1 monotonicity/no-reuse rules for the mailbox name, including delete/recreate and rename/recreate. Verify stale APPEND receipts do not return obsolete destination-epoch acknowledgements. - **Define tombstone expiry.** For an older MODSEQ than retained detail, use a correct complete missing-UID reconciliation or safe epoch reset; do not silently return only recent tombstones. RFC 7162 §5 discusses bounded retention. This must coexist with offline devices and bounded memory/disk. - **IDLE is a per-connection command for the selected mailbox.** RFC 2177 requires the continuation/DONE/tagged completion flow and correct queued responses. Cover the SELECT→IDLE subscription race, DONE→next command, reconnect, 29-minute refresh behavior, slow-reader backpressure, cancellation/revocation and sequence-number-safe unsolicited responses. In particular, test an edit to a folder that the device has **not selected**. One active IDLE does not promise all-folder two-second delivery. - **Use more than one ordinary client library as the correctness oracle.** `async-imap` socket tests are useful; add scripted raw-wire cases for invalid literals, pipelining, partial FETCH/HEADER sections, empty UID sets, CLOSE, failed writes, capability negotiation and reconnect. Captured Apple replay plus RFC invariants is stronger than merely accepting one library's successful requests. ## Security and isolation assessment The per-Installation Notes-only app-password design, TLS, live revocation, and user/account-scoped lookup keys are appropriate. Make the two-user/two-account test cover every command and code path: authentication, LIST/LSUB/STATUS, SEARCH predicates and counts, FETCH headers/sections/attachments, APPEND UUID aliases, COPY/MOVE, flags/expunge, partial replies, receipts, reconnect, recovery and submission compatibility. A Notes credential must not gain web/Mail/Contacts access or another account through an alias, shared hash or cache hit. Revocation must fence in-flight reads and mutations as well as wake an idle connection. Invalid foreign identities must not expose titles, paths, blob availability or another user’s deduplication result. The signed base, if used, must be verified under the authenticated account; an Apple UUID is a claim, not authorization. R1 is the remaining concrete hidden-content blocker. ## Failure-table additions / required outcomes The existing A–E crash rows and preserved-conflict policy are useful. Extend them with these concrete sequences and assertions: | Scenario | Required assertion | | --- | --- | | Weeks offline, deltas expired, then edit and delete | Full reconciliation does not establish a newer edit base. Retain unknown-base writing and deletion proposals; never resurrect a retired/hidden note. Bound candidate-base work. | | Two devices edit; one immediately deletes/moves | Disjoint merge is applied once; overlapping writing survives privately; delayed cleanup affects only its proven message/membership. Follow with a genuine new delete to catch overbroad receipts. | | UIDVALIDITY reset during active selection or lost APPEND reply | Old session UIDs cannot address new messages. Retries preserve the canonical action without inventing a valid old-epoch APPENDUID. | | Crash between APPEND, STORE and EXPUNGE, including across folders | Exact flags, membership and canonical identity agree after repeated recovery. Failure to publish/ack must not permit later cleanup to trash surviving work. | | Hide/unhide while offline or while a literal streams | Current policy controls every served generation; no post-fence protected output, stale resurrection or hidden-only change signal. Previously downloaded bytes remain outside recall guarantees. | | Device mass-deletes while edits continue | Guard each target against its expected revision; partial command/batch completion is represented correctly. Grouped restore retains UUIDs, folder metadata, all companions and attachments, including moved notes. | | Quota full, physical disk/WAL full, or Home unavailable | Admission reserves capacity; no acknowledged half-generation or endless global replay lock. Failed input is not falsely described as retained. Prior canonical content remains recoverable. | | Huge/slow attachment, disconnect and retry | Bound raw/decoded/staging/queue memory and deadlines; remove abandoned staging safely; retain verified referenced originals. Another user and ordinary note writes remain responsive. | | Broken HTML/MIME, duplicate identity/CID, charset failure | Distinguish safe but unrepresentable edits from unsafe/over-limit input. Explicit response, no partial canonical mutation; no script/remote fetch/path traversal. | | State loss / Home-only rebuild, renderer upgrade, or companion outside edit | Preserve current source and opaque data, aliases/membership and authorization; rotate the UID epoch when the ledger cannot be proved. Old sent bases remain usable only when actually retained; missing history cannot be guessed from current files. | ## Cost, scale and simpler delivery The budgets are correctly called **gates, not results**. The plan does require scale tests in the **first code PR** (lines 101 and 105), which directly addresses the earlier small-fixture failure. Keep that requirement and make the harness non-vacuous: - Add H (retained candidate bases/history) and C (connections/selected mailboxes) to N/S/B/A/K/D/Δ. Fan-out and memory depend on actual sessions/subscriptions, not physical device count alone. Bound per-Installation, per-user and global connections, queue bytes and slow-client work; coalesce wake hints and recover via persistent cursors. - First-PR tests must cover 100k notes/account, two concurrently active accounts, four devices each, 10k-line/1 MiB notes with repeated content, 100 attachments and 100 MiB decoded attachment totals, long histories, and 52 text edits with changed boundaries/CIDs/base64 wrapping. Assert raw-MIME, layout, companion, database/WAL and blob amplification, not just the original attachment directory size. Existing vCard PHOTO chunking is not a general MIME solution. - The first slice can measure mapping/storage work without a listener. Require its scale measurements before merge; add server/conformance and native measurements in the slices where those exist. Do not label the pure format harness an end-to-end 100k-note Apple acceptance pass. - Treat 1 TiB as an account-storage/streaming/initial-transfer test with an explicit fixture policy; do not allocate or materialize it in memory. State whether memory ceilings include database cache, DOM/source trees, TLS buffers, active room and page-cache effects; record incremental bridge and total-process memory. - The 60-second gate is for metadata, not full content delivery. Publish initial-indexing and first-usable-note criteria separately. MIME base64 expands large payloads before line wrapping; a 100 MiB decoded edit is roughly 137 MiB on the wire. Meeting the reserved half-second network budget for even one full such transfer requires roughly 2.3 Gbit/s of payload throughput. Record bandwidth and concurrent-device traffic rather than silently excluding heavy fixtures. Real device rendering is an independent gate. - Avoid parsing all notes into per-account or per-connection trees. Persist compact metadata/offset indexes and load bounded active notes; test cancellation and concurrent foreground writes during rebuild/SEARCH. A diff timeout alone does not bound parser, allocation, history lookup or verification time. A simpler first vertical slice is: **native provenance experiment → one Notes mailbox → safe text/style edits on known bases, source-preserving display of unsupported structures, base64 verified attachments, durable conflicts and hidden enforcement → two-device/crash/scale acceptance**. Keep true two-way editing as the milestone; defer nested-folder mutation, identity-changing COPY, broad structural edits and unproved media affordances until their evidence exists. QRESYNC can land after a correct persistent UID/CONDSTORE/IDLE path if not required by the captured client; advertise it only when complete. This reduces moving parts without changing the owner's final scope or reintroducing public IDs for every block. ## What must be verified on macos-01-std first Before committing to the mapper/receipt architecture, capture redacted wire plus original/returned MIME for: 1. Custom signed base headers and binding attributes across successive edits, reselect, restart, reconnect and offline save; establish whether ordinary automatic merges remain possible without them. 2. Native edit/replacement, move, copy, delete, undo/cancel-delete and failed APPEND cleanup, including multiple connections and the UID 118/119/120 pending-edit ordering from #543. Confirm flags and UIDVALIDITY/MODSEQ behavior for the advertised capability sets; do not assume Apple enables QRESYNC. 3. Base64 ASCII/binary PDF and image byte retention after a text edit, non-image downloadable files, CID rewriting and part omission/reordering. Add audio/video/drawing only if that support is promised. 4. Multi-item/nested lists, actual table-cell/code/quote/heading/link edits, repeated content and 10k-line notes. Record which source forms are editable and which preserve only an inert view. 5. IDLE pickup in selected and non-selected folders, account/profile setup on implicit TLS 993, whether authenticated submission validation is needed, and revocation during quiet IDLE and FETCH. 6. Hide removal from Notes and Spotlight UI, then Siri on kayg's real iPhone. The #543 real-iPhone Siri positive result is valid; the VM's negative `mdfind` result establishes neither UI search nor removal. Keep the existing cleanup/reservation rules. These are required experiments to put into the plan, not tests run by this writing job. ## Review evidence and validation Read issue #543 and all 21 currently returned comments, including its plan announcement; downloaded/read the macOS 27 report and examined the supplied raw MIME archive. Read ADRs 0028, 0056, 0062–0064 and the relevant 0066–0068 format/sync contracts; examined current notes/search/history/companion/attachment/action code, the unmerged `origin/issue/carddav-read` projection, and earlier `calternal-v0` IMAP/DAV source/research at `ebaa86165145f871441efe5ae950c7cfc94cc953`. Read the available global and v0 `CLAUDE.md` contracts; there is no tracked `CLAUDE.md` in the reviewed rewrite tree. Checked RFC 3501/9051 message/command/session rules, RFC 7162 CONDSTORE/QRESYNC, RFC 2177 IDLE, and RFC 4315 UIDPLUS. Worktree: `/home/agent/wt/review-766`, freshly created detached at `origin/docs/imap-notes-plan`, verified head `a4230326f54f1131bc8eb172df779b09c3d10e40`. Review artifact: `/home/agent/codex-jobs/review-766/review.md`. Evidence-check output: `/home/agent/codex-jobs/review-766/evidence-check.txt`. No product code changed. No build, heavy run, native feature run, deploy, push or merge was performed. The native checks above remain unverified requirements. This review requests concrete plan edits; it does not claim measured implementation failures for code that does not yet exist.

kayg commented

2026-10-29 17:36:33 +00:00

Author

Owner

Round 2 revision pushed as a9eeb2c9 (documentation only), addressing review 69752:

R1 — visibility/publication: §1 makes the shared visibility/marker module a prerequisite, defines whole-note/block forms and fail-closed malformed input, filters the whole authorized projection, separates private source revisions from wire revisions, and specifies a socket-write generation fence/connection abort. Partial hides replace immutable UIDs; hide/unhide races and identical-export hidden-content equivalence are acceptance vectors.
R2 — native operations: slice 0 now requires real Apple command transcripts for create/edit/move/copy/delete/cancel, attachment changes, rejected APPEND cleanup, offline saves, disconnect boundaries and two-client races. The older destination-APPEND→source-delete trace is evidence, not an assumption about current Notes. §3 adds all requested transitions, exact cleanup scope, raw UID versus canonical identity, acknowledgements/notifications, receipt expiry/recovery and subsequent genuine operations. Every admitted APPEND gets a fresh UID; canonical dedupe never fabricates an old APPENDUID. Ambiguous move/copy/delete intent stays a proposal; first usable scope is one mailbox.
R3 — provenance/work: spike results gate signed-header/binding use. §2 defines the complete possible-delivery ledger, semantic/identity patch equality, H-dependent work, and combined candidate/text/scratch/elapsed limits. Incomplete evidence is never sampled into unanimity. Repeated ordinary auto-applied edits after long histories/offline reconnect remain a release gate.
R4 — capacity/durability: §4 reserves full logical quota through one authority, including raw/layout/history/proposals/staging, with explicit transport/parser/concurrency limits. Its failure table specifies pre-B, post-B, C→D, D→E, Home and recovery failures, safe complete-generation reads and affected-action backpressure. Users see quota-full for capacity failures; physical causes go only to admins. Undurable rejected input is never described as retained; native recoverability is required. ADR 0062's one-second Home boundary remains explicit.
R5 — semantics/owners: §2 inventories protected identity/relationship/owner/task/event/frontmatter/table/reference semantics, requires owning-module validation inside the generation guard, and makes multi-owner companion atomicity a prerequisite. Owner-claimed notes are inert initially; concurrent DAV/task/calendar edits and hidden children have explicit preservation rules. No mapping-driven public IDs.

Also expanded the mapping/transport vectors, rev1/CONDSTORE/ENABLE-QRESYNC/IDLE/epoch/tombstone contract, all-command account isolation/revocation checks, and failure sequences. Scale includes H/C, bounded queues/connections, whole-store growth/shared attachment payloads across 52 edits, the full first-code-PR fixtures, explicit 1 TiB fixture policy, total/incremental memory, initial-use versus metadata/full delivery, and actual wire bandwidth/device rendering. Delivery is verification spike → pure format/storage scale harness → guarded visibility/actions → one-mailbox usable two-way slice → evidence-gated breadth.

Validation: git diff --check, local Markdown links, review-item coverage self-check, and a five-page A4 print (10pt prose, 9.25pt tables, 12.7mm margins). Only docs/plans/0543-imap-notes.md changed. No product code, native experiment, heavy test or deploy was run; slice-0 observations remain required work, not claimed results. Ready for the next independent review (maximum 3 rounds).

Round 2 revision pushed as `a9eeb2c9` (documentation only), addressing review [69752](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-69752): - **R1 — visibility/publication:** §1 makes the shared visibility/marker module a prerequisite, defines whole-note/block forms and fail-closed malformed input, filters the whole authorized projection, separates private source revisions from wire revisions, and specifies a socket-write generation fence/connection abort. Partial hides replace immutable UIDs; hide/unhide races and identical-export hidden-content equivalence are acceptance vectors. - **R2 — native operations:** slice 0 now requires real Apple command transcripts for create/edit/move/copy/delete/cancel, attachment changes, rejected APPEND cleanup, offline saves, disconnect boundaries and two-client races. The older destination-APPEND→source-delete trace is evidence, not an assumption about current Notes. §3 adds all requested transitions, exact cleanup scope, raw UID versus canonical identity, acknowledgements/notifications, receipt expiry/recovery and subsequent genuine operations. Every admitted APPEND gets a fresh UID; canonical dedupe never fabricates an old APPENDUID. Ambiguous move/copy/delete intent stays a proposal; first usable scope is one mailbox. - **R3 — provenance/work:** spike results gate signed-header/binding use. §2 defines the complete possible-delivery ledger, semantic/identity patch equality, H-dependent work, and combined candidate/text/scratch/elapsed limits. Incomplete evidence is never sampled into unanimity. Repeated ordinary auto-applied edits after long histories/offline reconnect remain a release gate. - **R4 — capacity/durability:** §4 reserves full logical quota through one authority, including raw/layout/history/proposals/staging, with explicit transport/parser/concurrency limits. Its failure table specifies pre-B, post-B, C→D, D→E, Home and recovery failures, safe complete-generation reads and affected-action backpressure. Users see quota-full for capacity failures; physical causes go only to admins. Undurable rejected input is never described as retained; native recoverability is required. ADR 0062's one-second Home boundary remains explicit. - **R5 — semantics/owners:** §2 inventories protected identity/relationship/owner/task/event/frontmatter/table/reference semantics, requires owning-module validation inside the generation guard, and makes multi-owner companion atomicity a prerequisite. Owner-claimed notes are inert initially; concurrent DAV/task/calendar edits and hidden children have explicit preservation rules. No mapping-driven public IDs. Also expanded the mapping/transport vectors, rev1/CONDSTORE/ENABLE-QRESYNC/IDLE/epoch/tombstone contract, all-command account isolation/revocation checks, and failure sequences. Scale includes H/C, bounded queues/connections, whole-store growth/shared attachment payloads across 52 edits, the full first-code-PR fixtures, explicit 1 TiB fixture policy, total/incremental memory, initial-use versus metadata/full delivery, and actual wire bandwidth/device rendering. Delivery is verification spike → pure format/storage scale harness → guarded visibility/actions → one-mailbox usable two-way slice → evidence-gated breadth. Validation: `git diff --check`, local Markdown links, review-item coverage self-check, and a **five-page A4 print** (10pt prose, 9.25pt tables, 12.7mm margins). Only `docs/plans/0543-imap-notes.md` changed. No product code, native experiment, heavy test or deploy was run; slice-0 observations remain required work, not claimed results. Ready for the next independent review (maximum 3 rounds).

kayg commented

2026-10-29 17:42:57 +00:00

Author

Owner

Independent design review — Apple Notes over IMAP, round 2

Verdict: approve with required edits. Round 2 of a maximum of 3.

Reviewed PR #766, docs/imap-notes-plan, head a9eeb2c9cdf88a637f1852eff9739ead3b899d03. Line references below are to docs/plans/0543-imap-notes.md at that head. Read the round-1 review, including its mapping/protocol/security/failure/cost sections, and the author's itemized response.

The revision closes the broad design gaps. Visibility now has a publication boundary; source preservation is separated from editable parity; cleanup is scoped to immutable messages; provenance work is bounded without sampling; capacity and owner composition have explicit prerequisites. Conditional support and private proposals are honest limits. I do not require implementation or completion of the native spike to approve this documentation PR. Two bounded plan edits remain; neither requires another architectural rewrite.

Exact required edits

E1 — P1: distinguish a definitively rejected APPEND from an uncertain completion

Lines 62, 73, 85, 89 and 93–97. The universal rule that all capacity failures return NO [OVERQUOTA] conflicts with the post-B recovery rows, which preserve redo and later retry the action. “No success” does not specify whether a tagged failure was sent.

Counterexample: B durably records an Apple edit; C commits its source/history; D fails on a full projection WAL. Line 89 permits NO [OVERQUOTA]. Apple retains/retries the failed edit or lets the user undo it. After repair, recovery publishes D from the rejected command. The user now has a late applied edit after a definite failure, potentially followed by another retry. The same problem exists if a B-only action receives NO and is subsequently committed by recovery.

RFC 3501 §6.3.11 requires an unsuccessful APPEND to restore the mailbox to its previous state; no partial appending is allowed. RFC 9051 §6.3.12 repeats that rule, allowing a UIDNEXT gap. Retaining a private rejected proposal is compatible with failure; replaying it automatically into the live note/mailbox is a different outcome.

Required edit: add terminal aborted / committed / indeterminate outcomes to §4 and qualify line 89:

Send tagged NO only after the operation is durably aborted, with no later automatic canonical mutation or mailbox publication from that attempt. Retained private evidence can remain available for explicit resolution.
If completion remains recoverable or cannot be safely ruled out after B/C, terminate the connection without a tagged APPEND completion. Keep the existing last-complete-generation fence and affected-action freeze; recovery can then determine/finish the uncertain operation. A durable successful publication can return OK [APPENDUID ...]; losing that reply remains uncertain to the client.
State that failure to persist an abort also takes the uncertain/connection-close path. Extend the existing A–E fault assertions to verify both received NO → never later applied and lost completion → safe recovery and subsequent fresh operation.

This uses the existing journal/publication mechanism. It does not require a second transaction system or rollback of acknowledged work. Capacity causes can still be hidden from ordinary users.

E2 — P2: make slice 0 observe the new reconciliation paths before committing to them

Lines 9–19, 62, 70–75 and 125. The spike covers native commands extensively, but it does not explicitly reproduce the new server behavior: admit exact raw MIME at UID R, then expunge R and publish canonical MIME at UID C, including when the incoming edit became only a private proposal. Line 75's deletion-loop check is currently later acceptance, although this behavior decides whether the chosen receipt/republication strategy is usable.

Counterexample: Apple receives APPENDUID R; the bridge retires R and republishes the unchanged canonical note at C because the edit conflicted. A dirty client treats C as another competing replacement, saves again, and cleans up C. The bridge preserves the note and republishes again. No source data is lost, yet the client and bridge can keep saving/republishing, or the Apple user can lose access to the pending local text after an apparently successful save. #543's ordinary external replacement and pending-edit race do not establish convergence of this new path.

Required edit: add a compact adversarial observation group to slice 0, with these explicit cases:

Successful raw-R → canonical-C reconciliation, then a second ordinary edit and reconnect; record which base fields and cleanup UIDs Apple actually uses.
Conflict/unknown-base admission followed by stale cleanup, and a standalone refused deletion followed by canonical republication; observe quiescence, pending-edit recoverability and a subsequent genuine delete on two clients.
The two E1 outcomes: definitive quota rejection versus dropped completion followed by repair/reconnect; distinguish Apple's retry and cleanup behavior.
UIDVALIDITY reset with an active client and an offline client, including reused numeric UIDs. Also run the existing long-history/offline ordinary-edit gate during the spike, with retained tokens and with tokens stripped and more than 64 plausible bases, so the fallback limit is exercised before mapper/ledger commitments. The over-budget stripped-token case must become a proposal; the ordinary automatic-edit gate must pass on whichever path is selected to ship.

For each, require exact wire/MIME and UI evidence, not just absence of a server crash. State the branch: no proven convergence/recoverability means defer the affected write/delete path or stop the two-way milestone; no guessed cleanup or new heuristic can substitute for evidence. These are plan additions, not a request to run the experiments in this writing job.

R1–R5 disposition

Round-1 item	Round-2 assessment
R1 — visibility	Resolved at design level, lines 23–27 and 103. Shared parser/search vectors, accepted whole-note/block forms, malformed-input fail-closed behavior, authorized-view-only metadata, socket-write fencing and immutable replacement are concrete mechanisms. Previously transported bytes remain outside recall guarantees. Raw admitted MIME must pass the same visibility/security rules (line 62); an old hidden payload cannot become readable merely because APPEND stores exact bytes.
R2 — native operation state machine	Resolved by mechanisms plus an honest limit, lines 58–75. Fresh raw UIDs are distinct from canonical action deduplication; P is scoped to proven alias/operation/source revisions; STORE is reversible; ambiguous move/copy/delete intent stays unresolved. Folder writes/COPY are deferred. A standalone native delete is not promised where intent cannot be established. E2 makes the remaining native convergence branch explicit.
R3 — base provenance and cost	Resolved, lines 11, 17–19 and 48–52. Signed tokens are conditional on measured preservation; the ledger includes possible deliveries across connections and repeated visible values; semantic/identity equality replaces text-only unanimity. Completeness, loading, mapping and verification share the 64-candidate/16 MiB/64 MiB/200 ms limits. With stripped tokens and 65 plausible bases, a proposal is the specified safe result, not a sampled merge. Repeated automatic ordinary edits remain a release gate; E2 moves that decisive experiment early.
R4 — capacity and durable stages	Substantially resolved; E1 remains required. Lines 81–97 reserve logical and physical capacity, bound staging/decoding, preserve reservations across recovery, isolate pending generations and distinguish Home degradation from a success promise. The remaining gap is the terminal wire outcome after B/C, not another missing quota mechanism.
R5 — protected semantics and owners	Resolved, lines 39–46. Protected identity/relationship/hidden/task/event/frontmatter/table/reference semantics are explicit. Initial edits are restricted to plain text/styles in unclaimed notes; owner-claimed notes remain inert. Enabling their edits requires owner validation inside current-generation guards and an atomic companion bundle. This avoids pretending today's single optional companion already composes multiple owners.

The mapping table now addresses the material round-1 counterexamples: real nested/repeated lists and cell edits, code whitespace, source-marker privacy, strict transport decoding, opaque-content edits, and attachment byte verification. None of those rows is evidence of native parity until the spike supplies it. That distinction is maintained correctly.

Failure table exercised on paper

Sequence attempted	Result under the revised rules / remaining check
Hidden toggled while a device is offline	Device cached visible V1; server hides it; device submits an old V1 edit on reconnect. Current visibility fences reads and mutations; exact old MIME cannot be admitted as a readable message. Preserve incoming writing privately if durable capacity permits, otherwise reject without claiming retention. No UID/canonical resurrection. If subsequently unhidden, publish current authorized content at a new UID; an old edit still needs a proven base and current semantic guard. Offline cached bytes cannot be recalled. Lines 17, 25–27, 50, 62 and 98 cover safety.
Apple edits an old version	Server has V3; Apple edits V1. A retained, verified V1 token/complete ledger permits only a guarded safe patch against current semantics. Overlap, incomplete history or exceeded H budget becomes a proposal. A later FETCH of V3 cannot promote it into the edit base. Returning to a prior visible value is not automatically deduped as a retry. Lines 48–52 close the stale-base overwrite.
Two devices	A and B edit the same base, then B cleans up A's newer UID. Guard/replan merges proven disjoint edits; overlap remains private. Cleanup cannot acquire canonical delete authority over a newer revision merely by targeting its UID. If raw expunge removes the surviving projection, canonical content is republished. Safety is specified; E2 must establish that two native clients settle and retain pending writing. Cross-device/untraceable base evidence stays incomplete.
UIDVALIDITY reset	An old connection has UID 42; rebuilt mailbox uses numeric 42 for something else. Close old selected sessions before accepting UID commands, rotate to a greater/non-reused epoch, and scope receipts by mailbox/epoch. A lost APPEND reply cannot be answered with an old-epoch APPENDUID. QRESYNC epoch mismatch falls back to a fresh selection; expired tombstones require complete missing-UID reconciliation or reset. Lines 73 and 109–113 cover server safety; E2 adds the native observation.
Crash mid-APPEND/EXPUNGE	Before B, no admission; after durable intent, replay exact operation and retain generation fences. STORE only flags the immutable message; expunge targets its exact epoch/UID and cannot consume a future revision's cleanup authority. Repeated recovery and a new genuine operation are explicitly tested. D without E means possible success with a lost reply; a fresh APPEND still gets a fresh raw UID. E1 must settle B/C failure responses before this is complete.
Quota-full mid-sync	Competing 100 MiB uploads cannot both spend the same reservation. Pre-B failure preserves prior content and returns failure; source cleanup still needs independent delete evidence even if failed-attempt P could not be recorded. Post-C projection failure freezes affected mutations, leaves complete generations readable and does not replay C twice. Reservations survive repair. E1 prevents a definite rejection from later becoming a successful write. Apple's failed/uncertain-edit recovery is E2 evidence, not an assumed client guarantee.
Attachment re-encode	Changed boundary/CID/name/base64 wrapping with the same decoded payload reuses the blob while retaining exact MIME/layout evidence. Different decoded bytes remain beside the original; they are not silently “repaired.” The 599→631-byte PDF is therefore a changed/corrupt candidate, not a no-op attachment. Fifty-two re-encoded text saves must bound whole-store amplification. Lines 43, 81 and 119 cover the mechanism; imported-base64 PDF survival through Apple save is still a required experiment.
Mass delete concurrent with edits	Guard each canonical target independently: unchanged notes can enter grouped recoverable Trash; concurrently changed/ambiguous notes survive with proposals and new projections as needed. Raw UID expunge completion is distinct from canonical Trash approval. Restore retains identities, membership, every companion and attachments. Grouped/partial outcomes must be visible through the canonical action/history result, rather than implying every note was trashed because IMAP returned OK. Lines 70, 75 and 99 provide that split; E2 covers the refusal/republication behavior on which it depends.

I found no additional blocker in the rev1/CONDSTORE/QRESYNC/IDLE contract. The plan correctly treats CONDSTORE as metadata concurrency, requires ENABLE before QRESYNC behavior, distinguishes VANISHED from EARLIER, preserves immutable UID epochs across active sessions and bounds tombstone retention. Sequence-safe notification restrictions still apply during reconciliation; IDLE cannot promise an unselected folder's delivery deadline. Test advertisement/grammar only for capabilities actually implemented. The proposed RFC 9051 use as guidance is appropriate for a rev1 server.

Account/Installation scoping, signed-token verification, all-command isolation, revocation during reads/mutations, private companions/proposals and restricted submission are sufficient design requirements. I am not reopening security work already made explicit in this revision.

Simplicity and the first slice

The one-mailbox, unclaimed-note, text/style boundary is a meaningful simplification. Deferring folder intent, COPY identity changes, owner-composed structure, QRESYNC and media until evidence exists is appropriate. Do not broaden that boundary merely because the vector table names richer formats.

The first post-spike slice is valuable as a feasibility harness, especially given the mandatory first-code-PR scale/amplification gates. It is not yet a small user-visible slice: the first usable milestone is still slice 3. That is acceptable for this plan, but the least costly implementation order is outbound authorized projection + immutable MIME/storage + byte-exact no-op companions first, then safe inbound editing. A one-way-out acceptance checkpoint within the protocol slice would establish real device value before enabling inbound merges; unsupported Apple writes must receive explicit, tested behavior. This is a sequencing recommendation, not a third required edit or permission to weaken the final two-way scope.

Keep the first-PR scale requirement and the distinction between pure-harness results and native acceptance. The H/C variables, bounded queues/trees, total-process memory, logical-versus-physical storage accounting, streaming 1 TiB fixture policy and measured ~137 MiB wire cost are now explicit. The two-second gate remains demanding for heavy fixtures; it is honestly conditional on measured network/device performance, not evidence already achieved.

Evidence and scope

Fresh detached worktree: /home/agent/wt/review-766b, created at origin/docs/imap-notes-plan and verified as a9eeb2c9cdf88a637f1852eff9739ead3b899d03 on 10by32-01-std. Review artifact: /home/agent/codex-jobs/review-766b/review.md.

Read #543 and all returned comments, the macOS 27 report and raw-MIME archive, the prior review and author response, the revised plan, relevant visibility/room/companion context and ADR 0062/0068. Re-fetched RFCs 3501, 9051, 7162, 2177 and supporting UIDPLUS RFC 4315 from the RFC Editor; checked APPEND failure/UID immutability, STORE/EXPUNGE/CLOSE, CONDSTORE/ENABLE/VANISHED/tombstone and IDLE rules. Re-fetched the report/archive; both match the round-1 copies byte-for-byte. Independently decoded the recorded PDF/PNG samples: PDF is 631 bytes with recorded hash b39b06fe…; both PNG paths retain 209 bytes and hash c29de0e1…. The conflict samples preserve the same UUID/creation date, not proof of a newer edit base.

No product code, plan file, commit, push, merge, deployment, build, heavy run or native feature test was performed. This approves the design subject only to E1 and E2; the native/scale gates remain unrun work, not review claims of implementation correctness.

# Independent design review — Apple Notes over IMAP, round 2 **Verdict: approve with required edits. Round 2 of a maximum of 3.** Reviewed [PR #766](https://git.kayg.org/kayg/calternal/pulls/766), `docs/imap-notes-plan`, head `a9eeb2c9cdf88a637f1852eff9739ead3b899d03`. Line references below are to `docs/plans/0543-imap-notes.md` at that head. Read the [round-1 review](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-69752), including its mapping/protocol/security/failure/cost sections, and the [author's itemized response](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-69758). The revision closes the broad design gaps. Visibility now has a publication boundary; source preservation is separated from editable parity; cleanup is scoped to immutable messages; provenance work is bounded without sampling; capacity and owner composition have explicit prerequisites. Conditional support and private proposals are honest limits. I do not require implementation or completion of the native spike to approve this documentation PR. Two bounded plan edits remain; neither requires another architectural rewrite. ## Exact required edits ### E1 — P1: distinguish a definitively rejected APPEND from an uncertain completion **Lines 62, 73, 85, 89 and 93–97.** The universal rule that all capacity failures return `NO [OVERQUOTA]` conflicts with the post-B recovery rows, which preserve redo and later retry the action. “No success” does not specify whether a tagged failure was sent. **Counterexample:** B durably records an Apple edit; C commits its source/history; D fails on a full projection WAL. Line 89 permits `NO [OVERQUOTA]`. Apple retains/retries the failed edit or lets the user undo it. After repair, recovery publishes D from the rejected command. The user now has a late applied edit after a definite failure, potentially followed by another retry. The same problem exists if a B-only action receives `NO` and is subsequently committed by recovery. [RFC 3501 §6.3.11](https://www.rfc-editor.org/rfc/rfc3501#section-6.3.11) requires an unsuccessful APPEND to restore the mailbox to its previous state; no partial appending is allowed. [RFC 9051 §6.3.12](https://www.rfc-editor.org/rfc/rfc9051#section-6.3.12) repeats that rule, allowing a UIDNEXT gap. Retaining a private rejected proposal is compatible with failure; replaying it automatically into the live note/mailbox is a different outcome. **Required edit:** add terminal `aborted / committed / indeterminate` outcomes to §4 and qualify line 89: - Send tagged `NO` only after the operation is durably aborted, with no later automatic canonical mutation or mailbox publication from that attempt. Retained private evidence can remain available for explicit resolution. - If completion remains recoverable or cannot be safely ruled out after B/C, terminate the connection without a tagged APPEND completion. Keep the existing last-complete-generation fence and affected-action freeze; recovery can then determine/finish the uncertain operation. A durable successful publication can return `OK [APPENDUID ...]`; losing that reply remains uncertain to the client. - State that failure to persist an abort also takes the uncertain/connection-close path. Extend the existing A–E fault assertions to verify both **received `NO` → never later applied** and **lost completion → safe recovery and subsequent fresh operation**. This uses the existing journal/publication mechanism. It does not require a second transaction system or rollback of acknowledged work. Capacity causes can still be hidden from ordinary users. ### E2 — P2: make slice 0 observe the new reconciliation paths before committing to them **Lines 9–19, 62, 70–75 and 125.** The spike covers native commands extensively, but it does not explicitly reproduce the new server behavior: admit exact raw MIME at UID R, then expunge R and publish canonical MIME at UID C, including when the incoming edit became only a private proposal. Line 75's deletion-loop check is currently later acceptance, although this behavior decides whether the chosen receipt/republication strategy is usable. **Counterexample:** Apple receives `APPENDUID R`; the bridge retires R and republishes the unchanged canonical note at C because the edit conflicted. A dirty client treats C as another competing replacement, saves again, and cleans up C. The bridge preserves the note and republishes again. No source data is lost, yet the client and bridge can keep saving/republishing, or the Apple user can lose access to the pending local text after an apparently successful save. #543's ordinary external replacement and pending-edit race do not establish convergence of this new path. **Required edit:** add a compact adversarial observation group to slice 0, with these explicit cases: 1. Successful raw-R → canonical-C reconciliation, then a second ordinary edit and reconnect; record which base fields and cleanup UIDs Apple actually uses. 2. Conflict/unknown-base admission followed by stale cleanup, and a standalone refused deletion followed by canonical republication; observe quiescence, pending-edit recoverability and a subsequent genuine delete on two clients. 3. The two E1 outcomes: definitive quota rejection versus dropped completion followed by repair/reconnect; distinguish Apple's retry and cleanup behavior. 4. UIDVALIDITY reset with an active client and an offline client, including reused numeric UIDs. Also run the existing long-history/offline ordinary-edit gate **during the spike**, with retained tokens and with tokens stripped and more than 64 plausible bases, so the fallback limit is exercised before mapper/ledger commitments. The over-budget stripped-token case must become a proposal; the ordinary automatic-edit gate must pass on whichever path is selected to ship. For each, require exact wire/MIME and UI evidence, not just absence of a server crash. State the branch: no proven convergence/recoverability means defer the affected write/delete path or stop the two-way milestone; no guessed cleanup or new heuristic can substitute for evidence. These are plan additions, not a request to run the experiments in this writing job. ## R1–R5 disposition | Round-1 item | Round-2 assessment | | --- | --- | | **R1 — visibility** | **Resolved at design level**, lines 23–27 and 103. Shared parser/search vectors, accepted whole-note/block forms, malformed-input fail-closed behavior, authorized-view-only metadata, socket-write fencing and immutable replacement are concrete mechanisms. Previously transported bytes remain outside recall guarantees. Raw admitted MIME must pass the same visibility/security rules (line 62); an old hidden payload cannot become readable merely because APPEND stores exact bytes. | | **R2 — native operation state machine** | **Resolved by mechanisms plus an honest limit**, lines 58–75. Fresh raw UIDs are distinct from canonical action deduplication; P is scoped to proven alias/operation/source revisions; STORE is reversible; ambiguous move/copy/delete intent stays unresolved. Folder writes/COPY are deferred. A standalone native delete is not promised where intent cannot be established. E2 makes the remaining native convergence branch explicit. | | **R3 — base provenance and cost** | **Resolved**, lines 11, 17–19 and 48–52. Signed tokens are conditional on measured preservation; the ledger includes possible deliveries across connections and repeated visible values; semantic/identity equality replaces text-only unanimity. Completeness, loading, mapping and verification share the 64-candidate/16 MiB/64 MiB/200 ms limits. With stripped tokens and 65 plausible bases, a proposal is the specified safe result, not a sampled merge. Repeated automatic ordinary edits remain a release gate; E2 moves that decisive experiment early. | | **R4 — capacity and durable stages** | **Substantially resolved; E1 remains required.** Lines 81–97 reserve logical and physical capacity, bound staging/decoding, preserve reservations across recovery, isolate pending generations and distinguish Home degradation from a success promise. The remaining gap is the terminal wire outcome after B/C, not another missing quota mechanism. | | **R5 — protected semantics and owners** | **Resolved**, lines 39–46. Protected identity/relationship/hidden/task/event/frontmatter/table/reference semantics are explicit. Initial edits are restricted to plain text/styles in unclaimed notes; owner-claimed notes remain inert. Enabling their edits requires owner validation inside current-generation guards and an atomic companion bundle. This avoids pretending today's single optional companion already composes multiple owners. | The mapping table now addresses the material round-1 counterexamples: real nested/repeated lists and cell edits, code whitespace, source-marker privacy, strict transport decoding, opaque-content edits, and attachment byte verification. None of those rows is evidence of native parity until the spike supplies it. That distinction is maintained correctly. ## Failure table exercised on paper | Sequence attempted | Result under the revised rules / remaining check | | --- | --- | | **Hidden toggled while a device is offline** | Device cached visible V1; server hides it; device submits an old V1 edit on reconnect. Current visibility fences reads and mutations; exact old MIME cannot be admitted as a readable message. Preserve incoming writing privately if durable capacity permits, otherwise reject without claiming retention. No UID/canonical resurrection. If subsequently unhidden, publish current authorized content at a new UID; an old edit still needs a proven base and current semantic guard. Offline cached bytes cannot be recalled. Lines 17, 25–27, 50, 62 and 98 cover safety. | | **Apple edits an old version** | Server has V3; Apple edits V1. A retained, verified V1 token/complete ledger permits only a guarded safe patch against current semantics. Overlap, incomplete history or exceeded H budget becomes a proposal. A later FETCH of V3 cannot promote it into the edit base. Returning to a prior visible value is not automatically deduped as a retry. Lines 48–52 close the stale-base overwrite. | | **Two devices** | A and B edit the same base, then B cleans up A's newer UID. Guard/replan merges proven disjoint edits; overlap remains private. Cleanup cannot acquire canonical delete authority over a newer revision merely by targeting its UID. If raw expunge removes the surviving projection, canonical content is republished. Safety is specified; E2 must establish that two native clients settle and retain pending writing. Cross-device/untraceable base evidence stays incomplete. | | **UIDVALIDITY reset** | An old connection has UID 42; rebuilt mailbox uses numeric 42 for something else. Close old selected sessions before accepting UID commands, rotate to a greater/non-reused epoch, and scope receipts by mailbox/epoch. A lost APPEND reply cannot be answered with an old-epoch APPENDUID. QRESYNC epoch mismatch falls back to a fresh selection; expired tombstones require complete missing-UID reconciliation or reset. Lines 73 and 109–113 cover server safety; E2 adds the native observation. | | **Crash mid-APPEND/EXPUNGE** | Before B, no admission; after durable intent, replay exact operation and retain generation fences. STORE only flags the immutable message; expunge targets its exact epoch/UID and cannot consume a future revision's cleanup authority. Repeated recovery and a new genuine operation are explicitly tested. D without E means possible success with a lost reply; a fresh APPEND still gets a fresh raw UID. E1 must settle B/C failure responses before this is complete. | | **Quota-full mid-sync** | Competing 100 MiB uploads cannot both spend the same reservation. Pre-B failure preserves prior content and returns failure; source cleanup still needs independent delete evidence even if failed-attempt P could not be recorded. Post-C projection failure freezes affected mutations, leaves complete generations readable and does not replay C twice. Reservations survive repair. E1 prevents a definite rejection from later becoming a successful write. Apple's failed/uncertain-edit recovery is E2 evidence, not an assumed client guarantee. | | **Attachment re-encode** | Changed boundary/CID/name/base64 wrapping with the same decoded payload reuses the blob while retaining exact MIME/layout evidence. Different decoded bytes remain beside the original; they are not silently “repaired.” The 599→631-byte PDF is therefore a changed/corrupt candidate, not a no-op attachment. Fifty-two re-encoded text saves must bound whole-store amplification. Lines 43, 81 and 119 cover the mechanism; imported-base64 PDF survival through Apple save is still a required experiment. | | **Mass delete concurrent with edits** | Guard each canonical target independently: unchanged notes can enter grouped recoverable Trash; concurrently changed/ambiguous notes survive with proposals and new projections as needed. Raw UID expunge completion is distinct from canonical Trash approval. Restore retains identities, membership, every companion and attachments. Grouped/partial outcomes must be visible through the canonical action/history result, rather than implying every note was trashed because IMAP returned OK. Lines 70, 75 and 99 provide that split; E2 covers the refusal/republication behavior on which it depends. | I found no additional blocker in the rev1/CONDSTORE/QRESYNC/IDLE contract. The plan correctly treats CONDSTORE as metadata concurrency, requires ENABLE before QRESYNC behavior, distinguishes VANISHED from EARLIER, preserves immutable UID epochs across active sessions and bounds tombstone retention. Sequence-safe notification restrictions still apply during reconciliation; IDLE cannot promise an unselected folder's delivery deadline. Test advertisement/grammar only for capabilities actually implemented. The proposed RFC 9051 use as guidance is appropriate for a rev1 server. Account/Installation scoping, signed-token verification, all-command isolation, revocation during reads/mutations, private companions/proposals and restricted submission are sufficient design requirements. I am not reopening security work already made explicit in this revision. ## Simplicity and the first slice The one-mailbox, unclaimed-note, text/style boundary is a meaningful simplification. Deferring folder intent, COPY identity changes, owner-composed structure, QRESYNC and media until evidence exists is appropriate. Do not broaden that boundary merely because the vector table names richer formats. The first post-spike slice is **valuable as a feasibility harness**, especially given the mandatory first-code-PR scale/amplification gates. It is not yet a small user-visible slice: the first usable milestone is still slice 3. That is acceptable for this plan, but the least costly implementation order is outbound authorized projection + immutable MIME/storage + byte-exact no-op companions first, then safe inbound editing. A one-way-out acceptance checkpoint within the protocol slice would establish real device value before enabling inbound merges; unsupported Apple writes must receive explicit, tested behavior. This is a sequencing recommendation, not a third required edit or permission to weaken the final two-way scope. Keep the first-PR scale requirement and the distinction between pure-harness results and native acceptance. The H/C variables, bounded queues/trees, total-process memory, logical-versus-physical storage accounting, streaming 1 TiB fixture policy and measured ~137 MiB wire cost are now explicit. The two-second gate remains demanding for heavy fixtures; it is honestly conditional on measured network/device performance, not evidence already achieved. ## Evidence and scope Fresh detached worktree: `/home/agent/wt/review-766b`, created at `origin/docs/imap-notes-plan` and verified as `a9eeb2c9cdf88a637f1852eff9739ead3b899d03` on **10by32-01-std**. Review artifact: `/home/agent/codex-jobs/review-766b/review.md`. Read #543 and all returned comments, the macOS 27 report and raw-MIME archive, the prior review and author response, the revised plan, relevant visibility/room/companion context and ADR 0062/0068. Re-fetched RFCs 3501, 9051, 7162, 2177 and supporting UIDPLUS RFC 4315 from the RFC Editor; checked APPEND failure/UID immutability, STORE/EXPUNGE/CLOSE, CONDSTORE/ENABLE/VANISHED/tombstone and IDLE rules. Re-fetched the report/archive; both match the round-1 copies byte-for-byte. Independently decoded the recorded PDF/PNG samples: PDF is 631 bytes with recorded hash `b39b06fe…`; both PNG paths retain 209 bytes and hash `c29de0e1…`. The conflict samples preserve the same UUID/creation date, not proof of a newer edit base. No product code, plan file, commit, push, merge, deployment, build, heavy run or native feature test was performed. This approves the design subject only to **E1 and E2**; the native/scale gates remain unrun work, not review claims of implementation correctness.

kayg commented

2026-10-29 17:44:48 +00:00

Author

Owner

Applied the two required round-2 edits from review 69759 in 0816b5ff72cec14b56e39eb0fbc24803f9788f21. Documentation only; no product code or native experiments.

E1 — §4 APPEND outcomes and retry reconciliation (quoted):

APPEND outcomes use the existing journal/publication marker: aborted records a durable replay prohibition: no canonical mutation/mailbox publication from that attempt, now or after repair; prior mailbox state remains (UIDNEXT gaps allowed). Retained private evidence is for explicit resolution only. Send tagged NO only after this abort is durable. committed records complete C/D publication and may return OK [APPENDUID ...]; a lost reply is still uncertain to Apple. indeterminate records unresolved intent/stages; if completion remains recoverable or cannot be ruled out after B/C, close without tagged APPEND completion. Failure to persist an abort also closes without completion. Keep last-complete-generation fences and affected-write freeze; recovery resolves/finishes uncertain intent, never replays an aborted attempt or rolls back acknowledged work.

A retried APPEND is a fresh wire command/UID on admission, never an old APPENDUID. Match §3's scoped receipt plus proven base/operation/source evidence to deduplicate only the canonical action; MIME/hash/UUID alone cannot prove a retry. Reconcile raw duplicates to the one canonical note at fresh UIDs. Without proof, retain a proposal against the claimed identity, not a second canonical note or guessed overwrite; a subsequent proven genuine edit remains a new action.

Capacity rejection is NO [OVERQUOTA] quota-full only after durable abort: logical quota, staging, physical disk/WAL or Home. Otherwise use the uncertain/connection-close path above. Cause/paths/disk-full go only to admins; other unsafe/limit rejections follow the same abort rule. Never claim retention of undurable input.

E2 — §1 slice-0 native reconciliation/recovery observations and branch decisions (quoted):

Slice-0 adversarial reconciliation observations (same wire/MIME and UI evidence for every case):

Admit exact raw MIME at UID R, expunge R and publish canonical MIME at C; make a second ordinary edit, then reconnect. Record surviving base fields, retry APPENDs and cleanup UIDs on every connection. This decides whether raw-receipt/canonical-republication and its base/cleanup bindings can ship.

Admit a conflicting/unknown-base edit as a private proposal, then stale cleanup; separately refuse canonical deletion and republish after raw expunge. On two clients record quiescence versus repeated saves/deletes, access to pending text, and a subsequent genuine delete. This decides replacement/proposal recovery and native-delete enablement.

Compare definitive quota NO with dropped APPEND completion at B/C/D, including crash/repair/reconnect. Record Apple's retries, retained local text, cleanup targets and next genuine edit; verify one canonical note/action or an explicit unresolved proposal. This decides whether §4's rejected/uncertain paths preserve usable native recovery.

Reset UIDVALIDITY with active and offline clients, reusing numeric UIDs; while offline also change/edit/delete canonical content on the server. Record reselect/refetch, local pending-text recovery, stale cleanup and subsequent ordinary edit/delete. This decides epoch-reset safety and offline reconciliation.

During this spike, run repeated automatic ordinary-text edits after long histories/offline reconnect with retained tokens, then stripped tokens and more than 64 plausible bases. Record actual base fields, candidate counts, wire/MIME and UI outcomes. Over-budget stripped-token edits must become proposals; the chosen shipping path must pass ordinary automatic edits. This decides token versus complete bounded-ledger feasibility before mapper/ledger commitments; conflict-only editing fails two-way acceptance.

For each branch, no proven convergence/pending-edit recoverability means defer the affected write/delete path or stop the two-way milestone; guessed cleanup or new heuristics cannot replace evidence.

Matching §3 command/failure rows now distinguish definitive rejection from lost completion. §4 failure rows cover durable abort before NO, B/C uncertainty, no replay after rejection, lost-reply recovery followed by a fresh operation, raw/canonical convergence, epoch resets and offline history budgets.

Validation: git diff --check passed; only docs/plans/0543-imap-notes.md changed; the existing five page sections/four explicit page breaks are retained. Native/scale gates remain planned experiments, not claimed results.

Applied the two required round-2 edits from [review 69759](https://git.kayg.org/kayg/calternal/pulls/766#issuecomment-69759) in `0816b5ff72cec14b56e39eb0fbc24803f9788f21`. Documentation only; no product code or native experiments. E1 — §4 APPEND outcomes and retry reconciliation (quoted): > **APPEND outcomes use the existing journal/publication marker:** `aborted` records a durable replay prohibition: no canonical mutation/mailbox publication from that attempt, now or after repair; prior mailbox state remains (UIDNEXT gaps allowed). Retained private evidence is for explicit resolution only. Send tagged `NO` only after this abort is durable. `committed` records complete C/D publication and may return `OK [APPENDUID ...]`; a lost reply is still uncertain to Apple. `indeterminate` records unresolved intent/stages; if completion remains recoverable or cannot be ruled out after B/C, close without tagged APPEND completion. Failure to persist an abort also closes without completion. Keep last-complete-generation fences and affected-write freeze; recovery resolves/finishes uncertain intent, never replays an aborted attempt or rolls back acknowledged work. > > A retried APPEND is a fresh wire command/UID on admission, never an old APPENDUID. Match §3's scoped receipt plus proven base/operation/source evidence to deduplicate only the canonical action; MIME/hash/UUID alone cannot prove a retry. Reconcile raw duplicates to the one canonical note at fresh UIDs. Without proof, retain a proposal against the claimed identity, not a second canonical note or guessed overwrite; a subsequent proven genuine edit remains a new action. > > **Capacity rejection is `NO [OVERQUOTA] quota-full` only after durable abort**: logical quota, staging, physical disk/WAL or Home. Otherwise use the uncertain/connection-close path above. Cause/paths/disk-full go only to admins; other unsafe/limit rejections follow the same abort rule. Never claim retention of undurable input. E2 — §1 slice-0 native reconciliation/recovery observations and branch decisions (quoted): > **Slice-0 adversarial reconciliation observations (same wire/MIME and UI evidence for every case):** > > 1. Admit exact raw MIME at UID R, expunge R and publish canonical MIME at C; make a second ordinary edit, then reconnect. Record surviving base fields, retry APPENDs and cleanup UIDs on every connection. This decides whether raw-receipt/canonical-republication and its base/cleanup bindings can ship. > 2. Admit a conflicting/unknown-base edit as a private proposal, then stale cleanup; separately refuse canonical deletion and republish after raw expunge. On two clients record quiescence versus repeated saves/deletes, access to pending text, and a subsequent genuine delete. This decides replacement/proposal recovery and native-delete enablement. > 3. Compare definitive quota `NO` with dropped APPEND completion at B/C/D, including crash/repair/reconnect. Record Apple's retries, retained local text, cleanup targets and next genuine edit; verify one canonical note/action or an explicit unresolved proposal. This decides whether §4's rejected/uncertain paths preserve usable native recovery. > 4. Reset UIDVALIDITY with active and offline clients, reusing numeric UIDs; while offline also change/edit/delete canonical content on the server. Record reselect/refetch, local pending-text recovery, stale cleanup and subsequent ordinary edit/delete. This decides epoch-reset safety and offline reconciliation. > 5. During this spike, run repeated automatic ordinary-text edits after long histories/offline reconnect with retained tokens, then stripped tokens and more than 64 plausible bases. Record actual base fields, candidate counts, wire/MIME and UI outcomes. Over-budget stripped-token edits must become proposals; the chosen shipping path must pass ordinary automatic edits. This decides token versus complete bounded-ledger feasibility before mapper/ledger commitments; conflict-only editing fails two-way acceptance. > > For each branch, no proven convergence/pending-edit recoverability means defer the affected write/delete path or stop the two-way milestone; guessed cleanup or new heuristics cannot replace evidence. Matching §3 command/failure rows now distinguish definitive rejection from lost completion. §4 failure rows cover durable abort before NO, B/C uncertainty, no replay after rejection, lost-reply recovery followed by a fresh operation, raw/canonical convergence, epoch resets and offline history budgets. Validation: `git diff --check` passed; only `docs/plans/0543-imap-notes.md` changed; the existing five page sections/four explicit page breaks are retained. Native/scale gates remain planned experiments, not claimed results.

kayg commented

2026-10-30 13:03:06 +00:00

Author

Owner

Slice 0 preflight on 6by32-compute01 is blocked; no native experiment ran. Read the approved plan at 0816b5ff72cec14b56e39eb0fbc24803f9788f21, both reviews and their responses before preflight.

Findings: docs/plans/766-slice0-findings.md, branch spike/imap-766-s0, commit 2b16363d.

macos-01-std.vpn.kayg.org resolves to 10.69.69.216; a FQDN TCP/22 probe returned SSH-2.0-NetBird-SSH-Server-0.82.0. This establishes listener reachability only. The inspected plan/current dev docs contain no macOS SSH username; no login was guessed and no authenticated SSH command was attempted. This is not an observed VM-to-VM SSH failure.
op is absent from PATH and ~/.config/1password/calternal.env is missing. The vault was not queried; whether a suitable disposable account exists remains unknown.
The findings answer each of the five baseline and five E2 adversarial groups as unrun/unknown, with consequences and exact resumption prerequisites. No mapper/receipt/token/delete/convergence decision is supported. Existing E1/E2 and native release gates remain unchanged.

Needed to resume: documented macOS SSH login and access to the specified 1Password setup on the execution host (or control-host execution with those available). The document specifies the disposable trusted-TLS IMAP account and controllable harness/two-client requirements; no secret should be supplied in comments.

Validation: staged git diff --check passed; only the findings document was added, based directly on the approved plan head. Branch pushed. No Rust builds, product changes, Mac lock/resources, GUI login/logout, trust/certificate changes, merge or deployment.

Slice 0 preflight on **6by32-compute01** is blocked; **no native experiment ran**. Read the approved plan at `0816b5ff72cec14b56e39eb0fbc24803f9788f21`, both reviews and their responses before preflight. Findings: [docs/plans/766-slice0-findings.md](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0/docs/plans/766-slice0-findings.md), branch `spike/imap-766-s0`, commit `2b16363d`. - `macos-01-std.vpn.kayg.org` resolves to `10.69.69.216`; a FQDN TCP/22 probe returned `SSH-2.0-NetBird-SSH-Server-0.82.0`. This establishes listener reachability only. The inspected plan/current dev docs contain no macOS SSH username; no login was guessed and no authenticated SSH command was attempted. This is **not** an observed VM-to-VM SSH failure. - `op` is absent from PATH and `~/.config/1password/calternal.env` is missing. The vault was not queried; whether a suitable disposable account exists remains unknown. - The findings answer each of the five baseline and five E2 adversarial groups as unrun/unknown, with consequences and exact resumption prerequisites. No mapper/receipt/token/delete/convergence decision is supported. Existing E1/E2 and native release gates remain unchanged. Needed to resume: documented macOS SSH login and access to the specified 1Password setup on the execution host (or control-host execution with those available). The document specifies the disposable trusted-TLS IMAP account and controllable harness/two-client requirements; no secret should be supplied in comments. Validation: staged `git diff --check` passed; only the findings document was added, based directly on the approved plan head. Branch pushed. No Rust builds, product changes, Mac lock/resources, GUI login/logout, trust/certificate changes, merge or deployment.

kayg commented

2026-10-30 13:09:03 +00:00

Author

Owner

Slice 0 control-host continuation: blocked on a suitable disposable IMAP account; no native experiment ran.

Continued spike/imap-766-s0 from 2b16363d; pushed documentation-only commit 7b02b7b. Updated findings contain metadata-only commands/output, all ten baseline/E2 groups marked unrun, and exact resumption requirements.

The specified 1Password setup now works. Apps / Calternal returned four items: YNAB API KEY, root@calternal-cloud, Unsplash API Key, and Apple Certificates. None is identified as a disposable IMAP/Notes test account. The cloud-server login is not authorization to experiment on a mail account. Stopped at the requested no-account gate; no credential values were printed or saved.

Needed: an explicitly disposable Apple Notes-capable IMAP account in that vault, separate from personal mail, with hostname, TLS port, username/password, Notes mailbox namespace and any required submission/profile details. TLS must validate against existing guest trust. The full spike also needs a controllable endpoint/harness for every-connection traces, MIME/base injection, capability variation, raw/canonical reconciliation, quota failures, dropped completions and UIDVALIDITY resets; two-client cases need a second real Apple installation. Put secrets only in 1Password.

The guest login/direct-IP route and Studio fallback are now supplied, superseding the old missing-login prerequisite, but neither was attempted because account preflight stopped first. No Mac lock/resources, SSH, GUI interaction, trust/certificate changes, Studio filesystem access, /mnt/mac access, product code, merge or deployment. Neither osascript nor cua-driver was tested. All feasibility gates remain unresolved.

Validation: git diff --check passed; only docs/plans/766-slice0-findings.md changed. No product builds/tests were needed for this documentation update.

Slice 0 control-host continuation: **blocked on a suitable disposable IMAP account; no native experiment ran**. Continued `spike/imap-766-s0` from `2b16363d`; pushed documentation-only commit `7b02b7b`. [Updated findings](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0/docs/plans/766-slice0-findings.md) contain metadata-only commands/output, all ten baseline/E2 groups marked unrun, and exact resumption requirements. The specified 1Password setup now works. `Apps / Calternal` returned four items: YNAB API KEY, root@calternal-cloud, Unsplash API Key, and Apple Certificates. None is identified as a disposable IMAP/Notes test account. The cloud-server login is not authorization to experiment on a mail account. Stopped at the requested no-account gate; no credential values were printed or saved. Needed: an explicitly disposable Apple Notes-capable IMAP account in that vault, separate from personal mail, with hostname, TLS port, username/password, Notes mailbox namespace and any required submission/profile details. TLS must validate against existing guest trust. The full spike also needs a controllable endpoint/harness for every-connection traces, MIME/base injection, capability variation, raw/canonical reconciliation, quota failures, dropped completions and UIDVALIDITY resets; two-client cases need a second real Apple installation. Put secrets only in 1Password. The guest login/direct-IP route and Studio fallback are now supplied, superseding the old missing-login prerequisite, but neither was attempted because account preflight stopped first. No Mac lock/resources, SSH, GUI interaction, trust/certificate changes, Studio filesystem access, /mnt/mac access, product code, merge or deployment. Neither osascript nor cua-driver was tested. All feasibility gates remain unresolved. Validation: `git diff --check` passed; only `docs/plans/766-slice0-findings.md` changed. No product builds/tests were needed for this documentation update.

kayg commented

2026-10-30 13:14:30 +00:00

Author

Owner

Disposable-endpoint continuation of slice 0: blocked before any native IMAP experiment. Pushed documentation-only commit 349ad3dd0e13a06e9da5491be23cdf3e378806d8 to spike/imap-766-s0.

Updated findings record commands/errors, cleanup, all ten groups still unrun, and resumption prerequisites. The vault-account prerequisite is superseded by the authorized disposable endpoint.

Server reached only as agent@6by32-compute01.vpn.kayg.org. Wildcard cert/key files exist with mode 0600. Podman is absent from PATH; Nix supplied Podman 5.8.9. Rootless initialization fails because privileged newuidmap/newgidmap helpers are missing. Adding Nix shadow to PATH instead fails newuidmap: write to uid_map failed: Operation not permitted, with Podman's setuid/filecap diagnostic. Subordinate UID/GID ranges exist. sudo -n true requires a password. An isolated unprivileged namespace probe reports Rootless=false; it was not used to deploy Dovecot.
Direct guest SSH rejects a changed host key; the authorized Studio/lume fallback works (macOS 27.0.1, build 26A434). BM1 holds /tmp/mac-bench.lock: calternal bm1-756 2026-10-30T13:08:30Z 90 (18:38:30 IST). Its lock was left intact. Waiting stopped once the independent server blocker was established.
T3 device tools work for calternal-01-std (iOS 27.0), but its full installed-app list contains no Notes or Mail; Settings search also finds no Notes. Multi-client cases need another Notes installation.

Needed next: install working privileged UID/GID mapping helpers on the Debian VM (normally uidmap), verify Podman runs rootless as agent, then provision/test trusted TLS and rawlogging before acquiring the Mac lock. No mapper/receipt/delete/recovery feasibility gate has passed.

No container/image/mailbox/account/profile/credential file was created; no IMAP rawlogs exist because no session started. The isolated Podman probe directory was removed. Nix shared-store downloads remain cached. No trust changes, GUI login/logout, product code, merge or deployment. No /mnt/mac access or direct Studio /Users shell access beyond the specified lume binary; T3 managed its own simulator runner/session artifacts. Validation: staged git diff --check passed; only the findings doc changed; remote branch SHA verified.

Disposable-endpoint continuation of slice 0: **blocked before any native IMAP experiment**. Pushed documentation-only commit `349ad3dd0e13a06e9da5491be23cdf3e378806d8` to `spike/imap-766-s0`. [Updated findings](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0/docs/plans/766-slice0-findings.md) record commands/errors, cleanup, all ten groups still unrun, and resumption prerequisites. The vault-account prerequisite is superseded by the authorized disposable endpoint. - Server reached only as `agent@6by32-compute01.vpn.kayg.org`. Wildcard cert/key files exist with mode 0600. Podman is absent from PATH; Nix supplied Podman 5.8.9. Rootless initialization fails because privileged `newuidmap`/`newgidmap` helpers are missing. Adding Nix shadow to PATH instead fails `newuidmap: write to uid_map failed: Operation not permitted`, with Podman's setuid/filecap diagnostic. Subordinate UID/GID ranges exist. `sudo -n true` requires a password. An isolated unprivileged namespace probe reports `Rootless=false`; it was not used to deploy Dovecot. - Direct guest SSH rejects a changed host key; the authorized Studio/lume fallback works (macOS 27.0.1, build 26A434). BM1 holds `/tmp/mac-bench.lock`: `calternal bm1-756 2026-10-30T13:08:30Z 90` (18:38:30 IST). Its lock was left intact. Waiting stopped once the independent server blocker was established. - T3 device tools work for `calternal-01-std` (iOS 27.0), but its full installed-app list contains no Notes or Mail; Settings search also finds no Notes. Multi-client cases need another Notes installation. Needed next: install working privileged UID/GID mapping helpers on the Debian VM (normally `uidmap`), verify Podman runs rootless as `agent`, then provision/test trusted TLS and rawlogging before acquiring the Mac lock. No mapper/receipt/delete/recovery feasibility gate has passed. No container/image/mailbox/account/profile/credential file was created; no IMAP rawlogs exist because no session started. The isolated Podman probe directory was removed. Nix shared-store downloads remain cached. No trust changes, GUI login/logout, product code, merge or deployment. No `/mnt/mac` access or direct Studio `/Users` shell access beyond the specified lume binary; T3 managed its own simulator runner/session artifacts. Validation: staged `git diff --check` passed; only the findings doc changed; remote branch SHA verified.

kayg commented

2026-10-30 17:27:27 +00:00

Author

Owner

Slice 0 (macOS spike) ran partly and is now paused at the orchestrator's request. Pushed f490ae6 to spike/imap-766-s0 (docs only): findings, redacted evidence. No product code, no merge.

Setup that worked: stock Dovecot 2.4.5 run as agent on 6by32-compute01 (no containers), IMAPS on 9993 with the wildcard cert. The macOS 26.5.2 guest (Notes 4.13) trusted it with no trust changes. Full rawlog plus a TLS tap for fault injection. Only one Apple client existed; two-client cases were not run.

Answered (with wire evidence)

Every Apple save = APPEND of the full MIME, then UID STORE \Deleted + UID EXPUNGE of the last UID Apple knows. Moves are APPEND(dest) + STORE/EXPUNGE(src), not UID MOVE. Delete = STORE/EXPUNGE with no trash copy.
Injected HTTP headers are dropped on the first Apple save; HTML attributes and comments survive (but get duplicated on block split / emptied on clear). display:none text leaks into the Notes list preview.
Attachments: binary types round-trip byte-exact (PDF comes back QP, CIDs and order change every save). text/* is rewritten (LF→CRLF, CRLF→CRCRLF) and empty files become AppleDouble.
E2-1: raw R → canonical C converged over two ordinary edits and a reconnect. E2-2: refused delete → the note reappears, no Apple-side loop, a genuine second delete works.
E2-3: definitive NO [OVERQUOTA] → Apple keeps the text locally (survives restart), does no cleanup, and sends the whole pending text with the next edit. A lost APPEND/STORE completion leaves permanent same-UUID duplicates / \Deleted leftovers that Apple never cleans.
E2-4 (offline variant): Apple does not check UIDVALIDITY before cleanup. After an epoch reset with a reused numeric UID, its UID STORE/EXPUNGE 27 deleted an unrelated note. The plan needs numeric UIDs never reused across epochs.
Apple opens ~7 connections at account load (plan ceiling is 8/Installation). It works with CONDSTORE/QRESYNC stripped (plain SELECT, NOOP polling). The setup form ignores host:port, needs an Outgoing server filled in, and never contacted my SMTP stub.

Not answered: nested-structure/10k-line round trips, UIDPLUS-off / IDLE-off / Unicode folders / literal grammar / revocation, hide + Spotlight + Siri, copy, undo, attachment removal, any two-client case, E2-2 proposal + stale cleanup, E2-3 EXPUNGE-drop and crash/repair, E2-4 active client, E2-5 (tokens vs ledger, >64 bases) entirely. No feasibility gate is closed by this run. Section 6 of the findings lists the 13 plan changes the observations support; none are applied to the plan doc.

Cleanup (verified): test account deleted in System Settings (Accounts DB holds only the pre-existing rows; no keychain items); Notes/Mail/Reminders/Settings quit; Mac job dir removed. On 6by32-compute01 all four imap-766* units stopped, ports 9993/9995/9465 closed, ~/imap-766 and temp files deleted. /tmp/mac-bench.lock released at 17:26:16Z (22:56 IST) and the dir is gone. Left behind: the shared Nix store copy of Dovecot; the lume user's one-time Automation approval for Notes (user scope); Mail's first-run privacy choice; control-host scratch in /tmp/imap766 (scripts + screenshots, no secrets).

Resume: the scripts in evidence/tools/ rebuild the rig (needs lib.py, a ~25-line imaplib helper that connects by IP with SNI because the server's local DNS is flaky). Next best slice: E2-5, a fast-second-edit race for E2-1, then the two-client cases with a scripted second imaplib connection.

Slice 0 (macOS spike) **ran partly and is now paused** at the orchestrator's request. Pushed `f490ae6` to `spike/imap-766-s0` (docs only): [findings](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0/docs/plans/766-slice0-findings.md), [redacted evidence](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0/docs/plans/766-slice0-evidence). No product code, no merge. **Setup that worked:** stock Dovecot 2.4.5 run as `agent` on 6by32-compute01 (no containers), IMAPS on 9993 with the wildcard cert. The macOS 26.5.2 guest (Notes 4.13) trusted it with **no trust changes**. Full rawlog plus a TLS tap for fault injection. Only one Apple client existed; two-client cases were not run. **Answered (with wire evidence)** - Every Apple save = APPEND of the full MIME, then `UID STORE \Deleted` + `UID EXPUNGE` of the **last UID Apple knows**. Moves are APPEND(dest) + STORE/EXPUNGE(src), **not UID MOVE**. Delete = STORE/EXPUNGE with no trash copy. - **Injected HTTP headers are dropped on the first Apple save; HTML attributes and comments survive** (but get duplicated on block split / emptied on clear). `display:none` text leaks into the Notes list preview. - Attachments: binary types round-trip byte-exact (PDF comes back QP, CIDs and order change every save). `text/*` is rewritten (LF→CRLF, CRLF→CRCRLF) and empty files become AppleDouble. - E2-1: raw R → canonical C converged over two ordinary edits and a reconnect. E2-2: refused delete → the note reappears, no Apple-side loop, a genuine second delete works. - E2-3: definitive `NO [OVERQUOTA]` → Apple keeps the text locally (survives restart), does no cleanup, and sends the whole pending text with the next edit. A lost APPEND/STORE completion leaves **permanent same-UUID duplicates / `\Deleted` leftovers** that Apple never cleans. - **E2-4 (offline variant): Apple does not check UIDVALIDITY before cleanup.** After an epoch reset with a reused numeric UID, its `UID STORE/EXPUNGE 27` deleted an unrelated note. The plan needs numeric UIDs never reused across epochs. - Apple opens ~7 connections at account load (plan ceiling is 8/Installation). It works with CONDSTORE/QRESYNC stripped (plain SELECT, NOOP polling). The setup form ignores `host:port`, needs an Outgoing server filled in, and never contacted my SMTP stub. **Not answered:** nested-structure/10k-line round trips, UIDPLUS-off / IDLE-off / Unicode folders / literal grammar / revocation, hide + Spotlight + Siri, copy, undo, attachment removal, any two-client case, E2-2 proposal + stale cleanup, E2-3 EXPUNGE-drop and crash/repair, E2-4 active client, **E2-5 (tokens vs ledger, >64 bases) entirely**. No feasibility gate is closed by this run. Section 6 of the findings lists the 13 plan changes the observations support; none are applied to the plan doc. **Cleanup (verified):** test account deleted in System Settings (Accounts DB holds only the pre-existing rows; no keychain items); Notes/Mail/Reminders/Settings quit; Mac job dir removed. On 6by32-compute01 all four `imap-766*` units stopped, ports 9993/9995/9465 closed, `~/imap-766` and temp files deleted. `/tmp/mac-bench.lock` released at 17:26:16Z (22:56 IST) and the dir is gone. Left behind: the shared Nix store copy of Dovecot; the lume user's one-time Automation approval for Notes (user scope); Mail's first-run privacy choice; control-host scratch in `/tmp/imap766` (scripts + screenshots, no secrets). Resume: the scripts in `evidence/tools/` rebuild the rig (needs `lib.py`, a ~25-line imaplib helper that connects by IP with SNI because the server's local DNS is flaky). Next best slice: E2-5, a fast-second-edit race for E2-1, then the two-client cases with a scripted second `imaplib` connection.

kayg referenced this pull request

2026-10-30 17:29:17 +00:00

Handover: Calternal orchestration paused 2026-10-30 (state, branches, open questions) #774

kayg referenced this pull request

2026-10-31 05:05:40 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg commented

2026-10-31 21:34:00 +00:00

Author

Owner

Slice 0 continuation (2026-10-31 UTC)

Case	Behavior and transcript excerpt	Design implication
E2-5 stripped-token reconnect	Ten raw and ten canonical saves completed; all ten edits remained in Notes. R1–R10 had no `cal-base` comment or `data-cal` carriers. The cut fired before edit 5's cleanup `UID STORE 341`; after reconnect, edit 6 appended UID 344 and cleaned canonical UID 343. The test actor left live canonicals 341 and 353 because it does not supersede earlier copies.	Without a body token, cleanup UID alone is not a base proof. Keep retained bounded history or return a proposal when ambiguous. This ten-edit reconnect does not close the 70-edit gate.
E2-1b	Part A (earlier run): naive mode left duplicates at 6/8 offsets; supersede mode left one copy at all 8. Part B at 4 s / 1 s missed the race: Apple sent R2 after C1. At 12 s / 1 s, Apple removed R1 UID 363 while the actor waited; `UID STORE 363` and `UID EXPUNGE 363` returned `OK`, the actor skipped R1 and published R2 UID 364 as C365. Notes showed both edits and one live copy. The 0.3 s / 0.35 s attempt also missed the R1 target: Apple cleaned C1 UID 368 after R2 UID 369.	A cleanup of an already-vanished UID is a harmless no-op. Reconcile the later raw save and explicitly collapse earlier same-UUID copies. The quick-gap attempt did not exercise cleanup of R1.
T1 (earlier run)	Mac A edited after B while selected. B appended UID 313 and cleaned UID 312 (`STORE` / `EXPUNGE` both `OK`). Final Notes-316 contained `P1 by B` and `A1`, with one live copy.	A selected client must incorporate B's edit before its next full-MIME save.
T2 (rerun)	With A offline, B appended UID 372 and cleaned 371 (`OK`); actor published C373. On reconnect, Mac appended R374 and again cleaned old UID 371 (`OK`); actor published C375. After five minutes, live UIDs 373 and 375 held disjoint `P2 by B` and `P1 by A`; Notes showed only A's text and no proposal.	Fail. Preserve concurrent text as an explicit proposal; do not expose two ordinary copies or lose B's edit from the Mac view.
T3 (earlier run)	B deleted UID 325 while A was offline. On reconnect, Mac returned raw UID 326 and the actor published C327 with `P3 by A`; the unrelated synthetic note remained. B's `STORE` / `EXPUNGE` both returned `OK`.	A stale delete must not discard pending text; reconcile and republish it while leaving unrelated notes alone.
T4 (rerun; B is scripted)	Mac delete removed UID 376; actor republished UID 377. B observed it and sent `UID STORE 377` / `UID EXPUNGE 377`, both `OK`. The UUID stayed absent for the five-minute poll; Notes showed zero copies. The actor logged no further republish after B's delete.	No republish loop was observed and the mailbox quiesced. This does not establish true two-Apple quiescence; B is an `imaplib` client, and the actor did not log a second cycle.
E2-4b	Before reset: UIDVALIDITY 1793423367, UIDNEXT 380. Reset only `S0 keep` / `S0 edit` to UIDs 400 / 401; new UIDVALIDITY 1793424367. Mac's offline edit appended UID 402; stale cleanup `UID STORE 379` / `UID EXPUNGE 379` both returned `OK`. Actor superseded 401 and published C403. Final live UIDs were 400 and 403; both notes and the pending `P3 by A` text remained.	Monotonic UIDs make old-epoch cleanup harmless. Collapse the duplicate edited copy after publishing the canonical message.
Change 9	`CAPABILITY False False`; `IFJN3 BAD Unsupported extension`, `IFJN4 BAD Unsupported extension`, `IFJN6 BAD Unsupported extension` for `ENABLE QRESYNC`, `SELECT ... (CONDSTORE)`, and `UID FETCH ... (CHANGEDSINCE 0)`.	Omit unsupported capabilities and explicitly reject their command forms.

Open and skipped items

The retained-token 70-edit gate remains open: the prior run's driver log does not map all planned edits, and R73–R76 lack the comment. The reconnect rerun above was the stripped-token case.
In E2-1b, 4 s / 1 s and 0.3 s / 0.35 s did not hit the intended pre-C cleanup window; the 12 s / 1 s run did.
Mac test-account removal was skipped. The Notes delete account call failed with -10000; System Events returned -1743 (Not authorized to send Apple events to System Events). The Example account's synthetic notes are deleted, but removing the account requires local GUI authorization in System Settings → Internet Accounts. I did not force that over SSH.

Cleanup

On dev-02, Dovecot and the tap are stopped; ports 9145 and 443 are closed. I removed /home/cua/imap-766, including throwaway users and credential files. On the Mac, all synthetic Notes are deleted, /tmp/imap766-job is removed, and the lock-holder shell exited through its EXIT trap. The empty Example test account remains for the GUI-removal reason above. No real mail account was touched.

### Slice 0 continuation (2026-10-31 UTC) | Case | Behavior and transcript excerpt | Design implication | | --- | --- | --- | | E2-5 stripped-token reconnect | Ten raw and ten canonical saves completed; all ten edits remained in Notes. R1–R10 had no `cal-base` comment or `data-cal` carriers. The cut fired before edit 5's cleanup `UID STORE 341`; after reconnect, edit 6 appended UID 344 and cleaned canonical UID 343. The test actor left live canonicals 341 and 353 because it does not supersede earlier copies. | Without a body token, cleanup UID alone is not a base proof. Keep retained bounded history or return a proposal when ambiguous. This ten-edit reconnect does not close the 70-edit gate. | | E2-1b | Part A (earlier run): naive mode left duplicates at 6/8 offsets; supersede mode left one copy at all 8. Part B at 4 s / 1 s missed the race: Apple sent R2 after C1. At 12 s / 1 s, Apple removed R1 UID 363 while the actor waited; `UID STORE 363` and `UID EXPUNGE 363` returned `OK`, the actor skipped R1 and published R2 UID 364 as C365. Notes showed both edits and one live copy. The 0.3 s / 0.35 s attempt also missed the R1 target: Apple cleaned C1 UID 368 after R2 UID 369. | A cleanup of an already-vanished UID is a harmless no-op. Reconcile the later raw save and explicitly collapse earlier same-UUID copies. The quick-gap attempt did not exercise cleanup of R1. | | T1 (earlier run) | Mac A edited after B while selected. B appended UID 313 and cleaned UID 312 (`STORE` / `EXPUNGE` both `OK`). Final Notes-316 contained `P1 by B` and `A1`, with one live copy. | A selected client must incorporate B's edit before its next full-MIME save. | | T2 (rerun) | With A offline, B appended UID 372 and cleaned 371 (`OK`); actor published C373. On reconnect, Mac appended R374 and again cleaned old UID 371 (`OK`); actor published C375. After five minutes, live UIDs 373 and 375 held disjoint `P2 by B` and `P1 by A`; Notes showed only A's text and no proposal. | **Fail.** Preserve concurrent text as an explicit proposal; do not expose two ordinary copies or lose B's edit from the Mac view. | | T3 (earlier run) | B deleted UID 325 while A was offline. On reconnect, Mac returned raw UID 326 and the actor published C327 with `P3 by A`; the unrelated synthetic note remained. B's `STORE` / `EXPUNGE` both returned `OK`. | A stale delete must not discard pending text; reconcile and republish it while leaving unrelated notes alone. | | T4 (rerun; B is scripted) | Mac delete removed UID 376; actor republished UID 377. B observed it and sent `UID STORE 377` / `UID EXPUNGE 377`, both `OK`. The UUID stayed absent for the five-minute poll; Notes showed zero copies. The actor logged no further republish after B's delete. | No republish loop was observed and the mailbox quiesced. This does not establish true two-Apple quiescence; B is an `imaplib` client, and the actor did not log a second cycle. | | E2-4b | Before reset: UIDVALIDITY 1793423367, UIDNEXT 380. Reset only `S0 keep` / `S0 edit` to UIDs 400 / 401; new UIDVALIDITY 1793424367. Mac's offline edit appended UID 402; stale cleanup `UID STORE 379` / `UID EXPUNGE 379` both returned `OK`. Actor superseded 401 and published C403. Final live UIDs were 400 and 403; both notes and the pending `P3 by A` text remained. | Monotonic UIDs make old-epoch cleanup harmless. Collapse the duplicate edited copy after publishing the canonical message. | | Change 9 | `CAPABILITY False False`; `IFJN3 BAD Unsupported extension`, `IFJN4 BAD Unsupported extension`, `IFJN6 BAD Unsupported extension` for `ENABLE QRESYNC`, `SELECT ... (CONDSTORE)`, and `UID FETCH ... (CHANGEDSINCE 0)`. | Omit unsupported capabilities and explicitly reject their command forms. | ### Open and skipped items - The retained-token 70-edit gate remains open: the prior run's driver log does not map all planned edits, and R73–R76 lack the comment. The reconnect rerun above was the stripped-token case. - In E2-1b, 4 s / 1 s and 0.3 s / 0.35 s did not hit the intended pre-C cleanup window; the 12 s / 1 s run did. - Mac test-account removal was skipped. The Notes `delete account` call failed with `-10000`; System Events returned `-1743` (`Not authorized to send Apple events to System Events`). The `Example` account's synthetic notes are deleted, but removing the account requires local GUI authorization in System Settings → Internet Accounts. I did not force that over SSH. ### Cleanup On dev-02, Dovecot and the tap are stopped; ports 9145 and 443 are closed. I removed `/home/cua/imap-766`, including throwaway users and credential files. On the Mac, all synthetic Notes are deleted, `/tmp/imap766-job` is removed, and the lock-holder shell exited through its EXIT trap. The empty `Example` test account remains for the GUI-removal reason above. No real mail account was touched.

kayg referenced this pull request

2026-10-31 21:40:04 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg referenced this pull request

2026-10-31 21:43:22 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg commented

2026-10-31 22:25:36 +00:00

Author

Owner

Fresh-mailbox recovery: scripted/native status (2026-10-31 UTC)

Three variants were exercised with scripted IMAP clients against disposable Dovecot 2.4.5. No Notes.app account was configured. See the findings note, redacted transcript, and summary.

Variant	Observation	Plan implication
Fresh mailbox beside old	`LIST` showed both names and all three UUIDs appeared in both. The fresh name entered `LSUB` only after explicit `SUBSCRIBE`. A scripted pending-edit APPEND to the old mailbox left the fresh copy at its base text.	Do not assume Notes discovers/subscribes to the new name or transfers local pending edits; duplicate folders/notes remain a native question.
Rename old to retired	New `SELECT` of the old name returned `NO [NONEXISTENT]`. A session already selected on it got `NO [NONEXISTENT]` for stale `STORE`, then its `UID EXPUNGE` failed with a relogin/session-state error. Deleting the retired name left the fresh server mailbox’s three notes intact.	Keep old names retired, without aliasing them to the fresh mailbox. This server-side result does not establish safe Notes cache removal.
Delete old, then replay stale UID after selecting fresh	The old name returned `NO [NONEXISTENT]`. In a writable fresh selection, replaying old UID 2 as UID 2 returned `OK` for `STORE`/`EXPUNGE` and removed a different synthetic note assigned UID 2 in the fresh mailbox.	A stale numeric UID can remove a different note if sent in the fresh UID namespace. Do not redirect old names; keep migration/recovery gated.

Native Notes behavior remains unknown: no Notes folders, duplicate presentation, re-download, stale-UID action, or local-only offline edit was observed. Setup stopped before account creation because temporary CA trust configuration returned SecTrustSettingsSetTrustSettings: The authorization was denied since no user interaction was possible. Keychain Access launched without a visible window. A scripted pending-edit MIME explicitly APPENDed to the fresh mailbox remained beside the base message, but this does not test local Notes cache handoff. The native recovery gate in round-2 review remains open; an interactive Keychain authorization path (or pre-trusted test CA) is needed to rerun it.

### Fresh-mailbox recovery: scripted/native status (2026-10-31 UTC) Three variants were exercised with scripted IMAP clients against disposable Dovecot 2.4.5. No Notes.app account was configured. See the [findings note](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0-cases/docs/plans/766-slice0-evidence/fresh-mailbox/findings.md), [redacted transcript](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0-cases/docs/plans/766-slice0-evidence/fresh-mailbox/fresh-mailbox-scripted-transcript.redacted.txt), and [summary](https://git.kayg.org/kayg/calternal/src/branch/spike/imap-766-s0-cases/docs/plans/766-slice0-evidence/fresh-mailbox/fresh-mailbox-scripted-summary.json). | Variant | Observation | Plan implication | | --- | --- | --- | | Fresh mailbox beside old | `LIST` showed both names and all three UUIDs appeared in both. The fresh name entered `LSUB` only after explicit `SUBSCRIBE`. A scripted pending-edit APPEND to the old mailbox left the fresh copy at its base text. | Do not assume Notes discovers/subscribes to the new name or transfers local pending edits; duplicate folders/notes remain a native question. | | Rename old to retired | New `SELECT` of the old name returned `NO [NONEXISTENT]`. A session already selected on it got `NO [NONEXISTENT]` for stale `STORE`, then its `UID EXPUNGE` failed with a relogin/session-state error. Deleting the retired name left the fresh server mailbox’s three notes intact. | Keep old names retired, without aliasing them to the fresh mailbox. This server-side result does not establish safe Notes cache removal. | | Delete old, then replay stale UID after selecting fresh | The old name returned `NO [NONEXISTENT]`. In a writable fresh selection, replaying old UID 2 as UID 2 returned `OK` for `STORE`/`EXPUNGE` and removed a different synthetic note assigned UID 2 in the fresh mailbox. | A stale numeric UID can remove a different note if sent in the fresh UID namespace. Do not redirect old names; keep migration/recovery gated. | Native Notes behavior remains unknown: no Notes folders, duplicate presentation, re-download, stale-UID action, or local-only offline edit was observed. Setup stopped before account creation because temporary CA trust configuration returned `SecTrustSettingsSetTrustSettings: The authorization was denied since no user interaction was possible.` Keychain Access launched without a visible window. A scripted pending-edit MIME explicitly APPENDed to the fresh mailbox remained beside the base message, but this does not test local Notes cache handoff. The native recovery gate in [round-2 review](https://git.kayg.org/kayg/calternal/pulls/775#issuecomment-71831) remains open; an interactive Keychain authorization path (or pre-trusted test CA) is needed to rerun it.

kayg referenced this pull request

2026-10-31 22:25:36 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg commented

2026-11-01 06:28:11 +00:00

Author

Owner

GUI prerequisites on macos-01-std are done (2026-11-01, IST morning):

Automation (Apple Events) allowed for the ssh login process (it shows up as "NetBird"): System Events, Contacts (control and data access), Mail, Notes. Checked over ssh with no prompt: System Events and Notes, Contacts and Mail scripts all return values.
Test CA trust: no fixed CA file exists. The CA is created fresh per run (mkcert.sh), and slice 0 used a public wildcard cert. Trusting needs no GUI: ~/bin/trust-test-ca.sh <ca.pem> on the Mac (sudo security add-trusted-cert -d -r trustRoot -p ssl -k /Library/Keychains/System.keychain). Proved with a throwaway CA: verify-cert NOT_TRUSTED before, "successful" after; removed again afterwards. Remove a run's CA at the end with sudo security remove-trusted-cert -d ca.pem.
The old slice-0 account ("Example", user imap766/t1) is deleted: Internet Accounts says "No accounts". A leftover certificate warning dialog in Notes was cancelled.

Screen Sharing is enabled on the VM. Reconnect: GUI clicks are easiest through the guest's cua-spacesd (MCP at 127.0.0.1:3211, token in ~/.cua/spacesd/token). The VNC framebuffer showed black, so VNC is not useful here.

GUI prerequisites on `macos-01-std` are done (2026-11-01, IST morning): 1. Automation (Apple Events) allowed for the ssh login process (it shows up as "NetBird"): System Events, Contacts (control and data access), Mail, Notes. Checked over ssh with no prompt: System Events and Notes, Contacts and Mail scripts all return values. 2. Test CA trust: no fixed CA file exists. The CA is created fresh per run (`mkcert.sh`), and slice 0 used a public wildcard cert. Trusting needs no GUI: `~/bin/trust-test-ca.sh <ca.pem>` on the Mac (`sudo security add-trusted-cert -d -r trustRoot -p ssl -k /Library/Keychains/System.keychain`). Proved with a throwaway CA: verify-cert NOT_TRUSTED before, "successful" after; removed again afterwards. Remove a run's CA at the end with `sudo security remove-trusted-cert -d ca.pem`. 3. The old slice-0 account ("Example", user imap766/t1) is deleted: Internet Accounts says "No accounts". A leftover certificate warning dialog in Notes was cancelled. Screen Sharing is enabled on the VM. Reconnect: GUI clicks are easiest through the guest's cua-spacesd (MCP at 127.0.0.1:3211, token in ~/.cua/spacesd/token). The VNC framebuffer showed black, so VNC is not useful here.

kayg commented

2026-11-01 06:57:19 +00:00

Author

Owner

Native fresh-mailbox recovery evidence, 1 November 2026

Pushed fbf69c2f2622ccd91f3f21049ce1c16f6d194f11 to docs/imap-766-plan-v2 (plan PR #775). Native report and redacted evidence; the parent evidence README is updated.

One real macOS 26.5.2 / Notes 4.13 client against disposable Dovecot 2.4.5 on dev-02, approximately 12:07–12:23 IST. Three synthetic UUIDs and throwaway credentials only. Scripted server transitions, native Apple saves/sync/cache observations; no Calternal reconciliation actor or second Apple client.

Native scenario	Observed result
Add account and sync	Internet Accounts setup succeeded with Notes enabled. Three notes downloaded. A native edit APPENDed successfully and cleaned old UID 1.
Fresh root mailbox alongside old, UIDs 1001–1003	Explicitly subscribed `Notes-766-active` was not discovered/downloaded, even after Notes relaunch. Notes showed one folder/three notes and listed `Notes/*`, not arbitrary root names.
Rename old to retired, then delete retired	Notes continued SELECTing old `Notes`, received `NO [NONEXISTENT]`, and kept its cached three notes and prior active edit. Fresh copies/bystanders were unchanged. Server removal did not migrate or remove the cached folder.
Same-name active reset with higher UIDs	New UIDVALIDITY and 10001–10003 re-downloaded after relaunch; another active reset to 20001–20003 reselected/fetched after the selected connection was explicitly kicked. Three notes, unchanged bystanders. This is a controlled complete-history probe, not an incomplete-history migration.
Offline edit, same-name reset to 30001–30003	Notes received the new UIDVALIDITY, APPENDed pending A as 30004, then sent `UID STORE 20002` / `UID EXPUNGE 20002`. Old cleanup was a no-op and bystanders survived. Server had four messages, including base/pending A with the same UUID; Notes showed three and exposed base A rather than the uploaded pending text. Raw pending text was retained, but hidden from this native view.
Offline edit, new root name at 40001–40003; old retired/deleted	Local-only text remained in cached old `Notes`, including after retired-folder deletion and Notes relaunch. Notes retried old SELECT/STATUS, never selected the fresh name, and did not transfer the pending text. Fresh server mailbox remained three base messages. Final native observation was about two minutes after retirement.

UID conclusion: higher numeric UIDs prevented the measured stale-cleanup collision. Changed UIDVALIDITY did not stop Notes using an old cleanup number, even after its new SELECT response. This supports never-reuse with complete independent issued/reserved-range authority; it does not prove safety for unknown restored history, or native cleanup after selecting a different mailbox name (Notes never selected the fresh roots here). No wrong-note edit or bystander deletion occurred.

Round 3 changes needed: transparent Notes-<generation> root-name recovery fails this client. Keep incomplete-history writes stopped until a tested account re-pairing path exposes the fresh account namespace at the expected mailbox; re-pairing itself is still untested. Keep visible pending-text/conflict reconciliation as an explicit native gate: server retention plus higher UIDs did not preserve visible access after the offline same-name reset. Export/reconcile pending text before account removal, which warns it removes Notes data. Do not close recovery or claim transparent migration from this run.

Validation on dev-02: nix develop -c bash -c 'python3 ~/imap-766/tools/analyze.py && python3 ~/imap-766/tools/export.py' passed. It checked 60 captured bystander instances unchanged (two distinct bystanders), increasing observed UID ranges, the duplicate raw A state, zero native fresh-root selections, and absence of throwaway passwords in evidence/archive members. git diff --check passed; pushed SHA verified. Evidence includes all-connection redacted wire/rawlogs, stored MIME, native folder/body snapshots, screenshots and exact harness/check scripts. No product code tests or native recovery release gate are claimed passing.

Cleanup verified: test account removed; Internet Accounts says No accounts, Notes reports only On My Mac; CA trust/certificate removed and verification no longer trusts it; no test endpoint/user keychain item; Mac tree and EXIT-trap lock gone. Both server units inactive, ports 993/9145 closed, disposable server/credential tree removed. Pending bodies were exported before account deletion. No production service or real account/note touched.

### Native fresh-mailbox recovery evidence, 1 November 2026 Pushed `fbf69c2f2622ccd91f3f21049ce1c16f6d194f11` to `docs/imap-766-plan-v2` ([plan PR #775](https://git.kayg.org/kayg/calternal/pulls/775)). [Native report and redacted evidence](https://git.kayg.org/kayg/calternal/src/commit/fbf69c2f2622ccd91f3f21049ce1c16f6d194f11/docs/plans/766-slice0-evidence/native-fresh-mailbox/README.md); the parent evidence README is updated. One real macOS 26.5.2 / Notes 4.13 client against disposable Dovecot 2.4.5 on dev-02, approximately 12:07–12:23 IST. Three synthetic UUIDs and throwaway credentials only. Scripted server transitions, native Apple saves/sync/cache observations; no Calternal reconciliation actor or second Apple client. | Native scenario | Observed result | | --- | --- | | Add account and sync | Internet Accounts setup succeeded with Notes enabled. Three notes downloaded. A native edit APPENDed successfully and cleaned old UID 1. | | Fresh root mailbox alongside old, UIDs 1001–1003 | Explicitly subscribed `Notes-766-active` was not discovered/downloaded, even after Notes relaunch. Notes showed one folder/three notes and listed `Notes/*`, not arbitrary root names. | | Rename old to retired, then delete retired | Notes continued SELECTing old `Notes`, received `NO [NONEXISTENT]`, and kept its cached three notes and prior active edit. Fresh copies/bystanders were unchanged. Server removal did not migrate or remove the cached folder. | | Same-name active reset with higher UIDs | New UIDVALIDITY and 10001–10003 re-downloaded after relaunch; another active reset to 20001–20003 reselected/fetched after the selected connection was explicitly kicked. Three notes, unchanged bystanders. This is a controlled complete-history probe, not an incomplete-history migration. | | Offline edit, same-name reset to 30001–30003 | Notes received the new UIDVALIDITY, APPENDed pending A as 30004, then sent `UID STORE 20002` / `UID EXPUNGE 20002`. Old cleanup was a no-op and bystanders survived. **Server had four messages, including base/pending A with the same UUID; Notes showed three and exposed base A rather than the uploaded pending text.** Raw pending text was retained, but hidden from this native view. | | Offline edit, new root name at 40001–40003; old retired/deleted | Local-only text remained in cached old `Notes`, including after retired-folder deletion and Notes relaunch. Notes retried old SELECT/STATUS, never selected the fresh name, and did not transfer the pending text. Fresh server mailbox remained three base messages. Final native observation was about two minutes after retirement. | **UID conclusion:** higher numeric UIDs prevented the measured stale-cleanup collision. Changed UIDVALIDITY did not stop Notes using an old cleanup number, even after its new SELECT response. This supports never-reuse with complete independent issued/reserved-range authority; it does not prove safety for unknown restored history, or native cleanup after selecting a different mailbox name (Notes never selected the fresh roots here). No wrong-note edit or bystander deletion occurred. **Round 3 changes needed:** transparent `Notes-<generation>` root-name recovery fails this client. Keep incomplete-history writes stopped until a tested account re-pairing path exposes the fresh account namespace at the expected mailbox; re-pairing itself is still untested. Keep visible pending-text/conflict reconciliation as an explicit native gate: server retention plus higher UIDs did not preserve visible access after the offline same-name reset. Export/reconcile pending text before account removal, which warns it removes Notes data. Do not close recovery or claim transparent migration from this run. Validation on dev-02: `nix develop -c bash -c 'python3 ~/imap-766/tools/analyze.py && python3 ~/imap-766/tools/export.py'` passed. It checked 60 captured bystander instances unchanged (two distinct bystanders), increasing observed UID ranges, the duplicate raw A state, zero native fresh-root selections, and absence of throwaway passwords in evidence/archive members. `git diff --check` passed; pushed SHA verified. Evidence includes all-connection redacted wire/rawlogs, stored MIME, native folder/body snapshots, screenshots and exact harness/check scripts. No product code tests or native recovery release gate are claimed passing. Cleanup verified: test account removed; Internet Accounts says `No accounts`, Notes reports only `On My Mac`; CA trust/certificate removed and verification no longer trusts it; no test endpoint/user keychain item; Mac tree and EXIT-trap lock gone. Both server units inactive, ports 993/9145 closed, disposable server/credential tree removed. Pending bodies were exported before account deletion. No production service or real account/note touched.

kayg referenced this pull request

2026-11-01 07:13:49 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg referenced this pull request

2026-11-01 07:18:48 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg referenced this pull request

2026-11-01 08:11:49 +00:00

feat: persist Apple Notes IMAP UID authority (#766, #775) #824

kayg referenced this pull request

2026-11-01 17:05:42 +00:00

WIP: docs: plan read-only IMAP Notes server (#766) #830

kayg referenced this pull request

2026-11-02 17:32:54 +00:00

feat: add bounded IMAPS authentication (#766) #874

kayg referenced this pull request

2026-11-02 18:12:51 +00:00

feat: serve read-only Apple Notes over IMAPS (#766) #879

kayg referenced this pull request

2026-11-04 13:06:24 +00:00

Open PR triage (Nov 4) #931

kayg referenced this pull request

2026-11-04 18:22:35 +00:00

Cleanup (Nov 4 night) #940

~~kayg referenced this pull request 2026-11-05 23:21:11 +00:00~~

Cleanup 2026-11-06 #978

kayg referenced this pull request

2026-11-05 23:29:33 +00:00

Cleanup 2026-11-06 #978

kayg referenced this pull request

2026-11-07 17:56:31 +00:00

Housekeeping 2026-11-07 #1009

kayg commented

2026-11-07 19:33:41 +00:00

Author

Owner

housekeeping #1009: rebased+green, left open for review.

rebased IMAP plan while retaining proposed recovery decisions.

head: 42a690e21af707b35256b1f411bc998a3d18f5e6. rebased onto dev cd7f84432febfc63b788fbb9b9f9950452982d74 and pushed with the original exact force-with-lease.

validation on the Mac VM, through Nix and the shared Mac lock/slot: root bun install --frozen-lockfile; in apps/web, bun run check, bun run lint, bun run build, bun run test; PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. exact commands, logs, and dev bundle control.

original-head missing-commit/patch inventory, file pr-766.md.

housekeeping #1009: rebased+green, left open for review. rebased IMAP plan while retaining proposed recovery decisions. head: `42a690e21af707b35256b1f411bc998a3d18f5e6`. rebased onto dev `cd7f84432febfc63b788fbb9b9f9950452982d74` and pushed with the original exact force-with-lease. validation on the Mac VM, through Nix and the shared Mac lock/slot: root `bun install --frozen-lockfile`; in `apps/web`, `bun run check`, `bun run lint`, `bun run build`, `bun run test`; `PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate`. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. [exact commands, logs, and dev bundle control](https://git.kayg.org/attachments/64244177-fd81-4992-860e-4d5d22ca01cd). [original-head missing-commit/patch inventory](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5), file `pr-766.md`.

kayg referenced this pull request

2026-11-07 19:33:42 +00:00

docs: revise IMAP Notes plan after slice 0 (#766) #775

kayg referenced this pull request

2026-11-07 21:18:43 +00:00

Housekeeping 2026-11-07 #1009

kayg referenced this pull request

2026-11-08 19:01:27 +00:00

Housekeeping 2026-11-07 #1009

kayg referenced this pull request

2026-11-08 19:04:24 +00:00

Housekeeping 2026-11-07 #1009

kayg referenced this pull request

2026-11-09 18:53:14 +00:00

Housekeeping 2026-11-09 night #1045

CI / server (pull_request) Successful in 1m31s

Details

CI / rust (pull_request) Successful in 15m23s

Details

CI / web (pull_request) Has been cancelled

Details

Merge conflict checking is in progress. Try again in few moments.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin docs/imap-notes-plan:docs/imap-notes-plan

git switch docs/imap-notes-plan

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev

git merge --no-ff docs/imap-notes-plan

git switch docs/imap-notes-plan

git rebase dev

git switch dev

git merge --ff-only docs/imap-notes-plan

git switch docs/imap-notes-plan

git rebase dev

git switch dev

git merge --no-ff docs/imap-notes-plan

git switch dev

git merge --squash docs/imap-notes-plan