kayg/calternal

Watch

Fork 0

Notes over IMAP: adding a line or list item on the Mac creates a '(conflict, Apple Notes)' copy instead of updating the Note #644

New issue

Open

opened 2026-10-01 18:48:04 +00:00 by kayg · 5 comments

kayg commented

2026-10-01 18:48:04 +00:00

Owner

Summary

In Apple Notes (calternal Notes account over the IMAP bridge, #428) a normal edit that adds a line or a list item to a Note does not update the Note. The bridge keeps the original unchanged and creates a new Note titled "

## Summary In Apple Notes (calternal Notes account over the IMAP bridge, #428) a normal edit that **adds a line or a list item** to a Note does not update the Note. The bridge keeps the original unchanged and creates a new Note titled "<title> (conflict, Apple Notes)". On the Mac the original then shows the old text again and the user's text lives only in the copy. Every further edit of that copy makes another copy ("… (conflict, Apple Notes) (conflict, Apple Notes)"). There was no concurrent edit on the calternal side. This is a sync collision, so it blocks a merge per CLAUDE.md. Found in the Apple interop run on the macOS 27 VM, 2026-10-01 (lab server from `dev` at `687ff7031`, profile-installed account, real Notes app, edits typed through the GUI). ## Repro 1. `POST /api/v1/notes {"title":"MDV web note","body":"First line from web\n\n- bullet one\n- bullet **two**\n"}` → 201. The Note appears in Notes on the Mac (folder Notes). 2. In Notes, open the Note, put the cursor at the end of "bullet two", press Return, type "Typed in Notes GUI". 3. About 30 s later: `Notes/20261002-mdv-web-note-594cfa8d.md` is unchanged. A new file `Notes/20261001-mdv-web-note-conflict-apple-notes-a332e9c5.md` holds the edit. Notes shows "MDV web note (conflict, Apple Notes)". 4. Same result for a Note created on the Mac ("MDV mac note"): adding a new paragraph → `…-mdv-mac-note-conflict-apple-notes-6e095605.md`. 5. Control: replacing one word inside an existing paragraph of the same Note (no new line) updates the Note in place (PASS). Evidence: the incoming APPEND for step 2 (saved raw MIME) carries ``` X-Universally-Unique-Identifier: 594CFA8D-295D-4F93-9D0E-220BEC418035 X-Mail-Created-Date: Fri, 2 Oct 2026 05:30:41 +0530 <li>bullet two</li><li>Typed in Notes GUI</li> ``` The echoed date equals a `note_imap_served.served_date` row for that Note, and the served content hash equals the current file hash (checked with b3sum), so the base revision is proven. The conflict therefore comes from `Projection::merge` not returning `Merge::Applied` for an inserted block (`crates/plugins/notes/src/imap.rs`, `append_note`, `conflict = merged.is_none()`). #428 comment noted the same for "a later edit removed a line boundary". ## Expected A proven-base Apple edit that adds, removes or splits paragraphs or list items is applied to the Note (DESIGN §9, #428: Apple edits change body text). A kept copy is only for a real concurrent edit or content the bridge cannot represent, and the user's own edit never "disappears" from the original on the Mac. ## Actual Ordinary typing on the Mac forks the Note on every save that changes the block structure. ## Suggested tests Replay tests with the captured Apple MIME: append a list item, add a paragraph, delete a paragraph, split a paragraph with Return, join two paragraphs. Each must update the original in place with a proven base.

mac-typed-list.png

255 KiB

mac-roundtrip.png

277 KiB

kayg commented

2026-10-01 19:29:29 +00:00

Author

Owner

Started #644, #645, #646 on job/notesbridge-644, base 687ff703136e71e89f8dfba139e93cd0788b25c1 (dev). Read the contract, bridge mapping and Mac lab evidence. First trace: the proven-base gate succeeds, but the text-span projection refuses inserted block boundaries. Task Notes are explicitly excluded and Daily notes are skipped without stable identities. Will fix these separately with replay tests and real-client verification. No push or deployment.

Started #644, #645, #646 on `job/notesbridge-644`, base `687ff703136e71e89f8dfba139e93cd0788b25c1` (dev). Read the contract, bridge mapping and Mac lab evidence. First trace: the proven-base gate succeeds, but the text-span projection refuses inserted block boundaries. Task Notes are explicitly excluded and Daily notes are skipped without stable identities. Will fix these separately with replay tests and real-client verification. No push or deployment.

kayg referenced this issue

2026-10-01 19:29:30 +00:00

Notes over IMAP: bold/italic from Apple Notes stored as raw HTML, then stripped on the next Mac edit #645

kayg referenced this issue

2026-10-01 19:29:31 +00:00

Notes over IMAP: Daily notes and Task Notes never appear in Apple Notes (Journal/Daily Notes always empty) #646

kayg commented

2026-10-01 19:42:21 +00:00

Author

Owner

Root cause confirmed: durable_base accepts the captured echoed Date and current hash, but Projection::merge rejects synthetic block boundaries. New replay tests now apply list insertion and paragraph add/delete/split/join while preserving unchanged Markdown. Projection slice committed as d4b7c9039; cargo test -p calternal-imap passed (all suites). APPEND now uses that handler and saves hash-checked served bases for disjoint three-way edits; Notes crate validation is running. Inline aliases are normalized to Markdown instead of raw /.

Root cause confirmed: `durable_base` accepts the captured echoed Date and current hash, but `Projection::merge` rejects synthetic block boundaries. New replay tests now apply list insertion and paragraph add/delete/split/join while preserving unchanged Markdown. Projection slice committed as `d4b7c9039`; `cargo test -p calternal-imap` passed (all suites). APPEND now uses that handler and saves hash-checked served bases for disjoint three-way edits; Notes crate validation is running. Inline aliases are normalized to Markdown instead of raw ``/``.

kayg commented

2026-10-01 21:19:28 +00:00

Author

Owner

The real Mac proof passes for #644 at head 5342586c594e895a7cf6430bd64029905e269bfc: typing a list item in Notes updates Note d3a4c902-d665-4fe1-bfc4-55addbaf75e4 in place. The API lists exactly one Note with its title, with no conflict copy. Native Notes screenshot.

The captured-session replay includes APPEND, old-UID deletion and EXPUNGE. Immutable served Markdown supplies a proven stale base. The merge keeps unchanged source bytes and supports disjoint edits, including two insertions in one paragraph. Tables, wikilinks, Tags and block identities require a lossless span edit when a block rewrite is ambiguous.

One existing test expectation changes because this job explicitly requires a three-way merge where possible: offline_mac_edit_after_fetching_newer_revision_keeps_both had web text inserted before : and Mac text inserted after it. They are disjoint. The renamed test now requires both texts on one identity. Overlapping and unproven-base tests still require conflict recovery. No other existing conflict assertion was relaxed.

Final IMAP tests pass (29 projection tests); Notes tests pass (167 unit tests and the Apple replay integration). The focused real-server IMAP and submission probes report zero findings. Full final report, formatting/Daily proof and gate output follow after the Mac session is restored.

The real Mac proof passes for #644 at head `5342586c594e895a7cf6430bd64029905e269bfc`: typing a list item in Notes updates Note `d3a4c902-d665-4fe1-bfc4-55addbaf75e4` in place. The API lists exactly one Note with its title, with no conflict copy. [Native Notes screenshot](https://git.kayg.org/attachments/27181261-792a-4bf6-81a7-cefb7e64cf54). The captured-session replay includes APPEND, old-UID deletion and EXPUNGE. Immutable served Markdown supplies a proven stale base. The merge keeps unchanged source bytes and supports disjoint edits, including two insertions in one paragraph. Tables, wikilinks, Tags and block identities require a lossless span edit when a block rewrite is ambiguous. One existing test expectation changes because this job explicitly requires a three-way merge where possible: `offline_mac_edit_after_fetching_newer_revision_keeps_both` had web text inserted before `:` and Mac text inserted after it. They are disjoint. The renamed test now requires both texts on one identity. Overlapping and unproven-base tests still require conflict recovery. No other existing conflict assertion was relaxed. Final IMAP tests pass (29 projection tests); Notes tests pass (167 unit tests and the Apple replay integration). The focused real-server IMAP and submission probes report zero findings. Full final report, formatting/Daily proof and gate output follow after the Mac session is restored.

kayg commented

2026-10-01 22:10:53 +00:00

Author

Owner

Real-client verification found one more #644 collision after the first list-edit proof. Notes saved Enter as an empty <li> before typing. Its captured X-Mail-Created-Date was still Fri, 2 Oct 2026 02:34:41 +0530, even though the server had already accepted the earlier added item. The current Note and incoming HTML both contained that item. The old block mapper treated the repeated insertion as concurrent and made a copy.

Commit f22054eab adds a shared-insertion merge. It retains equal inserted blocks once and can fill an empty item. The actual captured MIME is in crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml. Its provider replay passes on a real Home with one Note identity. Different nonempty additions still conflict. A follow-up handles a typing pause by requiring a lossless text extension with all current characters and inline marks retained.

The comments no longer assume a refetch always changes Apple's Created-Date. The updated probe and benchmark now cover Enter, partial text, full text and another item with one initial Date. Final gates are running before a fresh Mac verification. The original lab server is restored and the Mac lock is free during builds. One unrelated lab Note touched after a native selection change was restored through its conditional API writer; the new input harness selects the exact test Note before each input step.

Real-client verification found one more #644 collision after the first list-edit proof. Notes saved Enter as an empty `<li>` before typing. Its captured `X-Mail-Created-Date` was still `Fri, 2 Oct 2026 02:34:41 +0530`, even though the server had already accepted the earlier added item. The current Note and incoming HTML both contained that item. The old block mapper treated the repeated insertion as concurrent and made a copy. Commit `f22054eab` adds a shared-insertion merge. It retains equal inserted blocks once and can fill an empty item. The actual captured MIME is in `crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml`. Its provider replay passes on a real Home with one Note identity. Different nonempty additions still conflict. A follow-up handles a typing pause by requiring a lossless text extension with all current characters and inline marks retained. The comments no longer assume a refetch always changes Apple's Created-Date. The updated probe and benchmark now cover Enter, partial text, full text and another item with one initial Date. Final gates are running before a fresh Mac verification. The original lab server is restored and the Mac lock is free during builds. One unrelated lab Note touched after a native selection change was restored through its conditional API writer; the new input harness selects the exact test Note before each input step.

kayg commented

2026-10-01 23:07:44 +00:00

Author

Owner

Final report for #644

Branch job/notesbridge-644; base 687ff703136e71e89f8dfba139e93cd0788b25c1; HEAD 3fae323267e63494f03d4631f13b60ef134a82b9. Atomic commits are complete. The required fetch and merge of origin/dev ran once: Already up to date. No push, deploy or merge to dev/main.

Built:

#644: use hash-checked immutable served Markdown as the proven base. Merge ordinary list and mark edits in place. Merge disjoint concurrent edits. Keep both sources when the merge cannot prove a safe change. Successive Apple saves may retain their initial Created-Date; equal inserted items appear once, empty items can be filled, and partial text can grow only through the lossless mapper.
#645: convert safe Apple inline marks, links, headings, lists and checklists to the existing Note grammar and back. Unchanged source bytes, tables, References, Tags, frontmatter and block IDs stay intact.
#646: give new and existing Daily notes stable UUIDs. Include Task Notes in their written Tag folders or Notes when untagged. Files owned-path rules remain separate from the protocol provider.

Files:

bench/notes-bridge.py
crates/calternal-imap/src/mailboxes.rs
crates/calternal-imap/src/mime.rs
crates/calternal-imap/src/projection.rs
crates/calternal-imap/tests/fixtures/macos27/README.md
crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml
crates/calternal-imap/tests/fixtures/macos27/append-list-item.eml
crates/calternal-imap/tests/fixtures/macos27/edit-inline-marks.eml
crates/calternal-imap/tests/mailboxes.rs
crates/calternal-imap/tests/projection.rs
crates/plugins/notes/src/imap.rs
crates/plugins/notes/src/lib.rs
tests/adversarial/notes_imap.py

Real Mac proof: the final Note acd80cd5-25bb-4203-8d92-54bc5b7b236e saved Enter, then a typed list item, then native inline formatting, then a calternal API formatting edit, then another Mac edit. The final API list contains one matching Note and no conflict copy. Native typed text inherited bold and italic together; the stored Markdown retains those combined marks. Separate API bold and italic render with the correct native HTML marks and survive the next Mac edit.

Gates (output below is verbatim):
cargo fmt --check: exit 0, no output. Python syntax and git diff --check: exit 0.

cargo clippy -p calternal-imap --all-targets -- -D warnings; cargo test -p calternal-imap (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 14s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings; cargo test -p calternal-plugin-notes (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 38s
test result: ok. 168 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 153.42s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings; cargo test -p calternal-server (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s
test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 37.07s

No web source changed. A real production web build supplied the local server's SPA for the probe.

Adversarial evidence:

Notes IMAP probe: 0 finding(s)

The expanded IMAP round and every completed benchmark cycle preserved one original Note with the exact expected body. The wrapper's later submission setup returned HTTP 403: creating App Passwords requires a recent passkey assertion, which expires after 300 seconds; the preceding benchmark took longer. Thus the final wrapper exit is 1, not a complete submission pass. The earlier submission round passed before the expanded benchmark. The Notes crate's first parallel test run hit its existing 10-second session-test timeout on the shared host; the unchanged test and complete crate passed with one test thread. No assertion was relaxed for that timeout.

Performance: local debug build on the shared host; perf VM was unreachable (No route to host). Load average [28.42, 27.83, 23.63]. One sample is four successive APPENDs with the original Date. No Notes IMAP metric exists in docs/perf/baseline.json, so this run cannot establish a baseline regression.

Profile	p50 / p95 ms	CPU seconds / average %	RSS before / after MiB
average: 64 rows, 12 samples, 1 worker(s)	15897.17 / 28443.36	50.27 / 19.79	447.76 / 515.64
worst_burst: 900 rows, 8 samples, 8 worker(s)	139297.23 / 152164.77	153.88 / 87.5	697.12 / 848.2

No samples hit the 60-second per-APPEND read limit in the final measured run. Latency includes four APPENDs, not one. Performance remains a periodic review item.

Decisions: retain at most 64 served bases per Note (at most 4 MiB of Markdown) under the existing User writer lock; use semantic block comparisons with byte patches rather than rewrite the whole Note; accept shared insertion growth only when it retains current characters and marks; use a per-User migration ledger for missing Daily UUIDs, with no numbered SQL migration. Underline uses the existing inert  grammar. No dependencies were added or upgraded.

Known gaps: ambiguous/missing/damaged base evidence and irreconcilable edits still make a recovery copy. Unsupported or invalid existing Daily frontmatter stays untouched for repair and migration retry. Three server tests remain ignored. Local shared-host performance is not release/perf-VM evidence. Final submission wrapper setup has the recent-auth timing limitation described above.

Real-client evidence: Mac list edit and later formatting round trip. All final saves kept UUID acd80cd5-25bb-4203-8d92-54bc5b7b236e; the final Note list had one matching item.

Cleanup: the original macdav-verify server is restored and /healthz responds. The Mac lock is free. cargo clean output:

Removed 17015 files, 9.8GiB total

Web build output was removed. The worktree is clean. Review screenshots remain in ignored artifacts and are attached above.

Final report for #644 Branch `job/notesbridge-644`; base `687ff703136e71e89f8dfba139e93cd0788b25c1`; HEAD `3fae323267e63494f03d4631f13b60ef134a82b9`. Atomic commits are complete. The required fetch and merge of `origin/dev` ran once: Already up to date. No push, deploy or merge to dev/main. Built: - #644: use hash-checked immutable served Markdown as the proven base. Merge ordinary list and mark edits in place. Merge disjoint concurrent edits. Keep both sources when the merge cannot prove a safe change. Successive Apple saves may retain their initial Created-Date; equal inserted items appear once, empty items can be filled, and partial text can grow only through the lossless mapper. - #645: convert safe Apple inline marks, links, headings, lists and checklists to the existing Note grammar and back. Unchanged source bytes, tables, References, Tags, frontmatter and block IDs stay intact. - #646: give new and existing Daily notes stable UUIDs. Include Task Notes in their written Tag folders or Notes when untagged. Files owned-path rules remain separate from the protocol provider. Files: ```text bench/notes-bridge.py crates/calternal-imap/src/mailboxes.rs crates/calternal-imap/src/mime.rs crates/calternal-imap/src/projection.rs crates/calternal-imap/tests/fixtures/macos27/README.md crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml crates/calternal-imap/tests/fixtures/macos27/append-list-item.eml crates/calternal-imap/tests/fixtures/macos27/edit-inline-marks.eml crates/calternal-imap/tests/mailboxes.rs crates/calternal-imap/tests/projection.rs crates/plugins/notes/src/imap.rs crates/plugins/notes/src/lib.rs tests/adversarial/notes_imap.py ``` Real Mac proof: the final Note `acd80cd5-25bb-4203-8d92-54bc5b7b236e` saved Enter, then a typed list item, then native inline formatting, then a calternal API formatting edit, then another Mac edit. The final API list contains one matching Note and no conflict copy. Native typed text inherited bold and italic together; the stored Markdown retains those combined marks. Separate API bold and italic render with the correct native HTML marks and survive the next Mac edit. Gates (output below is verbatim): `cargo fmt --check`: exit 0, no output. Python syntax and `git diff --check`: exit 0. `cargo clippy -p calternal-imap --all-targets -- -D warnings`; `cargo test -p calternal-imap` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 14s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`; `cargo test -p calternal-plugin-notes` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 38s test result: ok. 168 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 153.42s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings`; `cargo test -p calternal-server` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 37.07s ``` No web source changed. A real production web build supplied the local server's SPA for the probe. Adversarial evidence: ```text Notes IMAP probe: 0 finding(s) ``` The expanded IMAP round and every completed benchmark cycle preserved one original Note with the exact expected body. The wrapper's later submission setup returned HTTP 403: creating App Passwords requires a recent passkey assertion, which expires after 300 seconds; the preceding benchmark took longer. Thus the final wrapper exit is 1, not a complete submission pass. The earlier submission round passed before the expanded benchmark. The Notes crate's first parallel test run hit its existing 10-second session-test timeout on the shared host; the unchanged test and complete crate passed with one test thread. No assertion was relaxed for that timeout. Performance: local debug build on the shared host; perf VM was unreachable (No route to host). Load average [28.42, 27.83, 23.63]. One sample is four successive APPENDs with the original Date. No Notes IMAP metric exists in `docs/perf/baseline.json`, so this run cannot establish a baseline regression. | Profile | p50 / p95 ms | CPU seconds / average % | RSS before / after MiB | | --- | --- | --- | --- | | average: 64 rows, 12 samples, 1 worker(s) | 15897.17 / 28443.36 | 50.27 / 19.79 | 447.76 / 515.64 | | worst_burst: 900 rows, 8 samples, 8 worker(s) | 139297.23 / 152164.77 | 153.88 / 87.5 | 697.12 / 848.2 | No samples hit the 60-second per-APPEND read limit in the final measured run. Latency includes four APPENDs, not one. Performance remains a periodic review item. Decisions: retain at most 64 served bases per Note (at most 4 MiB of Markdown) under the existing User writer lock; use semantic block comparisons with byte patches rather than rewrite the whole Note; accept shared insertion growth only when it retains current characters and marks; use a per-User migration ledger for missing Daily UUIDs, with no numbered SQL migration. Underline uses the existing inert `` grammar. No dependencies were added or upgraded. Known gaps: ambiguous/missing/damaged base evidence and irreconcilable edits still make a recovery copy. Unsupported or invalid existing Daily frontmatter stays untouched for repair and migration retry. Three server tests remain ignored. Local shared-host performance is not release/perf-VM evidence. Final submission wrapper setup has the recent-auth timing limitation described above. Real-client evidence: [Mac list edit and later formatting round trip](https://git.kayg.org/attachments/677890b8-8afe-4afc-8833-d7bdf9646601). All final saves kept UUID `acd80cd5-25bb-4203-8d92-54bc5b7b236e`; the final Note list had one matching item. Cleanup: the original macdav-verify server is restored and /healthz responds. The Mac lock is free. `cargo clean` output: ```text Removed 17015 files, 9.8GiB total ``` Web build output was removed. The worktree is clean. Review screenshots remain in ignored artifacts and are attached above.

kayg referenced this issue

2026-10-01 23:07:44 +00:00

Notes over IMAP: Daily notes and Task Notes never appear in Apple Notes (Journal/Daily Notes always empty) #646

kayg referenced this issue

2026-10-01 23:07:44 +00:00

Notes over IMAP: bold/italic from Apple Notes stored as raw HTML, then stripped on the next Mac edit #645

kayg referenced this issue

2026-10-02 13:10:35 +00:00

PERF: Notes IMAP SELECT and STATUS scan every Note under the User writer lock #780

kayg referenced this issue

2026-10-02 13:12:48 +00:00

PERF: Notes IMAP SELECT and refresh retain full mailbox sources per connection (#663) #808