kayg/calternal

Watch

Fork 0

Notes over IMAP: bold/italic from Apple Notes stored as raw HTML, then stripped on the next Mac edit #645

New issue

Open

opened 2026-10-01 18:48:05 +00:00 by kayg · 3 comments

kayg commented

2026-10-01 18:48:05 +00:00

Owner

Summary

Bold and italic made in Apple Notes are lost. A Note created on the Mac stores its inline marks as raw HTML in Markdown (<b>bold</b> and <i>italic</i>, not **bold** and *italic*). The next Mac edit anywhere in the Note (here: one word in another paragraph) rewrites that line as plain bold and italic. The Mac then receives the plain text too, so the formatting is gone on both sides. This is quiet data loss of the user's formatting.

Found in the Apple interop run on the macOS 27 VM, 2026-10-01 (lab server from dev at 687ff7031, Notes over IMAP, #428).

Repro

On the Mac create a Note in the calternal Notes account with body <h1>MDV mac note</h1><div>Line from Mac ✓ café</div><div><b>bold</b> and <i>italic</i></div><ul><li>item A</li><li>item B</li></ul> (AppleScript make new note, or type it with ⌘B/⌘I).
Markdown written by the bridge:

# MDV mac note

Line from Mac ✓ café

<b>bold</b> and <i>italic</i>

- item A
- item B

In Notes (GUI) double-click "café" and type "bistro". The Apple APPEND still contains <div><b>bold</b> and <i>italic</i></div>.
The merged Markdown now reads Line from Mac ✓ bistro and bold and italic — the tags are gone. After the next server change the Mac shows <p>bold and italic</p> without bold or italic.

Expected

New Apple Notes convert inline marks to Markdown (**bold**, *italic*, and the same for underline/strike where calternal has a form), per the #428 mapping table ("Paragraphs and inline bold, italic, underline, strike").
A changed-text-only merge keeps the marks of unchanged paragraphs byte for byte (the #428 rule: "Apply changed text only; keep original Markdown delimiters").

Actual

Raw HTML on create; marks stripped on the first later edit.

Tests

Replay: Apple create with <b>/<i> → Markdown uses Markdown marks; Apple edit of another paragraph keeps them; projection back to Apple shows <b>/<i> (or <strong>/<em>).

## Summary Bold and italic made in Apple Notes are lost. A Note created on the Mac stores its inline marks as raw HTML in Markdown (`<b>bold</b> and <i>italic</i>`, not `**bold**` and `*italic*`). The next Mac edit anywhere in the Note (here: one word in another paragraph) rewrites that line as plain `bold and italic`. The Mac then receives the plain text too, so the formatting is gone on both sides. This is quiet data loss of the user's formatting. Found in the Apple interop run on the macOS 27 VM, 2026-10-01 (lab server from `dev` at `687ff7031`, Notes over IMAP, #428). ## Repro 1. On the Mac create a Note in the calternal Notes account with body `<h1>MDV mac note</h1><div>Line from Mac ✓ café</div><div><b>bold</b> and <i>italic</i></div><ul><li>item A</li><li>item B</li></ul>` (AppleScript `make new note`, or type it with ⌘B/⌘I). 2. Markdown written by the bridge: ``` # MDV mac note Line from Mac ✓ café <b>bold</b> and <i>italic</i> - item A - item B ``` 3. In Notes (GUI) double-click "café" and type "bistro". The Apple APPEND still contains `<div><b>bold</b> and <i>italic</i></div>`. 4. The merged Markdown now reads `Line from Mac ✓ bistro` and `bold and italic` — the tags are gone. After the next server change the Mac shows `<p>bold and italic</p>` without bold or italic. ## Expected - New Apple Notes convert inline marks to Markdown (`**bold**`, `*italic*`, and the same for underline/strike where calternal has a form), per the #428 mapping table ("Paragraphs and inline bold, italic, underline, strike"). - A changed-text-only merge keeps the marks of unchanged paragraphs byte for byte (the #428 rule: "Apply changed text only; keep original Markdown delimiters"). ## Actual Raw HTML on create; marks stripped on the first later edit. ## Tests Replay: Apple create with `<b>`/`<i>` → Markdown uses Markdown marks; Apple edit of another paragraph keeps them; projection back to Apple shows `<b>`/`<i>` (or `<strong>`/`<em>`).

mac-typed-marks.png

266 KiB

mac-typed-marks.png

265 KiB

mac-web-marks.png

272 KiB

mac-roundtrip.png

277 KiB

kayg referenced this issue

2026-10-01 19:29:29 +00:00

Notes over IMAP: adding a line or list item on the Mac creates a '(conflict, Apple Notes)' copy instead of updating the Note #644

kayg commented

2026-10-01 19:29:30 +00:00

Author

Owner

Started #644, #645, #646 on job/notesbridge-644, base 687ff703136e71e89f8dfba139e93cd0788b25c1 (dev). Read the contract, bridge mapping and Mac lab evidence. First trace: the proven-base gate succeeds, but the text-span projection refuses inserted block boundaries. Task Notes are explicitly excluded and Daily notes are skipped without stable identities. Will fix these separately with replay tests and real-client verification. No push or deployment.

Started #644, #645, #646 on `job/notesbridge-644`, base `687ff703136e71e89f8dfba139e93cd0788b25c1` (dev). Read the contract, bridge mapping and Mac lab evidence. First trace: the proven-base gate succeeds, but the text-span projection refuses inserted block boundaries. Task Notes are explicitly excluded and Daily notes are skipped without stable identities. Will fix these separately with replay tests and real-client verification. No push or deployment.

kayg referenced this issue

2026-10-01 19:29:31 +00:00

Notes over IMAP: Daily notes and Task Notes never appear in Apple Notes (Journal/Daily Notes always empty) #646

kayg commented

2026-10-01 19:42:22 +00:00

Author

Owner

Root cause confirmed: durable_base accepts the captured echoed Date and current hash, but Projection::merge rejects synthetic block boundaries. New replay tests now apply list insertion and paragraph add/delete/split/join while preserving unchanged Markdown. Projection slice committed as d4b7c9039; cargo test -p calternal-imap passed (all suites). APPEND now uses that handler and saves hash-checked served bases for disjoint three-way edits; Notes crate validation is running. Inline aliases are normalized to Markdown instead of raw <b>/<i>.

Root cause confirmed: `durable_base` accepts the captured echoed Date and current hash, but `Projection::merge` rejects synthetic block boundaries. New replay tests now apply list insertion and paragraph add/delete/split/join while preserving unchanged Markdown. Projection slice committed as `d4b7c9039`; `cargo test -p calternal-imap` passed (all suites). APPEND now uses that handler and saves hash-checked served bases for disjoint three-way edits; Notes crate validation is running. Inline aliases are normalized to Markdown instead of raw `<b>`/`<i>`.

kayg referenced this issue

2026-10-01 23:07:44 +00:00

Notes over IMAP: adding a line or list item on the Mac creates a '(conflict, Apple Notes)' copy instead of updating the Note #644

kayg referenced this issue

2026-10-01 23:07:44 +00:00

Notes over IMAP: Daily notes and Task Notes never appear in Apple Notes (Journal/Daily Notes always empty) #646

kayg commented

2026-10-01 23:07:44 +00:00

Author

Owner

Final report for #645

Branch job/notesbridge-644; base 687ff703136e71e89f8dfba139e93cd0788b25c1; HEAD 3fae323267e63494f03d4631f13b60ef134a82b9. Atomic commits are complete. The required fetch and merge of origin/dev ran once: Already up to date. No push, deploy or merge to dev/main.

Built:

#644: use hash-checked immutable served Markdown as the proven base. Merge ordinary list and mark edits in place. Merge disjoint concurrent edits. Keep both sources when the merge cannot prove a safe change. Successive Apple saves may retain their initial Created-Date; equal inserted items appear once, empty items can be filled, and partial text can grow only through the lossless mapper.
#645: convert safe Apple inline marks, links, headings, lists and checklists to the existing Note grammar and back. Unchanged source bytes, tables, References, Tags, frontmatter and block IDs stay intact.
#646: give new and existing Daily notes stable UUIDs. Include Task Notes in their written Tag folders or Notes when untagged. Files owned-path rules remain separate from the protocol provider.

Files:

bench/notes-bridge.py
crates/calternal-imap/src/mailboxes.rs
crates/calternal-imap/src/mime.rs
crates/calternal-imap/src/projection.rs
crates/calternal-imap/tests/fixtures/macos27/README.md
crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml
crates/calternal-imap/tests/fixtures/macos27/append-list-item.eml
crates/calternal-imap/tests/fixtures/macos27/edit-inline-marks.eml
crates/calternal-imap/tests/mailboxes.rs
crates/calternal-imap/tests/projection.rs
crates/plugins/notes/src/imap.rs
crates/plugins/notes/src/lib.rs
tests/adversarial/notes_imap.py

Real Mac proof: the final Note acd80cd5-25bb-4203-8d92-54bc5b7b236e saved Enter, then a typed list item, then native inline formatting, then a calternal API formatting edit, then another Mac edit. The final API list contains one matching Note and no conflict copy. Native typed text inherited bold and italic together; the stored Markdown retains those combined marks. Separate API bold and italic render with the correct native HTML marks and survive the next Mac edit.

Gates (output below is verbatim):
cargo fmt --check: exit 0, no output. Python syntax and git diff --check: exit 0.

cargo clippy -p calternal-imap --all-targets -- -D warnings; cargo test -p calternal-imap (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 14s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings; cargo test -p calternal-plugin-notes (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 38s
test result: ok. 168 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 153.42s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings; cargo test -p calternal-server (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s
test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 37.07s

No web source changed. A real production web build supplied the local server's SPA for the probe.

Adversarial evidence:

Notes IMAP probe: 0 finding(s)

The expanded IMAP round and every completed benchmark cycle preserved one original Note with the exact expected body. The wrapper's later submission setup returned HTTP 403: creating App Passwords requires a recent passkey assertion, which expires after 300 seconds; the preceding benchmark took longer. Thus the final wrapper exit is 1, not a complete submission pass. The earlier submission round passed before the expanded benchmark. The Notes crate's first parallel test run hit its existing 10-second session-test timeout on the shared host; the unchanged test and complete crate passed with one test thread. No assertion was relaxed for that timeout.

Performance: local debug build on the shared host; perf VM was unreachable (No route to host). Load average [28.42, 27.83, 23.63]. One sample is four successive APPENDs with the original Date. No Notes IMAP metric exists in docs/perf/baseline.json, so this run cannot establish a baseline regression.

Profile	p50 / p95 ms	CPU seconds / average %	RSS before / after MiB
average: 64 rows, 12 samples, 1 worker(s)	15897.17 / 28443.36	50.27 / 19.79	447.76 / 515.64
worst_burst: 900 rows, 8 samples, 8 worker(s)	139297.23 / 152164.77	153.88 / 87.5	697.12 / 848.2

No samples hit the 60-second per-APPEND read limit in the final measured run. Latency includes four APPENDs, not one. Performance remains a periodic review item.

Decisions: retain at most 64 served bases per Note (at most 4 MiB of Markdown) under the existing User writer lock; use semantic block comparisons with byte patches rather than rewrite the whole Note; accept shared insertion growth only when it retains current characters and marks; use a per-User migration ledger for missing Daily UUIDs, with no numbered SQL migration. Underline uses the existing inert <u> grammar. No dependencies were added or upgraded.

Known gaps: ambiguous/missing/damaged base evidence and irreconcilable edits still make a recovery copy. Unsupported or invalid existing Daily frontmatter stays untouched for repair and migration retry. Three server tests remain ignored. Local shared-host performance is not release/perf-VM evidence. Final submission wrapper setup has the recent-auth timing limitation described above.

Real-client evidence: native combined bold/italic, separate calternal bold and italic in Notes, marks after another Mac edit.

Cleanup: the original macdav-verify server is restored and /healthz responds. The Mac lock is free. cargo clean output:

Removed 17015 files, 9.8GiB total

Web build output was removed. The worktree is clean. Review screenshots remain in ignored artifacts and are attached above.

Final report for #645 Branch `job/notesbridge-644`; base `687ff703136e71e89f8dfba139e93cd0788b25c1`; HEAD `3fae323267e63494f03d4631f13b60ef134a82b9`. Atomic commits are complete. The required fetch and merge of `origin/dev` ran once: Already up to date. No push, deploy or merge to dev/main. Built: - #644: use hash-checked immutable served Markdown as the proven base. Merge ordinary list and mark edits in place. Merge disjoint concurrent edits. Keep both sources when the merge cannot prove a safe change. Successive Apple saves may retain their initial Created-Date; equal inserted items appear once, empty items can be filled, and partial text can grow only through the lossless mapper. - #645: convert safe Apple inline marks, links, headings, lists and checklists to the existing Note grammar and back. Unchanged source bytes, tables, References, Tags, frontmatter and block IDs stay intact. - #646: give new and existing Daily notes stable UUIDs. Include Task Notes in their written Tag folders or Notes when untagged. Files owned-path rules remain separate from the protocol provider. Files: ```text bench/notes-bridge.py crates/calternal-imap/src/mailboxes.rs crates/calternal-imap/src/mime.rs crates/calternal-imap/src/projection.rs crates/calternal-imap/tests/fixtures/macos27/README.md crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml crates/calternal-imap/tests/fixtures/macos27/append-list-item.eml crates/calternal-imap/tests/fixtures/macos27/edit-inline-marks.eml crates/calternal-imap/tests/mailboxes.rs crates/calternal-imap/tests/projection.rs crates/plugins/notes/src/imap.rs crates/plugins/notes/src/lib.rs tests/adversarial/notes_imap.py ``` Real Mac proof: the final Note `acd80cd5-25bb-4203-8d92-54bc5b7b236e` saved Enter, then a typed list item, then native inline formatting, then a calternal API formatting edit, then another Mac edit. The final API list contains one matching Note and no conflict copy. Native typed text inherited bold and italic together; the stored Markdown retains those combined marks. Separate API bold and italic render with the correct native HTML marks and survive the next Mac edit. Gates (output below is verbatim): `cargo fmt --check`: exit 0, no output. Python syntax and `git diff --check`: exit 0. `cargo clippy -p calternal-imap --all-targets -- -D warnings`; `cargo test -p calternal-imap` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 14s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`; `cargo test -p calternal-plugin-notes` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 38s test result: ok. 168 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 153.42s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings`; `cargo test -p calternal-server` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 37.07s ``` No web source changed. A real production web build supplied the local server's SPA for the probe. Adversarial evidence: ```text Notes IMAP probe: 0 finding(s) ``` The expanded IMAP round and every completed benchmark cycle preserved one original Note with the exact expected body. The wrapper's later submission setup returned HTTP 403: creating App Passwords requires a recent passkey assertion, which expires after 300 seconds; the preceding benchmark took longer. Thus the final wrapper exit is 1, not a complete submission pass. The earlier submission round passed before the expanded benchmark. The Notes crate's first parallel test run hit its existing 10-second session-test timeout on the shared host; the unchanged test and complete crate passed with one test thread. No assertion was relaxed for that timeout. Performance: local debug build on the shared host; perf VM was unreachable (No route to host). Load average [28.42, 27.83, 23.63]. One sample is four successive APPENDs with the original Date. No Notes IMAP metric exists in `docs/perf/baseline.json`, so this run cannot establish a baseline regression. | Profile | p50 / p95 ms | CPU seconds / average % | RSS before / after MiB | | --- | --- | --- | --- | | average: 64 rows, 12 samples, 1 worker(s) | 15897.17 / 28443.36 | 50.27 / 19.79 | 447.76 / 515.64 | | worst_burst: 900 rows, 8 samples, 8 worker(s) | 139297.23 / 152164.77 | 153.88 / 87.5 | 697.12 / 848.2 | No samples hit the 60-second per-APPEND read limit in the final measured run. Latency includes four APPENDs, not one. Performance remains a periodic review item. Decisions: retain at most 64 served bases per Note (at most 4 MiB of Markdown) under the existing User writer lock; use semantic block comparisons with byte patches rather than rewrite the whole Note; accept shared insertion growth only when it retains current characters and marks; use a per-User migration ledger for missing Daily UUIDs, with no numbered SQL migration. Underline uses the existing inert `<u>` grammar. No dependencies were added or upgraded. Known gaps: ambiguous/missing/damaged base evidence and irreconcilable edits still make a recovery copy. Unsupported or invalid existing Daily frontmatter stays untouched for repair and migration retry. Three server tests remain ignored. Local shared-host performance is not release/perf-VM evidence. Final submission wrapper setup has the recent-auth timing limitation described above. Real-client evidence: [native combined bold/italic](https://git.kayg.org/attachments/be00146a-363b-4d05-937e-4db3e89746b3), [separate calternal bold and italic in Notes](https://git.kayg.org/attachments/7ad83334-3225-4260-b508-1ee6904f5b86), [marks after another Mac edit](https://git.kayg.org/attachments/b9f834b9-4b5c-4286-af48-04a21880135c). Cleanup: the original macdav-verify server is restored and /healthz responds. The Mac lock is free. `cargo clean` output: ```text Removed 17015 files, 9.8GiB total ``` Web build output was removed. The worktree is clean. Review screenshots remain in ignored artifacts and are attached above.

kayg referenced this issue

2026-10-03 12:51:32 +00:00

Issue hygiene sweep 2026-10-03 #1008

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#645

No description provided.

Rows
Columns