kayg/calternal

Watch

Fork 0

Notes over IMAP: Daily notes and Task Notes never appear in Apple Notes (Journal/Daily Notes always empty) #646

New issue

Open

opened 2026-10-01 18:48:06 +00:00 by kayg · 3 comments

kayg commented

2026-10-01 18:48:06 +00:00

Owner

Summary

Daily notes and Task Notes never appear in Apple Notes over the IMAP bridge. The Journal/Daily Notes folder is listed but always empty, and Task Notes are not in any folder. The #428 bridge design (docs/research/apple-notes-imap.md, "Proposed bridge mapping", and the decision "Interpretation of issue #428's 'all Notes': include task Notes") says both are exposed.

Found in the Apple interop run on the macOS 27 VM, 2026-10-01 (lab server from dev at 687ff7031).

Evidence

Home had three Daily notes (written by the Log API and by CalDAV event PUTs) and four Task Notes (written by Reminders PUTs and the Tasks API). An authenticated IMAP client (same App Password as the Mac profile) sees:

"Notes" 0 messages            (before any plain Note existed)
"Notes/Journal" 0
"Notes/Journal/Daily Notes" 0

Plain Notes created later through POST /api/v1/notes appeared at once (PASS), so the listener and account work.

Index rows: note_imap_changes has rows for the Daily notes with note_id = 'path:Notes/20260928-dailynote.md' and for the Task Notes with their calternal-id.

Causes found in code

Daily notes written by the Log writer have no calternal-id frontmatter, so NotesImapStore::snapshot skips them: "Legacy path-only Notes have no stable identity yet. Never expose a path as a protocol identity" (crates/plugins/notes/src/imap.rs). A Daily note created today by calternal itself is "legacy" by this rule, so it can never reach the Mac. The Daily note frontmatter does carry tags: ["journal/daily-notes"], so the folder exists.
calternal_imap::mailboxes::folders_for_note returns no folders for is_task_note(source), and append_note refuses Task Notes ("Task notes are not exposed by IMAP"). This is the old provisional D6 exclusion that the research file says was replaced.

Expected

Either (a) Daily notes get a stable calternal-id when the Log writer creates them (plus the lossless per-User migration for existing ones that the design mentions) and Task Notes are exposed in their tag folders with frontmatter untouched by Apple edits, or (b) the design is changed by the owner and the empty Journal/Daily Notes folder is not advertised. Owner decision needed on (b) only if the design is to change.

Actual

Folder shown, never filled; Task Notes invisible.

## Summary Daily notes and Task Notes never appear in Apple Notes over the IMAP bridge. The `Journal/Daily Notes` folder is listed but always empty, and Task Notes are not in any folder. The #428 bridge design (docs/research/apple-notes-imap.md, "Proposed bridge mapping", and the decision "Interpretation of issue #428's 'all Notes': include task Notes") says both are exposed. Found in the Apple interop run on the macOS 27 VM, 2026-10-01 (lab server from `dev` at `687ff7031`). ## Evidence Home had three Daily notes (written by the Log API and by CalDAV event PUTs) and four Task Notes (written by Reminders PUTs and the Tasks API). An authenticated IMAP client (same App Password as the Mac profile) sees: ``` "Notes" 0 messages (before any plain Note existed) "Notes/Journal" 0 "Notes/Journal/Daily Notes" 0 ``` Plain Notes created later through `POST /api/v1/notes` appeared at once (PASS), so the listener and account work. Index rows: `note_imap_changes` has rows for the Daily notes with `note_id = 'path:Notes/20260928-dailynote.md'` and for the Task Notes with their `calternal-id`. ## Causes found in code 1. Daily notes written by the Log writer have no `calternal-id` frontmatter, so `NotesImapStore::snapshot` skips them: "Legacy path-only Notes have no stable identity yet. Never expose a path as a protocol identity" (`crates/plugins/notes/src/imap.rs`). A Daily note created today by calternal itself is "legacy" by this rule, so it can never reach the Mac. The Daily note frontmatter does carry `tags: ["journal/daily-notes"]`, so the folder exists. 2. `calternal_imap::mailboxes::folders_for_note` returns no folders for `is_task_note(source)`, and `append_note` refuses Task Notes ("Task notes are not exposed by IMAP"). This is the old provisional D6 exclusion that the research file says was replaced. ## Expected Either (a) Daily notes get a stable `calternal-id` when the Log writer creates them (plus the lossless per-User migration for existing ones that the design mentions) and Task Notes are exposed in their tag folders with frontmatter untouched by Apple edits, or (b) the design is changed by the owner and the empty `Journal/Daily Notes` folder is not advertised. Owner decision needed on (b) only if the design is to change. ## Actual Folder shown, never filled; Task Notes invisible.

mac-task.png

226 KiB

mac-daily.png

158 KiB

kayg referenced this issue

2026-10-01 19:29:29 +00:00

Notes over IMAP: adding a line or list item on the Mac creates a '(conflict, Apple Notes)' copy instead of updating the Note #644

kayg referenced this issue

2026-10-01 19:29:30 +00:00

Notes over IMAP: bold/italic from Apple Notes stored as raw HTML, then stripped on the next Mac edit #645

kayg commented

2026-10-01 19:29:31 +00:00

Author

Owner

Started #644, #645, #646 on job/notesbridge-644, base 687ff703136e71e89f8dfba139e93cd0788b25c1 (dev). Read the contract, bridge mapping and Mac lab evidence. First trace: the proven-base gate succeeds, but the text-span projection refuses inserted block boundaries. Task Notes are explicitly excluded and Daily notes are skipped without stable identities. Will fix these separately with replay tests and real-client verification. No push or deployment.

Started #644, #645, #646 on `job/notesbridge-644`, base `687ff703136e71e89f8dfba139e93cd0788b25c1` (dev). Read the contract, bridge mapping and Mac lab evidence. First trace: the proven-base gate succeeds, but the text-span projection refuses inserted block boundaries. Task Notes are explicitly excluded and Daily notes are skipped without stable identities. Will fix these separately with replay tests and real-client verification. No push or deployment.

kayg commented

2026-10-01 20:03:30 +00:00

Author

Owner

Implemented #646 in 1f52ed958: new Daily notes receive a stable UUID before their first write. A separate per-User migration adds only missing IDs to existing Daily notes through the conditional calternal-fs writer. The moved-Daily-note CRLF test checks unchanged body, metadata and repeated migration. Removed the provisional Task Note exclusion; their tag-folder edits preserve status, priority, due and custom fields. Notes crate: test result: ok. 167 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 114.90s. The #606 owned-path rule is scoped to the web Notes view; the #428 Apple mapping explicitly includes Daily and Task Notes. Server gates and real Mac verification remain.

Implemented #646 in `1f52ed958`: new Daily notes receive a stable UUID before their first write. A separate per-User migration adds only missing IDs to existing Daily notes through the conditional calternal-fs writer. The moved-Daily-note CRLF test checks unchanged body, metadata and repeated migration. Removed the provisional Task Note exclusion; their tag-folder edits preserve status, priority, due and custom fields. Notes crate: `test result: ok. 167 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 114.90s`. The #606 owned-path rule is scoped to the web Notes view; the #428 Apple mapping explicitly includes Daily and Task Notes. Server gates and real Mac verification remain.

kayg referenced this issue

2026-10-01 23:07:44 +00:00

Notes over IMAP: adding a line or list item on the Mac creates a '(conflict, Apple Notes)' copy instead of updating the Note #644

kayg commented

2026-10-01 23:07:44 +00:00

Author

Owner

Final report for #646

Branch job/notesbridge-644; base 687ff703136e71e89f8dfba139e93cd0788b25c1; HEAD 3fae323267e63494f03d4631f13b60ef134a82b9. Atomic commits are complete. The required fetch and merge of origin/dev ran once: Already up to date. No push, deploy or merge to dev/main.

Built:

#644: use hash-checked immutable served Markdown as the proven base. Merge ordinary list and mark edits in place. Merge disjoint concurrent edits. Keep both sources when the merge cannot prove a safe change. Successive Apple saves may retain their initial Created-Date; equal inserted items appear once, empty items can be filled, and partial text can grow only through the lossless mapper.
#645: convert safe Apple inline marks, links, headings, lists and checklists to the existing Note grammar and back. Unchanged source bytes, tables, References, Tags, frontmatter and block IDs stay intact.
#646: give new and existing Daily notes stable UUIDs. Include Task Notes in their written Tag folders or Notes when untagged. Files owned-path rules remain separate from the protocol provider.

Files:

bench/notes-bridge.py
crates/calternal-imap/src/mailboxes.rs
crates/calternal-imap/src/mime.rs
crates/calternal-imap/src/projection.rs
crates/calternal-imap/tests/fixtures/macos27/README.md
crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml
crates/calternal-imap/tests/fixtures/macos27/append-list-item.eml
crates/calternal-imap/tests/fixtures/macos27/edit-inline-marks.eml
crates/calternal-imap/tests/mailboxes.rs
crates/calternal-imap/tests/projection.rs
crates/plugins/notes/src/imap.rs
crates/plugins/notes/src/lib.rs
tests/adversarial/notes_imap.py

Real Mac proof: the final Note acd80cd5-25bb-4203-8d92-54bc5b7b236e saved Enter, then a typed list item, then native inline formatting, then a calternal API formatting edit, then another Mac edit. The final API list contains one matching Note and no conflict copy. Native typed text inherited bold and italic together; the stored Markdown retains those combined marks. Separate API bold and italic render with the correct native HTML marks and survive the next Mac edit.

Gates (output below is verbatim):
cargo fmt --check: exit 0, no output. Python syntax and git diff --check: exit 0.

cargo clippy -p calternal-imap --all-targets -- -D warnings; cargo test -p calternal-imap (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 14s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings; cargo test -p calternal-plugin-notes (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 38s
test result: ok. 168 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 153.42s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings; cargo test -p calternal-server (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s
test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 37.07s

No web source changed. A real production web build supplied the local server's SPA for the probe.

Adversarial evidence:

Notes IMAP probe: 0 finding(s)

The expanded IMAP round and every completed benchmark cycle preserved one original Note with the exact expected body. The wrapper's later submission setup returned HTTP 403: creating App Passwords requires a recent passkey assertion, which expires after 300 seconds; the preceding benchmark took longer. Thus the final wrapper exit is 1, not a complete submission pass. The earlier submission round passed before the expanded benchmark. The Notes crate's first parallel test run hit its existing 10-second session-test timeout on the shared host; the unchanged test and complete crate passed with one test thread. No assertion was relaxed for that timeout.

Performance: local debug build on the shared host; perf VM was unreachable (No route to host). Load average [28.42, 27.83, 23.63]. One sample is four successive APPENDs with the original Date. No Notes IMAP metric exists in docs/perf/baseline.json, so this run cannot establish a baseline regression.

Profile	p50 / p95 ms	CPU seconds / average %	RSS before / after MiB
average: 64 rows, 12 samples, 1 worker(s)	15897.17 / 28443.36	50.27 / 19.79	447.76 / 515.64
worst_burst: 900 rows, 8 samples, 8 worker(s)	139297.23 / 152164.77	153.88 / 87.5	697.12 / 848.2

No samples hit the 60-second per-APPEND read limit in the final measured run. Latency includes four APPENDs, not one. Performance remains a periodic review item.

Decisions: retain at most 64 served bases per Note (at most 4 MiB of Markdown) under the existing User writer lock; use semantic block comparisons with byte patches rather than rewrite the whole Note; accept shared insertion growth only when it retains current characters and marks; use a per-User migration ledger for missing Daily UUIDs, with no numbered SQL migration. Underline uses the existing inert <u> grammar. No dependencies were added or upgraded.

Known gaps: ambiguous/missing/damaged base evidence and irreconcilable edits still make a recovery copy. Unsupported or invalid existing Daily frontmatter stays untouched for repair and migration retry. Three server tests remain ignored. Local shared-host performance is not release/perf-VM evidence. Final submission wrapper setup has the recent-auth timing limitation described above.

Real-client evidence: Daily note open under Journal → Daily Notes, Task Note visible in Notes. Native Daily row UUID e4e62be3-dfa9-4f1a-91b2-a791c07c91fb matches the API. The task has the unique API title NB646 final task proof. AppleScript could not resolve the nested Daily folder alias; the native folder list opened it correctly and showed seven Daily notes. The Mac settings and accounts were not changed.

Cleanup: the original macdav-verify server is restored and /healthz responds. The Mac lock is free. cargo clean output:

Removed 17015 files, 9.8GiB total

Web build output was removed. The worktree is clean. Review screenshots remain in ignored artifacts and are attached above.

Final report for #646 Branch `job/notesbridge-644`; base `687ff703136e71e89f8dfba139e93cd0788b25c1`; HEAD `3fae323267e63494f03d4631f13b60ef134a82b9`. Atomic commits are complete. The required fetch and merge of `origin/dev` ran once: Already up to date. No push, deploy or merge to dev/main. Built: - #644: use hash-checked immutable served Markdown as the proven base. Merge ordinary list and mark edits in place. Merge disjoint concurrent edits. Keep both sources when the merge cannot prove a safe change. Successive Apple saves may retain their initial Created-Date; equal inserted items appear once, empty items can be filled, and partial text can grow only through the lossless mapper. - #645: convert safe Apple inline marks, links, headings, lists and checklists to the existing Note grammar and back. Unchanged source bytes, tables, References, Tags, frontmatter and block IDs stay intact. - #646: give new and existing Daily notes stable UUIDs. Include Task Notes in their written Tag folders or Notes when untagged. Files owned-path rules remain separate from the protocol provider. Files: ```text bench/notes-bridge.py crates/calternal-imap/src/mailboxes.rs crates/calternal-imap/src/mime.rs crates/calternal-imap/src/projection.rs crates/calternal-imap/tests/fixtures/macos27/README.md crates/calternal-imap/tests/fixtures/macos27/append-empty-list-item.eml crates/calternal-imap/tests/fixtures/macos27/append-list-item.eml crates/calternal-imap/tests/fixtures/macos27/edit-inline-marks.eml crates/calternal-imap/tests/mailboxes.rs crates/calternal-imap/tests/projection.rs crates/plugins/notes/src/imap.rs crates/plugins/notes/src/lib.rs tests/adversarial/notes_imap.py ``` Real Mac proof: the final Note `acd80cd5-25bb-4203-8d92-54bc5b7b236e` saved Enter, then a typed list item, then native inline formatting, then a calternal API formatting edit, then another Mac edit. The final API list contains one matching Note and no conflict copy. Native typed text inherited bold and italic together; the stored Markdown retains those combined marks. Separate API bold and italic render with the correct native HTML marks and survive the next Mac edit. Gates (output below is verbatim): `cargo fmt --check`: exit 0, no output. Python syntax and `git diff --check`: exit 0. `cargo clippy -p calternal-imap --all-targets -- -D warnings`; `cargo test -p calternal-imap` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 14s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`; `cargo test -p calternal-plugin-notes` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 38s test result: ok. 168 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 153.42s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings`; `cargo test -p calternal-server` (exit 0): ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s test result: ok. 107 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 37.07s ``` No web source changed. A real production web build supplied the local server's SPA for the probe. Adversarial evidence: ```text Notes IMAP probe: 0 finding(s) ``` The expanded IMAP round and every completed benchmark cycle preserved one original Note with the exact expected body. The wrapper's later submission setup returned HTTP 403: creating App Passwords requires a recent passkey assertion, which expires after 300 seconds; the preceding benchmark took longer. Thus the final wrapper exit is 1, not a complete submission pass. The earlier submission round passed before the expanded benchmark. The Notes crate's first parallel test run hit its existing 10-second session-test timeout on the shared host; the unchanged test and complete crate passed with one test thread. No assertion was relaxed for that timeout. Performance: local debug build on the shared host; perf VM was unreachable (No route to host). Load average [28.42, 27.83, 23.63]. One sample is four successive APPENDs with the original Date. No Notes IMAP metric exists in `docs/perf/baseline.json`, so this run cannot establish a baseline regression. | Profile | p50 / p95 ms | CPU seconds / average % | RSS before / after MiB | | --- | --- | --- | --- | | average: 64 rows, 12 samples, 1 worker(s) | 15897.17 / 28443.36 | 50.27 / 19.79 | 447.76 / 515.64 | | worst_burst: 900 rows, 8 samples, 8 worker(s) | 139297.23 / 152164.77 | 153.88 / 87.5 | 697.12 / 848.2 | No samples hit the 60-second per-APPEND read limit in the final measured run. Latency includes four APPENDs, not one. Performance remains a periodic review item. Decisions: retain at most 64 served bases per Note (at most 4 MiB of Markdown) under the existing User writer lock; use semantic block comparisons with byte patches rather than rewrite the whole Note; accept shared insertion growth only when it retains current characters and marks; use a per-User migration ledger for missing Daily UUIDs, with no numbered SQL migration. Underline uses the existing inert `<u>` grammar. No dependencies were added or upgraded. Known gaps: ambiguous/missing/damaged base evidence and irreconcilable edits still make a recovery copy. Unsupported or invalid existing Daily frontmatter stays untouched for repair and migration retry. Three server tests remain ignored. Local shared-host performance is not release/perf-VM evidence. Final submission wrapper setup has the recent-auth timing limitation described above. Real-client evidence: [Daily note open under Journal → Daily Notes](https://git.kayg.org/attachments/d1fb7080-133d-416d-ad4c-280e637226ab), [Task Note visible in Notes](https://git.kayg.org/attachments/98a80860-c587-4469-b612-369ee9cf8e0a). Native Daily row UUID `e4e62be3-dfa9-4f1a-91b2-a791c07c91fb` matches the API. The task has the unique API title `NB646 final task proof`. AppleScript could not resolve the nested Daily folder alias; the native folder list opened it correctly and showed seven Daily notes. The Mac settings and accounts were not changed. Cleanup: the original macdav-verify server is restored and /healthz responds. The Mac lock is free. `cargo clean` output: ```text Removed 17015 files, 9.8GiB total ``` Web build output was removed. The worktree is clean. Review screenshots remain in ignored artifacts and are attached above.

kayg referenced this issue

2026-10-01 23:07:44 +00:00

Notes over IMAP: bold/italic from Apple Notes stored as raw HTML, then stripped on the next Mac edit #645

kayg referenced this issue

2026-10-03 12:51:32 +00:00

Issue hygiene sweep 2026-10-03 #1008

kayg referenced this issue

2026-10-03 15:50:51 +00:00

COLLAB: live document history — benchmark the cheapest footprint, then build (§61) #975