kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

feat: serve read-only Apple Notes over IMAPS (#766) #879

Open
kayg wants to merge 10 commits from feat/imap-notes-readonly into feat/imap-notes-auth
pull from: feat/imap-notes-readonly
merge into: kayg:feat/imap-notes-auth
Conversation 1 Commits 10 Files changed 14 +73 -3
kayg commented 2026-11-02 18:12:51 +00:00
Owner
Copy link

Plan: implementation PR 2, Notes-serving projection

This stack layer consumes the authenticated listener from #874, which is based on #873 and #824 (f1c1aa96). It adds one stable Notes mailbox with read-only SELECT/EXAMINE, deterministic Apple Notes MIME, immutable disk-backed messages, bounded FETCH, and explicit NO [READ-ONLY] mutation refusals. It does not add write reconciliation, arbitrary folders, SMTP, QRESYNC, or CONDSTORE.

The current change begins the serving slice by requiring CALTERNAL_IMAP_UID_AUTHORITY_ROOT to be an absolute, separately managed host path and validating the configured service From address. The rest of the implementation will stay within #830's estimate: 10–16 files and 1,200–1,800 changed lines, including protocol/MIME/auth integration tests.

Storage and complexity

  • Keep #824's mailbox ID, fixed UIDVALIDITY, exact tuple, and monotonic UID authority. Persist only a versioned blob reference, exact byte length, digest, and serializer version in the UID map.
  • Publish a complete immutable MIME blob before committing its UID reference. Stream with 64 KiB buffers; keep bodies off heap. Open the blob before releasing the UID-map lock so a concurrent retirement cannot remove an in-flight fetch.
  • Mailbox metadata is O(N + A); LIST, STATUS, SELECT and indexed UID lookup are O(1); SEARCH ALL is O(N); one FETCH is O(K + total B) time with O(64 KiB) application transfer memory per active literal. Disk usage tracks serialized bytes referenced by live UIDs.
  • Reconcile each note revision to a new UID. Retire the old revision only by its exact mailbox/UIDVALIDITY/UID tuple. Never render changed bytes under an issued UID.

Failure table

Failure Effect Detection Recovery / test
Missing, stale or mismatched UID authority SELECT unavailable; no UID guessed or issued Authority status and startup diagnostic Admin recovery per #824; missing/stale authority gate
Blob missing, wrong length/digest or read failure No FETCH success; close after any started literal Blob reference and read diagnostic Restore source and publish a new UID; recovery and injected mid-stream failure gates
Oversize MIME or missing attachment Retire stale current UID; publish no partial replacement Projection error keyed by note UUID Correct source then publish a new revision; aggregate cap and missing attachment gates
Mutation or queued offline write NO [READ-ONLY], or close before APPEND literal; no state change Refusal response plus native UI/server state Mandatory Mac preservation gate before general enablement
Certificate generation change or invalid material Reload current generation; reject service if invalid, HTTP remains up TLS fingerprint/handshake and loader diagnostic Restore a valid generation; symlink rotation and bad/missing cert gates

Gate plan

  • Scripted IMAP subset: auth boundary, command states, read-only SELECT, LIST/NAMESPACE/STATUS/SEARCH/FETCH, all mutation refusals, literal framing, identity isolation and permit caps.
  • UID/MIME: U1 to U2 after editing one note, immutable repeat FETCH across reconnect/restart, exact MIME bytes/digest, Unicode, attachments, aggregate-size failures and blob corruption.
  • TLS: implicit TLS only on 9993, mounted certificate directory with current generation symlink reload, and trusted hostname validation.
  • Native gate on the disposable Mac profile only: seed an offline edit, concurrently change the same note on Calternal, verify selection/refusal preserves exact pending text across sync and relaunch, watch a server update arrive, and record the bounded retry window. Keep general enrollment unavailable until that gate passes.

Deploy contract

Calternal terminates implicit TLS on 9993. Production publishes 10.70.3.162:9993:9993, mounts /etc/calternal/imap-tls read-only as a directory, and reloads when current changes. The installed host socket forwarder already maps 10.70.3.162:993 to :9993. The listener remains disabled by default; this PR will test only on the NetBird-only dev-02 staging host.

## Plan: implementation PR 2, Notes-serving projection This stack layer consumes the authenticated listener from #874, which is based on #873 and #824 (`f1c1aa96`). It adds one stable `Notes` mailbox with read-only SELECT/EXAMINE, deterministic Apple Notes MIME, immutable disk-backed messages, bounded FETCH, and explicit `NO [READ-ONLY]` mutation refusals. It does not add write reconciliation, arbitrary folders, SMTP, QRESYNC, or CONDSTORE. The current change begins the serving slice by requiring `CALTERNAL_IMAP_UID_AUTHORITY_ROOT` to be an absolute, separately managed host path and validating the configured service `From` address. The rest of the implementation will stay within #830's estimate: 10–16 files and 1,200–1,800 changed lines, including protocol/MIME/auth integration tests. ### Storage and complexity - Keep #824's mailbox ID, fixed UIDVALIDITY, exact tuple, and monotonic UID authority. Persist only a versioned blob reference, exact byte length, digest, and serializer version in the UID map. - Publish a complete immutable MIME blob before committing its UID reference. Stream with 64 KiB buffers; keep bodies off heap. Open the blob before releasing the UID-map lock so a concurrent retirement cannot remove an in-flight fetch. - Mailbox metadata is O(N + A); LIST, STATUS, SELECT and indexed UID lookup are O(1); SEARCH ALL is O(N); one FETCH is O(K + total B) time with O(64 KiB) application transfer memory per active literal. Disk usage tracks serialized bytes referenced by live UIDs. - Reconcile each note revision to a new UID. Retire the old revision only by its exact mailbox/UIDVALIDITY/UID tuple. Never render changed bytes under an issued UID. ### Failure table | Failure | Effect | Detection | Recovery / test | | --- | --- | --- | --- | | Missing, stale or mismatched UID authority | SELECT unavailable; no UID guessed or issued | Authority status and startup diagnostic | Admin recovery per #824; missing/stale authority gate | | Blob missing, wrong length/digest or read failure | No FETCH success; close after any started literal | Blob reference and read diagnostic | Restore source and publish a new UID; recovery and injected mid-stream failure gates | | Oversize MIME or missing attachment | Retire stale current UID; publish no partial replacement | Projection error keyed by note UUID | Correct source then publish a new revision; aggregate cap and missing attachment gates | | Mutation or queued offline write | `NO [READ-ONLY]`, or close before APPEND literal; no state change | Refusal response plus native UI/server state | Mandatory Mac preservation gate before general enablement | | Certificate generation change or invalid material | Reload current generation; reject service if invalid, HTTP remains up | TLS fingerprint/handshake and loader diagnostic | Restore a valid generation; symlink rotation and bad/missing cert gates | ### Gate plan - Scripted IMAP subset: auth boundary, command states, read-only SELECT, LIST/NAMESPACE/STATUS/SEARCH/FETCH, all mutation refusals, literal framing, identity isolation and permit caps. - UID/MIME: U1 to U2 after editing one note, immutable repeat FETCH across reconnect/restart, exact MIME bytes/digest, Unicode, attachments, aggregate-size failures and blob corruption. - TLS: implicit TLS only on 9993, mounted certificate directory with `current` generation symlink reload, and trusted hostname validation. - Native gate on the disposable Mac profile only: seed an offline edit, concurrently change the same note on Calternal, verify selection/refusal preserves exact pending text across sync and relaunch, watch a server update arrive, and record the bounded retry window. Keep general enrollment unavailable until that gate passes. ### Deploy contract Calternal terminates implicit TLS on 9993. Production publishes `10.70.3.162:9993:9993`, mounts `/etc/calternal/imap-tls` read-only as a directory, and reloads when `current` changes. The installed host socket forwarder already maps `10.70.3.162:993` to `:9993`. The listener remains disabled by default; this PR will test only on the NetBird-only dev-02 staging host.
feat(imap): require a separate UID authority root
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
2d084ef4f5
kayg commented 2026-11-07 20:32:09 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+failing, left open for review.

restacked readonly IMAP on #874, retaining current UID authority methods and docs. inherits foundation and TLS fixture repairs. moved the later FETCH layer’s safe blob rename and MIME lint fixes down to readonly; MIME fixtures use atomic writes.

head: 6c5f99563cb35f57ad4a75ecfe28c26e2f97e172; based on dev cd7f84432febfc63b788fbb9b9f9950452982d74, pushed with the original exact force-with-lease.

frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B).

Rust gate results (Mac VM):

exit command
0 cargo fmt -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -- --check
0 cargo check -j4 --locked -p calternal --bins
101 cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap --all-targets
101 cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap --all-targets -- -D warnings
0 cargo test -j4 --locked -p calternal-auth --lib
0 cargo test -j4 --locked -p calternal-imap --lib
101 cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap
0 cargo check -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets
0 cargo clippy -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets -- -D warnings
0 cargo test -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --lib

server-wide all-target checks/tests are blocked on macOS by storage_startup.rs:147 borrowing Child immutably; clippy also reports unused state in storage.rs:265. both reproduce on dev. scoped auth/IMAP component checks, clippy and unit tests passed. native Apple Notes client qualification and perf qualification remain unrun.

all executed gates used the VM Nix environment, shared Mac lock and slot. exact commands, source heads, logs and dev baseline. frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head.

original-head cherry/patch inventory, pr-879.md; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.

housekeeping #1009: rebased+failing, left open for review. restacked readonly IMAP on #874, retaining current UID authority methods and docs. inherits foundation and TLS fixture repairs. moved the later FETCH layer’s safe blob rename and MIME lint fixes down to readonly; MIME fixtures use atomic writes. head: `6c5f99563cb35f57ad4a75ecfe28c26e2f97e172`; based on dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, pushed with the original exact force-with-lease. frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B). Rust gate results (Mac VM): | exit | command | |---|---| | 0 | `cargo fmt -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -- --check` | | 0 | `cargo check -j4 --locked -p calternal --bins` | | 101 | `cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap --all-targets` | | 101 | `cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap --all-targets -- -D warnings` | | 0 | `cargo test -j4 --locked -p calternal-auth --lib` | | 0 | `cargo test -j4 --locked -p calternal-imap --lib` | | 101 | `cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap` | | 0 | `cargo check -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets` | | 0 | `cargo clippy -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets -- -D warnings` | | 0 | `cargo test -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --lib` | server-wide all-target checks/tests are blocked on macOS by `storage_startup.rs:147` borrowing Child immutably; clippy also reports unused `state` in `storage.rs:265`. both reproduce on dev. scoped auth/IMAP component checks, clippy and unit tests passed. native Apple Notes client qualification and perf qualification remain unrun. all executed gates used the VM Nix environment, shared Mac lock and slot. [exact commands, source heads, logs and dev baseline](https://git.kayg.org/attachments/0aee78b2-28de-4cba-890a-830a5086e373). frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head. [original-head cherry/patch inventory](https://git.kayg.org/attachments/e05a38dc-c028-4d77-8655-efa9d5c8ea50), `pr-879.md`; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
This pull request is broken due to missing fork information.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/imap-notes-readonly:feat/imap-notes-readonly
git switch feat/imap-notes-readonly

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch feat/imap-notes-auth
git merge --no-ff feat/imap-notes-readonly
git switch feat/imap-notes-readonly
git rebase feat/imap-notes-auth
git switch feat/imap-notes-auth
git merge --ff-only feat/imap-notes-readonly
git switch feat/imap-notes-readonly
git rebase feat/imap-notes-auth
git switch feat/imap-notes-auth
git merge --no-ff feat/imap-notes-readonly
git switch feat/imap-notes-auth
git merge --squash feat/imap-notes-readonly
git switch feat/imap-notes-auth
git merge --ff-only feat/imap-notes-readonly
git switch feat/imap-notes-auth
git merge feat/imap-notes-readonly
git push origin feat/imap-notes-auth
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!879
No description provided.