kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

feat: project Notes into read-only IMAP #881

Open
kayg wants to merge 16 commits from feat/imap-notes-projection into feat/imap-notes-readonly
pull from: feat/imap-notes-projection
merge into: kayg:feat/imap-notes-readonly
Conversation 1 Commits 16 Files changed 20 +1312 -45
kayg commented 2026-11-02 19:14:37 +00:00
Owner
Copy link

Plan: read-only Notes projection

This stack layer consumes #879 (durable blob references and deterministic Apple Notes MIME, including attachment parts) and #874 (the disabled-by-default IMAPS listener and mail.notes.read authentication). It adds the per-user Notes mailbox projection, journal-driven updates, read-only SELECT/EXAMINE, STATUS, LIST, SEARCH ALL, IDLE, and explicit write refusal. Bounded FETCH is the next stack layer; keep this draft stacked until that layer lands.

Size and complexity

This layer changes 1,312 lines in and 45 lines out, about 1,357 changed lines across six files. That is within #830's 1,200–1,800 line estimate for the Notes-serving slice.

For N notes, A attachment references, B bytes in one serialized message, K fetched messages, and C active connections: mailbox metadata is O(N + A); LIST, STATUS, SELECT, and indexed UID lookup are O(1); SEARCH ALL is O(N) time with bounded-page process memory; FETCH is O(K + total B) with a 64 KiB transfer buffer per active connection. Bodies remain disk-backed.

Failure table

Failure Effect Detection Recovery / gate
Missing, stale, or mismatched UID authority SELECT is unavailable; no UID is guessed or issued Authority diagnostic without note content Follow #824 recovery; test missing and stale authority
Missing or unreadable note or attachment Retire the old current UID; publish no partial replacement Projection diagnostic keyed by note UUID Correct the source and publish a new UID; test missing attachments
Blob publication or metadata commit fails No successful new revision is exposed Storage error and exact UID map Reopen from durable references; test restart and orphan cleanup
Mutation or queued offline write Tagged NO [READ-ONLY], or close before an APPEND literal; no state change Refusal response and native UI state Preserve pending text; mandatory disposable Mac gate
Journal fan-out falls behind Close the affected IDLE session or reconcile from the Notes index Bounded broadcast lag handling Reconnect and reconcile; test updates across IDLE and reconnect

Gate tests

  • Scripted IMAP subset: authentication boundary, LIST/NAMESPACE/STATUS/SELECT/EXAMINE, read-only mutation refusals, SEARCH ALL, IDLE updates, and no pre-auth mailbox lookup.
  • UID and MIME: publish U1, edit the same note, require U2 greater than U1 with new immutable bytes; compare deterministic MIME including attachments across reconnect and restart.
  • Limits and failures: missing authority/blob, aggregate MIME limit, journal lag, command bounds, and per-IP/process/Installation/user permits.
  • Native gate on the disposable Mac profile: preserve a seeded offline edit through read-only selection, a server revision, sync cycles, and relaunch; record refusals and retry bounds. Keep general enrollment disabled until it passes.
  • TLS gate: implicit TLS on 9993, hostname validation, and certificate reload after rotating the current symlink generation.

Staging and deployment contract

Test only on NetBird-only calternal-dev-02-std. The app listens on 9993 with implicit TLS. Mount /etc/calternal/imap-tls read-only as a directory and reload when the current symlink generation changes. Production's Podman mapping is 10.70.3.162:9993:9993; the installed 993-to-9993 forwarder is not used by this staging test. Do not deploy this stack to production.

## Plan: read-only Notes projection This stack layer consumes #879 (durable blob references and deterministic Apple Notes MIME, including attachment parts) and #874 (the disabled-by-default IMAPS listener and mail.notes.read authentication). It adds the per-user Notes mailbox projection, journal-driven updates, read-only SELECT/EXAMINE, STATUS, LIST, SEARCH ALL, IDLE, and explicit write refusal. Bounded FETCH is the next stack layer; keep this draft stacked until that layer lands. ## Size and complexity This layer changes 1,312 lines in and 45 lines out, about 1,357 changed lines across six files. That is within #830's 1,200–1,800 line estimate for the Notes-serving slice. For N notes, A attachment references, B bytes in one serialized message, K fetched messages, and C active connections: mailbox metadata is O(N + A); LIST, STATUS, SELECT, and indexed UID lookup are O(1); SEARCH ALL is O(N) time with bounded-page process memory; FETCH is O(K + total B) with a 64 KiB transfer buffer per active connection. Bodies remain disk-backed. ## Failure table | Failure | Effect | Detection | Recovery / gate | | --- | --- | --- | --- | | Missing, stale, or mismatched UID authority | SELECT is unavailable; no UID is guessed or issued | Authority diagnostic without note content | Follow #824 recovery; test missing and stale authority | | Missing or unreadable note or attachment | Retire the old current UID; publish no partial replacement | Projection diagnostic keyed by note UUID | Correct the source and publish a new UID; test missing attachments | | Blob publication or metadata commit fails | No successful new revision is exposed | Storage error and exact UID map | Reopen from durable references; test restart and orphan cleanup | | Mutation or queued offline write | Tagged NO [READ-ONLY], or close before an APPEND literal; no state change | Refusal response and native UI state | Preserve pending text; mandatory disposable Mac gate | | Journal fan-out falls behind | Close the affected IDLE session or reconcile from the Notes index | Bounded broadcast lag handling | Reconnect and reconcile; test updates across IDLE and reconnect | ## Gate tests - Scripted IMAP subset: authentication boundary, LIST/NAMESPACE/STATUS/SELECT/EXAMINE, read-only mutation refusals, SEARCH ALL, IDLE updates, and no pre-auth mailbox lookup. - UID and MIME: publish U1, edit the same note, require U2 greater than U1 with new immutable bytes; compare deterministic MIME including attachments across reconnect and restart. - Limits and failures: missing authority/blob, aggregate MIME limit, journal lag, command bounds, and per-IP/process/Installation/user permits. - Native gate on the disposable Mac profile: preserve a seeded offline edit through read-only selection, a server revision, sync cycles, and relaunch; record refusals and retry bounds. Keep general enrollment disabled until it passes. - TLS gate: implicit TLS on 9993, hostname validation, and certificate reload after rotating the current symlink generation. ## Staging and deployment contract Test only on NetBird-only calternal-dev-02-std. The app listens on 9993 with implicit TLS. Mount /etc/calternal/imap-tls read-only as a directory and reload when the current symlink generation changes. Production's Podman mapping is 10.70.3.162:9993:9993; the installed 993-to-9993 forwarder is not used by this staging test. Do not deploy this stack to production.
feat(imap): project Notes into a read-only mailbox
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
e5aea4d7e5
kayg commented 2026-11-07 20:32:10 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+failing, left open for review.

restacked note projection on #879. kept dev restart behavior rather than restoring obsolete Notes::page; retained new Notes::entry. inherits foundation and TLS fixture repairs. inherits readonly storage/lint fixes.

head: e9150e4534a690e7baabb32e1281cc4c5c147f51; based on dev cd7f84432febfc63b788fbb9b9f9950452982d74, pushed with the original exact force-with-lease.

frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B).

Rust gate results (Mac VM):

exit command
0 cargo fmt -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes -- --check
0 cargo check -j4 --locked -p calternal --bins
101 cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets
101 cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets -- -D warnings
0 cargo test -j4 --locked -p calternal-auth --lib
0 cargo test -j4 --locked -p calternal-imap --lib
101 cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes
0 cargo check -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets
101 cargo clippy -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets -- -D warnings
0 cargo test -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --lib

server-wide all-target checks/tests are blocked on macOS by storage_startup.rs:147 borrowing Child immutably; clippy also reports unused state in storage.rs:265. both reproduce on dev. auth/IMAP component check and unit tests passed, but component clippy still fails on unused staged mailbox fields/methods, rx length comparison and PathBuf argument. native Apple Notes and perf qualification remain unrun.

all executed gates used the VM Nix environment, shared Mac lock and slot. exact commands, source heads, logs and dev baseline. frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head.

original-head cherry/patch inventory, pr-881.md; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.

housekeeping #1009: rebased+failing, left open for review. restacked note projection on #879. kept dev restart behavior rather than restoring obsolete Notes::page; retained new Notes::entry. inherits foundation and TLS fixture repairs. inherits readonly storage/lint fixes. head: `e9150e4534a690e7baabb32e1281cc4c5c147f51`; based on dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, pushed with the original exact force-with-lease. frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B). Rust gate results (Mac VM): | exit | command | |---|---| | 0 | `cargo fmt -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes -- --check` | | 0 | `cargo check -j4 --locked -p calternal --bins` | | 101 | `cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets` | | 101 | `cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets -- -D warnings` | | 0 | `cargo test -j4 --locked -p calternal-auth --lib` | | 0 | `cargo test -j4 --locked -p calternal-imap --lib` | | 101 | `cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes` | | 0 | `cargo check -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets` | | 101 | `cargo clippy -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets -- -D warnings` | | 0 | `cargo test -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --lib` | server-wide all-target checks/tests are blocked on macOS by `storage_startup.rs:147` borrowing Child immutably; clippy also reports unused `state` in `storage.rs:265`. both reproduce on dev. auth/IMAP component check and unit tests passed, but component clippy still fails on unused staged mailbox fields/methods, rx length comparison and PathBuf argument. native Apple Notes and perf qualification remain unrun. all executed gates used the VM Nix environment, shared Mac lock and slot. [exact commands, source heads, logs and dev baseline](https://git.kayg.org/attachments/0aee78b2-28de-4cba-890a-830a5086e373). frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head. [original-head cherry/patch inventory](https://git.kayg.org/attachments/e05a38dc-c028-4d77-8655-efa9d5c8ea50), `pr-881.md`; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
This pull request is broken due to missing fork information.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/imap-notes-projection:feat/imap-notes-projection
git switch feat/imap-notes-projection

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch feat/imap-notes-readonly
git merge --no-ff feat/imap-notes-projection
git switch feat/imap-notes-projection
git rebase feat/imap-notes-readonly
git switch feat/imap-notes-readonly
git merge --ff-only feat/imap-notes-projection
git switch feat/imap-notes-projection
git rebase feat/imap-notes-readonly
git switch feat/imap-notes-readonly
git merge --no-ff feat/imap-notes-projection
git switch feat/imap-notes-readonly
git merge --squash feat/imap-notes-projection
git switch feat/imap-notes-readonly
git merge --ff-only feat/imap-notes-projection
git switch feat/imap-notes-readonly
git merge feat/imap-notes-projection
git push origin feat/imap-notes-readonly
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!881
No description provided.