kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

feat: stream immutable Notes messages over IMAP #882

Open
kayg wants to merge 19 commits from feat/imap-notes-fetch into feat/imap-notes-projection
pull from: feat/imap-notes-fetch
merge into: kayg:feat/imap-notes-projection
Conversation 4 Commits 19 Files changed 20 +545 -137
kayg commented 2026-11-02 19:19:21 +00:00
Owner
Copy link

Plan: bounded read-only FETCH

This layer completes the Notes-serving stack on #881 and #879. It adds sequence and UID FETCH, observed flags/date/size/UID fields, BODY.PEEK full/header/text/header-fields sections, RFC822 variants, BODYSTRUCTURE, SEARCH ALL, and bounded literal streaming. It never marks a note seen or accepts a write.

Size and complexity

FETCH uses a pinned immutable blob file, validates exact length before announcing each literal, and streams in 64 KiB chunks. For K fetched messages and B total serialized bytes, work is O(K + B); application transfer memory is O(64 KiB) per active literal plus bounded headers. Search and fetch enumerate bounded mailbox pages. The message body is never copied into an in-process cache.

Failure table

Failure Effect Detection Recovery / gate
Missing reference/blob or length mismatch No FETCH success; close if a literal has begun Reference and read error Repair source and publish a new UID; inject pre-literal failure
Read fails after literal starts Close without tagged success Socket result and blob error Reconnect and fetch immutable current revision; inject mid-stream failure
UID changes during a paged command Do not report a false success Mailbox generation mismatch Reconnect and consume queued update; edit during SEARCH/FETCH
Large message or slow transfer Stop at configured bound or deadline Size/deadline error Publish no partial message; exercise 96 MiB and transfer limits
Unsupported FETCH item or malformed range Tagged BAD; no mutation Protocol response Retry with the supported Apple Notes profile

Gate tests

  • Scripted subset: FETCH/UID FETCH across sequence and UID ranges, UID insertion, header fields, BODY.PEEK, BODYSTRUCTURE, RFC822.SIZE, SEARCH ALL, all mutation refusals, and repeated fetch bytes.
  • Immutable revision: fetch U1, update the same note, require U2 greater than U1 and new bytes; verify reconnect and restart preserve the bytes for every live UID.
  • Streaming failures: missing blob, wrong length/digest at recovery, and injected read failure after a literal begins; no tagged success after a partial literal.
  • TLS reload: connect with the staging hostname, rotate current to a new valid certificate generation, and verify a new connection uses the new certificate.
  • Native Mac preservation: disposable Notes profile only, with seeded offline edit, server-side revision, refusal observation, update visibility, sync cycles, and relaunch. Keep general enrollment disabled until this passes.

Staging and deployment contract

Use only NetBird-only calternal-dev-02-std. The app listens on 9993 with implicit TLS and mounts /etc/calternal/imap-tls as a read-only directory. Reload the current symlink generation. Production publishes 10.70.3.162:9993:9993 and already has a 993-to-9993 socket forwarder; do not deploy this stack to production.

## Plan: bounded read-only FETCH This layer completes the Notes-serving stack on #881 and #879. It adds sequence and UID FETCH, observed flags/date/size/UID fields, BODY.PEEK full/header/text/header-fields sections, RFC822 variants, BODYSTRUCTURE, SEARCH ALL, and bounded literal streaming. It never marks a note seen or accepts a write. ## Size and complexity FETCH uses a pinned immutable blob file, validates exact length before announcing each literal, and streams in 64 KiB chunks. For K fetched messages and B total serialized bytes, work is O(K + B); application transfer memory is O(64 KiB) per active literal plus bounded headers. Search and fetch enumerate bounded mailbox pages. The message body is never copied into an in-process cache. ## Failure table | Failure | Effect | Detection | Recovery / gate | | --- | --- | --- | --- | | Missing reference/blob or length mismatch | No FETCH success; close if a literal has begun | Reference and read error | Repair source and publish a new UID; inject pre-literal failure | | Read fails after literal starts | Close without tagged success | Socket result and blob error | Reconnect and fetch immutable current revision; inject mid-stream failure | | UID changes during a paged command | Do not report a false success | Mailbox generation mismatch | Reconnect and consume queued update; edit during SEARCH/FETCH | | Large message or slow transfer | Stop at configured bound or deadline | Size/deadline error | Publish no partial message; exercise 96 MiB and transfer limits | | Unsupported FETCH item or malformed range | Tagged BAD; no mutation | Protocol response | Retry with the supported Apple Notes profile | ## Gate tests - Scripted subset: FETCH/UID FETCH across sequence and UID ranges, UID insertion, header fields, BODY.PEEK, BODYSTRUCTURE, RFC822.SIZE, SEARCH ALL, all mutation refusals, and repeated fetch bytes. - Immutable revision: fetch U1, update the same note, require U2 greater than U1 and new bytes; verify reconnect and restart preserve the bytes for every live UID. - Streaming failures: missing blob, wrong length/digest at recovery, and injected read failure after a literal begins; no tagged success after a partial literal. - TLS reload: connect with the staging hostname, rotate current to a new valid certificate generation, and verify a new connection uses the new certificate. - Native Mac preservation: disposable Notes profile only, with seeded offline edit, server-side revision, refusal observation, update visibility, sync cycles, and relaunch. Keep general enrollment disabled until this passes. ## Staging and deployment contract Use only NetBird-only calternal-dev-02-std. The app listens on 9993 with implicit TLS and mounts /etc/calternal/imap-tls as a read-only directory. Reload the current symlink generation. Production publishes 10.70.3.162:9993:9993 and already has a 993-to-9993 socket forwarder; do not deploy this stack to production.
feat(imap): stream immutable Notes messages
Some checks failed
CI / web (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
8d7acada30
kayg commented 2026-11-02 20:39:30 +00:00
Author
Owner
Copy link

Staging gates at 8674270389e2c670c6f0dda5a121a54095814688

This stack is based on #824 head f1c1aa961e169ef99e132063a97ccbd9a89a5def, not dev.

PR Head
#873 7e9034a6b6b6fe0eb62fc3a204540bd423bfdc51
#874 67d4d06609f84340e290b4661e6440e28f8096dc
#879 3ed2f4c6d53ec970069be5bf51aa2127e7d2054d
#881 859869db8d99d372e9c6da37dbb82807f59e86b1
#882 8674270389e2c670c6f0dda5a121a54095814688

All commands below ran on calternal-dev-02-std in the worktree at this head, through ~/bin/slot and nix develop.

  • cargo fmt --all -- --check: PASS.
  • cargo test -p calternal-auth --quiet: PASS.
  • cargo test -p calternal-imap --quiet: PASS, 19 passed.
  • cargo test -p calternal-imap-uid --quiet: PASS, 15 passed.
  • cargo clippy -p calternal-auth -p calternal-imap-uid -p calternal-imap -p calternal --all-targets -- -D warnings: PASS.
  • cargo build -p calternal --quiet: PASS.

The staging IMAPS endpoint is imap-staging.calternal.cloud:9993 at NetBird-only 10.69.69.222, using a DNS-01 Let’s Encrypt certificate. The active unit is calternal-imap-staging-r6.service. Disposable test user: imap-staging-test; its credential remains private on dev-02.

  • python3 -u ~/imap-staging/run/protocol_gate.py: PASS for TLS hostname verification, pre-auth boundary, LOGIN and PLAIN, mailbox reads, MIME/BODYSTRUCTURE, UID SEARCH, explicit write refusals, unchanged mailbox after APPEND refusal/reconnect, IDLE/DONE, NOOP and CHECK.
  • python3 -u ~/imap-staging/run/tls_reload_gate.py: PASS. Served certificate fingerprints changed from cd64f454eadadce1777c62ee93d9c78b5a05586d53257ead0b2797f43d9d3251 to a temporary generation and back; trusted TLS and IMAP greeting passed after restoration. current is back on generations/letsencrypt-1.
  • A loopback notes.append action against the disposable user produced U2 12289 for the same note; U1 2 retired. UIDVALIDITY stayed 2568110071; bystander UID 1 and MIME bytes stayed unchanged. After a normal-mode server restart, UIDs 1,12289 and both MIME hashes remained stable.

The mandatory native Apple Notes preservation gate is still pending. The Mac lock is currently held by another task (holder PID 1171), so I have not added the account or changed any note content. Production was not touched. I will post a follow-up with the native gate result when the exclusive Mac lock is available.

## Staging gates at `8674270389e2c670c6f0dda5a121a54095814688` This stack is based on #824 head `f1c1aa961e169ef99e132063a97ccbd9a89a5def`, not `dev`. | PR | Head | | --- | --- | | #873 | `7e9034a6b6b6fe0eb62fc3a204540bd423bfdc51` | | #874 | `67d4d06609f84340e290b4661e6440e28f8096dc` | | #879 | `3ed2f4c6d53ec970069be5bf51aa2127e7d2054d` | | #881 | `859869db8d99d372e9c6da37dbb82807f59e86b1` | | #882 | `8674270389e2c670c6f0dda5a121a54095814688` | All commands below ran on `calternal-dev-02-std` in the worktree at this head, through `~/bin/slot` and `nix develop`. - `cargo fmt --all -- --check`: PASS. - `cargo test -p calternal-auth --quiet`: PASS. - `cargo test -p calternal-imap --quiet`: PASS, 19 passed. - `cargo test -p calternal-imap-uid --quiet`: PASS, 15 passed. - `cargo clippy -p calternal-auth -p calternal-imap-uid -p calternal-imap -p calternal --all-targets -- -D warnings`: PASS. - `cargo build -p calternal --quiet`: PASS. The staging IMAPS endpoint is `imap-staging.calternal.cloud:9993` at NetBird-only `10.69.69.222`, using a DNS-01 Let’s Encrypt certificate. The active unit is `calternal-imap-staging-r6.service`. Disposable test user: `imap-staging-test`; its credential remains private on dev-02. - `python3 -u ~/imap-staging/run/protocol_gate.py`: PASS for TLS hostname verification, pre-auth boundary, LOGIN and PLAIN, mailbox reads, MIME/BODYSTRUCTURE, UID SEARCH, explicit write refusals, unchanged mailbox after APPEND refusal/reconnect, IDLE/DONE, NOOP and CHECK. - `python3 -u ~/imap-staging/run/tls_reload_gate.py`: PASS. Served certificate fingerprints changed from `cd64f454eadadce1777c62ee93d9c78b5a05586d53257ead0b2797f43d9d3251` to a temporary generation and back; trusted TLS and IMAP greeting passed after restoration. `current` is back on `generations/letsencrypt-1`. - A loopback `notes.append` action against the disposable user produced U2 `12289` for the same note; U1 `2` retired. UIDVALIDITY stayed `2568110071`; bystander UID 1 and MIME bytes stayed unchanged. After a normal-mode server restart, UIDs `1,12289` and both MIME hashes remained stable. The mandatory native Apple Notes preservation gate is still pending. The Mac lock is currently held by another task (holder PID 1171), so I have not added the account or changed any note content. Production was not touched. I will post a follow-up with the native gate result when the exclusive Mac lock is available.
kayg commented 2026-11-02 22:42:56 +00:00
Author
Owner
Copy link

Native Apple Notes gate update

The staging service is active on dev-02 at imap-staging.calternal.cloud:9993 (NetBird-only, 10.69.69.222), current unit calternal-imap-staging-r7.service; disposable user imap-staging-test remains isolated there.

The mandatory native preservation gate is pending. /tmp/mac-bench.lock is still held by codex-1793657411-profile-read, so I have not installed or opened the staging profile. No Mac account or note state and no production data have been changed. I will continue once the exclusive Mac lock is available.

## Native Apple Notes gate update The staging service is active on dev-02 at `imap-staging.calternal.cloud:9993` (NetBird-only, `10.69.69.222`), current unit `calternal-imap-staging-r7.service`; disposable user `imap-staging-test` remains isolated there. The mandatory native preservation gate is pending. `/tmp/mac-bench.lock` is still held by `codex-1793657411-profile-read`, so I have not installed or opened the staging profile. No Mac account or note state and no production data have been changed. I will continue once the exclusive Mac lock is available.
kayg commented 2026-11-03 00:58:40 +00:00
Author
Owner
Copy link

Native Apple Notes gate result

The Mac reservation wait did not acquire the lock. One remote shell used an atomic mkdir /tmp/mac-bench.lock, retried every 5 seconds for 7,200 seconds, and emitted a minute heartbeat. It exited with LOCK_TIMEOUT after 7200s (status 75) before acquisition, so no holder file was written and no GUI, profile, account, or note operation ran. An immediate atomic retry returned LOCK_BUSY holder=codex-1793659951-profile-read.

Result: native preservation gate not run. No Mac account or Notes data was changed, and production was not touched. Rerun the native gate when the Mac lock is available.

## Native Apple Notes gate result The Mac reservation wait did not acquire the lock. One remote shell used an atomic `mkdir /tmp/mac-bench.lock`, retried every 5 seconds for 7,200 seconds, and emitted a minute heartbeat. It exited with `LOCK_TIMEOUT after 7200s` (status 75) before acquisition, so no holder file was written and no GUI, profile, account, or note operation ran. An immediate atomic retry returned `LOCK_BUSY holder=codex-1793659951-profile-read`. **Result: native preservation gate not run.** No Mac account or Notes data was changed, and production was not touched. Rerun the native gate when the Mac lock is available.
kayg commented 2026-11-07 20:26:01 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+failing, left open for review.

restacked FETCH on #881. native Apple Notes preservation qualification remains outstanding, irrespective of cargo results. kept atomic TLS fixture writes together with canonical fixture paths. moved shared blob/MIME fixes earlier without changing the final source tree.

head: d9793d9a42be4eed3cedb2a4a2337544248995f6; based on dev cd7f84432febfc63b788fbb9b9f9950452982d74, pushed with the original exact force-with-lease.

frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B).

Rust gate results (Mac VM):

exit command
0 cargo fmt -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes -- --check
0 cargo check -j4 --locked -p calternal --bins
101 cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets
101 cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets -- -D warnings
0 cargo test -j4 --locked -p calternal-auth --lib
0 cargo test -j4 --locked -p calternal-imap --lib
101 cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes
0 cargo check -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets
0 cargo clippy -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets -- -D warnings
0 cargo test -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --lib

server-wide all-target checks/tests are blocked on macOS by storage_startup.rs:147 borrowing Child immutably; clippy also reports unused state in storage.rs:265. both reproduce on dev. scoped auth/IMAP component checks, clippy and unit tests passed. native Apple Notes client qualification and perf qualification remain unrun.

all executed gates used the VM Nix environment, shared Mac lock and slot. exact commands, source heads, logs and dev baseline. frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head.

original-head cherry/patch inventory, pr-882.md; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.

housekeeping #1009: rebased+failing, left open for review. restacked FETCH on #881. native Apple Notes preservation qualification remains outstanding, irrespective of cargo results. kept atomic TLS fixture writes together with canonical fixture paths. moved shared blob/MIME fixes earlier without changing the final source tree. head: `d9793d9a42be4eed3cedb2a4a2337544248995f6`; based on dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, pushed with the original exact force-with-lease. frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B). Rust gate results (Mac VM): | exit | command | |---|---| | 0 | `cargo fmt -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes -- --check` | | 0 | `cargo check -j4 --locked -p calternal --bins` | | 101 | `cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets` | | 101 | `cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes --all-targets -- -D warnings` | | 0 | `cargo test -j4 --locked -p calternal-auth --lib` | | 0 | `cargo test -j4 --locked -p calternal-imap --lib` | | 101 | `cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-imap-uid -p calternal-imap -p calternal-notes` | | 0 | `cargo check -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets` | | 0 | `cargo clippy -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --all-targets -- -D warnings` | | 0 | `cargo test -j4 --locked -p calternal-auth -p calternal-imap -p calternal-imap-uid --lib` | server-wide all-target checks/tests are blocked on macOS by `storage_startup.rs:147` borrowing Child immutably; clippy also reports unused `state` in `storage.rs:265`. both reproduce on dev. scoped auth/IMAP component checks, clippy and unit tests passed. native Apple Notes client qualification and perf qualification remain unrun. all executed gates used the VM Nix environment, shared Mac lock and slot. [exact commands, source heads, logs and dev baseline](https://git.kayg.org/attachments/bdb8d284-b80d-4185-b0be-6fa74f48a191). frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head. [original-head cherry/patch inventory](https://git.kayg.org/attachments/e05a38dc-c028-4d77-8655-efa9d5c8ea50), `pr-882.md`; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.
Some checks failed
CI / web (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
This pull request is broken due to missing fork information.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/imap-notes-fetch:feat/imap-notes-fetch
git switch feat/imap-notes-fetch

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch feat/imap-notes-projection
git merge --no-ff feat/imap-notes-fetch
git switch feat/imap-notes-fetch
git rebase feat/imap-notes-projection
git switch feat/imap-notes-projection
git merge --ff-only feat/imap-notes-fetch
git switch feat/imap-notes-fetch
git rebase feat/imap-notes-projection
git switch feat/imap-notes-projection
git merge --no-ff feat/imap-notes-fetch
git switch feat/imap-notes-projection
git merge --squash feat/imap-notes-fetch
git switch feat/imap-notes-projection
git merge --ff-only feat/imap-notes-fetch
git switch feat/imap-notes-projection
git merge feat/imap-notes-fetch
git push origin feat/imap-notes-projection
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!882
No description provided.