Attachments: upload, sniffing, limits, dated paths (#539) #563

Closed

kayg wants to merge 0 commits from issue/539-attachments into dev

kayg commented

2026-10-17 18:44:57 +00:00

Owner

Closes the backend half of #539 (unblocks #322). Decisions: B7 (2026-10-13), G12 (2026-10-14), ADR 0027, ADR 0028.

What it does

A new plugin, crates/attachments (id attachments):

Upload route: POST /api/v1/attachments?name=<file>&date=<YYYY-MM-DD>, raw file as the body. It needs a session, X-Requested-With: calternal and an Idempotency-Key. The answer has the same shape as an action, {result, token, replayed}: 201 when the file was kept, 200 when it was already uploaded or this is a replay.
One action pipeline: the HTTP adapter streams the bytes into a staged upload, keyed by user and idempotency key. It then runs attachments.add through the executor. That journals attachments.added, so views stay live, and stores the result with the key, so a retry never makes a second copy. attachments.check {hashes} lets an app skip uploading files you already have; it only looks in your own Home.
Type from content: the type comes from infer 0.22, plus our own check for HEIF brands it misses (mif1/msf1/hevc bursts) and Canon CR3. The extension only chooses between text types. Files are never renamed. HTML is stored as text/plain. Unknown bytes are application/octet-stream.
Dated paths (G12): pictures and videos go to Photos/YYYY/YYYY-MM-DD/, everything else to Documents/YYYY/YYYY-MM-DD/. The original name is kept, made safe (no separators, no control characters, no leading dots, at most 200 bytes). A taken name becomes name 2.ext, and an existing file is never replaced (RENAME_NOREPLACE).
Limits while streaming: Content-Length is checked before anything is read. The bytes are counted as they arrive, so a chunked or lying body stops the moment it passes the limit. The default limit is 2 GiB, set with CALTERNAL_ATTACHMENT_MAX_MB. Nothing is decoded on the server, so image bombs and HEIC parser bugs can't reach it; pixel sizes are read from the header only.
Already uploaded (ADR 0027): each Home has a blake3 index at users/<id>/attachments/index.db. A hit is checked against the file on disk (it must exist with the same size) before it is believed. The same bytes return already: true with the existing path, and no second copy is kept. This is per person only: Bob uploading Alice's bytes keeps his own copy, so nothing leaks about what Alice has.
Serving: GET /api/v1/attachments/<path>, only Photos/… and Documents/…, never hidden names. The file is opened through the new calternal_fsio::beneath module, which uses openat2 with RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS, falls back to a per-component O_NOFOLLOW walk, and refuses non-regular files. The type is sniffed again from the content on every request. Every response sends nosniff, Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox (PDF gets the same without sandbox) and CORP: same-origin. HTML, SVG, XML and unknown types are served as Content-Disposition: attachment. Range and ETag/304 are supported, and the body streams from disk.
Durability: the body goes into a temporary file in the target folder, written in fsync'd 1 MiB steps (paced). The file is fsync'd, renamed into place, and then the folder is fsync'd, the same promise as notes. Before the temporary file exists, a crash marker naming it is written to attachments/pending/<boot>-<id>. A background sweep at startup removes earlier runs' leftovers. It never touches the current run's uploads and never removes anything that isn't a hidden temporary file.

Tests

crates/attachments: 19 tests covering sniffing (PDF named .jpg, SVG named .png, HTML by content, HEIF and CR3 brands, truncated UTF-8), safe names, strict days, the limit with and without a length, a broken-off connection leaving nothing, retries and idempotency conflicts, "Already uploaded", traversal (.., %2e%2e, %2F, symlinks out, Notes.md, index.db, temporary names), auth and CSRF checks, the sweep, and ranges.
crates/fsio: beneath tests covering paths and symlinks that climb out (both with openat2 and with the walk), and temporary files that persist without replacing anything.
isolation.rs: the all-actions sweep now attacks the two new actions with Alice's real hash. A new test, bob_cannot_reach_alices_attachments, runs through app::wire exactly as main does. Bob gets the same 404 for Alice's paths as for made-up ones, traversal toward her Home fails, check finds nothing, the same bytes give Bob his own copy, and attachments.add with her key fails.
tests/attachments_crash.rs uses the real binary:
- kill -9 in three rounds (before the temporary file exists, and twice with 4+ MiB on disk). No file ever appears under a real name, the next start removes the temporary file and marker, and uploads work afterwards.
- 1 GiB in bounded memory: the peak above idle stays under 16 MiB, the same as a 64 MiB upload's (measured 6 to 11 MiB).

Gates

Commit a2a20c4, on this host (debug build):

== cargo fmt --all --check
fmt ok
== cargo clippy --workspace --all-targets --locked -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 21.93s   (no warnings)
== cargo test --workspace --locked --no-fail-fast
test exit: 0 — 57 test binaries, every one "test result: ok. … 0 failed"
  calternal (bin, incl. isolation + contract): 12 passed
  attachments_crash: 2 passed
  calternal-attachments: 19 passed
  calternal-fsio: 12 passed
  calternal-notes: 202 passed, 1 ignored (unchanged)
upload 1024 MiB in 2.86 s = 358 MiB/s; RSS before 27528 KiB, peak during 35796 KiB, after 33076 KiB, high-water 35140 KiB

Benchmarks

Benchmarks (tests/attachments_crash.rs::a_big_upload_streams_in_bounded_memory; real binary, raw HTTP/1.1 client on loopback, 1 MiB writes; on this dev host, under load from trains and other builds):

build	target	size	time	throughput	peak RSS above idle (that upload / a 64 MiB one)
debug	tmpfs	1 GiB	2.86–5.41 s	189–358 MiB/s	6.4–10.8 MiB / 5.8–9.4 MiB
debug	disk (sda, ext4)	1 GiB	34.6 s	30 MiB/s	6.4 / 9.4 MiB
debug	disk	2 GiB	62.3 s	33 MiB/s	9.8 / 5.8 MiB
release	tmpfs	1 GiB	3.12 s	328 MiB/s	7.8 / 8.8 MiB
release	disk	1 GiB	21.5 s	48 MiB/s	7.2 / 8.1 MiB
release	disk	2 GiB	43.1 s	47 MiB/s	10.0 / 6.4 MiB

Memory is bounded and flat with size. A 64 MiB upload and a 2 GiB one peak the same, 6–11 MiB above idle. That is the 1 MiB fsync'd disk step, up to 4 queued network chunks, and allocator slack. Steady RSS after an upload is +3–4 MiB, and malloc_trim takes it back after a quiet spell. The test asserts an absolute bound of < 16 MiB; the relative check was noise (±4 MiB).
Throughput: on tmpfs, ~330 MiB/s in release. On disk, the host's shared sda while trains ran: ~48 MiB/s. That is bound by the deliberate fsync per 1 MiB step from #298 (paced), so note saves never wait behind an upload. The perf VM is the place for real numbers (zvol layouts).
Kill -9: kill_nine_mid_upload_leaves_nothing_half_written passes 3/3 runs, three kills each.

Commit 3d3dfea swaps the flaky relative memory assertion for the absolute bound. Re-verified there: fmt clean, clippy clean on the workspace, attachments_crash 3/3.

Extension points (not built)

tus and the IETF resumable-upload draft (F4, LQ3) only replace the staging step: they stage the same way, then run the same attachments.add.
Cross-user dedupe (F11, reflinks/FIDEDUPERANGE) can run after the fact on the blake3 index.
Thumbnails and previews (PH22, the sandboxed worker) read the files. Nothing in this PR decodes media.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WCcFhwwgMK66k9LMVqogNJ

Closes the backend half of #539 (unblocks #322). Decisions: B7 (2026-10-13), G12 (2026-10-14), ADR 0027, ADR 0028. ## What it does A new plugin, `crates/attachments` (id `attachments`): - **Upload route:** `POST /api/v1/attachments?name=<file>&date=<YYYY-MM-DD>`, raw file as the body. It needs a session, `X-Requested-With: calternal` and an `Idempotency-Key`. The answer has the same shape as an action, `{result, token, replayed}`: 201 when the file was kept, 200 when it was already uploaded or this is a replay. - **One action pipeline:** the HTTP adapter streams the bytes into a staged upload, keyed by user and idempotency key. It then runs `attachments.add` through the executor. That journals `attachments.added`, so views stay live, and stores the result with the key, so a retry never makes a second copy. `attachments.check {hashes}` lets an app skip uploading files you already have; it only looks in your own Home. - **Type from content:** the type comes from `infer` 0.22, plus our own check for HEIF brands it misses (mif1/msf1/hevc bursts) and Canon CR3. The extension only chooses between text types. Files are never renamed. HTML is stored as `text/plain`. Unknown bytes are `application/octet-stream`. - **Dated paths (G12):** pictures and videos go to `Photos/YYYY/YYYY-MM-DD/`, everything else to `Documents/YYYY/YYYY-MM-DD/`. The original name is kept, made safe (no separators, no control characters, no leading dots, at most 200 bytes). A taken name becomes `name 2.ext`, and an existing file is never replaced (`RENAME_NOREPLACE`). - **Limits while streaming:** `Content-Length` is checked before anything is read. The bytes are counted as they arrive, so a chunked or lying body stops the moment it passes the limit. The default limit is 2 GiB, set with `CALTERNAL_ATTACHMENT_MAX_MB`. Nothing is decoded on the server, so image bombs and HEIC parser bugs can't reach it; pixel sizes are read from the header only. - **Already uploaded (ADR 0027):** each Home has a blake3 index at `users/<id>/attachments/index.db`. A hit is checked against the file on disk (it must exist with the same size) before it is believed. The same bytes return `already: true` with the existing path, and no second copy is kept. This is per person only: Bob uploading Alice's bytes keeps his own copy, so nothing leaks about what Alice has. - **Serving:** `GET /api/v1/attachments/<path>`, only `Photos/…` and `Documents/…`, never hidden names. The file is opened through the new `calternal_fsio::beneath` module, which uses `openat2` with `RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS`, falls back to a per-component `O_NOFOLLOW` walk, and refuses non-regular files. The type is sniffed again from the content on every request. Every response sends `nosniff`, `Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox` (PDF gets the same without `sandbox`) and `CORP: same-origin`. HTML, SVG, XML and unknown types are served as `Content-Disposition: attachment`. `Range` and `ETag`/304 are supported, and the body streams from disk. - **Durability:** the body goes into a temporary file in the target folder, written in fsync'd 1 MiB steps (`paced`). The file is fsync'd, renamed into place, and then the folder is fsync'd, the same promise as notes. Before the temporary file exists, a crash marker naming it is written to `attachments/pending/<boot>-<id>`. A background sweep at startup removes earlier runs' leftovers. It never touches the current run's uploads and never removes anything that isn't a hidden temporary file. ## Tests - `crates/attachments`: 19 tests covering sniffing (PDF named `.jpg`, SVG named `.png`, HTML by content, HEIF and CR3 brands, truncated UTF-8), safe names, strict days, the limit with and without a length, a broken-off connection leaving nothing, retries and idempotency conflicts, "Already uploaded", traversal (`..`, `%2e%2e`, `%2F`, symlinks out, `Notes.md`, `index.db`, temporary names), auth and CSRF checks, the sweep, and ranges. - `crates/fsio`: `beneath` tests covering paths and symlinks that climb out (both with `openat2` and with the walk), and temporary files that persist without replacing anything. - `isolation.rs`: the all-actions sweep now attacks the two new actions with Alice's real hash. A new test, `bob_cannot_reach_alices_attachments`, runs through `app::wire` exactly as `main` does. Bob gets the same 404 for Alice's paths as for made-up ones, traversal toward her Home fails, `check` finds nothing, the same bytes give Bob his own copy, and `attachments.add` with her key fails. - `tests/attachments_crash.rs` uses the real binary: - kill -9 in three rounds (before the temporary file exists, and twice with 4+ MiB on disk). No file ever appears under a real name, the next start removes the temporary file and marker, and uploads work afterwards. - 1 GiB in bounded memory: the peak above idle stays under 16 MiB, the same as a 64 MiB upload's (measured 6 to 11 MiB). ## Gates Commit a2a20c4, on this host (debug build): ``` == cargo fmt --all --check fmt ok == cargo clippy --workspace --all-targets --locked -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 21.93s (no warnings) == cargo test --workspace --locked --no-fail-fast test exit: 0 — 57 test binaries, every one "test result: ok. … 0 failed" calternal (bin, incl. isolation + contract): 12 passed attachments_crash: 2 passed calternal-attachments: 19 passed calternal-fsio: 12 passed calternal-notes: 202 passed, 1 ignored (unchanged) upload 1024 MiB in 2.86 s = 358 MiB/s; RSS before 27528 KiB, peak during 35796 KiB, after 33076 KiB, high-water 35140 KiB ``` ## Benchmarks **Benchmarks** (`tests/attachments_crash.rs::a_big_upload_streams_in_bounded_memory`; real binary, raw HTTP/1.1 client on loopback, 1 MiB writes; on this dev host, under load from trains and other builds): | build | target | size | time | throughput | peak RSS above idle (that upload / a 64 MiB one) | |---|---|---|---|---|---| | debug | tmpfs | 1 GiB | 2.86–5.41 s | 189–358 MiB/s | 6.4–10.8 MiB / 5.8–9.4 MiB | | debug | disk (sda, ext4) | 1 GiB | 34.6 s | 30 MiB/s | 6.4 / 9.4 MiB | | debug | disk | 2 GiB | 62.3 s | 33 MiB/s | 9.8 / 5.8 MiB | | release | tmpfs | 1 GiB | 3.12 s | 328 MiB/s | 7.8 / 8.8 MiB | | release | disk | 1 GiB | 21.5 s | 48 MiB/s | 7.2 / 8.1 MiB | | release | disk | 2 GiB | 43.1 s | 47 MiB/s | 10.0 / 6.4 MiB | - **Memory is bounded and flat with size.** A 64 MiB upload and a 2 GiB one peak the same, 6–11 MiB above idle. That is the 1 MiB fsync'd disk step, up to 4 queued network chunks, and allocator slack. Steady RSS after an upload is +3–4 MiB, and `malloc_trim` takes it back after a quiet spell. The test asserts an absolute bound of < 16 MiB; the relative check was noise (±4 MiB). - **Throughput:** on tmpfs, ~330 MiB/s in release. On disk, the host's shared sda while trains ran: ~48 MiB/s. That is bound by the deliberate fsync per 1 MiB step from #298 (`paced`), so note saves never wait behind an upload. The perf VM is the place for real numbers (zvol layouts). - **Kill -9:** `kill_nine_mid_upload_leaves_nothing_half_written` passes 3/3 runs, three kills each. Commit 3d3dfea swaps the flaky relative memory assertion for the absolute bound. Re-verified there: fmt clean, clippy clean on the workspace, `attachments_crash` 3/3. ## Extension points (not built) - **tus and the IETF resumable-upload draft (F4, LQ3)** only replace the staging step: they stage the same way, then run the same `attachments.add`. - **Cross-user dedupe (F11, reflinks/FIDEDUPERANGE)** can run after the fact on the blake3 index. - **Thumbnails and previews (PH22, the sandboxed worker)** read the files. Nothing in this PR decodes media. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01WCcFhwwgMK66k9LMVqogNJ

kayg added 1 commit

2026-10-17 18:44:57 +00:00

Attachments: upload, sniffing, limits, dated paths (#539 )

CI / server (pull_request) Successful in 26s

Details

CI / rust (pull_request) Failing after 35s

Details

CI / web (pull_request) Failing after 6s

Details

a2a20c4153

A new `attachments` plugin. Uploads stream to a hidden temporary file in
`Photos/YYYY/YYYY-MM-DD/` or `Documents/YYYY/YYYY-MM-DD/` (G12), with the
type taken from the content signature (`infer` plus HEIF/CR3 brands; the
extension only picks between text types; HTML kept as text/plain), the
size limit enforced while streaming, and a blake3 hash for "Already
uploaded" (ADR 0027/0028). The upload then runs `attachments.add` through
the one action pipeline, so it is journaled and idempotent.

Files are opened through a new `calternal_fsio::beneath` module
(`openat2` with RESOLVE_BENEATH), so no path can leave the Home. Unsafe
types are served as downloads with nosniff and a sandboxing CSP. Crash
markers let the next start remove temporary files a kill -9 left.

Tests: sniffing, names, days, limits while streaming, retries, traversal,
isolation (Bob vs Alice) in isolation.rs, kill -9 mid-upload through the
real binary, and a 1 GiB upload in bounded memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WCcFhwwgMK66k9LMVqogNJ

kayg referenced this pull request

2026-10-17 18:55:17 +00:00

Attachments backend: upload, content-type detection, limits, dated paths (ADR 0028) #539

kayg referenced this pull request

2026-10-17 20:40:55 +00:00

Attachments backend: upload, content-type detection, limits, dated paths (ADR 0028) #539

kayg referenced this pull request

2026-10-18 02:42:17 +00:00

Train z: attachments backend, [[ suggestions, Vim mode (#539, #448, #538) #567

kayg commented

2026-10-18 02:42:24 +00:00

Author