#339 - WEBDAV: file WebDAV for Home (RFC 4918 class 1) — Finder, iOS Files, PhotoSync, rclone - kayg/calternal

kayg commented

2026-09-28 13:13:43 +00:00

Owner

Decided on #306 (research: docs/research/file-protocols.md). Depends on #328 (scoped app passwords): start after it merges.
Build:

/dav/files/{user-id}/: a thin adapter over the Files service (the server is the single writer; every path goes through calternal-fs; no string path building). RFC 4918 class 1: OPTIONS, PROPFIND (depth 0/1; depth infinity refused with a 403 precondition), GET/HEAD with Range, PUT (streamed, bounded memory, If-Match/If-None-Match for conflicts), DELETE (→ Trash), MKCOL, COPY, MOVE (Overwrite header). Strong ETags. No fake LOCK (class 2 only if a client test proves it is needed).
Auth: Basic with scoped app passwords (#328): protocol=webdav; access full/read/upload-only; an optional Home prefix. Upload-only: create under the prefix, list names under the prefix, no GET of contents, no DELETE, no MOVE out; a same-name file with different content gets a numbered name (never overwrite).
Photos: uploads under Photos/ feed the Photos pipeline (capture-date foldering per DESIGN, Live Photo/RAW pairs, stacks); nothing is moved after upload; atomic writes stay invisible (#305 lesson).
Real-client proof (the owner rule): macOS Finder (Connect to Server, on the macOS VM, the orchestrator verifies), rclone (CI), cadaver/litmus WebDAV conformance (CI); PhotoSync from the owner's iPhone after that. Measure large-video upload throughput and reliability through the k3s edge (a 2 GB file, interrupted once); if it is poor, propose Nextcloud chunking v2 compatibility with numbers.
Adversarial round: traversal and Unicode path tricks, huge PROPFIND, depth bombs, lock/ETag races, upload-only escalation, cross-user hrefs (#331 matrix).

Decided on #306 (research: docs/research/file-protocols.md). **Depends on #328 (scoped app passwords)**: start after it merges. Build: - `/dav/files/{user-id}/`: a thin adapter over the Files service (the server is the single writer; every path goes through calternal-fs; no string path building). RFC 4918 class 1: OPTIONS, PROPFIND (depth 0/1; depth infinity refused with a 403 precondition), GET/HEAD with Range, PUT (streamed, bounded memory, If-Match/If-None-Match for conflicts), DELETE (→ Trash), MKCOL, COPY, MOVE (Overwrite header). Strong ETags. No fake LOCK (class 2 only if a client test proves it is needed). - Auth: Basic with **scoped app passwords** (#328): protocol=webdav; access full/read/upload-only; an optional Home prefix. Upload-only: create under the prefix, list names under the prefix, no GET of contents, no DELETE, no MOVE out; a same-name file with different content gets a numbered name (never overwrite). - Photos: uploads under `Photos/` feed the Photos pipeline (capture-date foldering per DESIGN, Live Photo/RAW pairs, stacks); nothing is moved after upload; atomic writes stay invisible (#305 lesson). - **Real-client proof (the owner rule): macOS Finder (Connect to Server, on the macOS VM, the orchestrator verifies), rclone (CI), cadaver/litmus WebDAV conformance (CI); PhotoSync from the owner's iPhone after that.** Measure large-video upload throughput and reliability through the k3s edge (a 2 GB file, interrupted once); if it is poor, propose Nextcloud chunking v2 compatibility with numbers. - Adversarial round: traversal and Unicode path tricks, huge PROPFIND, depth bombs, lock/ETag races, upload-only escalation, cross-user hrefs (#331 matrix).

kayg commented

2026-09-28 13:18:58 +00:00

Author

Owner

Owner (2026-09-28): 'is there a very performant one for rust that we can improve upon by adding chunked uploads, parallelisation, special handling for long uploads or high latency connections?' Scope added as phase 2 (after class 1 works and is measured):

One upload staging engine (refactor the Files tus subset into it: private staging, quota reservation, chunk bookkeeping, expiry, atomic publish through calternal-fs/CAS with streaming hashing). Expose it through three thin adapters: our tus (existing), Nextcloud chunking v2 (/remote.php/dav/uploads/<user>/<id>/<n> MKCOL/PUT/MOVE .file, parallel chunks, resume; used by rclone's --webdav-nextcloud-chunk-size, the Nextcloud apps and PhotoSync's Nextcloud target), and IETF Resumable Uploads for HTTP (RUFH; Apple URLSession supports it natively since iOS 17, for the future native app). No per-protocol staging code.
Long/high-latency uploads (India ↔ the Germany VM): make sure Traefik does not buffer request bodies (streaming passthrough, raised read/idle timeouts for /dav and upload routes), HTTP/2 enabled; on the VM, BBR congestion control + larger TCP buffers (sysctl in /etc/sysctl.d/99-zz-calternal.conf; the orchestrator applies host changes); report the throughput before/after from an Indian client profile (tc netem 150 ms RTT, 1% loss, in CI).
Many small files (the owner's real case: ~100k HEIC/JPG with XMP/AAE sidecars): the server must sustain parallel PUTs: no fsync per file beyond atomic-rename correctness, batched index/feed writes, a PROPFIND that is O(entries) with no per-entry stat storm. Benchmark: 10k small files via rclone --transfers 16 and via PhotoSync-like sequential PUTs; report files/s and p95 per PUT.
Library: keep dav-server 0.11 (Apache-2.0, already a dependency) with our own DavFileSystem adapter over the Files service; contribute fixes upstream where they are generic.

Owner (2026-09-28): 'is there a very performant one for rust that we can improve upon by adding chunked uploads, parallelisation, special handling for long uploads or high latency connections?' Scope added as **phase 2** (after class 1 works and is measured): 1. **One upload staging engine** (refactor the Files tus subset into it: private staging, quota reservation, chunk bookkeeping, expiry, atomic publish through calternal-fs/CAS with streaming hashing). Expose it through three thin adapters: our **tus** (existing), **Nextcloud chunking v2** (`/remote.php/dav/uploads/<user>/<id>/<n>` MKCOL/PUT/MOVE .file, parallel chunks, resume; used by rclone's `--webdav-nextcloud-chunk-size`, the Nextcloud apps and PhotoSync's Nextcloud target), and **IETF Resumable Uploads for HTTP** (RUFH; Apple URLSession supports it natively since iOS 17, for the future native app). No per-protocol staging code. 2. **Long/high-latency uploads (India ↔ the Germany VM):** make sure Traefik does not buffer request bodies (streaming passthrough, raised read/idle timeouts for /dav and upload routes), HTTP/2 enabled; on the VM, BBR congestion control + larger TCP buffers (sysctl in /etc/sysctl.d/99-zz-calternal.conf; the orchestrator applies host changes); report the throughput before/after from an Indian client profile (tc netem 150 ms RTT, 1% loss, in CI). 3. **Many small files** (the owner's real case: ~100k HEIC/JPG with XMP/AAE sidecars): the server must sustain parallel PUTs: no fsync per file beyond atomic-rename correctness, batched index/feed writes, a PROPFIND that is O(entries) with no per-entry stat storm. Benchmark: 10k small files via rclone `--transfers 16` and via PhotoSync-like sequential PUTs; report files/s and p95 per PUT. Library: keep **dav-server 0.11** (Apache-2.0, already a dependency) with our own DavFileSystem adapter over the Files service; contribute fixes upstream where they are generic.

kayg referenced this issue

2026-09-28 13:26:13 +00:00

DOCUMENTS: a Documents plugin with paperless-ngx features — files and plain text, not a database (grill first) #341

kayg referenced this issue

2026-09-28 17:49:32 +00:00

APP PASSWORDS: scopes (protocol, read/write/upload-only, folder prefix, expiry, last used) — the foundation for WebDAV, CalDAV, the HTTP API, MCP and Money automation #328

kayg commented

2026-09-28 18:46:48 +00:00

Author

Owner

Started WebDAV phase 1 on branch job/webdav, based on dev at fba83527f2cccf2334934bb1fd0932be7c0e209b. I am tracing the existing DAV, Files, auth, and calternal-fs paths before adding the class 1 file collection and client tests.

Started WebDAV phase 1 on branch `job/webdav`, based on `dev` at `fba83527f2cccf2334934bb1fd0932be7c0e209b`. I am tracing the existing DAV, Files, auth, and calternal-fs paths before adding the class 1 file collection and client tests.

kayg commented

2026-09-28 18:51:13 +00:00

Author

Owner

Finding: the checked-out dev commit (fba83527f2cccf2334934bb1fd0932be7c0e209b) does not yet contain #328's scoped app-password contract. crates/calternal-auth/migrations/ ends at 0007_read_only_agent_scope.sql; app_passwords still stores no protocol/access/prefix, and verify_app_password returns only User. The #328 worktree is separate at job/app-pw-scopes (acc3796a). I am building the DAV file adapter so it requires an explicit WebDAV authorization scope and will integrate the #328 authority when it arrives; I will not let the legacy CalDAV credential gain Home access by default.

Finding: the checked-out `dev` commit (`fba83527f2cccf2334934bb1fd0932be7c0e209b`) does not yet contain #328's scoped app-password contract. `crates/calternal-auth/migrations/` ends at `0007_read_only_agent_scope.sql`; `app_passwords` still stores no protocol/access/prefix, and `verify_app_password` returns only `User`. The #328 worktree is separate at `job/app-pw-scopes` (`acc3796a`). I am building the DAV file adapter so it requires an explicit WebDAV authorization scope and will integrate the #328 authority when it arrives; I will not let the legacy CalDAV credential gain Home access by default.

kayg commented

2026-09-28 19:43:48 +00:00

Author

Owner

The scripted path/scope probe caught an Axum wildcard extraction bug before integration: /dav/files/{user}/{*path} supplies two path arguments, while the handler extracted one. Nested requests returned 500 before scope checks. The handler now validates the raw URI path segments itself. Regression evidence: traversal GETs return 400 and an upload-only GET returns 403 in cargo test -p calternal-dav.

The scripted path/scope probe caught an Axum wildcard extraction bug before integration: `/dav/files/{user}/{*path}` supplies two path arguments, while the handler extracted one. Nested requests returned 500 before scope checks. The handler now validates the raw URI path segments itself. Regression evidence: traversal GETs return 400 and an upload-only GET returns 403 in `cargo test -p calternal-dav`.

kayg commented

2026-09-28 20:59:52 +00:00

Author

Owner

Root-level PUT exposed a Tus metadata edge case: the adapter sent an empty path value, which the existing metadata parser rejects. WebDAV now omits that field for files directly in Home, and the nested PUT test creates its parent collection first. Regression evidence: cargo test -p calternal-plugin-files — 126 passed; 0 failed.

Root-level PUT exposed a Tus metadata edge case: the adapter sent an empty `path` value, which the existing metadata parser rejects. WebDAV now omits that field for files directly in Home, and the nested PUT test creates its parent collection first. Regression evidence: `cargo test -p calternal-plugin-files` — `126 passed; 0 failed`.

kayg commented

2026-09-28 21:04:56 +00:00

Author

Owner

Root-level PUT exposed a Tus metadata edge case: the adapter sent an empty path value, which the existing metadata parser rejects. WebDAV now omits that field for files directly in Home, and the nested PUT test creates its parent collection first. Regression evidence: the Files crate test run passed all 126 tests.

kayg commented

2026-09-28 21:18:15 +00:00

Author

Owner

After merging dev at 20d10868, I verified that calternal-auth still returns only a User from app-password verification; it has no protocol, access, or Home-prefix metadata. The File WebDAV route requires DavFilesIdentity and returns 403 when it is absent. I kept the route fail-closed because mapping legacy CalDAV credentials to Home access would widen their authority. Production WebDAV authentication needs the scoped app-password support from issue 328 before it can grant File access.

After merging dev at 20d10868, I verified that calternal-auth still returns only a User from app-password verification; it has no protocol, access, or Home-prefix metadata. The File WebDAV route requires DavFilesIdentity and returns 403 when it is absent. I kept the route fail-closed because mapping legacy CalDAV credentials to Home access would widen their authority. Production WebDAV authentication needs the scoped app-password support from issue 328 before it can grant File access.

kayg commented

2026-09-28 22:45:58 +00:00

Author

Owner

Phase 1 implementation is committed on job/webdav and pushed. HEAD: 9923c3e34aebe98af8121cefb9b5aeafde303f40. git push origin job/webdav returned Everything up-to-date (exit 0). I merged dev once at 20d10868; it merged without conflicts.

Built RFC 4918 class 1 File WebDAV with the calternal-dav adapter, production Files provider/route, and a scripted adversarial client. The loopback test starts the production Axum DAV route and provider with a test-only authenticated identity. The hostile-input probes passed, including traversal/NUL paths, Unicode names, malformed/oversized XML, huge/infinity PROPFIND depth, unsupported LOCK, conditional requests, Range, COPY/MOVE, Trash-backed DELETE, parallel requests, and cross-user paths. Litmus is not installed.

The official rclone v1.75.1 binary was checksum-verified and used against the local server. It transferred 10,000 files / 340,000 bytes in 3,253.09 seconds: 3.07 files/s and about 104 bytes/s on the shared host. This is a SLOW-only result.

Gates and validation:

cargo fmt --all -- --check: no output; exit 0.
cargo clippy --all-targets -- -D warnings: stopped at the four-hour job limit; exit 130. Last output was Checking dtoa v1.0.11; no diagnostics had appeared. This is incomplete, not a pass.
Full-workspace cargo test: not run before the job limit. cargo test -p calternal-plugin-files passed: test result: ok. 126 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 136.58s (doc tests: 0 passed, 0 failed).
bun run --cwd apps/web check: svelte-check found 0 errors and 0 warnings.
bun run --cwd apps/web test: Test Files 114 passed (114) and Tests 751 passed (751).
cargo clean: Removed 12324 files, 4.3GiB total.

Known gaps and decisions:

The merged dev app-password verifier still returns only a User; it does not provide the protocol/access/Home-prefix scopes from #328. The production DAV route therefore fails closed with 403 until that support lands. The local proof injects a test identity and does not claim to verify production app-password scope integration.
Recursive COPY over an existing collection returns Conflict; collection overwrite is not implemented.
PROPFIND is bounded to 50,000 entries and returns TooLarge above that bound to limit memory use.
I kept production auth fail-closed because enabling legacy credentials without the agreed scopes would widen access. The 50,000-entry bound is the implementation limit used for bounded listing work.

Phase 1 implementation is committed on `job/webdav` and pushed. HEAD: `9923c3e34aebe98af8121cefb9b5aeafde303f40`. `git push origin job/webdav` returned `Everything up-to-date` (exit 0). I merged `dev` once at `20d10868`; it merged without conflicts. Built RFC 4918 class 1 File WebDAV with the `calternal-dav` adapter, production Files provider/route, and a scripted adversarial client. The loopback test starts the production Axum DAV route and provider with a test-only authenticated identity. The hostile-input probes passed, including traversal/NUL paths, Unicode names, malformed/oversized XML, huge/infinity PROPFIND depth, unsupported LOCK, conditional requests, Range, COPY/MOVE, Trash-backed DELETE, parallel requests, and cross-user paths. Litmus is not installed. The official rclone v1.75.1 binary was checksum-verified and used against the local server. It transferred 10,000 files / 340,000 bytes in 3,253.09 seconds: 3.07 files/s and about 104 bytes/s on the shared host. This is a SLOW-only result. Gates and validation: - `cargo fmt --all -- --check`: no output; exit 0. - `cargo clippy --all-targets -- -D warnings`: stopped at the four-hour job limit; exit 130. Last output was `Checking dtoa v1.0.11`; no diagnostics had appeared. This is incomplete, not a pass. - Full-workspace `cargo test`: not run before the job limit. `cargo test -p calternal-plugin-files` passed: `test result: ok. 126 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 136.58s` (doc tests: 0 passed, 0 failed). - `bun run --cwd apps/web check`: `svelte-check found 0 errors and 0 warnings`. - `bun run --cwd apps/web test`: `Test Files 114 passed (114)` and `Tests 751 passed (751)`. - `cargo clean`: `Removed 12324 files, 4.3GiB total`. Known gaps and decisions: - The merged `dev` app-password verifier still returns only a `User`; it does not provide the protocol/access/Home-prefix scopes from #328. The production DAV route therefore fails closed with 403 until that support lands. The local proof injects a test identity and does not claim to verify production app-password scope integration. - Recursive COPY over an existing collection returns Conflict; collection overwrite is not implemented. - PROPFIND is bounded to 50,000 entries and returns TooLarge above that bound to limit memory use. - I kept production auth fail-closed because enabling legacy credentials without the agreed scopes would widen access. The 50,000-entry bound is the implementation limit used for bounded listing work.

kayg commented

2026-09-28 22:48:50 +00:00

Author

Owner

Review of job/webdav `9923c3e3` (Claude): not merged, performance round

The owner's real case is ~100k photos with sidecars. 3.07 files/s is about 9 hours for 100k files, which does not meet the issue's "many small files" requirement (comment 3). The shared-host load explains part of it, not a 300 ms per-file cost.

Profile one small PUT end to end (tracing spans with timings): auth/verifier (a password hash per request? cache the verified app password per connection or for a short TTL, with revocation checked through the session store), quota check, atomic write + fsync, Index row, change-feed row, SSE fan-out, Search enqueue, thumbnail enqueue. Report the per-stage breakdown.
Fix the dominant costs: batch Index and feed writes (group commit), never run a password-hash verification per request, defer derived work (search/thumbnail) to the queue, one fsync per file plus a directory fsync batched per folder.
Measure parallel clients as real clients use them: rclone copy --transfers 16 --checkers 16, 10k files of 30 KB and 1k files of 3 MB, on a quiet moment (check load average <= 8 and record it), with interleaved before/after runs. Target: at least 200 small files/s on this host, or a stated, measured bound that explains why not.
Finish the gates: clippy (OPENSSL_NO_VENDOR=1 is preset now) and cargo test -p calternal-dav -p calternal-plugin-files -p calternal-server.
Production auth: keep fail-closed until #328 merges. Then wire the protocol/access/Home-prefix scopes: merge dev when #328 lands and finish it in the same branch.

## Review of job/webdav 9923c3e3 (Claude): not merged, performance round The owner's real case is ~100k photos with sidecars. 3.07 files/s is about 9 hours for 100k files, which does not meet the issue's "many small files" requirement (comment 3). The shared-host load explains part of it, not a 300 ms per-file cost. 1. **Profile one small PUT end to end** (tracing spans with timings): auth/verifier (a password hash per request? cache the verified app password per connection or for a short TTL, with revocation checked through the session store), quota check, atomic write + fsync, Index row, change-feed row, SSE fan-out, Search enqueue, thumbnail enqueue. Report the per-stage breakdown. 2. **Fix the dominant costs:** batch Index and feed writes (group commit), never run a password-hash verification per request, defer derived work (search/thumbnail) to the queue, one fsync per file plus a directory fsync batched per folder. 3. **Measure parallel clients** as real clients use them: `rclone copy --transfers 16 --checkers 16`, 10k files of 30 KB and 1k files of 3 MB, on a quiet moment (check load average <= 8 and record it), with interleaved before/after runs. Target: at least 200 small files/s on this host, or a stated, measured bound that explains why not. 4. **Finish the gates:** clippy (OPENSSL_NO_VENDOR=1 is preset now) and `cargo test -p calternal-dav -p calternal-plugin-files -p calternal-server`. 5. Production auth: keep fail-closed until #328 merges. Then wire the protocol/access/Home-prefix scopes: merge dev when #328 lands and finish it in the same branch.

kayg commented

2026-09-28 22:49:57 +00:00

Author

Owner

Starting the performance round on branch job/webdav at 9923c3e34a; its last dev merge is 20d10868. Current dev is deb16f61 and #328 remains open, so production WebDAV auth stays fail-closed while I profile and optimize the Files path.

Starting the performance round on branch job/webdav at 9923c3e34aebe98af8121cefb9b5aeafde303f40; its last dev merge is 20d10868. Current dev is deb16f61 and #328 remains open, so production WebDAV auth stays fail-closed while I profile and optimize the Files path.

kayg commented

2026-09-28 22:57:25 +00:00

Author

Owner

Finding: the latest load average at the start of measurement was 47.84 / 55.60 / 48.34, above the requested <=8 quiet-host limit. The shared job rule says not to wait for a quiet host, so I will record load alongside one time-boxed run and treat the result as load-bound. No rclone binary is installed in PATH; I am restoring the previously used pinned v1.75.1 from the official release with its published checksum.

kayg referenced this issue

2026-09-28 23:02:09 +00:00

APP PASSWORDS: scopes (protocol, read/write/upload-only, folder prefix, expiry, last used) — the foundation for WebDAV, CalDAV, the HTTP API, MCP and Money automation #328

kayg commented

2026-09-28 23:24:48 +00:00

Author

Owner

Finding: the opt-in tracing profile of the Files loopback route showed a single 30 KB DAV PUT split into two provider chunk writes. Their durable staging writes took 53.4 ms and 14.0 ms, followed by two offset commits at 6.4 ms and 6.1 ms. Final atomic install took 74.5 ms; the Index UPSERT plus its feed trigger took 1.1 ms; thumbnail enqueue took 0.03 ms. The owner PUT also committed two identical Files events (0.9 ms and 0.5 ms). was 157.7 ms. This run was under the recorded load average >47, and the loopback injects test identity, so it does not include production password verification. I am fixing the extra chunk fsync/DB transaction from body-frame splitting and the duplicate same-owner event before measuring throughput again.

kayg commented

2026-09-28 23:24:59 +00:00

Author

Owner

Correction to the profile report: the finalization total was 157.7 ms. The previous comment had shell quoting around the stage name, but its measurement is unchanged.

kayg commented

2026-09-28 23:38:34 +00:00

Author

Owner

Updated profile under a second high-load sample: one 30,000-byte loopback PUT took 288.59 ms end-to-end. The two staging writes took 64.3 ms and 14.8 ms; their two offset commits took 1.1 ms and 9.1 ms; final atomic install took 102.4 ms; Index plus feed trigger took 0.82 ms; thumbnail enqueue took 0.02 ms. Finalization took 162.7 ms and still made two identical owner event commits (0.61 ms and 2.65 ms). Load average was 42.97 / 47.99 / 46.38 before and 41.99 / 47.53 / 46.26 after. This is another SLOW-only profile, not a quiet-host throughput result.

kayg commented

2026-09-28 23:57:56 +00:00

Author

Owner

WebDAV PUT profile update (HEAD fd721a44): DavPutBuffer coalesces body frames into bounded 1 MiB writes, and drains the final buffer only after checking the declared length. The new split-frame and exact-boundary tests pass; cargo test -p calternal-dav passed (15 unit tests, 8 replay tests, 0 doc tests).

The 30,000-byte profile now emits one staging_write_and_fsync (46.885 ms) and one upload_offset_commit (0.679 ms), compared with two of each in the earlier sample (79.1 ms of staging writes and 10.2 ms of offset commits combined). End-to-end was 374.92 ms in this run; finalization was 255.3 ms versus 162.7 ms in the earlier sample. This profile overlapped the baseline 16-transfer rclone run, so the end-to-end comparison is load-confounded; I am retaining the stage counts and timings as the evidence for the specific optimization.

#328 remains open, so production DAV authentication is still fail-closed and no scopes are wired from it.

WebDAV PUT profile update (HEAD `fd721a44`): `DavPutBuffer` coalesces body frames into bounded 1 MiB writes, and drains the final buffer only after checking the declared length. The new split-frame and exact-boundary tests pass; `cargo test -p calternal-dav` passed (15 unit tests, 8 replay tests, 0 doc tests). The 30,000-byte profile now emits one `staging_write_and_fsync` (46.885 ms) and one `upload_offset_commit` (0.679 ms), compared with two of each in the earlier sample (79.1 ms of staging writes and 10.2 ms of offset commits combined). End-to-end was 374.92 ms in this run; finalization was 255.3 ms versus 162.7 ms in the earlier sample. This profile overlapped the baseline 16-transfer rclone run, so the end-to-end comparison is load-confounded; I am retaining the stage counts and timings as the evidence for the specific optimization. `#328` remains open, so production DAV authentication is still fail-closed and no scopes are wired from it.

kayg commented

2026-09-29 00:08:54 +00:00

Author

Owner

Lease safety finding and fix (a1f92cdf): Files renews the 15-minute idle upload lease when it commits a chunk. Buffering DAV frames without periodic checkpoints could let cleanup expire a slow but active stream. DAV now durably flushes pending bytes after five minutes since the last committed chunk, and the new boundary test verifies the pending tail stays buffered until that point. cargo test -p calternal-dav passed: 16 unit tests and 8 replay tests.

Decision not stated in DESIGN: the checkpoint interval is five minutes, below Files' 15-minute idle lease. The periodic checkpoint keeps slow streams active while preserving frame coalescing for normal transfers.

Lease safety finding and fix (`a1f92cdf`): Files renews the 15-minute idle upload lease when it commits a chunk. Buffering DAV frames without periodic checkpoints could let cleanup expire a slow but active stream. DAV now durably flushes pending bytes after five minutes since the last committed chunk, and the new boundary test verifies the pending tail stays buffered until that point. `cargo test -p calternal-dav` passed: 16 unit tests and 8 replay tests. Decision not stated in DESIGN: the checkpoint interval is five minutes, below Files' 15-minute idle lease. The periodic checkpoint keeps slow streams active while preserving frame coalescing for normal transfers.

kayg commented

2026-09-29 00:32:26 +00:00

Author

Owner

Parallel rclone baseline (saved pre-coalescing binary from 670c1c2f): 16 transfers, 10,000 files × 30,000 bytes. The 45-minute cap expired with 7,111 files installed (exit 124); the test's completion summary was not emitted because the process was interrupted. Observed load averages during the run included 40.51 / 44.43 / 45.31 at 1:22, 45.95 / 43.90 / 44.85 at 4:22, and 30.71 / 30.78 / 32.15 at the cap. This is a partial, load-confounded baseline; I will use the same transfer count and time cap for the optimized comparison.

Parallel rclone baseline (saved pre-coalescing binary from `670c1c2f`): 16 transfers, 10,000 files × 30,000 bytes. The 45-minute cap expired with 7,111 files installed (exit 124); the test's completion summary was not emitted because the process was interrupted. Observed load averages during the run included 40.51 / 44.43 / 45.31 at 1:22, 45.95 / 43.90 / 44.85 at 4:22, and 30.71 / 30.78 / 32.15 at the cap. This is a partial, load-confounded baseline; I will use the same transfer count and time cap for the optimized comparison.

kayg commented

2026-09-29 02:08:56 +00:00

Author

Owner

Parallel rclone result after PUT buffering:

Same workload as baseline: 10,000 files × 30,000 bytes, 16 transfers and checkers, 45-minute cap.
The cap expired at 45:00 (exit 124). The server had installed 5,671 files. The test completion summary was not emitted because timeout interrupted the test.
The adversarial tests/adversarial/webdav.py preamble passed; the test reached rclone.
Load average at run start was 25.87 / 33.74 / 36.55; the in-test rclone start sample was 26.20 / 33.68 / 36.51. The sampled 1-minute load peak was 61.38 / 45.68 / 40.62 at 03:38:03. Load at the 45-minute cap was 16.08 / 23.97 / 31.78.
The saved pre-buffer baseline installed 7,111 files at its 45-minute cap, with recorded load samples 40.51 / 44.43 / 45.31 at 1:22 and 45.95 / 43.90 / 44.85 at 4:22.

The two partial results are load-confounded and do not establish an end-to-end speedup or regression. No further rclone rounds are planned. Minute-by-minute samples and the exact test output are in target/tmp/issue-339-rclone-after-1790644973-load.log and target/tmp/issue-339-rclone-after-1790644973.log.

Parallel rclone result after PUT buffering: - Same workload as baseline: 10,000 files × 30,000 bytes, 16 transfers and checkers, 45-minute cap. - The cap expired at 45:00 (exit 124). The server had installed 5,671 files. The test completion summary was not emitted because timeout interrupted the test. - The adversarial `tests/adversarial/webdav.py` preamble passed; the test reached rclone. - Load average at run start was 25.87 / 33.74 / 36.55; the in-test rclone start sample was 26.20 / 33.68 / 36.51. The sampled 1-minute load peak was 61.38 / 45.68 / 40.62 at 03:38:03. Load at the 45-minute cap was 16.08 / 23.97 / 31.78. - The saved pre-buffer baseline installed 7,111 files at its 45-minute cap, with recorded load samples 40.51 / 44.43 / 45.31 at 1:22 and 45.95 / 43.90 / 44.85 at 4:22. The two partial results are load-confounded and do not establish an end-to-end speedup or regression. No further rclone rounds are planned. Minute-by-minute samples and the exact test output are in `target/tmp/issue-339-rclone-after-1790644973-load.log` and `target/tmp/issue-339-rclone-after-1790644973.log`.

kayg commented

2026-09-29 03:00:08 +00:00

Author

Owner

Final report

Branch job/webdav is pushed. Head: bec90c0b997157abf4c7b1d3e84f4eff5b47f87a.

Built

Merged dev at 8a068b615b1ff7cb1811cc7ce932eec3f386c19d and wired WebDAV app-password scopes through Basic and Bearer authentication into the Files DAV identity. WebDAV credentials without a valid WebDAV scope remain denied. The Files boundary enforces the canonical Home prefix.
Added ReadUploadOnly, which allows file reads and create-only writes. Existing names are preserved by renaming the new upload. Write allows reads, PUT and MKCOL; destructive DAV operations stay Full-only.
Kept the buffered PUT path and its five-minute idle flush that renews the upload lease. This reduces staging commits for small PUTs and still renews leases for slow active streams.
Corrected the merged theme test’s indexed match access so svelte-check can prove it is present. Its expectation is unchanged.

Directly changed files

crates/calternal-server/src/wire.rs
crates/calternal-dav/src/files.rs
crates/plugins/files/src/dav.rs
crates/calternal-dav/tests/apple_replay.rs
apps/web/src/lib/themes.test.ts

The dev merge also contains its upstream changes across the workspace. Commits: b348deaa (merge and WebDAV scope wiring), bec90c0b (web type narrowing).

Performance round

The small PUT stage profile reduced staging writes from two to one and offset commits from two to one. The post-buffer 30 KB sample was 374.92 ms versus 288.59 ms before; its run overlapped a noisy host, so this is not an end-to-end speedup claim.
Parallel rclone used 16 transfers and checkers with 10,000 × 30 KB files and the same 45 minute cap. Baseline installed 7,111 files; optimized installed 5,671 files before exit 124. The optimized run’s sampled 1 minute load reached a low sample of 19.14 / 29.80 / 36.11 and a peak of 61.38 / 45.68 / 40.62, then ended at 16.08 / 23.97 / 31.78. This comparison is load-confounded and establishes neither a regression nor an improvement. The adversarial tests/adversarial/webdav.py preamble passed before rclone started. No more benchmark rounds were run.

Gates

cargo fmt --check emitted no output and exited 0.

cargo clippy --all-targets -- -D warnings output:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 8m 12s

cargo test output:

Finished `test` profile [unoptimized + debuginfo] target(s) in 13m 29s
test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 128 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 196.11s
test result: ok. 76 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 15.89s

Across the workspace output: 76 test-result groups, 1,501 passed, 0 failed, 13 ignored. Cargo exited 0.

bun run check output:

$ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/webdav/apps/web
Getting Svelte diagnostics...
svelte-check found 0 errors and 0 warnings

bun run test output:

Test Files  117 passed (117)
     Tests  764 passed (764)
Start at  04:50:43
Duration  122.87s (transform 49%, environment 21%, import 14%, tests 11%, setup 5%)

Vitest also printed Could not parse CSS stylesheet and jsdom scrollTo() notices; all 764 tests passed.

cargo clean output:

Removed 18052 files, 15.0GiB total

The generated apps/web/build, apps/web/.svelte-kit, rclone download, and benchmark transfer trees were removed. The working tree is clean.

Decisions not set by DESIGN

When one credential has both WebDAV Read and UploadOnly scopes, combine them as ReadUploadOnly: reads are allowed, creates are allowed, and replacements are renamed. This follows the API’s ability to store multiple scope pairs.
WebDAV Write follows the Files DAV adapter: it permits reads, create, and replace, but not DELETE, COPY, or MOVE. Only Full permits those destructive methods.
Continue flushing pending PUT bytes after five idle minutes. This bounds lease renewal for slow active streams while small PUTs still use one staging commit.
During merge resolution, retained dev commit b7d6174f’s valid Apple calendar-color PROPPATCH behavior and its updated 200 expectation. No unrelated test expectation was changed beyond that upstream behavior change.

## Final report Branch `job/webdav` is pushed. Head: `bec90c0b997157abf4c7b1d3e84f4eff5b47f87a`. ### Built - Merged `dev` at `8a068b615b1ff7cb1811cc7ce932eec3f386c19d` and wired WebDAV app-password scopes through Basic and Bearer authentication into the Files DAV identity. WebDAV credentials without a valid WebDAV scope remain denied. The Files boundary enforces the canonical Home prefix. - Added `ReadUploadOnly`, which allows file reads and create-only writes. Existing names are preserved by renaming the new upload. Write allows reads, PUT and MKCOL; destructive DAV operations stay Full-only. - Kept the buffered PUT path and its five-minute idle flush that renews the upload lease. This reduces staging commits for small PUTs and still renews leases for slow active streams. - Corrected the merged theme test’s indexed match access so `svelte-check` can prove it is present. Its expectation is unchanged. ### Directly changed files - `crates/calternal-server/src/wire.rs` - `crates/calternal-dav/src/files.rs` - `crates/plugins/files/src/dav.rs` - `crates/calternal-dav/tests/apple_replay.rs` - `apps/web/src/lib/themes.test.ts` The `dev` merge also contains its upstream changes across the workspace. Commits: `b348deaa` (merge and WebDAV scope wiring), `bec90c0b` (web type narrowing). ### Performance round - The small PUT stage profile reduced staging writes from two to one and offset commits from two to one. The post-buffer 30 KB sample was 374.92 ms versus 288.59 ms before; its run overlapped a noisy host, so this is not an end-to-end speedup claim. - Parallel rclone used 16 transfers and checkers with 10,000 × 30 KB files and the same 45 minute cap. Baseline installed 7,111 files; optimized installed 5,671 files before exit 124. The optimized run’s sampled 1 minute load reached a low sample of 19.14 / 29.80 / 36.11 and a peak of 61.38 / 45.68 / 40.62, then ended at 16.08 / 23.97 / 31.78. This comparison is load-confounded and establishes neither a regression nor an improvement. The adversarial `tests/adversarial/webdav.py` preamble passed before rclone started. No more benchmark rounds were run. ### Gates `cargo fmt --check` emitted no output and exited 0. `cargo clippy --all-targets -- -D warnings` output: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 8m 12s ``` `cargo test` output: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 13m 29s test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 128 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 196.11s test result: ok. 76 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 15.89s ``` Across the workspace output: 76 test-result groups, 1,501 passed, 0 failed, 13 ignored. Cargo exited 0. `bun run check` output: ```text $ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/webdav/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run test` output: ```text Test Files 117 passed (117) Tests 764 passed (764) Start at 04:50:43 Duration 122.87s (transform 49%, environment 21%, import 14%, tests 11%, setup 5%) ``` Vitest also printed `Could not parse CSS stylesheet` and jsdom `scrollTo()` notices; all 764 tests passed. `cargo clean` output: ```text Removed 18052 files, 15.0GiB total ``` The generated `apps/web/build`, `apps/web/.svelte-kit`, rclone download, and benchmark transfer trees were removed. The working tree is clean. ### Decisions not set by DESIGN - When one credential has both WebDAV Read and UploadOnly scopes, combine them as `ReadUploadOnly`: reads are allowed, creates are allowed, and replacements are renamed. This follows the API’s ability to store multiple scope pairs. - WebDAV Write follows the Files DAV adapter: it permits reads, create, and replace, but not DELETE, COPY, or MOVE. Only Full permits those destructive methods. - Continue flushing pending PUT bytes after five idle minutes. This bounds lease renewal for slow active streams while small PUTs still use one staging commit. - During merge resolution, retained `dev` commit `b7d6174f`’s valid Apple calendar-color PROPPATCH behavior and its updated 200 expectation. No unrelated test expectation was changed beyond that upstream behavior change.

kayg referenced this issue

2026-09-29 03:01:11 +00:00

PERF ROUND: benchmark + optimise average and worst-case CPU/memory (after current merges) #367

kayg commented

2026-09-29 03:02:05 +00:00

Author

Owner

Starting final integration on job/webdav at bec90c0b99. Branch base (merge-base with dev): 8a068b615b; dev HEAD: 5afb191be1.

Starting final integration on job/webdav at bec90c0b997157abf4c7b1d3e84f4eff5b47f87a. Branch base (merge-base with dev): 8a068b615b1ff7cb1811cc7ce932eec3f386c19d; dev HEAD: 5afb191be1dcf192c606a1071b750e640edc2ef1.

kayg referenced this issue

2026-09-29 04:15:56 +00:00

Fix WebDAV status codes for traversal and stale If-Match requests #390

kayg commented

2026-09-29 04:16:52 +00:00

Author

Owner

Finished

Merged dev once into job/webdav and pushed the branch. The push reported Everything up-to-date; the remote already had this head.

Head SHA: 47b07b47fc34a7aa1e15eaca54784e4b3df379c6.

The merge keeps both sides' work in crates/calternal-dav/tests/apple_replay.rs and crates/calternal-server/src/wire.rs. I took dev's version of apps/web/src/lib/themes.test.ts. The Apple Reminders replay coverage and dev's quota, profile signing, MCP, and jobs routes/tests remain present alongside the WebDAV tests and routes.

Gates

All final gates below exited 0. Output excerpts are verbatim.

cargo fmt --all --check

(stdout empty)

cargo test -p calternal-dav

running 18 tests
18 passed; 0 failed
running 13 tests
13 passed; 0 failed

cargo test -p calternal-plugin-files

test result: ok. 128 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 216.34s

cargo check -p calternal-server

Finished `dev` profile [unoptimized + debuginfo] target(s) in 24.22s

The first server check found that apps/web/build/ was absent. I ran the production web build, then reran the server check successfully. The build output included:

✓ built in 1m 41s
Wrote site to "build"

cargo clippy --all-targets -- -D warnings

Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 38s

bun run check (from apps/web)

$ node scripts/check-type-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Text sizes use shared role tokens.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/webdav/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

Adversarial round

The real-server WebDAV probe denied encoded traversal with 404 and an empty body; the probe expected 400. A concurrent pair of PUT requests with one strong If-Match ETag returned [204, 405]: one update succeeded and the stale update was rejected with the wrong status. I filed both status issues in #390. No unauthorized data was returned and no data loss was observed.

Two results from the temporary upload-only probe were invalid checks: Photos already existed, so MKCOL correctly returned 405; the numbered-name listing assertion searched for a literal space in an href. I removed the temporary probe after the one adversarial round. I did not change any test expectations.

Files and decisions

The conflict-resolution files are apps/web/src/lib/themes.test.ts, crates/calternal-dav/tests/apple_replay.rs, and crates/calternal-server/src/wire.rs. The merged branch also contains the Files WebDAV adapter and its tests in crates/plugins/files/src/dav.rs, the DAV protocol implementation in crates/calternal-dav/src/files.rs, and the WebDAV route integration in crates/calternal-server/src/wire.rs.

Decisions: used dev's theme test as directed, and retained both branches' Apple replay tests and wire routes/tests as directed. No additional product or API design decision was needed. Throughput work stays in #367.

Cleanup: removed the generated web build output and ran cargo clean (Removed 17503 files, 8.7GiB total). The worktree is clean.

## Finished Merged `dev` once into `job/webdav` and pushed the branch. The push reported `Everything up-to-date`; the remote already had this head. Head SHA: `47b07b47fc34a7aa1e15eaca54784e4b3df379c6`. The merge keeps both sides' work in `crates/calternal-dav/tests/apple_replay.rs` and `crates/calternal-server/src/wire.rs`. I took `dev`'s version of `apps/web/src/lib/themes.test.ts`. The Apple Reminders replay coverage and dev's quota, profile signing, MCP, and jobs routes/tests remain present alongside the WebDAV tests and routes. ## Gates All final gates below exited 0. Output excerpts are verbatim. `cargo fmt --all --check` ```text (stdout empty) ``` `cargo test -p calternal-dav` ```text running 18 tests 18 passed; 0 failed running 13 tests 13 passed; 0 failed ``` `cargo test -p calternal-plugin-files` ```text test result: ok. 128 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 216.34s ``` `cargo check -p calternal-server` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 24.22s ``` The first server check found that `apps/web/build/` was absent. I ran the production web build, then reran the server check successfully. The build output included: ```text ✓ built in 1m 41s Wrote site to "build" ``` `cargo clippy --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 38s ``` `bun run check` (from `apps/web`) ```text $ node scripts/check-type-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Text sizes use shared role tokens. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/webdav/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` ## Adversarial round The real-server WebDAV probe denied encoded traversal with 404 and an empty body; the probe expected 400. A concurrent pair of PUT requests with one strong `If-Match` ETag returned `[204, 405]`: one update succeeded and the stale update was rejected with the wrong status. I filed both status issues in #390. No unauthorized data was returned and no data loss was observed. Two results from the temporary upload-only probe were invalid checks: `Photos` already existed, so MKCOL correctly returned 405; the numbered-name listing assertion searched for a literal space in an href. I removed the temporary probe after the one adversarial round. I did not change any test expectations. ## Files and decisions The conflict-resolution files are `apps/web/src/lib/themes.test.ts`, `crates/calternal-dav/tests/apple_replay.rs`, and `crates/calternal-server/src/wire.rs`. The merged branch also contains the Files WebDAV adapter and its tests in `crates/plugins/files/src/dav.rs`, the DAV protocol implementation in `crates/calternal-dav/src/files.rs`, and the WebDAV route integration in `crates/calternal-server/src/wire.rs`. Decisions: used `dev`'s theme test as directed, and retained both branches' Apple replay tests and wire routes/tests as directed. No additional product or API design decision was needed. Throughput work stays in #367. Cleanup: removed the generated web build output and ran `cargo clean` (`Removed 17503 files, 8.7GiB total`). The worktree is clean.

kayg commented

2026-09-29 04:28:24 +00:00

Author

Owner

Phase 1 merged into dev by Claude and deploying. Throughput work continues in #367 (quiet host).

kayg referenced this issue from a commit

2026-09-29 04:28:25 +00:00

Merge job/webdav (#339 phase 1: file WebDAV for Home, class 1, app-password scopes incl. upload-only; throughput target in perf round #367)

kayg referenced this issue

2026-09-29 07:32:14 +00:00

PERF: WebDAV small files and large file on the low-spec perf-test VM, local and from the Mac at home #409

kayg referenced this issue