#306 - RESEARCH: standard file-transfer protocols for Home (WebDAV first; SFTP/FTPS/S3/others) — rootless, PhotoSync-ready (NO implementation) - kayg/calternal

kayg commented

2026-09-28 07:14:45 +00:00

Owner

Owner (2026-09-28): 'we need to expose webdav or ftps/sftp or some sort of proper/standard protocol that can run in user-mode without requiring root for our app. i ask because I use PhotoSync on iOS to upload my files through immich via a make-shift watcher → uploader from a vps. it would be much nicer if I could now upload all my newer photos/videos into calternal instead. … nextcloud does webdav so we can also do webdav so keep it standard but I think we should be able to support a couple more protocols as well?'

State today: crates/calternal-dav is a CalDAV projection only (/dav, Log entries). There is no file WebDAV. Uploads go through the web/PWA and the calternal CLI/sync (calternald). Deployment: a rootless podman container on the calternal-cloud VM behind the k3s Traefik edge (HTTPS on 443; the container listens on 8080; only HTTP(S) is routed today). Security rule (CLAUDE.md): the server is the single writer; all filesystem access goes through calternal-fs (openat2 RESOLVE_BENEATH); never build paths from user input; per-user isolation. Auth: passkeys, sessions, and app passwords (table app_passwords exists) for non-browser clients. The owner rule is to be opinionated and easy to use.

Research (web search; primary sources, RFCs, client docs):

Client compatibility matrix: PhotoSync (iOS/Android: exactly which protocols and features it supports: WebDAV, SFTP, FTP/FTPS, SMB, S3, Nextcloud/ownCloud, its 'auto-transfer' and background upload, how it handles Live Photos, HEIC/HEVC + .AAE sidecars, RAW, originals vs edited, date-based folders, duplicate detection), iOS Files and macOS Finder (native WebDAV/SMB/SFTP/FTPS support), Windows Explorer, Linux GVFS/KDE, rclone, Cyberduck/Transmit, Obsidian mobile sync plugins, Joplin, Zotero, Nextcloud clients, Immich-style CLI uploaders, the S3 ecosystem.
Protocols: WebDAV (RFC 4918, class 1/2 locking, Nextcloud's chunked upload v2 and its OCS headers that many apps expect, resumable uploads with the IETF draft 'Resumable Uploads for HTTP' and tus), SFTP (SSH; a rootless listener on a high port; how the edge would route raw TCP: Traefik TCP routers with SNI do not apply to SSH, so a separate port/LoadBalancer on the k3s host or a direct port on the VM), FTPS (explicit/implicit, passive port ranges through NAT, a poor fit), SMB (heavy; Samba licences; kernel/root concerns), the S3 API (MinIO/Garage-style; used by many backup apps), plus anything newer that fits.
Rust implementations (AGPL-compatible licences only): dav-server/webdav-handler, russh/thrussh + an SFTP subsystem (russh-sftp), libunftp/unftp (FTPS, pluggable storage back ends), s3s (the S3 server framework), tus servers. How each would sit on top of calternal-fs as a storage back end (no direct paths), streaming large files (4K video) with bounded memory, and the content-addressed store/dedup (.cas).
Semantics: mapping to Home and Photos (does an upload to Photos/ trigger the Photos pipeline, capture-date foldering per DESIGN Photos/YYYY/YYYY-MM-DD/, pairs/stacks); an 'inbox' drop folder vs direct paths; conflicts with sync and the collab editor; atomic writes (see #305: temp files must stay invisible); quotas; deletes going to Trash; versions.
Security: app passwords scoped per protocol and per folder (e.g. a PhotoSync password that can only write to Photos/, no read, no delete), rate limits, brute-force lockout, TLS, audit events, and the adversarial surface (path tricks, huge uploads, slowloris, zip/rename races).
Ops: rootless ports, the k3s Traefik edge (HTTP for WebDAV/S3/tus is easy; SFTP/FTPS need TCP exposure: describe the options and the firewall changes on the VM, /etc/calternal-firewall.nft), resource cost on the 4-vCPU VM.
Output: docs/research/file-protocols.md (ASD-STE100) with a comparison table, a recommendation (the expected shape is 'WebDAV (Nextcloud-compatible where cheap) first, then one or two more'), a PhotoSync setup walkthrough for the recommended protocol, and numbered grill questions (grilling format). Do NOT implement. Commit on the branch, push, and post a summary on this issue.

Owner (2026-09-28): 'we need to expose webdav or ftps/sftp or some sort of proper/standard protocol that can run in user-mode without requiring root for our app. i ask because I use PhotoSync on iOS to upload my files through immich via a make-shift watcher → uploader from a vps. it would be much nicer if I could now upload all my newer photos/videos into calternal instead. … nextcloud does webdav so we can also do webdav so keep it standard but I think we should be able to support a couple more protocols as well?' State today: `crates/calternal-dav` is a **CalDAV projection only** (/dav, Log entries). There is no file WebDAV. Uploads go through the web/PWA and the calternal CLI/sync (`calternald`). Deployment: a rootless podman container on the calternal-cloud VM behind the k3s Traefik edge (HTTPS on 443; the container listens on 8080; only HTTP(S) is routed today). Security rule (CLAUDE.md): the server is the single writer; all filesystem access goes through calternal-fs (openat2 RESOLVE_BENEATH); never build paths from user input; per-user isolation. Auth: passkeys, sessions, and **app passwords** (table `app_passwords` exists) for non-browser clients. The owner rule is to be opinionated and easy to use. Research (web search; primary sources, RFCs, client docs): 1. **Client compatibility matrix**: PhotoSync (iOS/Android: exactly which protocols and features it supports: WebDAV, SFTP, FTP/FTPS, SMB, S3, Nextcloud/ownCloud, its 'auto-transfer' and background upload, how it handles Live Photos, HEIC/HEVC + .AAE sidecars, RAW, originals vs edited, date-based folders, duplicate detection), iOS Files and macOS Finder (native WebDAV/SMB/SFTP/FTPS support), Windows Explorer, Linux GVFS/KDE, rclone, Cyberduck/Transmit, Obsidian mobile sync plugins, Joplin, Zotero, Nextcloud clients, Immich-style CLI uploaders, the S3 ecosystem. 2. **Protocols**: WebDAV (RFC 4918, class 1/2 locking, Nextcloud's chunked upload v2 and its OCS headers that many apps expect, resumable uploads with the IETF draft 'Resumable Uploads for HTTP' and tus), SFTP (SSH; a rootless listener on a high port; how the edge would route raw TCP: Traefik TCP routers with SNI do not apply to SSH, so a separate port/LoadBalancer on the k3s host or a direct port on the VM), FTPS (explicit/implicit, passive port ranges through NAT, a poor fit), SMB (heavy; Samba licences; kernel/root concerns), the S3 API (MinIO/Garage-style; used by many backup apps), plus anything newer that fits. 3. **Rust implementations** (AGPL-compatible licences only): dav-server/webdav-handler, russh/thrussh + an SFTP subsystem (russh-sftp), libunftp/unftp (FTPS, pluggable storage back ends), s3s (the S3 server framework), tus servers. How each would sit on top of calternal-fs as a storage back end (no direct paths), streaming large files (4K video) with bounded memory, and the content-addressed store/dedup (.cas). 4. **Semantics**: mapping to Home and Photos (does an upload to `Photos/` trigger the Photos pipeline, capture-date foldering per DESIGN `Photos/YYYY/YYYY-MM-DD/`, pairs/stacks); an 'inbox' drop folder vs direct paths; conflicts with sync and the collab editor; atomic writes (see #305: temp files must stay invisible); quotas; deletes going to Trash; versions. 5. **Security**: app passwords scoped per protocol and per folder (e.g. a PhotoSync password that can only write to `Photos/`, no read, no delete), rate limits, brute-force lockout, TLS, audit events, and the adversarial surface (path tricks, huge uploads, slowloris, zip/rename races). 6. **Ops**: rootless ports, the k3s Traefik edge (HTTP for WebDAV/S3/tus is easy; SFTP/FTPS need TCP exposure: describe the options and the firewall changes on the VM, /etc/calternal-firewall.nft), resource cost on the 4-vCPU VM. Output: docs/research/file-protocols.md (ASD-STE100) with a comparison table, a recommendation (the expected shape is 'WebDAV (Nextcloud-compatible where cheap) first, then one or two more'), a PhotoSync setup walkthrough for the recommended protocol, and numbered grill questions (grilling format). Do NOT implement. Commit on the branch, push, and post a summary on this issue.

kayg commented

2026-09-28 07:15:44 +00:00

Author

Owner

Starting research on branch job/file-protocols. Base/dev SHA: c9ec6aff84.

Starting research on branch job/file-protocols. Base/dev SHA: c9ec6aff84a9ec810124a2cfbdd07c189eea1b9c.

kayg commented

2026-09-28 07:21:39 +00:00

Author

Owner

Owner decision (2026-09-28, Q-N): the credential for every external integration is called App passwords (one concept; the existing app_passwords table). They cover WebDAV/files, CalDAV, the CLI/API, automations and any other external integration, with a name, a scope (protocol + read/write/full), optional folder limits, an optional expiry, 'last used' and one-click revoke, under Settings → Security. No separate 'API keys' concept. Use this in the research doc's recommendations.

Owner decision (2026-09-28, Q-N): the credential for every external integration is called **App passwords** (one concept; the existing `app_passwords` table). They cover WebDAV/files, CalDAV, the CLI/API, automations and any other external integration, with a name, a scope (protocol + read/write/full), optional folder limits, an optional expiry, 'last used' and one-click revoke, under Settings → Security. No separate 'API keys' concept. Use this in the research doc's recommendations.

kayg commented

2026-09-28 07:41:47 +00:00

Author

Owner

Finding: the existing DAV service is CalDAV-only, while ordinary uploads flow through Files and calternal-fs. Nextcloud chunking v2 uses its own DAV upload path and headers, so it should be treated as an optional compatibility layer rather than assumed WebDAV behavior.

kayg commented

2026-09-28 08:56:13 +00:00

Author

Owner

Completed research in docs/research/file-protocols.md. Recommendation: WebDAV over HTTPS, class 1 with strong ETags and preconditions, as an adapter to the Files service and calternal-fs. Keep tus for Calternal clients and add Nextcloud chunking only after a named client trace. Add SFTP on a high TCP port as the second protocol if public port 2222 is acceptable. Defer S3 until there is a concrete client need; do not expose SMB or plain FTP publicly.

The document covers the client matrix and PhotoSync setup, Home/Photos and Version/Trash semantics, app-password scopes, streaming, security, deployment, and adversarial cases. It includes twelve numbered owner questions. Decisions not in DESIGN: proposed /dav/files/{user-id}/ route, direct Home paths without a default Inbox, upload-only Photos/ app password, class 1 first, and tus remains the only resumable upload protocol.

Commit and push: 0f0c4219f... on job/file-protocols. git diff --cached --check exited 0 with no output. Cargo and Bun build gates were not run because this issue is research-only and explicitly prohibits builds. No endpoint was available for an adversarial test. The open choices are in the grill questions.

Completed research in docs/research/file-protocols.md. Recommendation: WebDAV over HTTPS, class 1 with strong ETags and preconditions, as an adapter to the Files service and calternal-fs. Keep tus for Calternal clients and add Nextcloud chunking only after a named client trace. Add SFTP on a high TCP port as the second protocol if public port 2222 is acceptable. Defer S3 until there is a concrete client need; do not expose SMB or plain FTP publicly. The document covers the client matrix and PhotoSync setup, Home/Photos and Version/Trash semantics, app-password scopes, streaming, security, deployment, and adversarial cases. It includes twelve numbered owner questions. Decisions not in DESIGN: proposed /dav/files/{user-id}/ route, direct Home paths without a default Inbox, upload-only Photos/ app password, class 1 first, and tus remains the only resumable upload protocol. Commit and push: 0f0c4219f... on job/file-protocols. git diff --cached --check exited 0 with no output. Cargo and Bun build gates were not run because this issue is research-only and explicitly prohibits builds. No endpoint was available for an adversarial test. The open choices are in the grill questions.

kayg commented

2026-09-28 09:02:11 +00:00

Author

Owner

Research is complete. The recommendation is WebDAV over HTTPS first, with class 1 behavior, strong ETags, and writes through the Files service and calternal-fs. Keep tus for Calternal clients. Add SFTP second only if public TCP port 2222 is acceptable. Defer S3 until a client need is named. Do not expose SMB or plain FTP publicly.

The document covers the client matrix, PhotoSync setup and AutoTransfer behavior, Home/Photos semantics, app-password scopes, streaming, security, deployment, and adversarial cases. It lists twelve owner questions. Decisions not covered by DESIGN: propose /dav/files/{user-id}/, use direct Home paths without a default Inbox, start with an upload-only Photos/ app password, start with WebDAV class 1, and keep tus as the only resumable upload protocol.

PhotoSync documents Nextcloud and ownCloud targets as WebDAV setups (Nextcloud, ownCloud).

Commit: 0f0c42199f on job/file-protocols. The branch is pushed. Cargo and Bun build gates were not run because issue #306 is research-only and prohibits builds. git diff --cached --check exited 0 with no output. cargo clean output: Removed 0 files. No protocol endpoint exists to attack. The remaining choices are listed in the grill questions.

Research is complete. The recommendation is WebDAV over HTTPS first, with class 1 behavior, strong ETags, and writes through the Files service and calternal-fs. Keep tus for Calternal clients. Add SFTP second only if public TCP port 2222 is acceptable. Defer S3 until a client need is named. Do not expose SMB or plain FTP publicly. The document covers the client matrix, PhotoSync setup and AutoTransfer behavior, Home/Photos semantics, app-password scopes, streaming, security, deployment, and adversarial cases. It lists twelve owner questions. Decisions not covered by DESIGN: propose /dav/files/{user-id}/, use direct Home paths without a default Inbox, start with an upload-only Photos/ app password, start with WebDAV class 1, and keep tus as the only resumable upload protocol. PhotoSync documents Nextcloud and ownCloud targets as WebDAV setups ([Nextcloud](https://www.photosync-app.com/support/cloud/answers/how-to-use-photosync-with-nextcloud), [ownCloud](https://www.photosync-app.com/support/cloud/answers/how-to-transfer-photos-to-owncloud)). Commit: 0f0c42199f6dedca4c7aa96ee3bafc2c4fc3ee14 on job/file-protocols. The branch is pushed. Cargo and Bun build gates were not run because issue #306 is research-only and prohibits builds. git diff --cached --check exited 0 with no output. cargo clean output: Removed 0 files. No protocol endpoint exists to attack. The remaining choices are listed in the grill questions.

kayg referenced this issue from a commit

2026-09-28 09:23:26 +00:00

Merge job/file-protocols (#306 research: WebDAV/SFTP/S3 for Home)

kayg referenced this issue

2026-09-28 10:06:16 +00:00

APP PASSWORDS: scopes (protocol, read/write/upload-only, folder prefix, expiry, last used) — the foundation for WebDAV, CalDAV, the HTTP API, MCP and Money automation #328

kayg commented

2026-09-28 13:13:42 +00:00

Author

Owner

Owner decisions (2026-09-28): P1 WebDAV only (no SFTP port for now). P2 yes, scoped app passwords (a separate feature, #328; PhotoSync gets an upload-only password under Photos/). P3 WebDAV deletes always go to Trash; upload-only passwords cannot delete. P4 standard RFC 4918 WebDAV is enough if it performs well. The only Nextcloud-specific extra that matters for speed is chunked or resumable upload of large files (Nextcloud chunking v2); calternal already has tus for its own clients. Add Nextcloud chunking only if a real client (PhotoSync, rclone) is measurably slower or less reliable on large videos without it. Measure that in the WebDAV build.

Owner decisions (2026-09-28): P1 **WebDAV only** (no SFTP port for now). P2 **yes, scoped app passwords** (a separate feature, #328; PhotoSync gets an upload-only password under Photos/). P3 **WebDAV deletes always go to Trash**; upload-only passwords cannot delete. P4 **standard RFC 4918 WebDAV** is enough if it performs well. The only Nextcloud-specific extra that matters for speed is chunked or resumable upload of large files (Nextcloud chunking v2); calternal already has tus for its own clients. Add Nextcloud chunking only if a real client (PhotoSync, rclone) is measurably slower or less reliable on large videos without it. Measure that in the WebDAV build.

kayg referenced this issue

2026-09-28 13:13:43 +00:00

WEBDAV: file WebDAV for Home (RFC 4918 class 1) — Finder, iOS Files, PhotoSync, rclone #339

kayg referenced this issue from a commit

2026-09-28 15:32:34 +00:00

copy(settings): 'App Passwords' instead of 'App passwords for calendar apps'

kayg commented

2026-10-03 11:55:39 +00:00

Author

Owner

Research complete in b74db998 (origin/dev); the protocol comparison and recommendation are recorded in the merged report.

Research complete in `b74db998` (origin/dev); the protocol comparison and recommendation are recorded in the merged report.

kayg closed this issue

2026-10-03 11:55:39 +00:00

kayg referenced this issue

2026-10-03 12:04:41 +00:00

Issue hygiene sweep 2026-10-03 #1008

Rows
Columns

RESEARCH: standard file-transfer protocols for Home (WebDAV first; SFTP/FTPS/S3/others) — rootless, PhotoSync-ready (NO implementation) #306