Photos: upload destination and ingest (web/PWA upload + calternald)

kayg commented

2026-09-24 14:45:05 +00:00

Owner

Owner decisions (round 13, P2 + P3):

Uploads made from the Photos view land in Photos/YYYY/YYYY-MM-DD/ by capture date (EXIF DateTimeOriginal, else file mtime, in the user's time zone). Name conflicts: keep both (name 2.ext). Nothing already on disk is ever moved by the photos plugin.
Ingest for the first photos release: web/PWA upload (drag-drop, multi-select, folder upload, resumable via the existing tus pipeline) and calternald syncing a Mac folder into a library root. No importers yet.
An Immich importer comes later (separate issue).

Acceptance: 300-photo drag-drop lands in the right date folders with correct capture dates; a HEIC+MOV Live Photo lands as a pair in the same folder; progress uses the shared Sonner-based upload toasts; data-loss guarantees of the files plugin apply unchanged.

Design: DESIGN §12, §24, §28.

Context for the owning job

Repo: kayg/calternal (~/Developer/calternal). Read CLAUDE.md, CONTEXT.md and docs/DESIGN.md first; this issue's section is cited below.
Owner rules that always apply: file over app (plain files are the truth, the DB is an index); the server is the single writer; data loss is unacceptable; performance first, never at the cost of finesse; UI is the calternal.js design system (copy components verbatim, compare side by side with calternal.js reference screenshots; Claude does visual review); never ship sample/mock data; atomic commits; adversarial testing after API work; good enough, not perfect (merge blockers: crash/DoS, data loss, security, sync collisions).
Comment on this issue when you start (branch, base SHA), on each finding, when blocked, and when finished (head SHA + gate output). Never close it.

Owner decisions (round 13, P2 + P3): - Uploads made **from the Photos view** land in `Photos/YYYY/YYYY-MM-DD/` by capture date (EXIF DateTimeOriginal, else file mtime, in the user's time zone). Name conflicts: keep both (`name 2.ext`). **Nothing already on disk is ever moved** by the photos plugin. - Ingest for the first photos release: web/PWA upload (drag-drop, multi-select, folder upload, resumable via the existing tus pipeline) and `calternald` syncing a Mac folder into a library root. No importers yet. - An Immich importer comes later (separate issue). Acceptance: 300-photo drag-drop lands in the right date folders with correct capture dates; a HEIC+MOV Live Photo lands as a pair in the same folder; progress uses the shared Sonner-based upload toasts; data-loss guarantees of the files plugin apply unchanged. Design: DESIGN §12, §24, §28. ## Context for the owning job - Repo: kayg/calternal (~/Developer/calternal). Read CLAUDE.md, CONTEXT.md and docs/DESIGN.md first; this issue's section is cited below. - Owner rules that always apply: file over app (plain files are the truth, the DB is an index); the server is the single writer; data loss is unacceptable; performance first, never at the cost of finesse; UI is the calternal.js design system (copy components verbatim, compare side by side with calternal.js reference screenshots; Claude does visual review); never ship sample/mock data; atomic commits; adversarial testing after API work; good enough, not perfect (merge blockers: crash/DoS, data loss, security, sync collisions). - Comment on this issue when you start (branch, base SHA), on each finding, when blocked, and when finished (head SHA + gate output). Never close it.

kayg referenced this issue

2026-09-24 17:57:38 +00:00

Photos: library scope — Photos/ plus opted-in folders #27

kayg referenced this issue

2026-09-24 17:57:38 +00:00

Photos: pairs and stacks (RAW+JPEG, Live Photos, bursts) #29

kayg referenced this issue

2026-09-24 17:57:38 +00:00

Photos: timeline, scrubber, zoom levels and fullscreen viewer — very fast at 100k+ items #33

kayg commented

2026-09-24 17:57:38 +00:00

Author

Owner

Starting server-side implementation for #27, #28, #29, and #33.

Branch: job/photos-core
Base SHA: 57118d9648582e682f0a0e1997fc8ad9f84bab35

I have read the issue bodies and repository design/context. I am mapping the existing Files upload, thumbnail, identity, and plugin registration APIs before defining the photos plugin boundary.

Starting server-side implementation for #27, #28, #29, and #33. Branch: `job/photos-core` Base SHA: `57118d9648582e682f0a0e1997fc8ad9f84bab35` I have read the issue bodies and repository design/context. I am mapping the existing Files upload, thumbnail, identity, and plugin registration APIs before defining the photos plugin boundary.

kayg commented

2026-09-24 18:04:29 +00:00

Author

Owner

Finding: Files tus accepts a destination at create time (crates/plugins/files/src/uploads.rs:343) and finish installs against that persisted destination (:645+). EXIF capture date is only available after all bytes arrive, so Photos needs a narrow destination resolver at tus finalization. The Files staging and atomic install path can remain shared; ordinary uploads keep their current destination behavior.

Finding: Files tus accepts a destination at create time (`crates/plugins/files/src/uploads.rs:343`) and `finish` installs against that persisted destination (`:645+`). EXIF capture date is only available after all bytes arrive, so Photos needs a narrow destination resolver at tus finalization. The Files staging and atomic install path can remain shared; ordinary uploads keep their current destination behavior.

kayg commented

2026-09-24 18:08:06 +00:00

Author

Owner

Second finding: calternald upload creation sends filename/path metadata but no source mtime (crates/calternal-sync/src/remote.rs:342-365), while Files installs a new inode through its normal writer. A synced photo without DateTimeOriginal therefore cannot use the Mac file's mtime on the server. Supporting that fallback end to end needs a sync-client mtime field and server-side mtime preservation, which is outside the listed photos/server files. A Photos-view web upload can carry File.lastModified; that does not fix Mac-folder sync.

Second finding: `calternald` upload creation sends filename/path metadata but no source mtime (`crates/calternal-sync/src/remote.rs:342-365`), while Files installs a new inode through its normal writer. A synced photo without `DateTimeOriginal` therefore cannot use the Mac file's mtime on the server. Supporting that fallback end to end needs a sync-client mtime field and server-side mtime preservation, which is outside the listed photos/server files. A Photos-view web upload can carry `File.lastModified`; that does not fix Mac-folder sync.

kayg referenced this issue

2026-09-24 18:20:21 +00:00

Photos: library scope — Photos/ plus opted-in folders #27

kayg commented

2026-09-24 18:20:22 +00:00

Author

Owner

Starting server-side implementation for #27, #28, #29, and #33.

Branch: job/photos-core
Base SHA: 41aa77499e0a090371ff252458722fd80b4e6a44

I read each issue body and comment thread, plus CLAUDE.md, docs/DESIGN.md, and CONTEXT.md. The approved scope includes a rebuildable Photos index keyed by Files item IDs, opt-in library roots in .calternal/settings.json, date-routed Photos uploads using the existing tus staging/install pipeline, pairing and compact paginated timeline APIs with ThumbHash placeholders, and source-mtime metadata from calternald through server installation.

Starting server-side implementation for #27, #28, #29, and #33. Branch: `job/photos-core` Base SHA: `41aa77499e0a090371ff252458722fd80b4e6a44` I read each issue body and comment thread, plus CLAUDE.md, docs/DESIGN.md, and CONTEXT.md. The approved scope includes a rebuildable Photos index keyed by Files item IDs, opt-in library roots in `.calternal/settings.json`, date-routed Photos uploads using the existing tus staging/install pipeline, pairing and compact paginated timeline APIs with ThumbHash placeholders, and source-mtime metadata from `calternald` through server installation.

kayg referenced this issue

2026-09-24 18:20:23 +00:00

Photos: pairs and stacks (RAW+JPEG, Live Photos, bursts) #29

kayg referenced this issue

2026-09-24 18:20:24 +00:00

Photos: timeline, scrubber, zoom levels and fullscreen viewer — very fast at 100k+ items #33

kayg commented

2026-09-24 18:41:11 +00:00

Author

Owner

Finding: Files deduplicates equal bytes through shared hardlinked Index blobs (calternal-fs/src/write.rs). Changing a synced file's mtime after installation would change the mtime of other files that share that inode. I added a mtime-aware atomic Root writer that sets mtime on a private staging inode before publication and skips shared-blob dedup for that write. Its regression test confirms the source mtime is preserved and an equal-content sibling keeps its own mtime and inode. This trades disk deduplication for correct per-file sync metadata only when a source mtime is supplied.

Finding: Files deduplicates equal bytes through shared hardlinked Index blobs (`calternal-fs/src/write.rs`). Changing a synced file's mtime after installation would change the mtime of other files that share that inode. I added a mtime-aware atomic Root writer that sets mtime on a private staging inode before publication and skips shared-blob dedup for that write. Its regression test confirms the source mtime is preserved and an equal-content sibling keeps its own mtime and inode. This trades disk deduplication for correct per-file sync metadata only when a source mtime is supplied.

kayg commented

2026-09-24 19:33:27 +00:00

Author

Owner

Capture date behavior is now explicit in the server slice: Photos uploads parse EXIF DateTimeOriginal/CreateDate from staged Tus chunks without joining a full upload into memory. Naive camera times use the request time-zone when valid, then the user's .calternal/settings.json timezone, then UTC. If EXIF has no valid capture time, the server uses the sync-provided source mtime; Files preserves that time on the installed inode. Shared Photos roots remain read-only.

The 3.8.0 nom-exif source exposes generic MakerNote values but skips proprietary Canon CR3 CMT3 MakerNotes, so Apple BurstUUID extraction can only work when a MakerNote payload is exposed. Fallback burst grouping remains same camera and within one second from the first frame.

Capture date behavior is now explicit in the server slice: Photos uploads parse EXIF `DateTimeOriginal`/`CreateDate` from staged Tus chunks without joining a full upload into memory. Naive camera times use the request time-zone when valid, then the user's `.calternal/settings.json` `timezone`, then UTC. If EXIF has no valid capture time, the server uses the sync-provided source mtime; Files preserves that time on the installed inode. Shared Photos roots remain read-only. The 3.8.0 `nom-exif` source exposes generic MakerNote values but skips proprietary Canon CR3 CMT3 MakerNotes, so Apple BurstUUID extraction can only work when a MakerNote payload is exposed. Fallback burst grouping remains same camera and within one second from the first frame.

kayg commented

2026-09-24 20:53:29 +00:00

Author

Owner

Finding in the first adversarial round: a photo upload with x-calternal-timezone: Europe/Berlin and source mtime 2024-06-01T22:30:00Z landed under Photos/2024/2024-06-02/, but the timeline indexed it under 2024-06-01 because the user's default Index time zone was UTC. The upload path and timeline therefore disagreed at the day boundary. I am making the Photos upload's validated Photos/YYYY/YYYY-MM-DD/ segment the timeline bucket for those routed uploads, while keeping capture seconds for ordering.

Finding in the first adversarial round: a photo upload with `x-calternal-timezone: Europe/Berlin` and source mtime `2024-06-01T22:30:00Z` landed under `Photos/2024/2024-06-02/`, but the timeline indexed it under `2024-06-01` because the user's default Index time zone was UTC. The upload path and timeline therefore disagreed at the day boundary. I am making the Photos upload's validated `Photos/YYYY/YYYY-MM-DD/` segment the timeline bucket for those routed uploads, while keeping capture seconds for ordering.

kayg commented

2026-09-24 21:37:24 +00:00

Author

Owner

Completed the Photos upload destination through Files Tus staging, with source mtime/timezone metadata carried from sync and preserved on install. Photos uploads use the validated Photos date folder as the timeline day; other paths use the viewer zone. Decision: timezone precedence is request header, then .calternal/settings.json, then UTC. Name conflicts use the Files rename policy.

The first workspace test attempt returned 413 because tempfile used /tmp with 916,078,592 bytes free, below the existing 1 GiB upload safety reserve. The focused Files test and full workspace suite passed with TMPDIR on the worktree filesystem; no product change was needed.

Head SHA: f77c8d135e.

Gate output (commands use CARGO_PROFILE_DEV_DEBUG=line-tables-only and CARGO_INCREMENTAL=0):

cargo fmt --all --check: exit 0, no output.
cargo clippy --all-targets -- -D warnings (exit 0):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 14.64s

cargo test (workspace, exit 0):

    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 25s
test result: ok. 13 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.04s

bash packages/api-client/check-generated.sh (exit 0, no generated diff):

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 40s
     Running `target/debug/calternal-server openapi`
$ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts
Resolving dependencies
Resolved, downloaded and extracted [24]
Saved lockfile
✨ openapi-typescript 7.13.0
🚀 ../../contracts/openapi.json → src/generated.ts [1.4s]

bash tests/adversarial/run.sh (exit 1 because Round 1 reported Calendar task latency; Round 2 reported zero findings):

==== FINDINGS 21
 - Task storm 3 :: SLOW 7.4s status 201
 - Task storm 4 :: SLOW 5.1s status 201
 - Task storm 5 :: SLOW 6.9s status 201
 - Task storm 6 :: SLOW 9.6s status 201
 - Task storm 7 :: SLOW 8.3s status 201
 - Task storm 8 :: SLOW 7.9s status 201
 - Task storm 9 :: SLOW 12.9s status 201
 - Task storm 10 :: SLOW 12.1s status 201
 - Task storm 11 :: SLOW 13.4s status 201
 - Task storm 12 :: SLOW 14.3s status 201
 - Task storm 13 :: SLOW 14.3s status 201
 - Task storm 14 :: SLOW 12.9s status 201
 - Task storm 15 :: SLOW 12.4s status 201
 - Task storm 16 :: SLOW 11.9s status 201
 - Task storm 17 :: SLOW 13.7s status 201
 - Task storm 18 :: SLOW 14.3s status 201
 - Task storm 19 :: SLOW 17.4s status 201
 - Task storm 20 :: SLOW 16.7s status 201
 - Task storm 21 :: SLOW 14.8s status 201
 - Task storm 22 :: SLOW 17.2s status 201
 - Task storm 23 :: SLOW 17.3s status 201
==== ROUND 2 FINDINGS 0

Completed the Photos upload destination through Files Tus staging, with source mtime/timezone metadata carried from sync and preserved on install. Photos uploads use the validated Photos date folder as the timeline day; other paths use the viewer zone. Decision: timezone precedence is request header, then .calternal/settings.json, then UTC. Name conflicts use the Files rename policy. The first workspace test attempt returned 413 because tempfile used /tmp with 916,078,592 bytes free, below the existing 1 GiB upload safety reserve. The focused Files test and full workspace suite passed with TMPDIR on the worktree filesystem; no product change was needed. Head SHA: f77c8d135eeec4f1a30da6034e61217df5b9a04b. Gate output (commands use CARGO_PROFILE_DEV_DEBUG=line-tables-only and CARGO_INCREMENTAL=0): - cargo fmt --all --check: exit 0, no output. - cargo clippy --all-targets -- -D warnings (exit 0): ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 14.64s ``` - cargo test (workspace, exit 0): ``` Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 25s test result: ok. 13 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.04s ``` - bash packages/api-client/check-generated.sh (exit 0, no generated diff): ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 40s Running `target/debug/calternal-server openapi` $ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts Resolving dependencies Resolved, downloaded and extracted [24] Saved lockfile ✨ openapi-typescript 7.13.0 🚀 ../../contracts/openapi.json → src/generated.ts [1.4s] ``` - bash tests/adversarial/run.sh (exit 1 because Round 1 reported Calendar task latency; Round 2 reported zero findings): ``` ==== FINDINGS 21 - Task storm 3 :: SLOW 7.4s status 201 - Task storm 4 :: SLOW 5.1s status 201 - Task storm 5 :: SLOW 6.9s status 201 - Task storm 6 :: SLOW 9.6s status 201 - Task storm 7 :: SLOW 8.3s status 201 - Task storm 8 :: SLOW 7.9s status 201 - Task storm 9 :: SLOW 12.9s status 201 - Task storm 10 :: SLOW 12.1s status 201 - Task storm 11 :: SLOW 13.4s status 201 - Task storm 12 :: SLOW 14.3s status 201 - Task storm 13 :: SLOW 14.3s status 201 - Task storm 14 :: SLOW 12.9s status 201 - Task storm 15 :: SLOW 12.4s status 201 - Task storm 16 :: SLOW 11.9s status 201 - Task storm 17 :: SLOW 13.7s status 201 - Task storm 18 :: SLOW 14.3s status 201 - Task storm 19 :: SLOW 17.4s status 201 - Task storm 20 :: SLOW 16.7s status 201 - Task storm 21 :: SLOW 14.8s status 201 - Task storm 22 :: SLOW 17.2s status 201 - Task storm 23 :: SLOW 17.3s status 201 ==== ROUND 2 FINDINGS 0 ```

kayg commented

2026-09-24 21:52:52 +00:00

Author

Owner

Starting the search window UI job for #57, #61, #63 and #64 (one job, one worktree). Branch: job/search-ui; base SHA: 8c216759f01d704e69ecf37ef8293f422abf5552.

Plan: extend the existing ⌘K dialog and registry in apps/web (no second palette) into one floating search window with pills, previews and an in-place expanded view; add Files ⌘F and Calendar contextual search; add a saved-search CRUD API (JSON files through calternal-fs) and smart folders. I will post findings and a finish comment here.

Starting the search window UI job for #57, #61, #63 and #64 (one job, one worktree). Branch: `job/search-ui`; base SHA: `8c216759f01d704e69ecf37ef8293f422abf5552`. Plan: extend the existing ⌘K dialog and registry in apps/web (no second palette) into one floating search window with pills, previews and an in-place expanded view; add Files ⌘F and Calendar contextual search; add a saved-search CRUD API (JSON files through `calternal-fs`) and smart folders. I will post findings and a finish comment here.

kayg referenced this issue

2026-09-24 21:53:08 +00:00

Photos: pairs and stacks (RAW+JPEG, Live Photos, bursts) #29

kayg commented

2026-09-24 21:53:10 +00:00

Author

Owner

Correction: the previous comment on this issue was posted by mistake (a stale file from the search-ui job). Please ignore it.

Starting the Photos mode UI (web) for #33, the UI half of #29 and the web upload of #28.

Branch: job/photos-ui
Base SHA: 4ba968912ea40c5f5ee90e169276fbb6b9e24420

Scope: virtualized justified timeline with day headers, scrubber and zoom levels; fullscreen viewer on the shared QuickLook (info panel, Live Photos, video, RAW toggle, stacks and key-photo choice); selection with tags, trash with Undo, download, share and Copy link; upload into Photos through the Photos destination; deep links /photos, /photos/<year>/<month> and /p/<item-id>. Playwright e2e and a 20k+ item scroll measurement follow.

Correction: the previous comment on this issue was posted by mistake (a stale file from the search-ui job). Please ignore it. Starting the Photos mode UI (web) for #33, the UI half of #29 and the web upload of #28. Branch: `job/photos-ui` Base SHA: `4ba968912ea40c5f5ee90e169276fbb6b9e24420` Scope: virtualized justified timeline with day headers, scrubber and zoom levels; fullscreen viewer on the shared QuickLook (info panel, Live Photos, video, RAW toggle, stacks and key-photo choice); selection with tags, trash with Undo, download, share and Copy link; upload into Photos through the Photos destination; deep links `/photos`, `/photos/<year>/<month>` and `/p/<item-id>`. Playwright e2e and a 20k+ item scroll measurement follow.

kayg referenced this issue

2026-09-24 21:53:15 +00:00

Photos: timeline, scrubber, zoom levels and fullscreen viewer — very fast at 100k+ items #33

kayg referenced this issue from a commit

2026-09-24 22:00:50 +00:00

Merge job/photos-core: Photos backend: library, pairing, stacks, timeline, upload ingest (#27, #28, #29, #33)

kayg commented

2026-09-25 04:40:33 +00:00

Author

Owner

Finished on job/photos-ui (head 9b0891e, rebased onto main f8e93b7; not merged, not pushed). Details are in the finish comment on #33.

Photos has an Upload button, and files can also be dropped on the window. Uploads go through the Photos destination into the capture-date folder. Progress shows in the shared Files upload toast.
Non-media files and XMP sidecars are skipped.
The upload sends the file's modified time as its source time.
The timeline refreshes from the files event stream after an upload.
The e2e covers the upload and the toast (19/19 PASS).

Finished on `job/photos-ui` (head `9b0891e`, rebased onto main `f8e93b7`; not merged, not pushed). Details are in the finish comment on #33. - Photos has an Upload button, and files can also be dropped on the window. Uploads go through the Photos destination into the capture-date folder. Progress shows in the shared Files upload toast. - Non-media files and XMP sidecars are skipped. - The upload sends the file's modified time as its source time. - The timeline refreshes from the files event stream after an upload. - The e2e covers the upload and the toast (19/19 PASS).

kayg referenced this issue from a commit

2026-09-25 05:01:28 +00:00

Merge job/photos-ui: Photos timeline, scrubber, zoom, viewer, stacks, upload (#33, #29, #28)

kayg referenced this issue

2026-10-01 05:08:38 +00:00

Photos: library scope — Photos/ plus opted-in folders #27

kayg commented

2026-10-01 05:08:40 +00:00

Author

Owner

Completed on dev in 9e3c59976f (Merge job/photos-ui: Photos timeline, scrubber, zoom, viewer, stacks, upload (#33, #29, #28)).

Completed on dev in 9e3c59976f163c640790d1d627b13a2bd925ffa9 (Merge job/photos-ui: Photos timeline, scrubber, zoom, viewer, stacks, upload (#33, #29, #28)).

kayg closed this issue

2026-10-01 05:08:40 +00:00

kayg referenced this issue

2026-10-01 05:08:41 +00:00

Photos: pairs and stacks (RAW+JPEG, Live Photos, bursts) #29

kayg referenced this issue

2026-10-01 05:08:43 +00:00

Photos: timeline, scrubber, zoom levels and fullscreen viewer — very fast at 100k+ items #33

Rows
Columns

Photos: upload destination and ingest (web/PWA upload + calternald) #28

Context for the owning job