Photos: incremental index refresh (a change rebuilds the whole library today)

kayg commented

2026-09-25 01:02:32 +00:00

Owner

Found by the Photos UI performance run (job/photos-ui, apps/web/e2e/photos-perf.mjs, 25,000 photos, release build).

What happens

crates/plugins/photos/src/index.rs refresh_user rebuilds the viewer's whole Photos index for any change: it reads every files_index row of every library root, deletes all of the viewer's photos_media, photos_groups, photos_group_members and photos_days rows, and inserts them again in one transaction. The background loop runs it after every coalesced burst of fs events (120 ms settle), and after any event-bus lag it runs refresh_all_users.

At 25k items one rebuild costs seconds of CPU and a long writer transaction. While it runs, timeline reads that normally take 80–180 ms took 5–7 s (p95 in the run), and ThumbHash values written by timeline reads meanwhile are lost when the rebuild writes its older copy back.

Two amplifiers are fixed on job/photos-ui:

The Home watcher treated inotify opens and read-only closes as changes, so every read of a photo (thumbnail job, download, EXIF read, video) triggered a full rebuild (Server: a read is not a Home change).
An XMP sidecar change was not detected; it is now keyed by the sidecar's Files Index hash (Photos: re-parse an item when its XMP sidecar changes).

Real changes still rebuild everything: a 300-photo upload rebuilds the whole library many times while it runs, and a 100k library makes each rebuild proportionally longer.

Proposal

Make the refresh incremental: re-parse and re-group only the items under the changed paths (pairing is local to one folder, and bursts to one camera and second, so the affected groups are bounded), update photos_days counts by delta, and keep the full rebuild for startup, root changes and lag. Measure with bun run perf:photos (it reports API p50/p95 and the load average).

Context for the owning job

Repo: kayg/calternal (~/Developer/calternal). Read CLAUDE.md, CONTEXT.md and docs/DESIGN.md (§12, §18 budgets, §28) first.
Owner rules: file over app (the index is derived and rebuildable); the server is the single writer; data loss is unacceptable; performance first, never at the cost of finesse; atomic commits; adversarial testing after API work; good enough, not perfect.
Comment on this issue when you start (branch, base SHA), on each finding, when blocked, and when finished (head SHA + gate output). Never close it.

Found by the Photos UI performance run (job/photos-ui, `apps/web/e2e/photos-perf.mjs`, 25,000 photos, release build). ## What happens `crates/plugins/photos/src/index.rs` `refresh_user` rebuilds the viewer's whole Photos index for any change: it reads every `files_index` row of every library root, deletes all of the viewer's `photos_media`, `photos_groups`, `photos_group_members` and `photos_days` rows, and inserts them again in one transaction. The background loop runs it after every coalesced burst of `fs` events (120 ms settle), and after any event-bus lag it runs `refresh_all_users`. At 25k items one rebuild costs seconds of CPU and a long writer transaction. While it runs, timeline reads that normally take 80–180 ms took 5–7 s (p95 in the run), and ThumbHash values written by timeline reads meanwhile are lost when the rebuild writes its older copy back. Two amplifiers are fixed on job/photos-ui: - The Home watcher treated inotify opens and read-only closes as changes, so every read of a photo (thumbnail job, download, EXIF read, video) triggered a full rebuild (`Server: a read is not a Home change`). - An XMP sidecar change was not detected; it is now keyed by the sidecar's Files Index hash (`Photos: re-parse an item when its XMP sidecar changes`). Real changes still rebuild everything: a 300-photo upload rebuilds the whole library many times while it runs, and a 100k library makes each rebuild proportionally longer. ## Proposal Make the refresh incremental: re-parse and re-group only the items under the changed paths (pairing is local to one folder, and bursts to one camera and second, so the affected groups are bounded), update `photos_days` counts by delta, and keep the full rebuild for startup, root changes and lag. Measure with `bun run perf:photos` (it reports API p50/p95 and the load average). ## Context for the owning job - Repo: kayg/calternal (~/Developer/calternal). Read CLAUDE.md, CONTEXT.md and docs/DESIGN.md (§12, §18 budgets, §28) first. - Owner rules: file over app (the index is derived and rebuildable); the server is the single writer; data loss is unacceptable; performance first, never at the cost of finesse; atomic commits; adversarial testing after API work; good enough, not perfect. - Comment on this issue when you start (branch, base SHA), on each finding, when blocked, and when finished (head SHA + gate output). Never close it.

kayg referenced this issue

2026-09-25 04:40:30 +00:00

Photos: timeline, scrubber, zoom levels and fullscreen viewer — very fast at 100k+ items #33

kayg commented

2026-09-25 05:11:07 +00:00

Author

Owner

Starting kayg/calternal#79 on branch job/photos-incremental, based at 4bbedae355f9ad0af625454b367e55b7a2306865.

Starting kayg/calternal#79 on branch `job/photos-incremental`, based at `4bbedae355f9ad0af625454b367e55b7a2306865`.

kayg commented

2026-09-25 05:17:10 +00:00

Author

Owner

Finding from index.rs and pairing_impl.rs: the filesystem event path currently enters refresh_user_if_changed(false), which still scans every configured root and clears/reinserts all four viewer tables if one version differs. group_media pairs by ContentIdentifier, same-folder basename, explicit burst ID, and same-camera capture times within one second; incremental updates must include old and new peers across those keys.

Finding from `index.rs` and `pairing_impl.rs`: the filesystem event path currently enters `refresh_user_if_changed(false)`, which still scans every configured root and clears/reinserts all four viewer tables if one version differs. `group_media` pairs by ContentIdentifier, same-folder basename, explicit burst ID, and same-camera capture times within one second; incremental updates must include old and new peers across those keys.

kayg commented

2026-09-25 05:54:56 +00:00

Author

Owner

A storage regression test confirmed another ThumbHash loss path: explicit reconciliation deleted photos_media and photos_groups before reinserting them. Full reconciliation now upserts current media and groups, preserving a read-generated ThumbHash when the content hash and display item are unchanged, and removes stale rows. The randomized DB consistency test compares media, groups, memberships and day buckets after each incremental change with an explicit full rebuild.

A storage regression test confirmed another ThumbHash loss path: explicit reconciliation deleted `photos_media` and `photos_groups` before reinserting them. Full reconciliation now upserts current media and groups, preserving a read-generated ThumbHash when the content hash and display item are unchanged, and removes stale rows. The randomized DB consistency test compares media, groups, memberships and day buckets after each incremental change with an explicit full rebuild.

kayg commented

2026-09-25 06:00:04 +00:00

Author

Owner

Workspace clippy found four production lints and six test-only modulo lints in the Photos incremental refresh change. These are mechanical style issues (collapsible_if, useless_format, iter_cloned_collect, filter_map_bool_then, and manual_is_multiple_of); I am fixing them before rerunning the gate.

Workspace clippy found four production lints and six test-only modulo lints in the Photos incremental refresh change. These are mechanical style issues (`collapsible_if`, `useless_format`, `iter_cloned_collect`, `filter_map_bool_then`, and `manual_is_multiple_of`); I am fixing them before rerunning the gate.

kayg commented

2026-09-25 06:05:19 +00:00

Author

Owner

The generated API gate could not start the server because calternal-server embeds apps/web/build, and that directory is absent in this worktree. The required web build also failed with /usr/bin/bash: line 1: vite: command not found. I am installing from the frozen Bun lockfile, building the web assets, and rerunning the gate.

The generated API gate could not start the server because `calternal-server` embeds `apps/web/build`, and that directory is absent in this worktree. The required web build also failed with `/usr/bin/bash: line 1: vite: command not found`. I am installing from the frozen Bun lockfile, building the web assets, and rerunning the gate.

kayg commented

2026-09-25 06:17:09 +00:00

Author

Owner

The adversarial run's second round reported: list shared folder: recipient listing wrote an unindexed child to the owner's Index synchronously. This is outside the owned Photos plugin and event-routing change. I am checking the probe's assertion and will report whether it reproduces as a Photos regression or a pre-existing Files/Share finding; the run is still active.

The adversarial run's second round reported: `list shared folder: recipient listing wrote an unindexed child to the owner's Index synchronously`. This is outside the owned Photos plugin and event-routing change. I am checking the probe's assertion and will report whether it reproduces as a Photos regression or a pre-existing Files/Share finding; the run is still active.

kayg commented

2026-09-25 06:19:22 +00:00

Author

Owner

The adversarial command exited 1. Round 1 reported FINDINGS 0. Round 2 reported exactly one finding: list shared folder :: recipient listing wrote an unindexed child to the owner's Index synchronously. The assertion deletes a Files Index row for AShared/unindexed.txt, calls the Files shared-folder listing endpoint, then checks whether that row was synchronously recreated. The Photos job changed no Files implementation or wire event routing. This is an out-of-scope Files/Share behavior gap; all other adversarial checks completed without findings, with the task storm reporting only its expected SLOW threshold.

The adversarial command exited 1. Round 1 reported `FINDINGS 0`. Round 2 reported exactly one finding: `list shared folder :: recipient listing wrote an unindexed child to the owner's Index synchronously`. The assertion deletes a Files Index row for `AShared/unindexed.txt`, calls the Files shared-folder listing endpoint, then checks whether that row was synchronously recreated. The Photos job changed no Files implementation or wire event routing. This is an out-of-scope Files/Share behavior gap; all other adversarial checks completed without findings, with the task storm reporting only its expected `SLOW` threshold.

kayg commented

2026-09-25 06:52:18 +00:00

Author

Owner

Release performance evidence (25,000 indexed Photos items): before uploads, GET /api/v1/photos/timeline?days=60&tiles_per_day=200 had p95 94.7 ms and buckets p95 33.6 ms. During 100 successful (201) Photos API uploads using 8 workers, timeline p95 rose to 2,380.7 ms and buckets p95 to 2,288.6 ms (29 read samples each). This exceeds the issue's 250 ms target. I am tracing the changed-path refresh path and will fix and rerun it.

Release performance evidence (25,000 indexed Photos items): before uploads, `GET /api/v1/photos/timeline?days=60&tiles_per_day=200` had p95 94.7 ms and buckets p95 33.6 ms. During 100 successful (`201`) Photos API uploads using 8 workers, timeline p95 rose to 2,380.7 ms and buckets p95 to 2,288.6 ms (29 read samples each). This exceeds the issue's 250 ms target. I am tracing the changed-path refresh path and will fix and rerun it.

kayg commented

2026-09-25 07:03:21 +00:00

Author

Owner

The release upload burst exposed a query-plan issue in incremental grouping: the folder-basename candidate lookup scanned all 25k owner Photos rows before checking each row against the Files parent index. EXPLAIN QUERY PLAN showed SEARCH p USING INDEX sqlite_autoindex_photos_media_1 (viewer_id=? AND owner_id=?) followed by a correlated Files lookup.

I changed the query to start from files_index parents and join Photos by stable item ID. The regression test checks this production query plan and passed; the Photos crate suite passed (31 passed, 2 ignored), and workspace clippy and formatting passed. I am rebuilding release now to repeat the 25k / 100-upload latency measurement.

The release upload burst exposed a query-plan issue in incremental grouping: the folder-basename candidate lookup scanned all 25k owner Photos rows before checking each row against the Files parent index. EXPLAIN QUERY PLAN showed `SEARCH p USING INDEX sqlite_autoindex_photos_media_1 (viewer_id=? AND owner_id=?)` followed by a correlated Files lookup. I changed the query to start from `files_index` parents and join Photos by stable item ID. The regression test checks this production query plan and passed; the Photos crate suite passed (31 passed, 2 ignored), and workspace clippy and formatting passed. I am rebuilding release now to repeat the 25k / 100-upload latency measurement.

kayg commented

2026-09-25 07:35:14 +00:00

Author

Owner

A 25k release burst found two remaining sources of work: 100 uploads advanced the CLIP reconcile generation 48 times, and the 120 ms event window split the burst into nearly one refresh per upload. Repeated full scans accumulated 13,946 CLIP jobs and timeline p95 reached 1,157.3 ms.

Incremental refreshes now queue bounded CLIP item jobs and do not change the full-scan generation. The regression test confirms one added photo keeps the generation stable and queues an item job. I also changed the event collector to a 250 ms quiet window, capped at 1 second, to combine active upload bursts. The Photos suite passed (31 passed, 2 ignored), workspace clippy and formatting passed. The next release burst measurement will verify this batching change.

A 25k release burst found two remaining sources of work: 100 uploads advanced the CLIP reconcile generation 48 times, and the 120 ms event window split the burst into nearly one refresh per upload. Repeated full scans accumulated 13,946 CLIP jobs and timeline p95 reached 1,157.3 ms. Incremental refreshes now queue bounded CLIP item jobs and do not change the full-scan generation. The regression test confirms one added photo keeps the generation stable and queues an item job. I also changed the event collector to a 250 ms quiet window, capped at 1 second, to combine active upload bursts. The Photos suite passed (31 passed, 2 ignored), workspace clippy and formatting passed. The next release burst measurement will verify this batching change.

kayg commented

2026-09-25 07:54:08 +00:00

Author

Owner

The new release run held the Photos full-scan generation at 1 during the upload burst, but timeline p95 was still 3,253.3 ms. The same database had 7,724 pending Files thumbnail jobs and 2,028 pending Photos CLIP jobs. Each wide timeline response could add up to 96 thumbnail jobs and 64 ThumbHash fills for offscreen tiles, so repeated benchmark reads added avoidable background work.

I capped each response at 16 thumbnail jobs and 16 ThumbHash fills, about one screenful. A regression test first failed at the old 96-job limit and passes at 16. Workspace clippy and formatting pass. I will rerun the burst on the final release build after this change.

The new release run held the Photos full-scan generation at 1 during the upload burst, but timeline p95 was still 3,253.3 ms. The same database had 7,724 pending Files thumbnail jobs and 2,028 pending Photos CLIP jobs. Each wide timeline response could add up to 96 thumbnail jobs and 64 ThumbHash fills for offscreen tiles, so repeated benchmark reads added avoidable background work. I capped each response at 16 thumbnail jobs and 16 ThumbHash fills, about one screenful. A regression test first failed at the old 96-job limit and passes at 16. Workspace clippy and formatting pass. I will rerun the burst on the final release build after this change.

kayg commented

2026-09-25 08:12:43 +00:00

Author

Owner

The capped-side-work release burst still measured timeline p95 at 707.0 ms. The timeline handler issued one day_tiles query per bucket, sequentially; its 60-day page can therefore make 60 database reads while uploads are writing.

I changed the timeline page and single-day route to share one per-day ranked query. The query returns the same newest-first limit for every selected day and retains the Files Index validity join. A regression test checks ordering and per-day limits. Photos tests pass (33 passed, 2 ignored), with workspace formatting and clippy passing. I will verify the release burst again.

The capped-side-work release burst still measured timeline p95 at 707.0 ms. The `timeline` handler issued one `day_tiles` query per bucket, sequentially; its 60-day page can therefore make 60 database reads while uploads are writing. I changed the timeline page and single-day route to share one per-day ranked query. The query returns the same newest-first limit for every selected day and retains the Files Index validity join. A regression test checks ordering and per-day limits. Photos tests pass (33 passed, 2 ignored), with workspace formatting and clippy passing. I will verify the release burst again.

kayg commented

2026-09-25 08:31:47 +00:00

Author

Owner

The latest release burst still misses the requested read target: the 100 upload run measured timeline p95 1,076 ms and buckets p95 1,248 ms, while the clean idle timeline p95 was 89.9 ms. During inspection the benchmark Photos server used about 125% CPU, with 5,509 files.thumbnail jobs and 2,082 photos.clip.index jobs pending. I am reducing local-root read work and pausing Photos-owned refresh work while the Photos upload route is active.

The latest release burst still misses the requested read target: the 100 upload run measured timeline p95 1,076 ms and buckets p95 1,248 ms, while the clean idle timeline p95 was 89.9 ms. During inspection the benchmark Photos server used about 125% CPU, with 5,509 `files.thumbnail` jobs and 2,082 `photos.clip.index` jobs pending. I am reducing local-root read work and pausing Photos-owned refresh work while the Photos upload route is active.

kayg commented

2026-09-25 08:42:46 +00:00

Author

Owner

The clean timeline baseline is 89.9 ms p95, but the 100-upload release burst reached 1,076 ms. A timeline root lookup also queried files_shares even when every configured root was local; a regression test reproduced an error when that unused table was absent. Local-only root resolution now skips the share query. Photos uploads now mark active work, the watcher drains and applies their changed paths after the uploads finish, timeline requests skip optional image side work, and CLIP jobs reschedule for 30 seconds with distinct dedup keys. Photos crate checks after these changes: 38 passed, 0 failed, 2 ignored; crate clippy passed.

The clean timeline baseline is 89.9 ms p95, but the 100-upload release burst reached 1,076 ms. A timeline root lookup also queried `files_shares` even when every configured root was local; a regression test reproduced an error when that unused table was absent. Local-only root resolution now skips the share query. Photos uploads now mark active work, the watcher drains and applies their changed paths after the uploads finish, timeline requests skip optional image side work, and CLIP jobs reschedule for 30 seconds with distinct dedup keys. Photos crate checks after these changes: 38 passed, 0 failed, 2 ignored; crate clippy passed.

kayg commented

2026-09-25 08:57:10 +00:00

Author

Owner

Upload coordination alone did not meet the requested p95: all 100 uploads returned 201, but timeline reads measured p95 1,481 ms and buckets p95 1,429 ms (burst 32.8 s). During this run the Photos server was using about 155% CPU; the Index had 12,730 Files thumbnail jobs and 4,519 Photos CLIP jobs pending. Idle release reads stayed below target at 72.9 ms timeline p95 and 59.9 ms buckets p95. I am limiting concurrent Photos upload work and will repeat the burst.

kayg commented

2026-09-25 08:58:57 +00:00

Author

Owner

Photos now caps active upload handling at two requests. The counter starts before the route waits for an upload slot, so incremental refresh and optional timeline work remain deferred across the full queued burst. The cap is a local Photos route limit and does not change Files APIs. Photos tests: 38 passed, 0 failed, 2 ignored; Photos clippy passed. Decision recorded for owner review: the issue did not set a concurrency value, so I chose two in flight to reduce shared Index writer pressure while keeping a pair of upload workers.

kayg commented

2026-09-25 09:07:26 +00:00

Author

Owner

The Photos route now limits uploads to two active Files writes. The release upload probe at the app's configured three upload workers completed all 100 uploads with HTTP 201; during the burst, timeline p95 was 116.7 ms and buckets p95 was 92.2 ms. A direct HTTP probe measured 66.0 ms timeline p95 and 51.7 ms bucket p95. The performance fixture began with 25,000 indexed items; prior probe runs added 300 uniquely named items before the final 100-upload browser run. The photos-perf.mjs release API baseline on the exact 25,000-item fixture was 72.9 ms timeline p95 and 59.9 ms buckets p95.

The Photos route now limits uploads to two active Files writes. The release upload probe at the app's configured three upload workers completed all 100 uploads with HTTP 201; during the burst, timeline p95 was 116.7 ms and buckets p95 was 92.2 ms. A direct HTTP probe measured 66.0 ms timeline p95 and 51.7 ms bucket p95. The performance fixture began with 25,000 indexed items; prior probe runs added 300 uniquely named items before the final 100-upload browser run. The `photos-perf.mjs` release API baseline on the exact 25,000-item fixture was 72.9 ms timeline p95 and 59.9 ms buckets p95.

kayg commented

2026-09-25 09:16:55 +00:00

Author

Owner

Round two of bash tests/adversarial/run.sh reproduced the existing Files share isolation finding: list shared folder reported that recipient listing wrote an unindexed child to the owner's Index synchronously. This code is in the Files plugin, outside the Photos files owned by issue #79. I am recording it as an external gate finding; Photos API changes did not trigger it.

Round two of `bash tests/adversarial/run.sh` reproduced the existing Files share isolation finding: `list shared folder` reported that recipient listing wrote an unindexed child to the owner's Index synchronously. This code is in the Files plugin, outside the Photos files owned by issue #79. I am recording it as an external gate finding; Photos API changes did not trigger it.

kayg commented

2026-09-25 09:20:09 +00:00

Author

Owner

Finished on branch job/photos-incremental at head 6110034d3201af544779a87db48dc5463eb1fa87.

Release performance: photos-perf.mjs --items 25000 indexed all 25,000 items in 134,421 ms. API p95 was 59.9 ms for buckets, 72.9 ms for a timeline page, and 74.2 ms for a mid-library page. With the app's three upload workers, 100 concurrent uploads all returned 201; during that burst timeline p95 was 116.7 ms and buckets p95 was 92.2 ms.

Gate output:

cargo fmt --check: exit 0, no output.

cargo clippy --workspace --all-targets -- -D warnings:

    Checking calternal-plugin-photos v0.0.1 (/home/kayg/Developer/calternal-wt/photos-incremental/crates/plugins/photos)
    Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/photos-incremental/crates/calternal-server)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.82s

cargo test --workspace: 55 test result: ok summaries; 957 passed, 0 failed, 9 ignored. Photos crate output:
```
test result: ok. 38 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 7.08s
```

bash packages/api-client/check-generated.sh:

✨ openapi-typescript 7.13.0
🚀 ../../contracts/openapi.json → src/generated.ts [456ms]

bash tests/adversarial/run.sh exited 1:
```
==== FINDINGS 0
==== ROUND 2 FINDINGS 1
 - list shared folder :: recipient listing wrote an unindexed child to the owner's Index synchronously
```
Round one reported no findings. Round two reproduced the existing Files share-listing finding; it is outside the Photos-owned files for this job.

Decisions recorded for owner review: limit Photos upload handling to two active requests; defer Photos CLIP jobs by 30 seconds during uploads; wait for active Photos uploads to finish before applying their coalesced Files events; skip Files share-root lookup when all configured Photos roots are local. The design document did not specify these values or read optimizations.

Finished on branch `job/photos-incremental` at head `6110034d3201af544779a87db48dc5463eb1fa87`. Release performance: `photos-perf.mjs --items 25000` indexed all 25,000 items in 134,421 ms. API p95 was 59.9 ms for buckets, 72.9 ms for a timeline page, and 74.2 ms for a mid-library page. With the app's three upload workers, 100 concurrent uploads all returned 201; during that burst timeline p95 was 116.7 ms and buckets p95 was 92.2 ms. Gate output: - `cargo fmt --check`: exit 0, no output. - `cargo clippy --workspace --all-targets -- -D warnings`: ``` Checking calternal-plugin-photos v0.0.1 (/home/kayg/Developer/calternal-wt/photos-incremental/crates/plugins/photos) Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/photos-incremental/crates/calternal-server) Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.82s ``` - `cargo test --workspace`: 55 `test result: ok` summaries; 957 passed, 0 failed, 9 ignored. Photos crate output: ``` test result: ok. 38 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 7.08s ``` - `bash packages/api-client/check-generated.sh`: ``` ✨ openapi-typescript 7.13.0 🚀 ../../contracts/openapi.json → src/generated.ts [456ms] ``` - `bash tests/adversarial/run.sh` exited 1: ``` ==== FINDINGS 0 ==== ROUND 2 FINDINGS 1 - list shared folder :: recipient listing wrote an unindexed child to the owner's Index synchronously ``` Round one reported no findings. Round two reproduced the existing Files share-listing finding; it is outside the Photos-owned files for this job. Decisions recorded for owner review: limit Photos upload handling to two active requests; defer Photos CLIP jobs by 30 seconds during uploads; wait for active Photos uploads to finish before applying their coalesced Files events; skip Files share-root lookup when all configured Photos roots are local. The design document did not specify these values or read optimizations.

kayg commented

2026-09-25 09:21:58 +00:00

Author

Owner

Additional design decisions for owner review (the design does not set these implementation details): migration 4 stores photos_groups.root_path and adds lookup indexes so day buckets and group updates stay scoped to a configured root; related pair/burst/stack groups are recomputed as the connected component touched by old and new records; the watcher coalesces for 250 ms of quiet, up to a 1 s event window, and continues draining while Photos uploads are active; timeline pages use one ranked query across requested days and cap optional thumbnail and ThumbHash work at 16 items per request; Photos CLIP work defers for 30 seconds during active uploads; Photos upload handling allows two simultaneous Files writes. The generated migration remains derived and rebuildable.

Additional design decisions for owner review (the design does not set these implementation details): migration 4 stores `photos_groups.root_path` and adds lookup indexes so day buckets and group updates stay scoped to a configured root; related pair/burst/stack groups are recomputed as the connected component touched by old and new records; the watcher coalesces for 250 ms of quiet, up to a 1 s event window, and continues draining while Photos uploads are active; timeline pages use one ranked query across requested days and cap optional thumbnail and ThumbHash work at 16 items per request; Photos CLIP work defers for 30 seconds during active uploads; Photos upload handling allows two simultaneous Files writes. The generated migration remains derived and rebuildable.

kayg commented

2026-09-25 09:38:04 +00:00

Author

Owner

Starting continuation on job/photos-incremental. Current dev base: af185c5740. Previous head: 6110034d3201af544779a87db48dc5463eb1fa87.

Starting continuation on job/photos-incremental. Current dev base: af185c5740a03077f7e48c482813878ab3fc1a29. Previous head: 6110034d3201af544779a87db48dc5463eb1fa87.

kayg commented

2026-09-25 09:45:10 +00:00

Author

Owner

Finding: the focused Rust regression test fails on the pre-fix code. A recipient GET to the shared folder registers an owner-side reconcile_folders task while the namespace lock is held; Photos refresh, parent lookups, and share-root validation are read-only. The shared Files listing therefore causes the later owner Index write. Fixing the listing trigger while retaining owner listing reconciliation.

Finding: the focused Rust regression test fails on the pre-fix code. A recipient GET to the shared folder registers an owner-side `reconcile_folders` task while the namespace lock is held; Photos refresh, parent lookups, and share-root validation are read-only. The shared Files listing therefore causes the later owner Index write. Fixing the listing trigger while retaining owner listing reconciliation.

kayg commented

2026-09-25 10:04:53 +00:00

Author

Owner

Finished #79 on job/photos-incremental.
Head: cc2b76116537c6e0c4570c373913042b6a798889.

Decision not covered by DESIGN: a shared read may register a wakeup on an already-running owner reconcile, but it never starts one; only an owner listing starts on-demand reconciliation. The focused Rust test failed before the fix when recipient listing registered the reconcile.

Gate output:

cargo fmt --check: exit 0, no output.
cargo clippy --workspace --all-targets -- -D warnings:
Finished dev profile [unoptimized + debuginfo] target(s) in 1m 08s
cargo test --workspace (exit 0):
test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.33s
test result: ok. 38 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 1.81s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.01s
bash packages/api-client/check-generated.sh:
🚀 ../../contracts/openapi.json → src/generated.ts [325.9ms]
bash tests/adversarial/run.sh:
==== FINDINGS 0
==== ROUND 2 FINDINGS 0
restart probe: 0 findings

Finished #79 on `job/photos-incremental`. Head: `cc2b76116537c6e0c4570c373913042b6a798889`. Decision not covered by DESIGN: a shared read may register a wakeup on an already-running owner reconcile, but it never starts one; only an owner listing starts on-demand reconciliation. The focused Rust test failed before the fix when recipient listing registered the reconcile. Gate output: - `cargo fmt --check`: exit 0, no output. - `cargo clippy --workspace --all-targets -- -D warnings`: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 08s` - `cargo test --workspace` (exit 0): `test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.33s` `test result: ok. 38 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 1.81s` `test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.01s` - `bash packages/api-client/check-generated.sh`: `🚀 ../../contracts/openapi.json → src/generated.ts [325.9ms]` - `bash tests/adversarial/run.sh`: `==== FINDINGS 0` `==== ROUND 2 FINDINGS 0` `restart probe: 0 findings`

kayg closed this issue

2026-09-25 10:08:14 +00:00

kayg referenced this issue from a commit

2026-09-25 10:08:15 +00:00

Merge job/photos-incremental: incremental Photos refresh, upload-aware scheduling; shared reads never reconcile owner Files (#79)

kayg referenced this issue

2026-09-25 10:08:24 +00:00

Collab: restart_epoch test flaked once in a full workspace run #84