Local adversarial run lacks the media sandbox for thumbnail checks #209

Closed
opened 2026-09-26 19:06:58 +00:00 by kayg · 18 comments
Owner

Finding

The real-server adversarial round on 2026-09-26 could not exercise public photo thumbnails on the local host. tests/adversarial/attack2.py uploaded crates/plugins/files/tests/black.jpg, waited 30 seconds, then observed has_thumbnail=false; public /thumb?s=256 and /thumb?s=1024 returned 404. The public-link burst also returned 404 for all thumbnail requests.

Evidence

MediaThumbnailer starts calternal-media-sandbox in crates/plugins/files/src/media.rs. On the test host, calternal-media-sandbox, calternal-media-sandbox-dropcaps, and bwrap are absent from PATH; vips and vipsheader are present. tests/adversarial/run.sh builds and starts target/debug/calternal-server directly and does not provide the runtime sandbox wrapper from deploy/Containerfile.runtime.

This is a local adversarial-runner/runtime setup gap. The probe did not establish whether thumbnails work in the packaged runtime. Make the local probe use the production media runtime, or clearly gate thumbnail assertions when that runtime is unavailable while retaining them in the packaged-runtime check.

## Finding The real-server adversarial round on 2026-09-26 could not exercise public photo thumbnails on the local host. `tests/adversarial/attack2.py` uploaded `crates/plugins/files/tests/black.jpg`, waited 30 seconds, then observed `has_thumbnail=false`; public `/thumb?s=256` and `/thumb?s=1024` returned 404. The public-link burst also returned 404 for all thumbnail requests. ## Evidence `MediaThumbnailer` starts `calternal-media-sandbox` in `crates/plugins/files/src/media.rs`. On the test host, `calternal-media-sandbox`, `calternal-media-sandbox-dropcaps`, and `bwrap` are absent from `PATH`; `vips` and `vipsheader` are present. `tests/adversarial/run.sh` builds and starts `target/debug/calternal-server` directly and does not provide the runtime sandbox wrapper from `deploy/Containerfile.runtime`. This is a local adversarial-runner/runtime setup gap. The probe did not establish whether thumbnails work in the packaged runtime. Make the local probe use the production media runtime, or clearly gate thumbnail assertions when that runtime is unavailable while retaining them in the packaged-runtime check.
Author
Owner

Additional post-merge reproduction on #156: tests/adversarial/run.sh completed its local server round with the same runtime limitation. Round 2 reported share setup: no thumbnail for the shared photo after 30 s, 404 for public 256 px and 1024 px thumbnails, missing revalidation headers, and 404 for every request in the gallery burst. media_uploads.py passed and the server remained alive. command -v calternal-media-sandbox returned no path in this worktree environment. The runner did not establish thumbnail behavior in the packaged runtime.

Additional post-merge reproduction on #156: `tests/adversarial/run.sh` completed its local server round with the same runtime limitation. Round 2 reported `share setup: no thumbnail for the shared photo after 30 s`, 404 for public 256 px and 1024 px thumbnails, missing revalidation headers, and 404 for every request in the gallery burst. `media_uploads.py` passed and the server remained alive. `command -v calternal-media-sandbox` returned no path in this worktree environment. The runner did not establish thumbnail behavior in the packaged runtime.
Author
Owner

Also blocks the dev e2e gates (Claude, 2026-09-26 night): share e2e 'thumbnails did not appear' on 2527f0d0. bubblewrap is now installed on the build host (apt). Remaining: make the local server and e2e harness find calternal-media-sandbox and calternal-media-sandbox-dropcaps (build them from deploy/ in the harness or document a one-time install), so thumbnails work outside the container. Same job fixes the calendar and search e2e after the #159 shortcut changes.

Also blocks the dev e2e gates (Claude, 2026-09-26 night): share e2e 'thumbnails did not appear' on 2527f0d0. bubblewrap is now installed on the build host (apt). Remaining: make the local server and e2e harness find calternal-media-sandbox and calternal-media-sandbox-dropcaps (build them from deploy/ in the harness or document a one-time install), so thumbnails work outside the container. Same job fixes the calendar and search e2e after the #159 shortcut changes.
Author
Owner

Additional reproduction from overlay-glass #157 on 2026-09-26: the post-merge tests/adversarial/run.sh completed with exit 1 because it counts findings. media_uploads.py passed, the server stayed alive, and the restart probe reported 0 findings. Round 1's 36 findings were all SLOW responses (200/201/412). Round 2 had the same 15 missing-thumbnail 404s recorded in this issue plus 5 SLOW Unicode writes. On this host, command -v calternal-media-sandbox returned no path, while command -v bwrap returned /usr/bin/bwrap; vips and vipsheader are also present. This confirms the local runtime wrapper gap; the probe did not establish thumbnail behavior in the packaged runtime.

Additional reproduction from overlay-glass #157 on 2026-09-26: the post-merge `tests/adversarial/run.sh` completed with exit 1 because it counts findings. `media_uploads.py` passed, the server stayed alive, and the restart probe reported 0 findings. Round 1's 36 findings were all SLOW responses (200/201/412). Round 2 had the same 15 missing-thumbnail 404s recorded in this issue plus 5 SLOW Unicode writes. On this host, `command -v calternal-media-sandbox` returned no path, while `command -v bwrap` returned `/usr/bin/bwrap`; vips and vipsheader are also present. This confirms the local runtime wrapper gap; the probe did not establish thumbnail behavior in the packaged runtime.
Author
Owner

Post-merge adversarial rerun for #152 on 2026-09-26:

  • tests/adversarial/run.sh reported no thumbnail generated within 10 seconds (media worker busy or missing). The shared-photo 256 px and 1024 px endpoints returned 404 JSON; ETag/cache headers were absent and the gallery burst returned 404.
  • The production web build and media upload probe passed, and the server remained alive.
  • command -v calternal-media-sandbox returned no path in this local worktree environment.

This repeats the local runtime limitation already recorded here. The round does not establish thumbnail behavior in the packaged runtime.

Post-merge adversarial rerun for #152 on 2026-09-26: - `tests/adversarial/run.sh` reported no thumbnail generated within 10 seconds (`media worker busy or missing`). The shared-photo 256 px and 1024 px endpoints returned 404 JSON; ETag/cache headers were absent and the gallery burst returned 404. - The production web build and media upload probe passed, and the server remained alive. - `command -v calternal-media-sandbox` returned no path in this local worktree environment. This repeats the local runtime limitation already recorded here. The round does not establish thumbnail behavior in the packaged runtime.
Author
Owner

Claude restored Search to ⌘K/Ctrl+K on dev (8ee9bb9a); merge dev: the search e2e's Control+k now matches the registry. Calendar zoom keys and the media sandbox for thumbnails remain for this job.

Claude restored Search to ⌘K/Ctrl+K on dev (8ee9bb9a); merge dev: the search e2e's Control+k now matches the registry. Calendar zoom keys and the media sandbox for thumbnails remain for this job.
Author
Owner

Production finding (Claude): on o2 (arm64) deploy/media-sandbox failed with 'Can't find source path /lib64', so all sandboxed media jobs failed after #151. Fixed in e0431587 (--ro-bind-try /lib64) and hot-patched in the live container. Please add a test that runs deploy/media-sandbox (probe and image modes) in the harness, including a case with no /lib64, so a bind that does not exist on one architecture fails CI instead of production.

Production finding (Claude): on o2 (arm64) deploy/media-sandbox failed with 'Can't find source path /lib64', so all sandboxed media jobs failed after #151. Fixed in e0431587 (--ro-bind-try /lib64) and hot-patched in the live container. Please add a test that runs deploy/media-sandbox (probe and image modes) in the harness, including a case with no /lib64, so a bind that does not exist on one architecture fails CI instead of production.
Author
Owner

Starting #209 on branch job/e2e-fix at 8a2a9417fc0b1408c17e4807bb71fb25941fb574 (branch base SHA). I am reproducing the production e2e failures, then will update each affected suite or app behavior in separate commits. I will merge current dev before final gates as requested.

Starting #209 on branch `job/e2e-fix` at `8a2a9417fc0b1408c17e4807bb71fb25941fb574` (branch base SHA). I am reproducing the production e2e failures, then will update each affected suite or app behavior in separate commits. I will merge current `dev` before final gates as requested.
Author
Owner

Resuming #209 after the VM restart on job/e2e-fix at d2259af7081f22296748b02b0463c4d89727ed0c; the checkpoint is based on dev at f6418fc72d750d941f23554ad16381f6dd400fd6. The worktree is clean. The checkpoint contains the Calendar grid zoom key update. I am continuing the local production media sandbox setup and regression coverage, then I will rerun the affected browser flows and final gates.

Resuming #209 after the VM restart on `job/e2e-fix` at `d2259af7081f22296748b02b0463c4d89727ed0c`; the checkpoint is based on `dev` at `f6418fc72d750d941f23554ad16381f6dd400fd6`. The worktree is clean. The checkpoint contains the Calendar grid zoom key update. I am continuing the local production media sandbox setup and regression coverage, then I will rerun the affected browser flows and final gates.
Author
Owner

Evidence after adding the staged local runtime: apps/web/e2e/share.mjs still times out waiting for thumbnails. Directly running the Files header probe through the same sandbox, vipsheader -a '[descriptor=0]' < crates/plugins/files/tests/black.jpg, returns VipsForeignLoad: file "" does not exist. The real sandbox image mode does produce a valid WebP from that fixture. This points to a separate failure in the Files image-header precheck before thumbnail generation; I am verifying it against the crate integration test before deciding the smallest fix.

Evidence after adding the staged local runtime: `apps/web/e2e/share.mjs` still times out waiting for thumbnails. Directly running the Files header probe through the same sandbox, `vipsheader -a '[descriptor=0]' < crates/plugins/files/tests/black.jpg`, returns `VipsForeignLoad: file "" does not exist`. The real sandbox image mode does produce a valid WebP from that fixture. This points to a separate failure in the Files image-header precheck before thumbnail generation; I am verifying it against the crate integration test before deciding the smallest fix.
Author
Owner

Root cause confirmed. With CALTERNAL_MEDIA_SANDBOX_NPROC=4096 so the shared host's existing threads do not make the sandbox availability check return early, cargo test -p calternal-plugin-files real_media_fixtures_generate_webp_thumbnails_with_sandboxed_decoders -- --nocapture fails with NotFound on the first missing thumbnail. vipsheader -a '[descriptor=0]' returns VipsForeignLoad: file "" does not exist; the same sandboxed command with /proc/self/fd/0 reports the image dimensions. The thumbnail worker's precheck used vips's operation-only descriptor syntax with the vipsheader utility, so it rejected valid images before the image-mode decoder ran.

Root cause confirmed. With `CALTERNAL_MEDIA_SANDBOX_NPROC=4096` so the shared host's existing threads do not make the sandbox availability check return early, `cargo test -p calternal-plugin-files real_media_fixtures_generate_webp_thumbnails_with_sandboxed_decoders -- --nocapture` fails with `NotFound` on the first missing thumbnail. `vipsheader -a '[descriptor=0]'` returns `VipsForeignLoad: file "" does not exist`; the same sandboxed command with `/proc/self/fd/0` reports the image dimensions. The thumbnail worker's precheck used vips's operation-only descriptor syntax with the `vipsheader` utility, so it rejected valid images before the image-mode decoder ran.
Author
Owner

The Files probe now opens the inherited input as /proc/self/fd/0. With the local staged runtime, the real-media integration test passes for JPEG, PNG, WebP, HEIC and AVIF. The local runtime does not expose sandboxed FFmpeg, so the video fixture is skipped with an explicit message; the packaged runtime includes FFmpeg and retains that video assertion. I am rebuilding the server before rerunning Share E2E.

The Files probe now opens the inherited input as `/proc/self/fd/0`. With the local staged runtime, the real-media integration test passes for JPEG, PNG, WebP, HEIC and AVIF. The local runtime does not expose sandboxed FFmpeg, so the video fixture is skipped with an explicit message; the packaged runtime includes FFmpeg and retains that video assertion. I am rebuilding the server before rerunning Share E2E.
Author
Owner

E2E follow-up finding: the Search E2E still sent Ctrl+F for Files and Calendar scoped search and Ctrl+Shift+K for command mode. The current app shortcut registry and DESIGN §34 use Ctrl+Shift+F and Ctrl+Shift+; respectively. The first run timed out waiting for the Files in:Design pill. After updating the scoped-search keys, Files and Calendar flows passed; the next run reached command mode and timed out waiting for the search window. The E2E calls now use the registered shortcuts, including the screenshot flow. I am rerunning Search against the production build.

E2E follow-up finding: the Search E2E still sent Ctrl+F for Files and Calendar scoped search and Ctrl+Shift+K for command mode. The current app shortcut registry and DESIGN §34 use Ctrl+Shift+F and Ctrl+Shift+; respectively. The first run timed out waiting for the Files `in:Design` pill. After updating the scoped-search keys, Files and Calendar flows passed; the next run reached command mode and timed out waiting for the search window. The E2E calls now use the registered shortcuts, including the screenshot flow. I am rerunning Search against the production build.
Author
Owner

Merged local adversarial round result: tests/adversarial/run.sh exited 1 because Round 1 counted three SLOW responses (Appearance concurrent PUT p95 1.067s, DAV move 6.9s with 200, Calendar Event creation 5.1s with 201). The authorization matrix completed 916/916 requests. Round 2 reported 0 findings, the server stayed alive, and the restart probe reported 0 findings. Round 2 logged thumbnail: none generated in 10 s (media worker busy or missing); header check skipped while several other worktrees were running adversarial probes. The local sandbox script passed real probe and image modes, the Files real-media test passed, and Share E2E passed with thumbnails, so this was a 10-second check skipped under host load rather than a confirmed sandbox failure. No non-SLOW adversarial finding was reported.

Merged local adversarial round result: `tests/adversarial/run.sh` exited 1 because Round 1 counted three SLOW responses (Appearance concurrent PUT p95 1.067s, DAV move 6.9s with 200, Calendar Event creation 5.1s with 201). The authorization matrix completed 916/916 requests. Round 2 reported 0 findings, the server stayed alive, and the restart probe reported 0 findings. Round 2 logged `thumbnail: none generated in 10 s (media worker busy or missing); header check skipped` while several other worktrees were running adversarial probes. The local sandbox script passed real probe and image modes, the Files real-media test passed, and Share E2E passed with thumbnails, so this was a 10-second check skipped under host load rather than a confirmed sandbox failure. No non-SLOW adversarial finding was reported.
Author
Owner

Calendar E2E finding after merging dev at 21ee397d: the preview-drag flow timed out at apps/web/e2e/calendar.mjs while waiting for .cal-popover.placed to have computed transform === 'none'. The shared resting style in packages/ui/src/tokens.css sets .floating-surface-shell.placed { transform: scale(1) }, which computes to an identity matrix, so that predicate could never pass. I updated the wait to check for an identity transform matrix before measuring the popover. The existing E2E failure reproduced this timeout on the merged build; I am rerunning it now.

Calendar E2E finding after merging `dev` at `21ee397d`: the preview-drag flow timed out at `apps/web/e2e/calendar.mjs` while waiting for `.cal-popover.placed` to have computed `transform === 'none'`. The shared resting style in `packages/ui/src/tokens.css` sets `.floating-surface-shell.placed { transform: scale(1) }`, which computes to an identity matrix, so that predicate could never pass. I updated the wait to check for an identity transform matrix before measuring the popover. The existing E2E failure reproduced this timeout on the merged build; I am rerunning it now.
Author
Owner

Calendar Agenda E2E finding after the transform wait was corrected: the flow then timed out waiting for the entry Inspector. AgendaList matches the shared app.inspector registry, which binds Ctrl/Cmd+Shift+Period, but the new E2E sent Control+i. Existing popover E2E coverage and apps/web/src/lib/shortcuts/registry.ts confirm the registered combo. I changed the Calendar flow to press Control+Shift+. and am rerunning it.

Calendar Agenda E2E finding after the transform wait was corrected: the flow then timed out waiting for the entry Inspector. `AgendaList` matches the shared `app.inspector` registry, which binds Ctrl/Cmd+Shift+Period, but the new E2E sent `Control+i`. Existing popover E2E coverage and `apps/web/src/lib/shortcuts/registry.ts` confirm the registered combo. I changed the Calendar flow to press `Control+Shift+.` and am rerunning it.
Author
Owner

Calendar Agenda E2E finding after the registered shortcut passed: opening ticket.pdf after following the entry deep link failed Playwright strict mode because the same action name existed in both the Agenda attachment deck and the selected Log entry preview. The flow is specifically checking the Agenda deck, so I scoped the click to .agenda and am rerunning the full Calendar E2E.

Calendar Agenda E2E finding after the registered shortcut passed: opening `ticket.pdf` after following the entry deep link failed Playwright strict mode because the same action name existed in both the Agenda attachment deck and the selected Log entry preview. The flow is specifically checking the Agenda deck, so I scoped the click to `.agenda` and am rerunning the full Calendar E2E.
Author
Owner

Finished #209 on branch job/e2e-fix.

Head SHA: ea8c5f81ea0aa8674e2a4c37b11ac09bbf8a6d0d. The branch includes the requested merge of dev at 21ee397d5c3d24f6355d90b17c2e36b2cf76615a.

Built

  • Added a staged local media runtime based on deploy/media-sandbox. The adversarial runner and real-server E2E harness now build and use the production wrapper and dropcaps binary.
  • Added checks for hosts without /lib64, and real probe and image modes. Production sandbox defaults stay fixed. The local wrapper bounds its outer process limit to fit the host and applies the production inner limit after namespace setup.
  • Fixed the Files thumbnail header probe to open the inherited input via /proc/self/fd/0. The old vipsheader '[descriptor=0]' form failed on valid images.
  • Fixed the Search shortcut E2E and Share thumbnail waits. After merging the latest Calendar Agenda work, fixed its transform wait, Inspector shortcut and attachment locator. Calendar E2E now passes all flows.

Files: deploy/media-sandbox; tests/adversarial/prepare-media-runtime.sh; tests/adversarial/test-media-sandbox.sh; tests/adversarial/run.sh; crates/plugins/files/src/thumbnails.rs; crates/plugins/files/src/lib.rs; apps/web/e2e/harness.mjs; apps/web/e2e/search.mjs; apps/web/e2e/share.mjs; apps/web/e2e/calendar.mjs.

Validation

  • tests/adversarial/test-media-sandbox.sh: passed. Files real-media fixture: passed. Share E2E: SHARE E2E PASSED. Search E2E: search e2e: ok. Calendar E2E: calendar e2e: all flows passed.
  • Adversarial server round: authorization matrix 916/916; Round 2 and restart probe found 0 issues; server stayed alive. Round 1 made the runner exit 1 for three SLOW responses (Appearance PUT p95 1.067s, DAV move 6.9s, Calendar Event create 5.1s). No non-SLOW finding was reported.

Final gates ran after the latest dev merge:

  • cargo fmt --check: exit 0; stdout and stderr were empty.
  • cargo clippy --all-targets -- -D warnings:
    Finished dev profile [unoptimized + debuginfo] target(s) in 10.69s
  • cargo test: exit 0; 69 result summaries, 1,236 passed, 0 failed, 12 ignored. One exact result line:
    test result: ok. 477 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.18s
  • bun run --cwd apps/web check:
    svelte-check found 0 errors and 0 warnings
  • bun run --cwd apps/web test:
    Test Files 69 passed (69)
    Tests 539 passed (539)
    Start at 09:23:18
    Duration 20.31s (transform 64%, import 15%, environment 12%, tests 8%, setup 2%)

Known gaps

The staged local sandbox has no sandboxed FFmpeg, so the local video thumbnail fixture is skipped. The packaged runtime includes FFmpeg and retains that check. Under shared-host load, Round 2's 10-second thumbnail probe skipped its header check because no thumbnail appeared in time; the real sandbox test, Files media fixture and Share E2E all passed.

Decisions not covered by DESIGN.md

The local test runtime accepts a staged root and a host-sized outer nproc limit; production keeps its fixed root and limits. The video fixture skips only when the local sandbox lacks FFmpeg tools, while packaged-runtime coverage keeps the assertion.

Cargo and web build outputs were cleaned after the gates.

Finished #209 on branch `job/e2e-fix`. Head SHA: `ea8c5f81ea0aa8674e2a4c37b11ac09bbf8a6d0d`. The branch includes the requested merge of `dev` at `21ee397d5c3d24f6355d90b17c2e36b2cf76615a`. ## Built - Added a staged local media runtime based on `deploy/media-sandbox`. The adversarial runner and real-server E2E harness now build and use the production wrapper and dropcaps binary. - Added checks for hosts without `/lib64`, and real probe and image modes. Production sandbox defaults stay fixed. The local wrapper bounds its outer process limit to fit the host and applies the production inner limit after namespace setup. - Fixed the Files thumbnail header probe to open the inherited input via `/proc/self/fd/0`. The old `vipsheader '[descriptor=0]'` form failed on valid images. - Fixed the Search shortcut E2E and Share thumbnail waits. After merging the latest Calendar Agenda work, fixed its transform wait, Inspector shortcut and attachment locator. Calendar E2E now passes all flows. Files: `deploy/media-sandbox`; `tests/adversarial/prepare-media-runtime.sh`; `tests/adversarial/test-media-sandbox.sh`; `tests/adversarial/run.sh`; `crates/plugins/files/src/thumbnails.rs`; `crates/plugins/files/src/lib.rs`; `apps/web/e2e/harness.mjs`; `apps/web/e2e/search.mjs`; `apps/web/e2e/share.mjs`; `apps/web/e2e/calendar.mjs`. ## Validation - `tests/adversarial/test-media-sandbox.sh`: passed. Files real-media fixture: passed. Share E2E: `SHARE E2E PASSED`. Search E2E: `search e2e: ok`. Calendar E2E: `calendar e2e: all flows passed`. - Adversarial server round: authorization matrix 916/916; Round 2 and restart probe found 0 issues; server stayed alive. Round 1 made the runner exit 1 for three SLOW responses (Appearance PUT p95 1.067s, DAV move 6.9s, Calendar Event create 5.1s). No non-SLOW finding was reported. Final gates ran after the latest `dev` merge: - `cargo fmt --check`: exit 0; stdout and stderr were empty. - `cargo clippy --all-targets -- -D warnings`: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.69s` - `cargo test`: exit 0; 69 result summaries, 1,236 passed, 0 failed, 12 ignored. One exact result line: `test result: ok. 477 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.18s` - `bun run --cwd apps/web check`: `svelte-check found 0 errors and 0 warnings` - `bun run --cwd apps/web test`: ` Test Files 69 passed (69)` ` Tests 539 passed (539)` ` Start at 09:23:18` ` Duration 20.31s (transform 64%, import 15%, environment 12%, tests 8%, setup 2%)` ## Known gaps The staged local sandbox has no sandboxed FFmpeg, so the local video thumbnail fixture is skipped. The packaged runtime includes FFmpeg and retains that check. Under shared-host load, Round 2's 10-second thumbnail probe skipped its header check because no thumbnail appeared in time; the real sandbox test, Files media fixture and Share E2E all passed. ## Decisions not covered by DESIGN.md The local test runtime accepts a staged root and a host-sized outer `nproc` limit; production keeps its fixed root and limits. The video fixture skips only when the local sandbox lacks FFmpeg tools, while packaged-runtime coverage keeps the assertion. Cargo and web build outputs were cleaned after the gates.
Author
Owner

Merged into dev at cf142cb4 (local media sandbox for e2e, thumbnail probe and sandbox cap fixes, e2e updated to the #159 registry). Deploy status on #203.

Merged into dev at cf142cb4 (local media sandbox for e2e, thumbnail probe and sandbox cap fixes, e2e updated to the #159 registry). Deploy status on #203.
kayg referenced this issue from a commit 2026-09-27 07:30:02 +00:00
kayg closed this issue 2026-09-27 07:30:03 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#209
No description provided.