Adversarial: distinguish pending Search and preview work from missing results

kayg commented

2026-10-04 08:52:05 +00:00

Owner

Found during #867, branch job/7bfix-photos. The round-3 Search and document preview messages can report pending background work as missing results. Both probe deadlines are also present at production dev 6074f71d1.

Evidence:

tests/adversarial/search_chaos.py:163 defaults wait_for_hit to 5 seconds. On a fresh local Instance, the uploaded unicodenfcsentinel appeared after 32 query attempts spaced 250 ms apart (about 8 seconds). It appeared after 2 attempts in another run. Both ASCII content markers were searchable. The canonical NFD query miss is a separate defect tracked by #1044.
tests/adversarial/media_uploads.py:185 checks the valid PDF 60 times with a 250 ms pause (15 seconds). It checks the two cache families for another 15 seconds. Round-3 had already run a large background workload. A local run after 120 uploads did not show either document preview in 40 seconds. An isolated run showed only the PDF within that window; a further isolated run showed both. The last run's captured native-tool stderr was empty. #988 owns the shared media failure diagnostics.

Expected: retain the existing deadlines as performance evidence. When a result is late, record SLOW and whether the relevant work is pending, leased, complete or failed. Report missing indexed content only after completion; report a renderer failure with its content-free cause. Do not suppress malformed-input, authorization or crash findings. Do not change the existing assertions just to make a run green.

Test idea: use mocked Search/Jobs responses to prove that pending work and completed-but-missing data get different reports, then run the focused checks on a real local server under the normal shared host load. Reuse the existing server_quiesce and Search integrity helpers. Keep the failed or incomplete run in the report.

The exact branch and production-dev live comparison will be recorded on #867 when complete. No User source bytes or Security state were lost in the completed checks.

Found during #867, branch job/7bfix-photos. The round-3 Search and document preview messages can report pending background work as missing results. Both probe deadlines are also present at production dev 6074f71d1. Evidence: - tests/adversarial/search_chaos.py:163 defaults wait_for_hit to 5 seconds. On a fresh local Instance, the uploaded unicodenfcsentinel appeared after 32 query attempts spaced 250 ms apart (about 8 seconds). It appeared after 2 attempts in another run. Both ASCII content markers were searchable. The canonical NFD query miss is a separate defect tracked by #1044. - tests/adversarial/media_uploads.py:185 checks the valid PDF 60 times with a 250 ms pause (15 seconds). It checks the two cache families for another 15 seconds. Round-3 had already run a large background workload. A local run after 120 uploads did not show either document preview in 40 seconds. An isolated run showed only the PDF within that window; a further isolated run showed both. The last run's captured native-tool stderr was empty. #988 owns the shared media failure diagnostics. Expected: retain the existing deadlines as performance evidence. When a result is late, record SLOW and whether the relevant work is pending, leased, complete or failed. Report missing indexed content only after completion; report a renderer failure with its content-free cause. Do not suppress malformed-input, authorization or crash findings. Do not change the existing assertions just to make a run green. Test idea: use mocked Search/Jobs responses to prove that pending work and completed-but-missing data get different reports, then run the focused checks on a real local server under the normal shared host load. Reuse the existing server_quiesce and Search integrity helpers. Keep the failed or incomplete run in the report. The exact branch and production-dev live comparison will be recorded on #867 when complete. No User source bytes or Security state were lost in the completed checks.

kayg referenced this issue

2026-10-04 09:08:04 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-04 09:42:07 +00:00

Author

Owner

Exact production dev 6074f71d1 comparison for #867: the original Calendar section found all 120 Photos in 18 pages; ASCII Search markers passed; NFC café found both files and NFD cafe + combining acute found none (#1044). After the ZIP fixture, media_uploads.py reported kind-specific thumbnail worker: PDF and text renderers did not both publish a preview. The valid one-page PDF check passed. The round binary's matching media section passed in its latest run, while earlier isolated runs varied. This confirms the two-renderer availability failure is pre-existing under the existing 15-second deadline. It does not establish a new renderer defect.

Exact production dev 6074f71d1 comparison for #867: the original Calendar section found all 120 Photos in 18 pages; ASCII Search markers passed; NFC café found both files and NFD cafe + combining acute found none (#1044). After the ZIP fixture, media_uploads.py reported `kind-specific thumbnail worker: PDF and text renderers did not both publish a preview`. The valid one-page PDF check passed. The round binary's matching media section passed in its latest run, while earlier isolated runs varied. This confirms the two-renderer availability failure is pre-existing under the existing 15-second deadline. It does not establish a new renderer defect.

kayg referenced this issue

2026-10-04 09:57:16 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 10:28:32 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-04 13:54:47 +00:00

Author

Owner

Additional #867 evidence from job/7bfix-adv at a1f3a0797: the two tz-days missing markers are both present in the retained August 14 Daily note, with America/Los_Angeles and Asia/Kolkata zone tokens. The retained Search manifest has the same byte count and nanosecond timestamp as that source file. A bounded Search regression with those two late-night times passes on a fresh Index, including date:2026-08-14. This narrows the failure to visibility or runtime indexing; it does not prove a time-zone conversion defect. Production live comparison and completion-state evidence remain unverified. No source file was lost in the retained evidence.

Additional #867 evidence from job/7bfix-adv at a1f3a0797: the two tz-days missing markers are both present in the retained August 14 Daily note, with America/Los_Angeles and Asia/Kolkata zone tokens. The retained Search manifest has the same byte count and nanosecond timestamp as that source file. A bounded Search regression with those two late-night times passes on a fresh Index, including date:2026-08-14. This narrows the failure to visibility or runtime indexing; it does not prove a time-zone conversion defect. Production live comparison and completion-state evidence remain unverified. No source file was lost in the retained evidence.

kayg referenced this issue

2026-10-04 14:01:23 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 15:49:17 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 15:51:17 +00:00

Search: canonically equivalent NFD query misses NFC file names #1044

kayg commented

2026-10-04 15:51:41 +00:00

Author

Owner

Started #1045 on branch job/pdfprev-1045, based at c39ffe5d90126527d7aacf2d8b79507929c80616. I am tracing the per-kind thumbnail publication path and will reproduce the PDF/text and one-page PDF failures before changing it.

Started #1045 on branch `job/pdfprev-1045`, based at `c39ffe5d90126527d7aacf2d8b79507929c80616`. I am tracing the per-kind thumbnail publication path and will reproduce the PDF/text and one-page PDF failures before changing it.

kayg commented

2026-10-04 16:29:22 +00:00

Author

Owner

Confirmed the failure path from the #988 evidence. In this branch, the document command runner discards stderr and maps every non-zero exit to None; the PDF header check and renderer then treat that as rejected content and write a permanent kind-specific failure marker. #988 recorded bwrap: Creating new namespace failed: Resource temporarily unavailable when the service User had 73 threads against the old outer limit of 64. The job and cache keys already include the file path/render kind, so I found no PDF/text key collision.

The focused media_uploads.py probe passed with the staged runtime's host-sized process allowance. I added a regression for the exact bwrap failure and a real worker test that concurrently renders the same one-page PDF bytes as PDF and text, then checks both separate WebP cache entries. The fix now in progress returns launcher failures to durable Job retry and keeps decoder rejection terminal.

Confirmed the failure path from the #988 evidence. In this branch, the document command runner discards stderr and maps every non-zero exit to `None`; the PDF header check and renderer then treat that as rejected content and write a permanent kind-specific failure marker. #988 recorded `bwrap: Creating new namespace failed: Resource temporarily unavailable` when the service User had 73 threads against the old outer limit of 64. The job and cache keys already include the file path/render kind, so I found no PDF/text key collision. The focused `media_uploads.py` probe passed with the staged runtime's host-sized process allowance. I added a regression for the exact `bwrap` failure and a real worker test that concurrently renders the same one-page PDF bytes as PDF and text, then checks both separate WebP cache entries. The fix now in progress returns launcher failures to durable Job retry and keeps decoder rejection terminal.

kayg commented

2026-10-04 17:33:27 +00:00

Author

Owner

Completed #1045 on job/pdfprev-1045.

Built

PDF and text-card rendering now distinguish a retryable sandbox/runtime launch failure from terminal decoder rejection. Retryable errors return to the durable Jobs retry path and do not write a failed-thumbnail marker. Invalid content and bounded-output rejection keep their terminal behavior.
Added launch-failure classification with bounded private stderr capture. User-facing errors contain only thumbnail runtime unavailable.
Added a worker regression for “retry without marker” and a real sandbox integration case that concurrently renders identical one-page PDF bytes indexed as both PDF and text. It verifies both 256 and 1024 WebP cache entries and that the two 256 previews differ.

Files

crates/plugins/files/src/media.rs
crates/plugins/files/src/thumbnails.rs
crates/plugins/files/src/lib.rs

Head
11455a3bf4fd9465d0d56fa8bd18deb465709bd2 (working tree clean).

Gates
cargo fmt --check exited 0 with no output.

cargo clippy -p calternal-plugin-files --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 40.22s

cargo test -p calternal-plugin-files -- --test-threads=4:

test result: ok. 162 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 206.81s

cargo clippy -p calternal-fs --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 50s

cargo test -p calternal-fs -- --test-threads=4:

test result: ok. 59 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.57s
test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.51s

cargo clippy -p calternal-server --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 9m 28s

cargo test -p calternal-server -- --test-threads=4:

test result: ok. 163 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 30.91s

OPENSSL_NO_VENDOR=1 cargo build -p calternal-server:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 09s

Built apps/web first with bun run build (exit 0). The focused post-fix ADVERSARIAL_MEDIA_ONLY=1 run against the updated server exited 0. Its relevant output was:

PASS: document namespace, private input, inherited limits and network policy
PASS: PDF link/script metadata and bounded SVG references/entities stay isolated
PASS: explicit PDF and SVG loaders produce WebP; PDF action rejects JPEG
HEIF/AVIF/PDF upload probe passed

cargo clean:

     Removed 18772 files, 11.9GiB total

Web build output and staged media runtime were removed.

Findings and known gaps
Issue #988 recorded bwrap: Creating new namespace failed: Resource temporarily unavailable; the old path collapsed this launcher error into terminal rejection and wrote a failed-kind marker. Queue identity already includes owner/path/hash, and thumbnail cache families are kind-specific, so this was not a queue-key or cache-key collision. The baseline live probe did not hit the intermittent namespace failure under its staged host-sized process allowance. New tests exercise the exact diagnostic and verify retry/no-marker behavior; the post-fix real-server probe and same-hash PDF/text worker test passed.

Decisions not specified in DESIGN

Treat sandbox launch/spawn failures as retryable, and decoder rejection, output overflow, and timeout as terminal.
Retain only a bounded stderr prefix for fixed runtime diagnostics; return a generic error and never expose native stderr.
The existing bench/pdf_thumbnail.py covers the PDF render path. I did not measure performance because this issue is a correctness repair and the verification policy reserves measurements for performance issues.

UX gaps closed / left: Not applicable; this is a backend thumbnail-worker repair.

For the merge round: run the combined-branch full web/e2e and adversarial matrices under the merge-round policy.

READY FOR MERGE: yes.

Completed #1045 on `job/pdfprev-1045`. **Built** - PDF and text-card rendering now distinguish a retryable sandbox/runtime launch failure from terminal decoder rejection. Retryable errors return to the durable Jobs retry path and do not write a failed-thumbnail marker. Invalid content and bounded-output rejection keep their terminal behavior. - Added launch-failure classification with bounded private stderr capture. User-facing errors contain only `thumbnail runtime unavailable`. - Added a worker regression for “retry without marker” and a real sandbox integration case that concurrently renders identical one-page PDF bytes indexed as both PDF and text. It verifies both 256 and 1024 WebP cache entries and that the two 256 previews differ. **Files** - `crates/plugins/files/src/media.rs` - `crates/plugins/files/src/thumbnails.rs` - `crates/plugins/files/src/lib.rs` **Head** `11455a3bf4fd9465d0d56fa8bd18deb465709bd2` (working tree clean). **Gates** `cargo fmt --check` exited 0 with no output. `cargo clippy -p calternal-plugin-files --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 40.22s ``` `cargo test -p calternal-plugin-files -- --test-threads=4`: ``` test result: ok. 162 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 206.81s ``` `cargo clippy -p calternal-fs --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 50s ``` `cargo test -p calternal-fs -- --test-threads=4`: ``` test result: ok. 59 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.57s test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.51s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 9m 28s ``` `cargo test -p calternal-server -- --test-threads=4`: ``` test result: ok. 163 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 30.91s ``` `OPENSSL_NO_VENDOR=1 cargo build -p calternal-server`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 09s ``` Built `apps/web` first with `bun run build` (exit 0). The focused post-fix `ADVERSARIAL_MEDIA_ONLY=1` run against the updated server exited 0. Its relevant output was: ``` PASS: document namespace, private input, inherited limits and network policy PASS: PDF link/script metadata and bounded SVG references/entities stay isolated PASS: explicit PDF and SVG loaders produce WebP; PDF action rejects JPEG HEIF/AVIF/PDF upload probe passed ``` `cargo clean`: ``` Removed 18772 files, 11.9GiB total ``` Web build output and staged media runtime were removed. **Findings and known gaps** Issue #988 recorded `bwrap: Creating new namespace failed: Resource temporarily unavailable`; the old path collapsed this launcher error into terminal rejection and wrote a failed-kind marker. Queue identity already includes owner/path/hash, and thumbnail cache families are kind-specific, so this was not a queue-key or cache-key collision. The baseline live probe did not hit the intermittent namespace failure under its staged host-sized process allowance. New tests exercise the exact diagnostic and verify retry/no-marker behavior; the post-fix real-server probe and same-hash PDF/text worker test passed. **Decisions not specified in DESIGN** - Treat sandbox launch/spawn failures as retryable, and decoder rejection, output overflow, and timeout as terminal. - Retain only a bounded stderr prefix for fixed runtime diagnostics; return a generic error and never expose native stderr. - The existing `bench/pdf_thumbnail.py` covers the PDF render path. I did not measure performance because this issue is a correctness repair and the verification policy reserves measurements for performance issues. **UX gaps closed / left**: Not applicable; this is a backend thumbnail-worker repair. **For the merge round**: run the combined-branch full web/e2e and adversarial matrices under the merge-round policy. **READY FOR MERGE: yes.**

kayg commented

2026-10-04 20:46:13 +00:00

Author

Owner

Merged in 87c345a12. Conflict resolution keeps the current shared calternal-plugin::media_sandbox process-group runner and adds a retry-aware output path for document previews; runtime diagnostics stay private and decoder rejection remains terminal. cargo clippy -p calternal-plugin --all-targets -- -D warnings and cargo clippy -p calternal-plugin-files --all-targets -- -D warnings pass. Plugin tests: 39 passed. Files tests: 238 passed, 3 ignored.

Merged in `87c345a12`. Conflict resolution keeps the current shared `calternal-plugin::media_sandbox` process-group runner and adds a retry-aware output path for document previews; runtime diagnostics stay private and decoder rejection remains terminal. `cargo clippy -p calternal-plugin --all-targets -- -D warnings` and `cargo clippy -p calternal-plugin-files --all-targets -- -D warnings` pass. Plugin tests: 39 passed. Files tests: 238 passed, 3 ignored.

kayg commented

2026-10-04 22:52:58 +00:00

Author

Owner

The real-server media probe found PDF thumbnail worker: valid one-page PDF did not produce a thumbnail after the document sandbox and explicit PDF/SVG isolation checks passed. I am tracing the worker response through the retryable preview path and will add a regression check for the valid-PDF success case.

The real-server media probe found `PDF thumbnail worker: valid one-page PDF did not produce a thumbnail` after the document sandbox and explicit PDF/SVG isolation checks passed. I am tracing the worker response through the retryable preview path and will add a regression check for the valid-PDF success case.

kayg commented

2026-10-04 23:04:33 +00:00

Author

Owner

Merge-round-8 PDF follow-up: I retained the real-server work directory and let the queued jobs finish after the 15 s probe window. The valid one-page PDF job completed 20.3 s after enqueue and produced both 256 px and 1024 px WebP previews; the same-byte PDF and text-card variants also completed. The hostile-page-count and malformed PDFs completed without previews. The zip-bomb fixture stayed within the configured input/output bounds, and the JavaScript fixture rendered as a thumbnail without escaping the document sandbox probe. The earlier finding was a probe timeout during the seven-document burst, not a persistent renderer failure, so I classify it as SLOW/load per the merge-round rule. The separate document sandbox and PDF/SVG loader checks passed.

kayg referenced this issue

2026-10-05 00:52:53 +00:00

Mobile: long pull to update the app; tell the User when a newer build exists #1059

kayg commented

2026-10-05 00:53:03 +00:00

Author

Owner

Merge-round 8 completed at 2b6c77c14be78e7d1e1030e23b14c63a6772fca7; the retryable PDF preview changes are integrated. READY FOR STAGING: yes.

The focused PDF adversarial probe's 15-second wait expired for one valid one-page PDF during a 7-document burst. Inspection of the retained real-server job state showed it completed successfully at 20.3s and produced 256px and 1024px WebP previews. Malformed PDFs and oversized-page PDFs produced no previews; bounded worker/input checks completed. This is a SLOW/load-only probe timeout. It did not reproduce as a stuck job or unsafe render.

Final relevant gates: cargo clippy -p calternal-plugin-files --all-targets -- -D warnings and cargo test -p calternal-plugin-files passed (238 passed, 3 ignored); calternal-server clippy and tests passed (212 unit tests, 9 ignored, plus perf-guard and private-index tests). API classification passed for 392 operations, 1101 tools and 47 reviewed admin operations.

No PDF-specific UX gap remains from this run. Preview latency under burst can exceed the probe's 15-second deadline; the successful job took 20.3s.

Merge-round 8 completed at `2b6c77c14be78e7d1e1030e23b14c63a6772fca7`; the retryable PDF preview changes are integrated. READY FOR STAGING: yes. The focused PDF adversarial probe's 15-second wait expired for one valid one-page PDF during a 7-document burst. Inspection of the retained real-server job state showed it completed successfully at 20.3s and produced 256px and 1024px WebP previews. Malformed PDFs and oversized-page PDFs produced no previews; bounded worker/input checks completed. This is a SLOW/load-only probe timeout. It did not reproduce as a stuck job or unsafe render. Final relevant gates: `cargo clippy -p calternal-plugin-files --all-targets -- -D warnings` and `cargo test -p calternal-plugin-files` passed (238 passed, 3 ignored); `calternal-server` clippy and tests passed (212 unit tests, 9 ignored, plus perf-guard and private-index tests). API classification passed for 392 operations, 1101 tools and 47 reviewed admin operations. No PDF-specific UX gap remains from this run. Preview latency under burst can exceed the probe's 15-second deadline; the successful job took 20.3s.

kayg commented

2026-10-05 01:13:49 +00:00

Author

Owner

Deployed to production 2026-10-05 03:12 CEST in round 8 (2b6c77c14). Staging healthy first; production healthy in 33 s; /api/v1/version reports the build ID; change events 0/30 s.

Deployed to production 2026-10-05 03:12 CEST in round 8 (2b6c77c14). Staging healthy first; production healthy in 33 s; `/api/v1/version` reports the build ID; change events 0/30 s.

kayg closed this issue

2026-10-05 01:13:52 +00:00

kayg referenced this issue

2026-10-05 01:45:29 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-05 01:49:30 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-05 01:49:30 +00:00

Author

Owner

Round 7c4 merge finding (#867): the #988/#1045 tests make incompatible assertions on media::run_media_output_retryable. crates/plugins/files/src/media.rs has retry_message_keeps_diagnostics_out_of_user_jobs expecting Err("Thumbnails are temporarily unavailable"), and document_launcher_failure_is_retryable_and_private plus document_launcher_spawn_failure_is_retryable expecting Err("thumbnail runtime unavailable"). Both namespace fixtures have the same launcher failure category. I retained every original assertion and the #988 plain-language message, as required by the owner rule. The shared runner preserves #1045 retry behavior and private diagnostics. The two upstream string assertions need an owner decision; tests are not weakened. Full per-crate gates will record the actual failures.

Round 7c4 merge finding (#867): the #988/#1045 tests make incompatible assertions on `media::run_media_output_retryable`. `crates/plugins/files/src/media.rs` has `retry_message_keeps_diagnostics_out_of_user_jobs` expecting `Err("Thumbnails are temporarily unavailable")`, and `document_launcher_failure_is_retryable_and_private` plus `document_launcher_spawn_failure_is_retryable` expecting `Err("thumbnail runtime unavailable")`. Both namespace fixtures have the same launcher failure category. I retained every original assertion and the #988 plain-language message, as required by the owner rule. The shared runner preserves #1045 retry behavior and private diagnostics. The two upstream string assertions need an owner decision; tests are not weakened. Full per-crate gates will record the actual failures.

kayg referenced this issue

2026-10-05 02:45:20 +00:00

Merge gate: agree on the thumbnail launcher retry message (#988/#1045) #1080

kayg referenced this issue

2026-10-05 03:10:23 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-05 03:13:35 +00:00

Merge gate: agree on the thumbnail launcher retry message (#988/#1045) #1080

kayg referenced this issue

2026-10-05 03:22:27 +00:00

Merge gate: agree on the thumbnail launcher retry message (#988/#1045) #1080