BLOCKER: reserve and bound active HLS output and media snapshot disk use

kayg commented

2026-10-02 13:10:34 +00:00

Owner

Found during the read-only sec-fs audit requested on #663. Source evidence, not a reproduced disk-full incident.

Context and evidence

Audited origin/dev at c4a61e8cf0 and origin/job/merge-round-7a at 2f4482ded0. DESIGN §5 and §26 require a protected free-space reserve for writes.

crates/plugins/video/src/transcode.rs:265–317: the job runs the complete transcode before it measures output, inserts the rendition row and evicts to the cache limit.
crates/plugins/video/src/transcode.rs:610–696: FFmpeg writes an event playlist and segments directly into the held output directory; the playlist keeps all segments. The job deadline is six hours (:31).
crates/calternal-fs/src/hls.rs:72–88: creating the output directory does not reserve output space or check the protected reserve.
deploy/media-sandbox:130: video mode mounts that directory writable. The launcher limits address space, CPU, descriptors and processes, but does not enforce a total output-byte budget.
Round 7a replaces the input with a private snapshot. It keeps this output behavior. Its large snapshot copy (crates/calternal-fs/src/thumbnails.rs:435–502 on 7a) also does not reserve free space before the copy.

Reasoned impact

A compressed input size limit does not bound its derived output size. An active rendition can consume the protected space before completion-time eviction runs. Derived output is outside Home quota; on a shared filesystem, one User's media work can make writes fail for every User. On a split system filesystem, it can affect the Index and security-state writes. This is a resource-exhaustion risk, not a SLOW-only latency finding. No peak output or ENOSPC measurement was made.

Concrete fix

Add a calternal-fs reservation for derived work on the correct filesystem. Bound each active rendition and the aggregate active output, account for completed cache entries before admission, and stop and remove incomplete output before the reserve is crossed. Include large on-disk snapshots in the reservation. A per-file limit alone does not bound a multi-segment rendition. Preserve the held-directory boundary and cancellation cleanup.

Defensive tests

Use an inert writer and a test free-space provider with small byte budgets. Assert that active output and snapshot admission stop at the bound, two jobs cannot over-reserve, cancellation releases the reservation, incomplete files are removed, and a second User can still perform a normal write. Use no large allocation or disk-filling fixture.

Duplicate search: all-state searches for HLS and quota. #333 concerns Home quota defaults and is closed; #32 adds HLS playback; #501 fixes input isolation. None covers active derived-output reservations.

Found during the read-only sec-fs audit requested on #663. Source evidence, not a reproduced disk-full incident. ## Context and evidence Audited `origin/dev` at c4a61e8cf090170f35b1bed3350d9de20c83ecd5 and `origin/job/merge-round-7a` at 2f4482ded066d9c5d9c59130377907f7fd2916c9. DESIGN §5 and §26 require a protected free-space reserve for writes. - `crates/plugins/video/src/transcode.rs:265–317`: the job runs the complete transcode before it measures output, inserts the rendition row and evicts to the cache limit. - `crates/plugins/video/src/transcode.rs:610–696`: FFmpeg writes an event playlist and segments directly into the held output directory; the playlist keeps all segments. The job deadline is six hours (`:31`). - `crates/calternal-fs/src/hls.rs:72–88`: creating the output directory does not reserve output space or check the protected reserve. - `deploy/media-sandbox:130`: video mode mounts that directory writable. The launcher limits address space, CPU, descriptors and processes, but does not enforce a total output-byte budget. - Round 7a replaces the input with a private snapshot. It keeps this output behavior. Its large snapshot copy (`crates/calternal-fs/src/thumbnails.rs:435–502` on 7a) also does not reserve free space before the copy. ## Reasoned impact A compressed input size limit does not bound its derived output size. An active rendition can consume the protected space before completion-time eviction runs. Derived output is outside Home quota; on a shared filesystem, one User's media work can make writes fail for every User. On a split system filesystem, it can affect the Index and security-state writes. This is a resource-exhaustion risk, not a SLOW-only latency finding. No peak output or ENOSPC measurement was made. ## Concrete fix Add a calternal-fs reservation for derived work on the correct filesystem. Bound each active rendition and the aggregate active output, account for completed cache entries before admission, and stop and remove incomplete output before the reserve is crossed. Include large on-disk snapshots in the reservation. A per-file limit alone does not bound a multi-segment rendition. Preserve the held-directory boundary and cancellation cleanup. ## Defensive tests Use an inert writer and a test free-space provider with small byte budgets. Assert that active output and snapshot admission stop at the bound, two jobs cannot over-reserve, cancellation releases the reservation, incomplete files are removed, and a second User can still perform a normal write. Use no large allocation or disk-filling fixture. Duplicate search: all-state searches for HLS and quota. #333 concerns Home quota defaults and is closed; #32 adds HLS playback; #501 fixes input isolation. None covers active derived-output reservations.

kayg referenced this issue

2026-10-02 13:14:15 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:17:21 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:25:26 +00:00

Author

Owner

Started mediafix on job/mediafix, base 2f4482ded066d9c5d9c59130377907f7fd2916c9 (round 7a). Own #779, #781, #801, #802, #816. Read contracts, issues and sec-fs audit. Initial origin/dev and origin/job/merge-round-7a merges: Already up to date. Parser for #816 exists only on pending origin/job/voicefiles-620, not the required base or dev. Working on present boundaries first. No push or deploy.

Started mediafix on `job/mediafix`, base `2f4482ded066d9c5d9c59130377907f7fd2916c9` (round 7a). Own #779, #781, #801, #802, #816. Read contracts, issues and sec-fs audit. Initial origin/dev and origin/job/merge-round-7a merges: Already up to date. Parser for #816 exists only on pending origin/job/voicefiles-620, not the required base or dev. Working on present boundaries first. No push or deploy.

kayg commented

2026-10-02 13:55:39 +00:00

Author

Owner

Finding and first slice: #802 cancellation regression failed on old code with Home usage left: 15, right: 8 (seven abandoned bytes). Commit 4d07d7c7a adds held-parent cleanup and journal ownership transfer, plus bounded startup cleanup. Cancellation, live recovery and copy/chunk I/O checks pass; per-crate clippy passed. Final gates remain pending.

Decision for #779: replace direct decoder output-directory access with a single-file HLS stream through a server-owned bounded sink. FFmpeg emits a contiguous byte-range playlist to private scratch space. A real inert 8-second fixture produced eight contiguous 1880-byte ranges on stdout. The playlist will publish last; unfinished work restarts. This makes the aggregate disk cap enforceable at the server sink.

#816: the parser exists only on pending voicefiles-620. Port its shared calternal-media crate (without unrelated callers or UI) and bound its finite moov/trak/mdia grammar plus total work. Two small malformed-header tests fail on the pending implementation and pass with the bounds. The pending voicefiles callers can reuse this crate when the orchestrator integrates that branch.

Finding and first slice: #802 cancellation regression failed on old code with Home usage `left: 15`, `right: 8` (seven abandoned bytes). Commit `4d07d7c7a` adds held-parent cleanup and journal ownership transfer, plus bounded startup cleanup. Cancellation, live recovery and copy/chunk I/O checks pass; per-crate clippy passed. Final gates remain pending. Decision for #779: replace direct decoder output-directory access with a single-file HLS stream through a server-owned bounded sink. FFmpeg emits a contiguous byte-range playlist to private scratch space. A real inert 8-second fixture produced eight contiguous 1880-byte ranges on stdout. The playlist will publish last; unfinished work restarts. This makes the aggregate disk cap enforceable at the server sink. #816: the parser exists only on pending voicefiles-620. Port its shared calternal-media crate (without unrelated callers or UI) and bound its finite moov/trak/mdia grammar plus total work. Two small malformed-header tests fail on the pending implementation and pass with the bounds. The pending voicefiles callers can reuse this crate when the orchestrator integrates that branch.

kayg referenced this issue

2026-10-02 14:24:25 +00:00

Performance: reduce HLS stream publication latency after bounded output #852

kayg commented

2026-10-02 14:37:38 +00:00

Author

Owner

#779 progress: #802 cleanup is committed at 4d07d7c7a; #816's bounded shared parser is committed at 262b0e0bb. No UI has changed.

The HLS sink now keeps progressive playback: fixed length frames carry atomic private-scratch playlists, and the server publishes only ranges already written and synced. A rendition holds its aggregate reservation through the completed Index entry. Large disk snapshots use the same reservation ledger as uploads. Unfinished work is removed on cancellation and restarted from its immutable snapshot.

#781's child starts through prlimit with a 512 MiB hard address-space cap, two-second CPU cap, and cleared environment. The hidden command also applies limits before allocation or runtime startup. Extracted text goes directly into the existing bounded text buffer.

Corrected native process-tree measurements showed a latency regression; follow-up #852 records the before/after values. Final progressive-protocol measurements and crate gates remain pending. The build host's load average is above 120; compilation continues without waiting for a quiet host.

#779 progress: #802 cleanup is committed at 4d07d7c7a; #816's bounded shared parser is committed at 262b0e0bb. No UI has changed. The HLS sink now keeps progressive playback: fixed length frames carry atomic private-scratch playlists, and the server publishes only ranges already written and synced. A rendition holds its aggregate reservation through the completed Index entry. Large disk snapshots use the same reservation ledger as uploads. Unfinished work is removed on cancellation and restarted from its immutable snapshot. #781's child starts through prlimit with a 512 MiB hard address-space cap, two-second CPU cap, and cleared environment. The hidden command also applies limits before allocation or runtime startup. Extracted text goes directly into the existing bounded text buffer. Corrected native process-tree measurements showed a latency regression; follow-up #852 records the before/after values. Final progressive-protocol measurements and crate gates remain pending. The build host's load average is above 120; compilation continues without waiting for a quiet host.

kayg referenced this issue

2026-10-02 14:57:10 +00:00

Performance: reduce HLS stream publication latency after bounded output #852

kayg commented

2026-10-02 14:59:23 +00:00

Author

Owner

Build status: the PDF hard-limit regression passed (test pdf::tests::pdf_child_inherits_hard_memory_limit_and_clean_environment ... ok). Its first per-crate build took 53m 52s on the shared host. The corrected file-drop regression and filesystem reservation tests are queued on the same job target lock. Web check is running. No push or deploy.

One additional #802 finding: the first cleanup slice used a fixed 16,384-entry startup pass, which could leave abandoned bytes in a large Home/cache. The follow-up streams directory cursors across the whole tree, ignores concurrent missing entries, and includes a test with more than that entry count. Cancellation still cleans through held-parent guards, while published journal entries retain recovery ownership.

Build status: the PDF hard-limit regression passed (`test pdf::tests::pdf_child_inherits_hard_memory_limit_and_clean_environment ... ok`). Its first per-crate build took `53m 52s` on the shared host. The corrected file-drop regression and filesystem reservation tests are queued on the same job target lock. Web check is running. No push or deploy. One additional #802 finding: the first cleanup slice used a fixed 16,384-entry startup pass, which could leave abandoned bytes in a large Home/cache. The follow-up streams directory cursors across the whole tree, ignores concurrent missing entries, and includes a test with more than that entry count. Cancellation still cleans through held-parent guards, while published journal entries retain recovery ownership.

kayg commented

2026-10-02 15:07:30 +00:00

Author

Owner

Committed native-test runtime repair: Debian BLAS/LAPACK sonames linked through host /etc/alternatives were unavailable in the isolated decoder. The test runtime now copies only the resolved libraries into its existing private library directory. The real decoder suite passes on the locked perf VM. No native boundary was relaxed.

kayg referenced this issue from a commit

2026-10-02 15:14:45 +00:00

test(media): stage resolved Debian libraries for isolated decoder checks (#779)

kayg commented

2026-10-02 15:29:16 +00:00

Author

Owner

Web check passed on the merged branch:

svelte-check found 0 errors and 0 warnings

Web test completed under load 136+ on the shared host. Every failed assertion report is a test timeout; one additional fork worker did not start before its timeout. No web source or existing test expectation changed.

 Test Files  21 failed | 132 passed (153)
      Tests  32 failed | 1039 passed (1071)
   Duration  942.91s (transform 51%, environment 17%, import 16%, tests 11%, setup 4%)
error: script "test" exited with code 1

Rust gates continue per crate. The existing native boundary checks passed on the locked perf VM. No push or deploy.

Web check passed on the merged branch: ``` svelte-check found 0 errors and 0 warnings ``` Web test completed under load 136+ on the shared host. Every failed assertion report is a test timeout; one additional fork worker did not start before its timeout. No web source or existing test expectation changed. ``` Test Files 21 failed | 132 passed (153) Tests 32 failed | 1039 passed (1071) Duration 942.91s (transform 51%, environment 17%, import 16%, tests 11%, setup 4%) error: script "test" exited with code 1 ``` Rust gates continue per crate. The existing native boundary checks passed on the locked perf VM. No push or deploy.

kayg referenced this issue from a commit

2026-10-02 16:31:05 +00:00

fix(fs): reserve bounded HLS and snapshot output on its actual device (#779)

kayg commented

2026-10-02 16:32:48 +00:00

Author

Owner

Committed temporary-name ownership and streamed recovery (#802), finite media grammar (#816), HLS/snapshot reservations, bounded progressive HLS output, and pre-runtime PDF limits. Current head: 7f3b8aa7d. Independent read-only review found no critical or important defects. Final Cargo build/gates are still running; host load has delayed compilation. cargo fmt --check and FS clippy passed. Web check: svelte-check found 0 errors and 0 warnings. Web tests reported 32 timeouts and one worker startup timeout, with 1039 tests passing; no assertion mismatch. Native sandbox probes passed on the locked perf VM. Before/after measurements and the small-clip latency regression are recorded in #852. The corrected Sidecar fixture now refreshes the folder Index after raw fixture writes, as a real owner mutation does; its final test is pending. No push or deploy.

Committed temporary-name ownership and streamed recovery (#802), finite media grammar (#816), HLS/snapshot reservations, bounded progressive HLS output, and pre-runtime PDF limits. Current head: 7f3b8aa7d. Independent read-only review found no critical or important defects. Final Cargo build/gates are still running; host load has delayed compilation. `cargo fmt --check` and FS clippy passed. Web check: `svelte-check found 0 errors and 0 warnings`. Web tests reported 32 timeouts and one worker startup timeout, with 1039 tests passing; no assertion mismatch. Native sandbox probes passed on the locked perf VM. Before/after measurements and the small-clip latency regression are recorded in #852. The corrected Sidecar fixture now refreshes the folder Index after raw fixture writes, as a real owner mutation does; its final test is pending. No push or deploy.

kayg referenced this issue from a commit

2026-10-02 16:45:49 +00:00

fix(video): broker capped HLS pipes and progressive byte-range playlists (#779)

kayg referenced this issue from a commit

2026-10-02 16:45:49 +00:00

chore(api): describe HLS byte-range responses in generated clients (#779)

kayg referenced this issue from a commit

2026-10-02 16:45:49 +00:00

docs(media): explain the server-owned HLS output invariant (#779)

kayg commented

2026-10-02 16:59:23 +00:00

Author

Owner

Head e3b5ebe9f0. All changes are committed. Final FS gates passed. Output:

cargo fmt --check: 0
cargo clippy -p calternal-fs --all-targets -- -D warnings: 0
cargo test -p calternal-fs: 0
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 34m 03s
test result: ok. 69 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 141.37s
test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 245.13s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Focused PDF tests, using the compiled test binary:


running 6 tests
test pdf::tests::child_timeout_is_reaped_and_next_extraction_can_finish ... ok
test pdf::tests::extracts_text_from_a_pdf_page ... ok
test pdf::tests::invalid_and_oversized_pdf_inputs_are_rejected ... ok
test pdf::tests::output_cap_keeps_unicode_boundaries_valid ... ok
test pdf::tests::parser_text_sink_stops_at_unicode_boundary ... ok
test pdf::tests::pdf_child_inherits_hard_memory_limit_and_clean_environment ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 38 filtered out; finished in 3.05s

The server build has reached its entry point. Other per-crate gates wait for its target lock. A bounded ordinary media smoke mode was added to the existing Tus probe; it has not run yet. No push or deploy.

Head e3b5ebe9f00a743715dfe99bf542d15a02a62239. All changes are committed. Final FS gates passed. Output: ``` cargo fmt --check: 0 cargo clippy -p calternal-fs --all-targets -- -D warnings: 0 cargo test -p calternal-fs: 0 Finished `dev` profile [unoptimized + debuginfo] target(s) in 34m 03s test result: ok. 69 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 141.37s test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 245.13s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Focused PDF tests, using the compiled test binary: ``` running 6 tests test pdf::tests::child_timeout_is_reaped_and_next_extraction_can_finish ... ok test pdf::tests::extracts_text_from_a_pdf_page ... ok test pdf::tests::invalid_and_oversized_pdf_inputs_are_rejected ... ok test pdf::tests::output_cap_keeps_unicode_boundaries_valid ... ok test pdf::tests::parser_text_sink_stops_at_unicode_boundary ... ok test pdf::tests::pdf_child_inherits_hard_memory_limit_and_clean_environment ... ok test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 38 filtered out; finished in 3.05s ``` The server build has reached its entry point. Other per-crate gates wait for its target lock. A bounded ordinary media smoke mode was added to the existing Tus probe; it has not run yet. No push or deploy.

kayg referenced this issue from a commit

2026-10-02 17:01:37 +00:00

fix(api): match the implicit empty HLS range error schema (#779)

kayg referenced this issue from a commit

2026-10-02 17:01:37 +00:00

test(media): reuse Tus probes for a bounded ordinary server smoke round (#779)

kayg commented

2026-10-02 17:21:12 +00:00

Author

Owner

Implementation is committed. Required validation is still incomplete; do not treat this report as a passing merge gate.

Branch: job/mediafix. Head: bd6e97e09b

Built the media/file fixes for #779, #781, #801, #802 and #816. No push or deploy. Merged origin/dev (440e19dce2) and origin/job/merge-round-7a (efe8323fe8) once before final gates. Independent read-only review found no critical or important defects.

Changes:

HLS Jobs reserve an aggregate output budget on the actual device. Completed cache and active budgets share admission. Capacity failures stay retryable. Disk snapshots reserve their full cap before copying.
Native HLS output uses bounded server-owned pipes. The decoder has no writable cache directory. Valid byte-range playlists publish as streamed bytes arrive. Completed work is retained only after its Index row is saved. Cancelled or failed work is removed. Sources and HLS segments share the existing range writer.
PDF children have hard address-space, CPU and descriptor limits before parser/runtime startup. Text writes directly into a capped UTF-8 sink.
File drop installs recheck Sidecar permission under the mutation lock after folder and collision-name resolution. Directory errors deny the edit. The check reuses Sidecar rules and directory pages.
Temporary-name guards keep cleanup ownership until durable journal handoff. Recovery streams all abandoned names rather than stopping at 16,384 entries.
The pending media parser uses a finite ISO container grammar and a shared node-work budget. Its port is included because this base did not yet contain that crate. No parser caller or UI is added.
Native probes, an ordinary Tus smoke mode, generated range contracts, and an HLS bench profile cover the changed paths.

Performance:
Held /root/perf.lock on root@10.69.69.63 and used bench/hdd-emu.sh run-limited. Load inside the lock: 0.13 0.52 0.51. Before → after, five eight-second samples: p50 165.25 → 287.39 ms; p95 166.17 → 394.12 ms; CPU 112 → 124 ms; mean peak process-tree RSS 78,558,822.4 → 80,855,040 bytes. Two-job, 300-second burst: p50 877.11 → 760.85 ms; p95 880.25 → 760.88 ms; CPU 600 → 610 ms; RSS 81,514,496 → 84,824,064 bytes. The baseline JSON has no HLS metric, so this compares the old round-7a native protocol with the new protocol on the same VM and fixture. #852 tracks the small-clip latency regression. This profile excludes HTTP/Index and progressive Rust fsync publication. It does not cover largest output volumes or a measured 1 GiB snapshot.

Regression evidence:

Temporary cancellation: old code left 15 bytes where 8 were expected; the fixed regression passed.
Recovery beyond the old entry cutoff: an independent driver left 11 temporary names against the old library and zero against the fixed library. The fixed Cargo test also passed.
Parser: old/new direct test runs show the grammar regression failing then passing; all six fixed parser tests passed.
PDF: an inert helper reported unlimited old-child memory and 524288 KiB with the fix. All six focused PDF tests passed, including normal extraction, Unicode cap and timeout recovery.
The final FS run passed 69 unit tests and 42 integration tests, including HLS budgets, progressive replacement credits, Index-failure cleanup and disk snapshot admission.
All eleven focused Files media tests passed, including framed metadata rejection and process-group cleanup.
The corrected Sidecar fixture refreshes the folder Index after raw fixture writes. The initial old-code attempts exposed fixture URI/token errors; they are not valid old-code permission evidence.

Decisions:

HLS uses one byte stream plus byte-range playlists, with an 8 MiB metadata cap/headroom. Incomplete output restarts from its immutable snapshot. Its reservation is min(cache limit, 4 GiB).
PDF child address space is capped at 512 MiB; decoded streams share this aggregate hard cap. There is no separate lopdf per-stream cap.
ISO parsing accepts only the finite moov/trak/mdia/hdlr hierarchy and shares a 4096-node budget.
Sidecar scans use existing 128-entry directory pages. Recovery has bounded memory but no entry-count cutoff.

UX gaps closed: the HLS transport now supports progressive playlists and byte-range reads needed by playback. No UI source changed.
UX gaps left: browser playback and Apple-client validation were not completed. No UI screenshots were required for this backend-only change.

Files:

Cargo.lock
bench/media_hls.py
contracts/openapi.json
crates/calternal-fs/src/blob.rs
crates/calternal-fs/src/file_ops.rs
crates/calternal-fs/src/hls.rs
crates/calternal-fs/src/journal.rs
crates/calternal-fs/src/lib.rs
crates/calternal-fs/src/quota.rs
crates/calternal-fs/src/root.rs
crates/calternal-fs/src/thumbnails.rs
crates/calternal-fs/src/trash.rs
crates/calternal-fs/src/uploads.rs
crates/calternal-fs/src/write.rs
crates/calternal-media/Cargo.toml
crates/calternal-media/src/lib.rs
crates/calternal-search/src/lib.rs
crates/calternal-search/src/pdf.rs
crates/calternal-server/src/main.rs
crates/plugins/files/src/lib.rs
crates/plugins/files/src/media.rs
crates/plugins/files/src/public.rs
crates/plugins/files/src/uploads.rs
crates/plugins/video/src/routes.rs
crates/plugins/video/src/transcode.rs
deploy/media-sandbox
packages/api-client/src/generated.ts
tests/adversarial/media-input-policy.c
tests/adversarial/media_sandbox_inputs.py
tests/adversarial/media_uploads.py
tests/adversarial/prepare-media-runtime.sh

Gate output, verbatim:

Final formatting: exit 0, no output.

FS clippy:

    Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/mediafix/crates/calternal-fs)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 34m 03s

FS tests:

test result: ok. 69 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 141.37s
test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 245.13s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Parser Cargo clippy and tests:

    Checking calternal-media v0.0.1 (/home/kayg/Developer/calternal-wt/mediafix/crates/calternal-media)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.23s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Focused PDF test binary:

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 38 filtered out; finished in 3.05s

Focused Files media test binary:

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 148 filtered out; finished in 9.32s

Web check:

svelte-check found 0 errors and 0 warnings

Web tests:

 Test Files  21 failed | 132 passed (153)
      Tests  32 failed | 1039 passed (1071)
     Errors  1 error
   Start at  17:11:00
   Duration  942.91s (transform 51%, environment 17%, import 16%, tests 11%, setup 4%)

error: script "test" exited with code 1

API client tests:

(pass) stops a byte response at the tool budget without waiting for EOF [0.90ms]

 17 pass
 1 fail
 49 expect() calls
Ran 18 tests across 1 file. [20.43s]
error: script "test" exited with code 1

Source-extracted HLS validator test bodies (not a Video Cargo gate):

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

Native probe:

PASS: bounded sealed image/video probes, thumbnails and HLS transcode

Known gaps:

The corrected Sidecar final-install regression still needs its final run and a valid old-code negative run.
Remaining Cargo clippy/test gates for calternal-search, calternal-plugin-files, calternal-plugin-video and calternal-server are incomplete. The server build and target lock delayed them.
The real local-server ordinary media round and authoritative server OpenAPI regeneration have not run. Contract responses were updated from the route annotations; TypeScript was regenerated with the package command.
The web run had 32 test timeouts and one worker-start timeout. The API client run had one timeout. No assertion mismatch was reported in either run.
Largest-output and 1 GiB snapshot measurements, progressive Rust publication performance, and the latency follow-up #852 remain.
The PDF decoded-stream boundary is an aggregate process cap. A separate per-stream limit is not provided by this parser API.

The server build and queued gates were stopped to keep the four-hour limit. Cleanup completed: cargo clean removed 10828 files (7.6 GiB). Both apps/web/build and apps/web/.svelte-kit/output were removed. The working tree is clean.

Evidence is retained under artifacts/mediafix in the worktree. No screenshots or review artifacts were committed.

Implementation is committed. Required validation is still incomplete; do not treat this report as a passing merge gate. Branch: job/mediafix. Head: bd6e97e09b56c8df6ae770f5bb440aaf2d9d8f36 Built the media/file fixes for #779, #781, #801, #802 and #816. No push or deploy. Merged origin/dev (440e19dce23040ac8ebaae88f0469b6535b1afcb) and origin/job/merge-round-7a (efe8323fe827e106d1e1b669308602f64e7fb7d1) once before final gates. Independent read-only review found no critical or important defects. Changes: - HLS Jobs reserve an aggregate output budget on the actual device. Completed cache and active budgets share admission. Capacity failures stay retryable. Disk snapshots reserve their full cap before copying. - Native HLS output uses bounded server-owned pipes. The decoder has no writable cache directory. Valid byte-range playlists publish as streamed bytes arrive. Completed work is retained only after its Index row is saved. Cancelled or failed work is removed. Sources and HLS segments share the existing range writer. - PDF children have hard address-space, CPU and descriptor limits before parser/runtime startup. Text writes directly into a capped UTF-8 sink. - File drop installs recheck Sidecar permission under the mutation lock after folder and collision-name resolution. Directory errors deny the edit. The check reuses Sidecar rules and directory pages. - Temporary-name guards keep cleanup ownership until durable journal handoff. Recovery streams all abandoned names rather than stopping at 16,384 entries. - The pending media parser uses a finite ISO container grammar and a shared node-work budget. Its port is included because this base did not yet contain that crate. No parser caller or UI is added. - Native probes, an ordinary Tus smoke mode, generated range contracts, and an HLS bench profile cover the changed paths. Performance: Held /root/perf.lock on root@10.69.69.63 and used bench/hdd-emu.sh run-limited. Load inside the lock: 0.13 0.52 0.51. Before → after, five eight-second samples: p50 165.25 → 287.39 ms; p95 166.17 → 394.12 ms; CPU 112 → 124 ms; mean peak process-tree RSS 78,558,822.4 → 80,855,040 bytes. Two-job, 300-second burst: p50 877.11 → 760.85 ms; p95 880.25 → 760.88 ms; CPU 600 → 610 ms; RSS 81,514,496 → 84,824,064 bytes. The baseline JSON has no HLS metric, so this compares the old round-7a native protocol with the new protocol on the same VM and fixture. #852 tracks the small-clip latency regression. This profile excludes HTTP/Index and progressive Rust fsync publication. It does not cover largest output volumes or a measured 1 GiB snapshot. Regression evidence: - Temporary cancellation: old code left 15 bytes where 8 were expected; the fixed regression passed. - Recovery beyond the old entry cutoff: an independent driver left 11 temporary names against the old library and zero against the fixed library. The fixed Cargo test also passed. - Parser: old/new direct test runs show the grammar regression failing then passing; all six fixed parser tests passed. - PDF: an inert helper reported unlimited old-child memory and 524288 KiB with the fix. All six focused PDF tests passed, including normal extraction, Unicode cap and timeout recovery. - The final FS run passed 69 unit tests and 42 integration tests, including HLS budgets, progressive replacement credits, Index-failure cleanup and disk snapshot admission. - All eleven focused Files media tests passed, including framed metadata rejection and process-group cleanup. - The corrected Sidecar fixture refreshes the folder Index after raw fixture writes. The initial old-code attempts exposed fixture URI/token errors; they are not valid old-code permission evidence. Decisions: - HLS uses one byte stream plus byte-range playlists, with an 8 MiB metadata cap/headroom. Incomplete output restarts from its immutable snapshot. Its reservation is min(cache limit, 4 GiB). - PDF child address space is capped at 512 MiB; decoded streams share this aggregate hard cap. There is no separate lopdf per-stream cap. - ISO parsing accepts only the finite moov/trak/mdia/hdlr hierarchy and shares a 4096-node budget. - Sidecar scans use existing 128-entry directory pages. Recovery has bounded memory but no entry-count cutoff. UX gaps closed: the HLS transport now supports progressive playlists and byte-range reads needed by playback. No UI source changed. UX gaps left: browser playback and Apple-client validation were not completed. No UI screenshots were required for this backend-only change. Files: ``` Cargo.lock bench/media_hls.py contracts/openapi.json crates/calternal-fs/src/blob.rs crates/calternal-fs/src/file_ops.rs crates/calternal-fs/src/hls.rs crates/calternal-fs/src/journal.rs crates/calternal-fs/src/lib.rs crates/calternal-fs/src/quota.rs crates/calternal-fs/src/root.rs crates/calternal-fs/src/thumbnails.rs crates/calternal-fs/src/trash.rs crates/calternal-fs/src/uploads.rs crates/calternal-fs/src/write.rs crates/calternal-media/Cargo.toml crates/calternal-media/src/lib.rs crates/calternal-search/src/lib.rs crates/calternal-search/src/pdf.rs crates/calternal-server/src/main.rs crates/plugins/files/src/lib.rs crates/plugins/files/src/media.rs crates/plugins/files/src/public.rs crates/plugins/files/src/uploads.rs crates/plugins/video/src/routes.rs crates/plugins/video/src/transcode.rs deploy/media-sandbox packages/api-client/src/generated.ts tests/adversarial/media-input-policy.c tests/adversarial/media_sandbox_inputs.py tests/adversarial/media_uploads.py tests/adversarial/prepare-media-runtime.sh ``` Gate output, verbatim: Final formatting: exit 0, no output. FS clippy: ``` Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/mediafix/crates/calternal-fs) Finished `dev` profile [unoptimized + debuginfo] target(s) in 34m 03s ``` FS tests: ``` test result: ok. 69 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 141.37s test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 245.13s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Parser Cargo clippy and tests: ``` Checking calternal-media v0.0.1 (/home/kayg/Developer/calternal-wt/mediafix/crates/calternal-media) Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.23s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Focused PDF test binary: ``` test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 38 filtered out; finished in 3.05s ``` Focused Files media test binary: ``` test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 148 filtered out; finished in 9.32s ``` Web check: ``` svelte-check found 0 errors and 0 warnings ``` Web tests: ``` Test Files 21 failed | 132 passed (153) Tests 32 failed | 1039 passed (1071) Errors 1 error Start at 17:11:00 Duration 942.91s (transform 51%, environment 17%, import 16%, tests 11%, setup 4%) error: script "test" exited with code 1 ``` API client tests: ``` (pass) stops a byte response at the tool budget without waiting for EOF [0.90ms] 17 pass 1 fail 49 expect() calls Ran 18 tests across 1 file. [20.43s] error: script "test" exited with code 1 ``` Source-extracted HLS validator test bodies (not a Video Cargo gate): ``` test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s ``` Native probe: ``` PASS: bounded sealed image/video probes, thumbnails and HLS transcode ``` Known gaps: - The corrected Sidecar final-install regression still needs its final run and a valid old-code negative run. - Remaining Cargo clippy/test gates for calternal-search, calternal-plugin-files, calternal-plugin-video and calternal-server are incomplete. The server build and target lock delayed them. - The real local-server ordinary media round and authoritative server OpenAPI regeneration have not run. Contract responses were updated from the route annotations; TypeScript was regenerated with the package command. - The web run had 32 test timeouts and one worker-start timeout. The API client run had one timeout. No assertion mismatch was reported in either run. - Largest-output and 1 GiB snapshot measurements, progressive Rust publication performance, and the latency follow-up #852 remain. - The PDF decoded-stream boundary is an aggregate process cap. A separate per-stream limit is not provided by this parser API. The server build and queued gates were stopped to keep the four-hour limit. Cleanup completed: `cargo clean` removed 10828 files (7.6 GiB). Both apps/web/build and apps/web/.svelte-kit/output were removed. The working tree is clean. Evidence is retained under artifacts/mediafix in the worktree. No screenshots or review artifacts were committed.

kayg referenced this issue

2026-10-02 17:21:21 +00:00

Filesystem: remove pre-journal temporary files on cancellation and I/O failure #802

kayg referenced this issue

2026-10-02 17:21:22 +00:00

BLOCKER: media track parser has no nesting bound #816

kayg referenced this issue

2026-10-02 17:21:23 +00:00

BLOCKER: cap PDF search extractor memory before parsing User content #781

kayg referenced this issue

2026-10-02 17:21:24 +00:00

BLOCKER: recheck file-drop Sidecar authorization at final installation #801