SECURITY: least-privilege runtime (drop SYS_ADMIN, /dev/fuse, /dev/net/tun from the web server container)

kayg commented

2026-09-27 16:19:56 +00:00

Owner

Owner request (2026-09-27): "simplify this and make it more secure. The container should get as few privileges as it can."

Status: needs an owner grill on the questions below before a job builds it (changes a DESIGN decision). Security-heavy: per owner rule, Luna Max first, Sol medium only if Luna fails.

Today (checked 2026-09-27)

The production container (deploy/cloud/calternal-cloud.container, deploy/nested-podman/run-outer.sh) runs with:

--cap-add=SYS_ADMIN, --device=/dev/fuse, --device=/dev/net/tun, --security-opt unmask=/proc/*, --cgroupns=private, a writable nested-Podman storage volume.

Why: DESIGN §20 (round 10) decided that per-user agent containers (Claude Code / Codex / calternal CLI) run in a rootless Podman nested inside the calternal container, "so the blast radius of the engine is the calternal container, not the host". Nested rootless Podman needs SYS_ADMIN (mounts in the inner user namespace), fuse-overlayfs (/dev/fuse), slirp/pasta networking (/dev/net/tun) and an unmasked /proc. DESIGN §9 also plans calternal mount (FUSE) inside agent containers. The media sandbox (bubblewrap + media-sandbox-dropcaps) needs user namespaces.

Consequence: the internet-facing web server process shares a container with a powerful container engine. A bug in the HTTP server lands in a container with SYS_ADMIN (namespaced, but a far larger kernel attack surface: mount, FUSE, tun, unmasked /proc). The owner's rule: minimal privileges.

Goal

The internet-facing server container runs with no added privileges: --cap-drop=ALL, no-new-privileges, default seccomp, no devices, read-only root filesystem (tmpfs for /tmp), only its two data volumes writable, AppArmor/SELinux default profile. Anything that needs more privilege moves out of it, behind a narrow interface.

Options to grill

Agent host-broker (DESIGN's named fallback). A separate, tiny calternal-agentd service runs as its own Linux user (e.g. calternal-agents) with its own rootless Podman. The web server talks to it over a Unix socket with a narrow, typed API (start/stop/exec-turn for user X with a scoped token), never a Podman socket. Agent containers stay one per user; the broker is the only place with container-engine privileges, and it has no network listener and no access to the data directories.
Keep nested Podman but split containers. Web server in a no-privilege container; the nested-Podman "agent runtime" in a second privileged container in the same pod, reachable only via a Unix socket. Smaller change, but the privileged container still sits next to the server.
No FUSE in agents. Replace calternal mount (FUSE) with the CLI/API only, or a server-side sync of a scoped working copy into the agent container, so agent containers need no /dev/fuse either.
Media sandbox. Check whether bubblewrap works in a no-caps container (needs unprivileged user namespaces allowed by seccomp); if not, move media processing into the broker too, or into a separate unprivileged helper container.
Stronger isolation for agents later (gVisor/Kata/Firecracker per user agent), relevant for managed accounts where untrusted users run agents.

Questions for the owner

Broker as a separate Linux user + service (option 1, recommended) vs a second privileged container in the same pod (option 2)?
Drop FUSE for agents (CLI/API only) or keep it inside the agent containers (then only the broker's side needs /dev/fuse)?
For managed accounts (untrusted users running agents on the shared instance): gVisor/Kata for agent containers from day one, or later?
Is losing "blast radius = the calternal container" acceptable if the broker has no network listener and no data access (the broker's blast radius is its own user)?

Acceptance (after the grill)

Web server container: podman inspect shows CapAdd empty, CapDrop ALL, NoNewPrivileges, ReadonlyRootfs, no devices; the app works end to end (upload, thumbnails/HEIF/video, search, notes, collab, Ask/agents).
Adversarial round extended with an "assume RCE in the web server" section: from inside the server container, prove it cannot mount, create namespaces with elevated caps, reach the broker except through its API, read other users' data outside the API, or reach the host Podman.
DESIGN §20 updated (ASD-STE100); deploy files and README updated.

Owner request (2026-09-27): "simplify this and make it more secure. The container should get as few privileges as it can." **Status:** needs an owner grill on the questions below before a job builds it (changes a DESIGN decision). Security-heavy: per owner rule, Luna Max first, Sol medium only if Luna fails. ## Today (checked 2026-09-27) The production container (`deploy/cloud/calternal-cloud.container`, `deploy/nested-podman/run-outer.sh`) runs with: - `--cap-add=SYS_ADMIN`, `--device=/dev/fuse`, `--device=/dev/net/tun`, `--security-opt unmask=/proc/*`, `--cgroupns=private`, a writable nested-Podman storage volume. Why: DESIGN §20 (round 10) decided that **per-user agent containers** (Claude Code / Codex / calternal CLI) run in a **rootless Podman nested inside the calternal container**, "so the blast radius of the engine is the calternal container, not the host". Nested rootless Podman needs SYS_ADMIN (mounts in the inner user namespace), fuse-overlayfs (`/dev/fuse`), slirp/pasta networking (`/dev/net/tun`) and an unmasked `/proc`. DESIGN §9 also plans `calternal mount` (FUSE) inside agent containers. The media sandbox (bubblewrap + `media-sandbox-dropcaps`) needs user namespaces. Consequence: the internet-facing web server process shares a container with a powerful container engine. A bug in the HTTP server lands in a container with SYS_ADMIN (namespaced, but a far larger kernel attack surface: mount, FUSE, tun, unmasked /proc). The owner's rule: minimal privileges. ## Goal The internet-facing server container runs with **no added privileges**: `--cap-drop=ALL`, `no-new-privileges`, default seccomp, no devices, read-only root filesystem (tmpfs for `/tmp`), only its two data volumes writable, AppArmor/SELinux default profile. Anything that needs more privilege moves out of it, behind a narrow interface. ## Options to grill 1. **Agent host-broker (DESIGN's named fallback).** A separate, tiny `calternal-agentd` service runs as its own Linux user (e.g. `calternal-agents`) with its own rootless Podman. The web server talks to it over a Unix socket with a narrow, typed API (start/stop/exec-turn for user X with a scoped token), never a Podman socket. Agent containers stay one per user; the broker is the only place with container-engine privileges, and it has no network listener and no access to the data directories. 2. **Keep nested Podman but split containers.** Web server in a no-privilege container; the nested-Podman "agent runtime" in a second privileged container in the same pod, reachable only via a Unix socket. Smaller change, but the privileged container still sits next to the server. 3. **No FUSE in agents.** Replace `calternal mount` (FUSE) with the CLI/API only, or a server-side sync of a scoped working copy into the agent container, so agent containers need no `/dev/fuse` either. 4. **Media sandbox.** Check whether bubblewrap works in a no-caps container (needs unprivileged user namespaces allowed by seccomp); if not, move media processing into the broker too, or into a separate unprivileged helper container. 5. **Stronger isolation for agents later** (gVisor/Kata/Firecracker per user agent), relevant for managed accounts where untrusted users run agents. ## Questions for the owner 1. Broker as a separate Linux user + service (option 1, recommended) vs a second privileged container in the same pod (option 2)? 2. Drop FUSE for agents (CLI/API only) or keep it inside the agent containers (then only the broker's side needs `/dev/fuse`)? 3. For managed accounts (untrusted users running agents on the shared instance): gVisor/Kata for agent containers from day one, or later? 4. Is losing "blast radius = the calternal container" acceptable if the broker has no network listener and no data access (the broker's blast radius is its own user)? ## Acceptance (after the grill) - Web server container: `podman inspect` shows CapAdd empty, CapDrop ALL, NoNewPrivileges, ReadonlyRootfs, no devices; the app works end to end (upload, thumbnails/HEIF/video, search, notes, collab, Ask/agents). - Adversarial round extended with an "assume RCE in the web server" section: from inside the server container, prove it cannot mount, create namespaces with elevated caps, reach the broker except through its API, read other users' data outside the API, or reach the host Podman. - DESIGN §20 updated (ASD-STE100); deploy files and README updated.

kayg commented

2026-09-29 04:37:35 +00:00

Author

Owner

Started job/least-priv from dev 00015e652a.

Findings: Agent turns currently connect directly to a nested Podman socket in the web container. The production image's Bubblewrap media wrapper succeeds with cap-drop=ALL and no-new-privileges only when /proc/* is unmasked; with the default masked /proc it fails at bwrap: Can't mount proc on /proc: Operation not permitted. I will split the Agent engine from the web container and retain the /proc exception for the media wrapper, then test the built image locally.

Started job/least-priv from dev 00015e652a153485d9e7adda1c5ee61bc17300db. Findings: Agent turns currently connect directly to a nested Podman socket in the web container. The production image's Bubblewrap media wrapper succeeds with cap-drop=ALL and no-new-privileges only when /proc/* is unmasked; with the default masked /proc it fails at `bwrap: Can't mount proc on /proc: Operation not permitted`. I will split the Agent engine from the web container and retain the /proc exception for the media wrapper, then test the built image locally.

kayg commented

2026-09-29 04:40:53 +00:00

Author

Owner

Implementation finding: the Agent plugin uses the nested Podman API for create/start/exec/stop. I moved that API into a second rootless container and added a length-bounded Unix protocol that exposes only Run and Cancel. The web Quadlet now drops all capabilities, sets no-new-privileges and a read-only root, and has no devices or Podman storage mount. It mounts the broker socket directory read-only. The Agent service has no /data or /userdata mount. The media sandbox still needs Unmask=/proc/*; cap-drop=ALL plus the default masked /proc fails Bubblewrap's private proc mount on local Podman 5.4.2.

Implementation finding: the Agent plugin uses the nested Podman API for create/start/exec/stop. I moved that API into a second rootless container and added a length-bounded Unix protocol that exposes only Run and Cancel. The web Quadlet now drops all capabilities, sets no-new-privileges and a read-only root, and has no devices or Podman storage mount. It mounts the broker socket directory read-only. The Agent service has no /data or /userdata mount. The media sandbox still needs `Unmask=/proc/*`; cap-drop=ALL plus the default masked /proc fails Bubblewrap's private proc mount on local Podman 5.4.2.

kayg commented

2026-09-29 04:45:53 +00:00

Author

Owner

Security boundary limit: the server remains the single writer for every User's Home, so its /userdata mount necessarily lets a compromised server process read User data directly. This job can remove kernel and container-engine privileges from that process, but it cannot prove that a server RCE cannot read other Users' files outside the API without changing the single-writer architecture. The Agent service itself has no /data or /userdata mount.

kayg commented

2026-09-29 04:50:16 +00:00

Author

Owner

Rollout finding: the old web container and new Agent service reuse /srv/calternal/podman. Starting the Agent service before stopping the old web container would put two Podman engines on one graphroot. The deploy script now stops the old web service first, then starts the Agent service, then restarts the web service. No deployment was performed.

Rollout finding: the old web container and new Agent service reuse `/srv/calternal/podman`. Starting the Agent service before stopping the old web container would put two Podman engines on one graphroot. The deploy script now stops the old web service first, then starts the Agent service, then restarts the web service. No deployment was performed.

kayg commented

2026-09-29 04:59:00 +00:00

Author

Owner

First atomic commit: 47e7d92b test(runtime): verify cap drop still permits media sandbox. tests/adversarial/least_priv.sh localhost/calternal-cloud:main output:

vips-8.16.1
least-priv boundary and media sandbox: PASS

The Rust feature compile and changed production image smoke are still in progress.

First atomic commit: 47e7d92b `test(runtime): verify cap drop still permits media sandbox`. `tests/adversarial/least_priv.sh localhost/calternal-cloud:main` output: ``` vips-8.16.1 least-priv boundary and media sandbox: PASS ``` The Rust feature compile and changed production image smoke are still in progress.

kayg commented

2026-09-29 05:03:57 +00:00

Author

Owner

Decision for the open design choice: use a second rootless Agent service container with a fixed Run/Cancel Unix protocol. This keeps nested Podman and the existing Agent container semantics, including future FUSE use, without adding a new host user or host-level Podman broker in this job. The web container gets cap-drop=ALL, no-new-privileges, a read-only root and no devices. Media processing stays in the web container with Bubblewrap's private user namespace and the required /proc/* unmask. The stronger separate-Linux-user broker and stronger Agent isolation remain future design choices for the owner; this job does not implement them.

Decision for the open design choice: use a second rootless Agent service container with a fixed Run/Cancel Unix protocol. This keeps nested Podman and the existing Agent container semantics, including future FUSE use, without adding a new host user or host-level Podman broker in this job. The web container gets cap-drop=ALL, no-new-privileges, a read-only root and no devices. Media processing stays in the web container with Bubblewrap's private user namespace and the required `/proc/*` unmask. The stronger separate-Linux-user broker and stronger Agent isolation remain future design choices for the owner; this job does not implement them.

kayg commented

2026-09-29 05:14:27 +00:00

Author

Owner

Local Podman Quadlet generator accepted both new service files. Generated web command has --cap-drop all, --security-opt=no-new-privileges, --read-only, no --device, and a read-only Agent socket mount. Generated Agent service command alone has /dev/fuse, /dev/net/tun, and --cap-add sys_admin; it has no data directory mounts or published port. Shell syntax, Python syntax, and git diff --check pass. Rust compilation is still running.

Local Podman Quadlet generator accepted both new service files. Generated web command has `--cap-drop all`, `--security-opt=no-new-privileges`, `--read-only`, no `--device`, and a read-only Agent socket mount. Generated Agent service command alone has `/dev/fuse`, `/dev/net/tun`, and `--cap-add sys_admin`; it has no data directory mounts or published port. Shell syntax, Python syntax, and `git diff --check` pass. Rust compilation is still running.

kayg commented

2026-09-29 05:53:34 +00:00

Author

Owner

Committed 6d44250b: separate Agent service with a bounded typed Run/Cancel socket, and a web Quadlet with all capabilities dropped, no devices, no new privileges, and a read-only root. cargo test -p calternal-plugin-ai passed 13/13 tests; cargo fmt --check, shell and Python syntax, and Quadlet generation passed. Merged dev once as 8d180ae2 with no conflicts. Building the merged production image for the rootless Podman smoke now.

Committed 6d44250b: separate Agent service with a bounded typed Run/Cancel socket, and a web Quadlet with all capabilities dropped, no devices, no new privileges, and a read-only root. `cargo test -p calternal-plugin-ai` passed 13/13 tests; `cargo fmt --check`, shell and Python syntax, and Quadlet generation passed. Merged `dev` once as 8d180ae2 with no conflicts. Building the merged production image for the rootless Podman smoke now.

kayg commented

2026-09-29 10:09:10 +00:00

Author

Owner

Rootless production-image finding: the web container had 70 tasks in its cgroup while deploy/media-sandbox limited the outer Bubblewrap launcher to RLIMIT_NPROC=64. The video ffprobe launch then failed with bwrap: Creating new namespace failed: Resource temporarily unavailable; failed HLS markers were present for both renditions. I repeated the same Bubblewrap/drop-caps invocation in the container: outer limit 64 returned 1, outer limit 256 returned 0, with the decoder's inner RLIMIT_NPROC=64 unchanged. I am raising only the bounded outer allowance and will rerun the smoke on a rebuilt production image.

Rootless production-image finding: the web container had 70 tasks in its cgroup while `deploy/media-sandbox` limited the outer Bubblewrap launcher to `RLIMIT_NPROC=64`. The video `ffprobe` launch then failed with `bwrap: Creating new namespace failed: Resource temporarily unavailable`; failed HLS markers were present for both renditions. I repeated the same Bubblewrap/drop-caps invocation in the container: outer limit 64 returned 1, outer limit 256 returned 0, with the decoder's inner `RLIMIT_NPROC=64` unchanged. I am raising only the bounded outer allowance and will rerun the smoke on a rebuilt production image.

kayg commented

2026-09-29 10:25:56 +00:00

Author

Owner

Second rootless production-image finding: after the outer process-limit fix, the exact MP4 header probe still exited 137. I fed the uploaded fixture to the production media wrapper through a regular inherited file descriptor. The wrapper with Bubblewrap --die-with-parent returned 137 with no output; the same invocation without that flag returned 320,240. The Rust media caller already places the wrapper in its own process group and kills the whole group on timeout or drop. I am removing this incompatible flag and will rerun the fresh-image smoke.

Second rootless production-image finding: after the outer process-limit fix, the exact MP4 header probe still exited 137. I fed the uploaded fixture to the production media wrapper through a regular inherited file descriptor. The wrapper with Bubblewrap `--die-with-parent` returned 137 with no output; the same invocation without that flag returned `320,240`. The Rust media caller already places the wrapper in its own process group and kills the whole group on timeout or drop. I am removing this incompatible flag and will rerun the fresh-image smoke.

kayg commented

2026-09-29 10:33:13 +00:00

Author

Owner

Status

The least-privilege runtime changes are committed and pushed on job/least-priv. Head: a666e5df26e584c8708b0b0a7256633971d35dc2. dev was merged once at 8d180ae2. No deployment or merge was performed.

Built

Moved the Agent engine into a separate rootless service with a bounded Run/Cancel Unix protocol. The web service drops all capabilities, has no devices, uses no-new-privileges and a read-only root, and sees only a read-only broker socket. The Agent service has no data-directory mount.
Updated the cloud Quadlets and rollout order. Added a production-image smoke harness.
Raised the media wrapper's outer process allowance to 256 while keeping the decoder limit at 64. Removed Bubblewrap --die-with-parent: on local rootless Podman, the exact uploaded MP4 probe exited 137 with that flag and returned 320,240 without it. The Rust caller retains process-group timeout/drop cleanup.

Local rootless Podman smoke

Privilege-boundary inspection and Agent broker hostile-request checks passed. Setup/authentication and 30/30 fixture passkey logins passed. JPG, HEIF and MP4 uploads passed (201/204); image and HEIF thumbnails passed (200); video source range passed (206).

HLS did not pass. Both renditions received .calternal-failed markers (decoder-failed-v1) and the API returned HTTP 202 through the 120-second poll window. The exact failure was:

AssertionError: video HLS did not finish in 120 seconds

The harness stops at HLS, so Search, CalDAV, WebDAV and the adversarial endpoint round were not run. The remaining FFmpeg transcode error is not logged by the current job path and is unresolved.

Checks and cleanup

bash -n deploy/media-sandbox and git diff --check completed with exit 0 and no output. The earlier cargo test -p calternal-plugin-ai run passed 13/13 tests. The full final gates (cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test, bun run check, bun run test) were not run because the job reached its four-hour time limit while the required video smoke still failed.

The test containers and temporary test data were removed. cargo clean output:

Removed 14562 files, 6.8GiB total

Web build output was removed. The smoke harness cleanup itself hit a PermissionError on root-owned files in nested Podman storage; I removed that directory with podman unshare. The harness cleanup should be updated to use that ownership context.

Decisions for owner review

Use a second rootless Agent service with the fixed Run/Cancel protocol; this avoids a host-level broker or new host user in this change.
Give Bubblewrap's outer launcher a 256-task allowance because the web service already had more than 64 tasks; retain the 64-task limit inside the decoder namespace.
Omit --die-with-parent in the media wrapper because it killed the inherited-input MP4 probe in the local rootless production image. Rust process-group cleanup remains active. This change still needs review with the unresolved HLS failure.

The stronger separate-Linux-user Agent isolation remains future work. No UI changed, so role-token and screenshot checks did not apply.

## Status The least-privilege runtime changes are committed and pushed on `job/least-priv`. Head: `a666e5df26e584c8708b0b0a7256633971d35dc2`. `dev` was merged once at `8d180ae2`. No deployment or merge was performed. ## Built - Moved the Agent engine into a separate rootless service with a bounded Run/Cancel Unix protocol. The web service drops all capabilities, has no devices, uses no-new-privileges and a read-only root, and sees only a read-only broker socket. The Agent service has no data-directory mount. - Updated the cloud Quadlets and rollout order. Added a production-image smoke harness. - Raised the media wrapper's outer process allowance to 256 while keeping the decoder limit at 64. Removed Bubblewrap `--die-with-parent`: on local rootless Podman, the exact uploaded MP4 probe exited 137 with that flag and returned `320,240` without it. The Rust caller retains process-group timeout/drop cleanup. ## Local rootless Podman smoke Privilege-boundary inspection and Agent broker hostile-request checks passed. Setup/authentication and 30/30 fixture passkey logins passed. JPG, HEIF and MP4 uploads passed (201/204); image and HEIF thumbnails passed (200); video source range passed (206). HLS did not pass. Both renditions received `.calternal-failed` markers (`decoder-failed-v1`) and the API returned HTTP 202 through the 120-second poll window. The exact failure was: ```text AssertionError: video HLS did not finish in 120 seconds ``` The harness stops at HLS, so Search, CalDAV, WebDAV and the adversarial endpoint round were not run. The remaining FFmpeg transcode error is not logged by the current job path and is unresolved. ## Checks and cleanup `bash -n deploy/media-sandbox` and `git diff --check` completed with exit 0 and no output. The earlier `cargo test -p calternal-plugin-ai` run passed 13/13 tests. The full final gates (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`, `bun run check`, `bun run test`) were not run because the job reached its four-hour time limit while the required video smoke still failed. The test containers and temporary test data were removed. `cargo clean` output: ```text Removed 14562 files, 6.8GiB total ``` Web build output was removed. The smoke harness cleanup itself hit a `PermissionError` on root-owned files in nested Podman storage; I removed that directory with `podman unshare`. The harness cleanup should be updated to use that ownership context. ## Decisions for owner review - Use a second rootless Agent service with the fixed Run/Cancel protocol; this avoids a host-level broker or new host user in this change. - Give Bubblewrap's outer launcher a 256-task allowance because the web service already had more than 64 tasks; retain the 64-task limit inside the decoder namespace. - Omit `--die-with-parent` in the media wrapper because it killed the inherited-input MP4 probe in the local rootless production image. Rust process-group cleanup remains active. This change still needs review with the unresolved HLS failure. The stronger separate-Linux-user Agent isolation remains future work. No UI changed, so role-token and screenshot checks did not apply.

kayg commented

2026-09-29 10:43:09 +00:00

Author

Owner

HLS smoke finding: the production image completed upload, JPEG/HEIF thumbnails, and video range reads, but both HLS renditions remained unavailable after 120 seconds. A direct ffmpeg HLS run inside the same no-capabilities, read-only image succeeded with the inherited directory descriptor mounted at /output. The launcher applies RLIMIT_NPROC=256 before Bubblewrap and then lowers it to 64 inside the namespace. The limit counts tasks for the mapped UID, including the server and concurrent media work. I removed only the redundant inner 64-task limit; the inherited 256-task limit and the two-process media semaphore remain. Rebuilding the image and repeating the smoke now.

kayg commented

2026-09-29 10:48:49 +00:00

Author

Owner

Correction to my preceding HLS hypothesis: the smoke still failed with the second RLIMIT_NPROC limit removed. A direct ffmpeg HLS run succeeded in the web container, but that did not prove the task limit caused the production failure. I reverted the allowance change. The worker currently writes a terminal failure marker without recording whether the source signature, FFprobe header, ffmpeg command, or playlist check failed. I added fixed-string stage logging that excludes User content and am rebuilding to identify the actual cause.

kayg commented

2026-09-29 10:50:41 +00:00

Author

Owner

Boundary test added and passed on the production image: a child sleep 600 ran inside the no-capabilities web container; after SIGKILL to container PID 1, the child's host PID disappeared within one second. The existing Rust process-group test covers timeout and task cancellation. Podman's container cgroup covers abrupt server death, when Rust Drop cannot run. The smoke harness also now removes nested Podman graphroot files through podman unshare; its teardown left no least-priv-smoke directory after the previous failed HLS run. Commits: 40137f4a (tests and cleanup), a027ad7b (shell lint fix).

Boundary test added and passed on the production image: a child `sleep 600` ran inside the no-capabilities web container; after SIGKILL to container PID 1, the child's host PID disappeared within one second. The existing Rust process-group test covers timeout and task cancellation. Podman's container cgroup covers abrupt server death, when Rust Drop cannot run. The smoke harness also now removes nested Podman graphroot files through `podman unshare`; its teardown left no least-priv-smoke directory after the previous failed HLS run. Commits: 40137f4a (tests and cleanup), a027ad7b (shell lint fix).

kayg commented

2026-09-29 10:56:40 +00:00

Author

Owner

The reordered production smoke reached Search and CalDAV: Search returned HTTP 200 with the uploaded HEIF found, and CalDAV PROPFIND returned HTTP 207. Its WebDAV request returned HTTP 403 because the fixture used a default CalDAV-only app password, as confirmed by the auth default scope and existing WebDAV scope tests. This is a smoke-harness credential error, not a server authorization defect. I updated the smoke to create a separate webdav/full app password in its private work directory and to use that only for File WebDAV probes. HLS remains under diagnosis.

The reordered production smoke reached Search and CalDAV: Search returned HTTP 200 with the uploaded HEIF found, and CalDAV PROPFIND returned HTTP 207. Its WebDAV request returned HTTP 403 because the fixture used a default CalDAV-only app password, as confirmed by the auth default scope and existing WebDAV scope tests. This is a smoke-harness credential error, not a server authorization defect. I updated the smoke to create a separate `webdav/full` app password in its private work directory and to use that only for File WebDAV probes. HLS remains under diagnosis.

kayg commented

2026-09-29 11:02:06 +00:00

Author

Owner

The corrected production smoke now reports Search HTTP 200 with the uploaded HEIF found, CalDAV PROPFIND HTTP 207, and File WebDAV PROPFIND HTTP 207 using a separately scoped password. I also ran tests/adversarial/webdav.py against that live container: WebDAV scripted probes passed. HLS still has both terminal markers and the master route stays HTTP 202. Commit 6abff45b records the scoped-password harness fix; the diagnostic server build is still running.

The corrected production smoke now reports Search HTTP 200 with the uploaded HEIF found, CalDAV PROPFIND HTTP 207, and File WebDAV PROPFIND HTTP 207 using a separately scoped password. I also ran `tests/adversarial/webdav.py` against that live container: `WebDAV scripted probes passed`. HLS still has both terminal markers and the master route stays HTTP 202. Commit 6abff45b records the scoped-password harness fix; the diagnostic server build is still running.

kayg commented

2026-09-29 11:05:59 +00:00

Author

Owner

HLS narrowing: the same uploaded MP4 has Index MIME video/mp4; the video thumbnail job produced WebP output, which uses the same signature and FFprobe header checks and the same media sandbox. HLS alone writes both failure markers. The thumbnail ffmpeg invocation caps decoder and filter threads at two; the HLS invocation left both at FFmpeg's host-CPU auto setting while the sandbox caps tasks at 64. I added two-thread decoder, filter, and encoder limits to HLS instead of raising the sandbox limit. This is the current root-cause hypothesis pending the instrumented production image run; stage-specific, fixed-string HLS warnings are in the same build to confirm it.

HLS narrowing: the same uploaded MP4 has Index MIME `video/mp4`; the video thumbnail job produced WebP output, which uses the same signature and FFprobe header checks and the same media sandbox. HLS alone writes both failure markers. The thumbnail ffmpeg invocation caps decoder and filter threads at two; the HLS invocation left both at FFmpeg's host-CPU auto setting while the sandbox caps tasks at 64. I added two-thread decoder, filter, and encoder limits to HLS instead of raising the sandbox limit. This is the current root-cause hypothesis pending the instrumented production image run; stage-specific, fixed-string HLS warnings are in the same build to confirm it.

kayg commented

2026-09-29 11:10:37 +00:00

Author

Owner

Confirmed HLS root cause from a temporary wrapper inside the real production smoke container: three MP4 ffprobe calls exited 0, both video thumbnail ffmpeg calls exited 0, but the HLS encoder-probe calls arrived with -hide_banner as the first command token and exited 125. The sandbox dispatch accepts only encoder-probe:ffmpeg, so no encoder was tried. available_encoders() omitted .arg("ffmpeg"). I added that argument in the video plugin. I removed the speculative thread change; no new media sandbox allowance is needed. The smoke HLS check itself is the end-to-end regression probe. The release rebuild is still running, then I will repeat it against the corrected image.

Confirmed HLS root cause from a temporary wrapper inside the real production smoke container: three MP4 `ffprobe` calls exited 0, both video thumbnail ffmpeg calls exited 0, but the HLS `encoder-probe` calls arrived with `-hide_banner` as the first command token and exited 125. The sandbox dispatch accepts only `encoder-probe:ffmpeg`, so no encoder was tried. `available_encoders()` omitted `.arg("ffmpeg")`. I added that argument in the video plugin. I removed the speculative thread change; no new media sandbox allowance is needed. The smoke HLS check itself is the end-to-end regression probe. The release rebuild is still running, then I will repeat it against the corrected image.

kayg commented

2026-09-29 11:14:07 +00:00

Author

Owner

I strengthened the parent-death regression from a generic child to the actual media process: the production container launched calternal-media-sandbox probe -- ffmpeg through Bubblewrap, podman top showed the ffmpeg host PID, then SIGKILL to container PID 1 removed both Bubblewrap processes and ffmpeg. The shell probe prints server parent death removed its decoder: PASS. Commit ed43b0cc contains that test. Rust's process-group guard still covers timeout/cancellation; the container cgroup covers abrupt server death.

I strengthened the parent-death regression from a generic child to the actual media process: the production container launched `calternal-media-sandbox probe -- ffmpeg` through Bubblewrap, `podman top` showed the ffmpeg host PID, then SIGKILL to container PID 1 removed both Bubblewrap processes and ffmpeg. The shell probe prints `server parent death removed its decoder: PASS`. Commit ed43b0cc contains that test. Rust's process-group guard still covers timeout/cancellation; the container cgroup covers abrupt server death.

kayg commented

2026-09-29 11:19:08 +00:00

Author

Owner

The full endpoint part of the production smoke completed on the current image: Search HTTP 200 found the uploaded HEIF; CalDAV and File WebDAV PROPFIND each returned HTTP 207; attack.py printed DAV Apple property, write-capability, MKCALENDAR and adversarial probes completed; webdav.py printed WebDAV scripted probes passed. The run still exited 1 only because the old server binary omits ffmpeg in encoder-probe and HLS remained HTTP 202. The new Rust binary is compiling for the corrected-image pass.

The full endpoint part of the production smoke completed on the current image: Search HTTP 200 found the uploaded HEIF; CalDAV and File WebDAV PROPFIND each returned HTTP 207; `attack.py` printed `DAV Apple property, write-capability, MKCALENDAR and adversarial probes completed`; `webdav.py` printed `WebDAV scripted probes passed`. The run still exited 1 only because the old server binary omits `ffmpeg` in encoder-probe and HLS remained HTTP 202. The new Rust binary is compiling for the corrected-image pass.

kayg commented

2026-09-29 11:26:53 +00:00

Author

Owner

The next preview run exposed the second call-site defect. With the encoder probe fixed, both encoder-probe calls exited 0 and found FFmpeg encoders. Each subsequent video sandbox call arrived with -hide_banner as its first command token and exited 125; this includes all hardware fallback attempts and software H.264. run_encoder() also omitted the required .arg("ffmpeg"). I added that token. No decoder resource or mount allowance caused either failure, and no sandbox flag changed. I am testing a preview that supplies both missing tokens, then will rebuild the actual server image.

The next preview run exposed the second call-site defect. With the encoder probe fixed, both `encoder-probe` calls exited 0 and found FFmpeg encoders. Each subsequent `video` sandbox call arrived with `-hide_banner` as its first command token and exited 125; this includes all hardware fallback attempts and software H.264. `run_encoder()` also omitted the required `.arg("ffmpeg")`. I added that token. No decoder resource or mount allowance caused either failure, and no sandbox flag changed. I am testing a preview that supplies both missing tokens, then will rebuild the actual server image.

kayg commented

2026-09-29 11:30:30 +00:00

Author

Owner

Both HLS call-site fixes are now confirmed by a local preview image that supplies exactly the two missing ffmpeg tokens to the old server binary while retaining the 64-task media sandbox. The full production-image smoke printed: video HLS: HTTP 200, production image API smoke: PASS, DAV Apple property, write-capability, MKCALENDAR and adversarial probes completed, WebDAV scripted probes passed, and least-priv production image smoke: PASS; exit code 0. Commit 7876b3e5 contains the matching Rust fix. The actual rebuilt server image is still compiling; I will repeat the smoke on that binary before final gates.

Both HLS call-site fixes are now confirmed by a local preview image that supplies exactly the two missing `ffmpeg` tokens to the old server binary while retaining the 64-task media sandbox. The full production-image smoke printed: `video HLS: HTTP 200`, `production image API smoke: PASS`, `DAV Apple property, write-capability, MKCALENDAR and adversarial probes completed`, `WebDAV scripted probes passed`, and `least-priv production image smoke: PASS`; exit code 0. Commit 7876b3e5 contains the matching Rust fix. The actual rebuilt server image is still compiling; I will repeat the smoke on that binary before final gates.

kayg commented

2026-09-29 11:36:50 +00:00

Author

Owner

The strengthened preview smoke now verifies each rendition, not only the master playlist. With the two missing ffmpeg tokens supplied, it printed video HLS: HTTP 200, video HLS 720p: HTTP 200, and video HLS 1080p: HTTP 200, then completed the DAV adversarial and WebDAV scripts; final exit code 0. Commit 66f78ac6 makes the two-rendition check permanent. The corrected Rust release binary is still building, so the preview is evidence for the exact call contract, with a final real-binary run pending.

The strengthened preview smoke now verifies each rendition, not only the master playlist. With the two missing `ffmpeg` tokens supplied, it printed `video HLS: HTTP 200`, `video HLS 720p: HTTP 200`, and `video HLS 1080p: HTTP 200`, then completed the DAV adversarial and WebDAV scripts; final exit code 0. Commit 66f78ac6 makes the two-rendition check permanent. The corrected Rust release binary is still building, so the preview is evidence for the exact call contract, with a final real-binary run pending.

kayg commented

2026-09-29 12:54:35 +00:00

Author

Owner

The actual rebuilt release image localhost/calternal-cloud:least-priv-final passed the full local production smoke with exit code 0. Output included video HLS: HTTP 200, video HLS 720p: HTTP 200, video HLS 1080p: HTTP 200, search: HTTP 200, uploaded HEIF found, CalDAV: HTTP 207, WebDAV: HTTP 207, DAV Apple property, write-capability, MKCALENDAR and adversarial probes completed, WebDAV scripted probes passed, and least-priv production image smoke: PASS. The separate boundary probe exited 0 and printed server parent death removed its decoder: PASS and least-priv boundary and media sandbox: PASS. Teardown left no least-priv smoke directory or container. I am merging dev once now, then running final crate and shell gates.

The actual rebuilt release image `localhost/calternal-cloud:least-priv-final` passed the full local production smoke with exit code 0. Output included `video HLS: HTTP 200`, `video HLS 720p: HTTP 200`, `video HLS 1080p: HTTP 200`, `search: HTTP 200, uploaded HEIF found`, `CalDAV: HTTP 207`, `WebDAV: HTTP 207`, `DAV Apple property, write-capability, MKCALENDAR and adversarial probes completed`, `WebDAV scripted probes passed`, and `least-priv production image smoke: PASS`. The separate boundary probe exited 0 and printed `server parent death removed its decoder: PASS` and `least-priv boundary and media sandbox: PASS`. Teardown left no least-priv smoke directory or container. I am merging `dev` once now, then running final crate and shell gates.

kayg commented

2026-09-29 13:15:00 +00:00

Author

Owner

Post-merge audit finding: git diff dev..HEAD still showed stale branch deletions in DAV area calendars, Notes, calendar vectors/tests, and the DAV adversarial probes after the clean textual merge. Those changes were outside #242 and would roll back newer dev behavior. Commit 6fc1bda2 restores those files exactly from dev; the remaining diff is limited to the Agent runtime, Video HLS, deploy, DESIGN, and least-privilege probes. cargo fmt --check passed again. Final per-crate gates are running on this corrected merged tree.

Post-merge audit finding: `git diff dev..HEAD` still showed stale branch deletions in DAV area calendars, Notes, calendar vectors/tests, and the DAV adversarial probes after the clean textual merge. Those changes were outside #242 and would roll back newer `dev` behavior. Commit 6fc1bda2 restores those files exactly from `dev`; the remaining diff is limited to the Agent runtime, Video HLS, deploy, DESIGN, and least-privilege probes. `cargo fmt --check` passed again. Final per-crate gates are running on this corrected merged tree.

kayg referenced this issue from a commit

2026-09-29 13:17:28 +00:00

fix(merge): preserve current dev DAV and Notes behavior

kayg commented

2026-09-29 14:31:07 +00:00

Author

Owner

Implemented #242 on job/least-priv; head 6fc1bda2 (merged dev once, then restored unrelated files to the dev versions). No push, deploy, or merge to dev.

Built: a separate Agent service with a typed, bounded Unix-socket protocol; the web container now drops all capabilities, has a read-only root, no Podman socket/devices, and no SYS_ADMIN. The Agent container alone owns nested Podman. Media sandbox allowances are documented. The HLS failure was caused by both FFmpeg callers omitting the required ffmpeg executable token, so the launcher treated -hide_banner as the program and returned 125; both calls now pass it. The smoke harness cleans subordinate-ID-owned graphroot with guarded podman unshare rm -rf, and verifies that killing server PID 1 leaves no Bubblewrap/FFmpeg process.

Files: crates/plugins/ai/src/{lib.rs,runtime.rs,bin/calternal-agentd.rs}, crates/plugins/video/src/transcode.rs, deploy/{Containerfile.runtime,deploy-cloud.sh,media-sandbox,media-sandbox-dropcaps.c,nested-podman/runtime-entrypoint,cloud/README.md,cloud/calternal-agentd.container,cloud/calternal-cloud.container}, docs/DESIGN.md, tests/adversarial/{least_priv.sh,least_priv_api_smoke.py,least_priv_smoke.sh}.

Production-image smoke before the dev merge: Agent broker typed cancellation and hostile/oversized rejection; upload, JPEG/HEIF thumbnails, video source range, HLS master and complete 720p/1080p playlists; Search found the uploaded HEIF; CalDAV and File WebDAV returned 207; DAV-only attack.py and webdav.py passed. Boundary probe passed with zero effective/bounding capabilities and no retained decoder after PID 1 was SIGKILLed. Real image tag was local localhost/calternal-cloud:least-priv-final.

Completed gate output, verbatim:

cargo fmt --check exit=0
bash -n exit=0
shellcheck exit=0
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 68m 01s
cargo clippy -p calternal-plugin-video --all-targets -- -D warnings exit=0

Pending at report time: AI/server clippy and per-crate tests, and a post-merge adversarial run on a server binary from the merged tree. AI clippy was started but stopped during dependency compilation at the four-hour job timebox (exit 130); no AI diagnostic appeared. The first cold clippy build spent over 20 minutes in xmp_toolkit's bundled C++ build on the shared host. These are test coverage gaps, not known product failures. No screenshots or UI changes. cargo clean completed (output: Removed 12599 files, 4.1GiB total); generated web build output was removed.

Decisions for owner review: (1) separate Agent service container and fixed Unix protocol, so the web process never opens nested Podman; (2) outer media-launcher NPROC allowance 256 because the web runtime UID already has more than 64 tasks, while the decoder remains capped at 64 inside Bubblewrap; (3) remove Bubblewrap --die-with-parent, which killed stdin-reading probes on rootless Podman/Bubblewrap 0.12. The Rust process-group guard kills descendants on timeout/cancel, and Podman's container cgroup kills them on abrupt PID 1 death. The parent-death test starts real FFmpeg through Bubblewrap, records both host PIDs, SIGKILLs PID 1, and verifies both processes are gone or zombies.

Implemented #242 on `job/least-priv`; head `6fc1bda2` (merged `dev` once, then restored unrelated files to the `dev` versions). No push, deploy, or merge to dev. Built: a separate Agent service with a typed, bounded Unix-socket protocol; the web container now drops all capabilities, has a read-only root, no Podman socket/devices, and no SYS_ADMIN. The Agent container alone owns nested Podman. Media sandbox allowances are documented. The HLS failure was caused by both FFmpeg callers omitting the required `ffmpeg` executable token, so the launcher treated `-hide_banner` as the program and returned 125; both calls now pass it. The smoke harness cleans subordinate-ID-owned graphroot with guarded `podman unshare rm -rf`, and verifies that killing server PID 1 leaves no Bubblewrap/FFmpeg process. Files: `crates/plugins/ai/src/{lib.rs,runtime.rs,bin/calternal-agentd.rs}`, `crates/plugins/video/src/transcode.rs`, `deploy/{Containerfile.runtime,deploy-cloud.sh,media-sandbox,media-sandbox-dropcaps.c,nested-podman/runtime-entrypoint,cloud/README.md,cloud/calternal-agentd.container,cloud/calternal-cloud.container}`, `docs/DESIGN.md`, `tests/adversarial/{least_priv.sh,least_priv_api_smoke.py,least_priv_smoke.sh}`. Production-image smoke before the dev merge: Agent broker typed cancellation and hostile/oversized rejection; upload, JPEG/HEIF thumbnails, video source range, HLS master and complete 720p/1080p playlists; Search found the uploaded HEIF; CalDAV and File WebDAV returned 207; DAV-only attack.py and webdav.py passed. Boundary probe passed with zero effective/bounding capabilities and no retained decoder after PID 1 was SIGKILLed. Real image tag was local `localhost/calternal-cloud:least-priv-final`. Completed gate output, verbatim: ``` cargo fmt --check exit=0 bash -n exit=0 shellcheck exit=0 Finished `dev` profile [unoptimized + debuginfo] target(s) in 68m 01s cargo clippy -p calternal-plugin-video --all-targets -- -D warnings exit=0 ``` Pending at report time: AI/server clippy and per-crate tests, and a post-merge adversarial run on a server binary from the merged tree. AI clippy was started but stopped during dependency compilation at the four-hour job timebox (exit 130); no AI diagnostic appeared. The first cold clippy build spent over 20 minutes in xmp_toolkit's bundled C++ build on the shared host. These are test coverage gaps, not known product failures. No screenshots or UI changes. `cargo clean` completed (output: `Removed 12599 files, 4.1GiB total`); generated web build output was removed. Decisions for owner review: (1) separate Agent service container and fixed Unix protocol, so the web process never opens nested Podman; (2) outer media-launcher NPROC allowance 256 because the web runtime UID already has more than 64 tasks, while the decoder remains capped at 64 inside Bubblewrap; (3) remove Bubblewrap `--die-with-parent`, which killed stdin-reading probes on rootless Podman/Bubblewrap 0.12. The Rust process-group guard kills descendants on timeout/cancel, and Podman's container cgroup kills them on abrupt PID 1 death. The parent-death test starts real FFmpeg through Bubblewrap, records both host PIDs, SIGKILLs PID 1, and verifies both processes are gone or zombies.

kayg referenced this issue

2026-10-02 13:11:51 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:12:42 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 17:38:05 +00:00

Author

Owner

Static audit evidence for the FUSE decision in DESIGN §§10, 13, 18 and 23:

DESIGN §§10/13/18 say the Agent uses calternal mount over FUSE, and §23 grants the Agent /dev/fuse and SYS_ADMIN. The later runtime document says the Agent gets neither and works through CLI/API: docs/ai-runtime.md:101-102. The CLI walkthrough says mount is absent at docs/cli-walkthrough.md:735, and TopCommand in crates/calternal-cli/src/main.rs:52-154 has no Mount command.

This issue already lists dropping FUSE for Agents as an owner decision. Please reconcile the conflicting DESIGN and runtime text here before assigning the implementation. Test idea after the decision: verify the chosen Agent capability set and the available CLI data path in a real Agent container.

Static audit evidence for the FUSE decision in DESIGN §§10, 13, 18 and 23: DESIGN §§10/13/18 say the Agent uses calternal mount over FUSE, and §23 grants the Agent /dev/fuse and SYS_ADMIN. The later runtime document says the Agent gets neither and works through CLI/API: docs/ai-runtime.md:101-102. The CLI walkthrough says mount is absent at docs/cli-walkthrough.md:735, and TopCommand in crates/calternal-cli/src/main.rs:52-154 has no Mount command. This issue already lists dropping FUSE for Agents as an owner decision. Please reconcile the conflicting DESIGN and runtime text here before assigning the implementation. Test idea after the decision: verify the chosen Agent capability set and the available CLI data path in a real Agent container.

kayg referenced this issue

2026-10-02 17:41:23 +00:00

Audit: DESIGN.md versus the code (decided but missing, built differently, OPEN items built) #864

kayg referenced this issue

2026-10-02 17:44:29 +00:00

Audit: DESIGN.md versus the code (decided but missing, built differently, OPEN items built) #864

kayg referenced this issue

2026-10-02 17:47:45 +00:00

Audit: DESIGN.md versus the code (decided but missing, built differently, OPEN items built) #864

kayg commented

2026-10-02 17:53:55 +00:00

Author

Owner

Owner decision (2026-10-02): no FUSE for the AI Agent. The Agent reads and writes only through the calternal CLI/API with its scoped permissions; no FUSE device, no SYS_ADMIN for the Agent container. DESIGN §§10, 13, 18, 23 must be updated to match docs/ai-runtime.md.

Owner decision (2026-10-02): **no FUSE for the AI Agent.** The Agent reads and writes only through the calternal CLI/API with its scoped permissions; no FUSE device, no SYS_ADMIN for the Agent container. DESIGN §§10, 13, 18, 23 must be updated to match docs/ai-runtime.md.

kayg referenced this issue from a commit

2026-10-03 06:38:00 +00:00

docs(design): record Canvas (§60) and live document history (§61); drop FUSE from §10

kayg referenced this issue

2026-10-03 12:56:40 +00:00

Issue hygiene sweep 2026-10-03 #1008