DEPLOY: separate app-state and user-data roots (two bind mounts via env vars)

kayg commented

2026-09-27 11:52:43 +00:00

Owner

Owner request (2026-09-27): the production container must take two separate bind mounts, set by env vars: one for app config/state (SSD), one for user data (HDD).

Decided

Keep CALTERNAL_SERVER__DATA_DIR (default /data) for instance state: .system/ (config, DB, search index, thumbnails/previews, backups, secrets).
Add CALTERNAL_SERVER__USER_DATA_DIR (default: same as DATA_DIR, so existing installs change nothing). It holds users/, groups/ and .cas/.
.cas/ MUST live on the same filesystem as users//groups/: dedup uses links/reflinks and a cross-device rename or link returns EXDEV. Add a startup check: if users/ and .cas/ are on different devices, fail with a clear error.
Every place that derives homes from data_dir (Root::open in wire.rs, watch_home_changes → users/, sync, share, backups, importer, adversarial/e2e harnesses, deploy files) must use the user data root. .system stays under DATA_DIR. Grep for data_dir and join("users")/"groups"/".cas" everywhere; no path string concatenation from user input (calternal-fs only).
Scheduled backups in .system/backups stay on DATA_DIR (SSD).
Startup: if DATA_DIR and USER_DATA_DIR differ, both must exist and be writable; log both at info level.
Containerfile.runtime: declare both /data and /userdata volume paths; document the env vars. Update deploy/ docs (ASD-STE100).
Regression tests: server boots with the two roots on two different tempdirs; a user's files land under USER_DATA_DIR only, .system under DATA_DIR only; upload + dedup + rename + trash + version work across the split; the EXDEV guard trips when .cas is on another device (simulate if needed, e.g. unit-test the device-compare function).
Extend tests/adversarial/ with a run in split mode.

Out of scope

Automatic small-file tiering (being decided separately).

Owner request (2026-09-27): the production container must take two separate bind mounts, set by env vars: one for app config/state (SSD), one for user data (HDD). ## Decided - Keep `CALTERNAL_SERVER__DATA_DIR` (default `/data`) for instance state: `.system/` (config, DB, search index, thumbnails/previews, backups, secrets). - Add `CALTERNAL_SERVER__USER_DATA_DIR` (default: same as DATA_DIR, so existing installs change nothing). It holds `users/`, `groups/` and `.cas/`. - `.cas/` MUST live on the same filesystem as `users/`/`groups/`: dedup uses links/reflinks and a cross-device rename or link returns EXDEV. Add a startup check: if `users/` and `.cas/` are on different devices, fail with a clear error. - Every place that derives homes from `data_dir` (Root::open in wire.rs, `watch_home_changes` → `users/`, sync, share, backups, importer, adversarial/e2e harnesses, deploy files) must use the user data root. `.system` stays under DATA_DIR. Grep for `data_dir` and `join("users")`/`"groups"`/`".cas"` everywhere; no path string concatenation from user input (calternal-fs only). - Scheduled backups in `.system/backups` stay on DATA_DIR (SSD). - Startup: if DATA_DIR and USER_DATA_DIR differ, both must exist and be writable; log both at info level. - Containerfile.runtime: declare both `/data` and `/userdata` volume paths; document the env vars. Update `deploy/` docs (ASD-STE100). - Regression tests: server boots with the two roots on two different tempdirs; a user's files land under USER_DATA_DIR only, `.system` under DATA_DIR only; upload + dedup + rename + trash + version work across the split; the EXDEV guard trips when `.cas` is on another device (simulate if needed, e.g. unit-test the device-compare function). - Extend `tests/adversarial/` with a run in split mode. ## Out of scope Automatic small-file tiering (being decided separately).

kayg commented

2026-09-27 11:53:48 +00:00

Author

Owner

Starting implementation on branch job/data-split, based on dev at 1701cef5bd. Worktree is clean.

Starting implementation on branch job/data-split, based on dev at 1701cef5bd8936076522eb18039c3c8caed66299. Worktree is clean.

kayg commented

2026-09-27 12:06:46 +00:00

Author

Owner

Finding: crates/calternal-fs/src/user_homes.rs placed deletion staging and Home archives under .system. Those journaled renames would cross devices when DATA_DIR and USER_DATA_DIR use separate mounts and fail with EXDEV. This slice moves them to reserved .user-data-internal/ under the user-data root, keeping the rename atomic and .system on DATA_DIR.

kayg referenced this issue

2026-09-27 12:07:04 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg referenced this issue

2026-09-27 12:42:14 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 14:09:05 +00:00

Author

Owner

Split-mode adversarial findings (round run at ea3e5d5b):

Root placement passed. At the end, the server reported instance-data entries ['.system'] and user-data entries ['.cas', 'groups', 'users']; server alive was True.
editor server restart with a live editor timed out because run.sh stopped polling for restart-ready-3 after 60 seconds, before the editor reached that case. Fixed in f087cceb; focused split-mode verification passed: PASS editor server restart with a live editor seed=25608414 ms=11160 and PASS editor all areas seed=25608414.
recovery key rotation returned 403. The endpoint requires fresh authentication; the probe reaches it after the long authorization matrix without refreshing the owner session.
bookmark capture storm had 16/16 requests time out at the probe's 10-second limit. The server remained alive at the end. The host had concurrent adversarial and build jobs during this run.
Slowloris probes kept a silent connection and a partial-header connection open after 35 seconds. The idle keep-alive check passed.
A 70 KiB Analytics query ended with Connection reset by peer; the server process remained alive.
AI undo for an oversized body returned 502 with local adversarial server is unavailable.
Share thumbnail checks returned 404 for the 1024-size derivative and for 10 paths before the limit. The media probe also reported that its worker had not generated a thumbnail by its 10-second deadline.

The remaining reports were explicitly tagged SLOW. The full script hit an EOF parse error at its final restart phase because I edited run.sh while it was still reading the file; the API rounds had completed before that error. The focused editor restart run after the fix passed. I am running the room-restart probe separately and will include the final gate output in the completion report.

Split-mode adversarial findings (round run at `ea3e5d5b`): - Root placement passed. At the end, the server reported instance-data entries `['.system']` and user-data entries `['.cas', 'groups', 'users']`; server alive was `True`. - `editor server restart with a live editor` timed out because `run.sh` stopped polling for `restart-ready-3` after 60 seconds, before the editor reached that case. Fixed in `f087cceb`; focused split-mode verification passed: `PASS editor server restart with a live editor seed=25608414 ms=11160` and `PASS editor all areas seed=25608414`. - `recovery key rotation` returned 403. The endpoint requires fresh authentication; the probe reaches it after the long authorization matrix without refreshing the owner session. - `bookmark capture storm` had 16/16 requests time out at the probe's 10-second limit. The server remained alive at the end. The host had concurrent adversarial and build jobs during this run. - Slowloris probes kept a silent connection and a partial-header connection open after 35 seconds. The idle keep-alive check passed. - A 70 KiB Analytics query ended with `Connection reset by peer`; the server process remained alive. - AI undo for an oversized body returned 502 with `local adversarial server is unavailable`. - Share thumbnail checks returned 404 for the 1024-size derivative and for 10 paths before the limit. The media probe also reported that its worker had not generated a thumbnail by its 10-second deadline. The remaining reports were explicitly tagged `SLOW`. The full script hit an EOF parse error at its final restart phase because I edited `run.sh` while it was still reading the file; the API rounds had completed before that error. The focused editor restart run after the fix passed. I am running the room-restart probe separately and will include the final gate output in the completion report.

kayg commented

2026-09-27 15:23:47 +00:00

Author

Owner

#230 complete

Branch: job/data-split
Head SHA: b53dc804b8e55523cac7a5dea615cb9f5d8c7173 (pushed to origin)

Built

Added CALTERNAL_SERVER__USER_DATA_DIR; it defaults to CALTERNAL_SERVER__DATA_DIR. Root routes users/, groups/, and .cas/ to user-data, while .system/ and backups stay in instance-data.
Added startup directory and write checks. users/, groups/, and .cas/ must be on the same filesystem.
Routed Home watchers, search indexing, collaboration, CLI commands, and user deletion staging/archive work to the user-data root.
Added split-root server and filesystem regressions, split-mode adversarial setup, and split-aware E2E fixtures.
Added /userdata runtime and O2 mounts plus migration and filesystem guidance.

Files

crates/calternal-fs/src/{error.rs,lib.rs,root.rs,user_homes.rs}, crates/calternal-path/src/lib.rs, crates/calternal-server/src/{main.rs,wire.rs}, crates/calternal-search/src/indexer.rs, crates/calternal-collab/src/session.rs, apps/web/e2e/{calendar-perf.mjs,calendar.mjs,composer.mjs,harness.mjs,photos-perf.mjs}, tests/adversarial/{attack.py,attack2.py,run.sh,run-split.sh}, deploy/Containerfile.runtime, and deploy/o2/{README.md,calternal-app.container,calternal-app.env.example}. Cargo.lock also includes the current dev merge.

Gates

cargo fmt --all --check: exit 0, no output.
cargo clippy --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 14m 59s

cargo test: exit 0; captured output had 72 suite summaries, 1,256 passed, 0 failed, and 12 ignored. Split-root output:

test tests::split_roots_keep_user_files_and_system_state_on_their_configured_roots ... ok
test result: ok. 35 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 47.06s

bun run --cwd apps/web check:

svelte-check found 0 errors and 0 warnings

bun run --cwd apps/web test:

 Test Files  80 passed (80)
      Tests  585 passed (585)
   Duration  84.03s (transform 65%, import 13%, environment 13%, tests 6%, setup 3%)

Adversarial split run

After merging dev, tests/adversarial/run-split.sh completed both attack rounds and the room-restart probe. Round one reported 4 findings, all tagged SLOW; round two reported 41, all tagged SLOW. The server stayed alive. Both disk checks showed instance-data ['.system'] and user-data ['.cas', 'groups', 'users']. The room restart ended with restart probe: 0 findings. The editor restart and offline reconciliation probes also passed. The non-SLOW observations from the earlier pre-merge run did not recur; they are documented in the preceding comment for tracking.

Decisions

Placed reserved .user-data-internal/ on the user-data filesystem for archive and deletion staging. This keeps journaled renames atomic and avoids EXDEV across mounts.
Included groups/ in the same-device check with users/ and .cas/ to keep all user-data roots on one filesystem.
Kept the O2 /userdata mount explicit in its environment example; an unset setting still preserves the single-root default.

Known gap: adversarial operations marked SLOW reflect host load and are recorded above; no non-SLOW finding recurred in the post-merge round.

## #230 complete Branch: `job/data-split` Head SHA: `b53dc804b8e55523cac7a5dea615cb9f5d8c7173` (pushed to origin) ### Built - Added `CALTERNAL_SERVER__USER_DATA_DIR`; it defaults to `CALTERNAL_SERVER__DATA_DIR`. `Root` routes `users/`, `groups/`, and `.cas/` to user-data, while `.system/` and backups stay in instance-data. - Added startup directory and write checks. `users/`, `groups/`, and `.cas/` must be on the same filesystem. - Routed Home watchers, search indexing, collaboration, CLI commands, and user deletion staging/archive work to the user-data root. - Added split-root server and filesystem regressions, split-mode adversarial setup, and split-aware E2E fixtures. - Added `/userdata` runtime and O2 mounts plus migration and filesystem guidance. ### Files `crates/calternal-fs/src/{error.rs,lib.rs,root.rs,user_homes.rs}`, `crates/calternal-path/src/lib.rs`, `crates/calternal-server/src/{main.rs,wire.rs}`, `crates/calternal-search/src/indexer.rs`, `crates/calternal-collab/src/session.rs`, `apps/web/e2e/{calendar-perf.mjs,calendar.mjs,composer.mjs,harness.mjs,photos-perf.mjs}`, `tests/adversarial/{attack.py,attack2.py,run.sh,run-split.sh}`, `deploy/Containerfile.runtime`, and `deploy/o2/{README.md,calternal-app.container,calternal-app.env.example}`. `Cargo.lock` also includes the current `dev` merge. ### Gates - `cargo fmt --all --check`: exit 0, no output. - `cargo clippy --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 14m 59s ``` - `cargo test`: exit 0; captured output had 72 suite summaries, 1,256 passed, 0 failed, and 12 ignored. Split-root output: ```text test tests::split_roots_keep_user_files_and_system_state_on_their_configured_roots ... ok test result: ok. 35 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 47.06s ``` - `bun run --cwd apps/web check`: ```text svelte-check found 0 errors and 0 warnings ``` - `bun run --cwd apps/web test`: ```text Test Files 80 passed (80) Tests 585 passed (585) Duration 84.03s (transform 65%, import 13%, environment 13%, tests 6%, setup 3%) ``` ### Adversarial split run After merging `dev`, `tests/adversarial/run-split.sh` completed both attack rounds and the room-restart probe. Round one reported 4 findings, all tagged `SLOW`; round two reported 41, all tagged `SLOW`. The server stayed alive. Both disk checks showed instance-data `['.system']` and user-data `['.cas', 'groups', 'users']`. The room restart ended with `restart probe: 0 findings`. The editor restart and offline reconciliation probes also passed. The non-`SLOW` observations from the earlier pre-merge run did not recur; they are documented in the preceding comment for tracking. ### Decisions - Placed reserved `.user-data-internal/` on the user-data filesystem for archive and deletion staging. This keeps journaled renames atomic and avoids `EXDEV` across mounts. - Included `groups/` in the same-device check with `users/` and `.cas/` to keep all user-data roots on one filesystem. - Kept the O2 `/userdata` mount explicit in its environment example; an unset setting still preserves the single-root default. Known gap: adversarial operations marked `SLOW` reflect host load and are recorded above; no non-`SLOW` finding recurred in the post-merge round.

kayg commented

2026-09-27 15:26:34 +00:00

Author

Owner

Merged in 895d3479.

Merged in 895d3479.

kayg closed this issue

2026-09-27 15:26:35 +00:00

kayg referenced this issue from a commit

2026-09-27 15:29:08 +00:00

Merge job/data-split (#230)

kayg commented