kayg commented

Owner

Decision (owner, 2026-09-29): one search index per User

"Q-index of course, yes, please please, no cross-user stuff! the less shared stuff we have, the fewer chances of there being leaks!"

Today Search is one shared Tantivy index filtered by User at query time. Replace it with one index per User (and per shared root where sharing needs it, scoped to its members), so isolation comes from structure, not from a filter.

Build

Per-User Tantivy index directories under the User's own data area (never a shared directory); one writer per index; the indexer routes each document to its owner's index. Shared folders: the recipient's index gets the shared items it may see (or a per-share index queried alongside the User's own) — pick the design with no cross-User reads and state it.
The same for every other shared derived store that holds User content: embeddings (calternal-embed text and CLIP stores), tags aggregate, analytics, notifications, photos index projections. Audit each and list which are per-User already; move the rest, or document why a shared table is safe (and add a matrix probe).
Migration: rebuild per-User indexes in the background through the Jobs registry (#315), with Search serving from the old index until each User's new index is ready; then delete the shared index.
Account deletion deletes the User's index directories.
Performance: keep the #258/#338 targets (measure p50/p95 per-User query latency, memory per open index, a cap on open index readers with LRU).
Tests: the #331 cross-User matrix extended to Search, embeddings and tags; a probe that searches as User B for User A's unique token returns nothing even with a forced filter bug (simulate by removing the filter in a test build).

## Decision (owner, 2026-09-29): one search index per User "Q-index of course, yes, please please, no cross-user stuff! the less shared stuff we have, the fewer chances of there being leaks!" Today Search is one shared Tantivy index filtered by User at query time. Replace it with **one index per User** (and per shared root where sharing needs it, scoped to its members), so isolation comes from structure, not from a filter. ## Build - Per-User Tantivy index directories under the User's own data area (never a shared directory); one writer per index; the indexer routes each document to its owner's index. Shared folders: the recipient's index gets the shared items it may see (or a per-share index queried alongside the User's own) — pick the design with no cross-User reads and state it. - The same for every other shared derived store that holds User content: embeddings (calternal-embed text and CLIP stores), tags aggregate, analytics, notifications, photos index projections. Audit each and list which are per-User already; move the rest, or document why a shared table is safe (and add a matrix probe). - Migration: rebuild per-User indexes in the background through the Jobs registry (#315), with Search serving from the old index until each User's new index is ready; then delete the shared index. - Account deletion deletes the User's index directories. - Performance: keep the #258/#338 targets (measure p50/p95 per-User query latency, memory per open index, a cap on open index readers with LRU). - Tests: the #331 cross-User matrix extended to Search, embeddings and tags; a probe that searches as User B for User A's unique token returns nothing even with a forced filter bug (simulate by removing the filter in a test build).

kayg commented

2026-09-29 05:02:40 +00:00

Author

Owner

Starting #400 on branch job/per-user-index from base 00015e652a. I am tracing Search, derived stores, Jobs, account deletion, and the cross-User matrix before implementation.

Starting #400 on branch job/per-user-index from base 00015e652a153485d9e7adda1c5ee61bc17300db. I am tracing Search, derived stores, Jobs, account deletion, and the cross-User matrix before implementation.

kayg commented

2026-09-29 05:03:13 +00:00

Author

Owner

Finding: Search has one SearchIndex writer and query reader (calternal-search/src/indexer.rs); its active/staging directories are fixed under .system/index. Text and CLIP embeddings use shared SQLite files with owner predicates. Tags already keys rows by owner_id. A complete change must preserve staged rebuild publication (#345) and Share results while moving private reads to User-specific stores. The existing User deletion Job stages/removes the Home, so an index placed under that Home follows its lifecycle.

Finding: Search has one `SearchIndex` writer and query reader (`calternal-search/src/indexer.rs`); its active/staging directories are fixed under `.system/index`. Text and CLIP embeddings use shared SQLite files with owner predicates. Tags already keys rows by `owner_id`. A complete change must preserve staged rebuild publication (#345) and Share results while moving private reads to User-specific stores. The existing User deletion Job stages/removes the Home, so an index placed under that Home follows its lifecycle.

kayg commented

2026-09-29 05:14:59 +00:00

Author

Owner

Filesystem slice committed as 0e5ad206. calternal-fs now opens, clears, and atomically exchanges a derived Search generation under .system/index/users/<validated User ID>/; account purge removes that User's derived tree using the existing restartable removal journal. The 39 calternal-fs unit tests passed, including disjoint User directories and deletion that preserves another User's index. I chose the private per-User directory below .system/index/users because Derived data cannot live in a Home (CONTEXT.md), while account purge explicitly removes it. Search indexing and query routing are still in progress.

Filesystem slice committed as 0e5ad206. `calternal-fs` now opens, clears, and atomically exchanges a derived Search generation under `.system/index/users/<validated User ID>/`; account purge removes that User's derived tree using the existing restartable removal journal. The 39 `calternal-fs` unit tests passed, including disjoint User directories and deletion that preserves another User's index. I chose the private per-User directory below `.system/index/users` because Derived data cannot live in a Home (CONTEXT.md), while account purge explicitly removes it. Search indexing and query routing are still in progress.

kayg commented

2026-09-29 05:51:08 +00:00

Author

Owner

Finding: the account deletion job has Archive, Transfer, and Purge actions. Only Purge called the staged Home purge helper, so a private Search directory would remain after Archive or Transfer. The job now removes the source User's derived Search directory after any completed Home action and before it finalizes account state. The removal uses the existing restartable filesystem journal and is idempotent.

Audit: Tags, Analytics, Notifications, and Photos projections have User or viewer keys in their shared Index tables. Text embeddings and CLIP vectors still use Instance-wide SQLite files with owner partitions and path checks. The new two-User matrix probes keyword/hybrid Search, Tags, and Photos Search; a CLIP unit test inserts equal vectors for A and B and checks that B receives only B's candidate. docs/audits/per-user-index-400.md records each boundary and the remaining physical-isolation gaps. Search compilation is still running on the shared host; no gate result is claimed yet.

Finding: the account deletion job has Archive, Transfer, and Purge actions. Only Purge called the staged Home purge helper, so a private Search directory would remain after Archive or Transfer. The job now removes the source User's derived Search directory after any completed Home action and before it finalizes account state. The removal uses the existing restartable filesystem journal and is idempotent. Audit: Tags, Analytics, Notifications, and Photos projections have User or viewer keys in their shared Index tables. Text embeddings and CLIP vectors still use Instance-wide SQLite files with owner partitions and path checks. The new two-User matrix probes keyword/hybrid Search, Tags, and Photos Search; a CLIP unit test inserts equal vectors for A and B and checks that B receives only B's candidate. `docs/audits/per-user-index-400.md` records each boundary and the remaining physical-isolation gaps. Search compilation is still running on the shared host; no gate result is claimed yet.

kayg commented

2026-09-29 06:05:47 +00:00

Author

Owner

Committed 9c998201: staged per-User Tantivy generations, a bounded 8-reader cache, private query routing after readiness, and dual writes during migration. Focused tests passed: a raw unfiltered query against Bob's index could not see Alice's document; a rebuild/query/update/delete test kept both Users separate. Server Jobs integration is compiling now. Structural gaps remain in the shared legacy fallback and shared derived SQLite stores; the audit names each store.

Committed 9c998201: staged per-User Tantivy generations, a bounded 8-reader cache, private query routing after readiness, and dual writes during migration. Focused tests passed: a raw unfiltered query against Bob's index could not see Alice's document; a rebuild/query/update/delete test kept both Users separate. Server Jobs integration is compiling now. Structural gaps remain in the shared legacy fallback and shared derived SQLite stores; the audit names each store.

kayg commented

2026-09-29 06:08:45 +00:00

Author

Owner

Review finding: a private rebuild currently scans only the owner's Home, while an authenticated Search view can include another User's Shared root. Routing every ready User to the private index would silently omit Shared content. I added a regression case and kept views with Shared roots on the old migration Index. This preserves current behavior but leaves structural isolation incomplete for Shared content; a recipient projection is needed before the shared generation can be retired.

kayg commented

2026-09-29 06:22:23 +00:00

Author

Owner

Committed 1d10fff2: Search keeps Shared-root views on the migration Index until recipient-side content projections exist. The focused two-User rebuild test now covers that path and passed. The matrix profiler now waits through one scheduler sweep (9b33d672). I am validating the CLIP owner partition and Jobs wiring next.

Committed 1d10fff2: Search keeps Shared-root views on the migration Index until recipient-side content projections exist. The focused two-User rebuild test now covers that path and passed. The matrix profiler now waits through one scheduler sweep (9b33d672). I am validating the CLIP owner partition and Jobs wiring next.

kayg commented

2026-09-29 06:38:24 +00:00

Author

Owner

Committed 5e5042f2: CLIP regression test inserts equal vectors for two Users and checks that a Bob-only query returns only Bob's item. cargo test -p calternal-embed --lib photo_candidates_from_another_user_never_reach_the_viewer passed (1 passed). The audit now distinguishes this owner-partition evidence from physical separation: text and CLIP SQLite files and several shared Index tables remain shared.

Committed 5e5042f2: CLIP regression test inserts equal vectors for two Users and checks that a Bob-only query returns only Bob's item. `cargo test -p calternal-embed --lib photo_candidates_from_another_user_never_reach_the_viewer` passed (1 passed). The audit now distinguishes this owner-partition evidence from physical separation: text and CLIP SQLite files and several shared Index tables remain shared.

kayg commented

2026-09-29 07:00:17 +00:00

Author

Owner

Committed 859a06ef: the Jobs registry now schedules one durable private Search rebuild per active User after startup reconciliation and on a 60-second sweep; admin rebuilds enqueue forced User rebuilds. The User deletion job purges the private derived Search directory for archive, transfer, and purge. Server binary tests are still compiling dependencies; I will report their result before final gates.

Committed 859a06ef: the Jobs registry now schedules one durable private Search rebuild per active User after startup reconciliation and on a 60-second sweep; admin rebuilds enqueue forced User rebuilds. The User deletion job purges the private derived Search directory for archive, transfer, and purge. Server binary tests are still compiling dependencies; I will report their result before final gates.

kayg commented

2026-09-29 07:31:32 +00:00

Author

Owner

Merged current dev as 6e3177073 with no conflicts. Final gates: cargo fmt --check passed with empty output; bun run check passed with 0 errors and 0 warnings; bun run test passed 123 files / 787 tests. The shared host's cold Rust dependency build made clippy exceed the time available for the live probe, so I stopped that attempt and started the full cargo test gate. I will report any incomplete gate explicitly in the final report.

Merged current dev as 6e3177073 with no conflicts. Final gates: `cargo fmt --check` passed with empty output; `bun run check` passed with 0 errors and 0 warnings; `bun run test` passed 123 files / 787 tests. The shared host's cold Rust dependency build made clippy exceed the time available for the live probe, so I stopped that attempt and started the full `cargo test` gate. I will report any incomplete gate explicitly in the final report.

kayg commented

2026-09-29 08:02:46 +00:00

Author

Owner

Gate status update: cargo fmt --check, bun run check, and bun run test passed. The shared host compiled third-party Rust crates for clippy and the full workspace cargo test for roughly half an hour each without reaching a result; I stopped both to reserve time for a current-branch server binary and one real two-User adversarial round. These Rust gates are incomplete, not passing. cargo build -p calternal-server is active now.

Gate status update: `cargo fmt --check`, `bun run check`, and `bun run test` passed. The shared host compiled third-party Rust crates for clippy and the full workspace `cargo test` for roughly half an hour each without reaching a result; I stopped both to reserve time for a current-branch server binary and one real two-User adversarial round. These Rust gates are incomplete, not passing. `cargo build -p calternal-server` is active now.

kayg commented

2026-09-29 08:25:04 +00:00

Author

Owner

Per-User Index job report

Head: 6e31770735 (job/per-user-index; merged current dev once, no push or merge into dev). Working tree clean.

Built:

Per-User Tantivy active and staging directories, durable readiness marker, validated directory-handle access, generation exchange, and deletion purge for purge/archive/transfer.
Search actor rebuilds one Home per User; ready Users route to a private Index with an 8-reader LRU cache. Shared migration generation continues to serve not-ready Users and views with Shared roots. Ready private generations receive owner-scoped upserts/deletes.
Durable system.search.user-rebuild Jobs are enqueued after startup reconciliation and on a 60-second sweep; admin rebuild enqueues forced User jobs.
Cross-User Search/Tags/Photos matrix extensions, an optional Search p50/p95 and RSS probe, direct forced-filter regression test, private rebuild/update/delete test, CLIP owner-partition test, and docs/audits/per-user-index-400.md.

Files: crates/calternal-fs/src/{path,root,user_homes}.rs, crates/calternal-search/src/{index,indexer,plugin}.rs, crates/calternal-server/src/wire.rs, crates/calternal-embed/src/clip_store.rs, tests/adversarial/{run.sh,xuser_matrix.py}, docs/audits/per-user-index-400.md.

Validation (verbatim gate excerpts):

$ cargo fmt --check
(no output; exit 0)
$ bun run check  # apps/web
Text sizes use shared role tokens.
svelte-check found 0 errors and 0 warnings
$ bun run test  # apps/web
 Test Files  123 passed (123)
      Tests  787 passed (787)

Focused calternal-fs library test: test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.52s.
Focused Search index test: test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 34 filtered out; finished in 2.07s.
Focused private rebuild test: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 3.90s.
Focused CLIP test: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 24 filtered out; finished in 0.11s.

Incomplete: full cargo clippy --all-targets -- -D warnings and cargo test remained in dependency compilation and were stopped near the four-hour job limit (exit 130). cargo build -p calternal-server was also stopped before a binary existed (exit 130). Therefore the required live two-User adversarial round and Search latency/RSS measurements did not run. cargo clean completed: Removed 11135 files, 6.2GiB total.

Known gaps: the old shared Tantivy generation still exists and handles Shared-root views; Shared content has no recipient-side private projection. Text and CLIP embeddings and Tags/Analytics/Notifications/Photos projections remain in shared SQLite tables with User-keyed rows. This does not yet meet the issue's full physical isolation target. Server Jobs wiring did not finish compiling in a combined server build, so integration remains unverified.

Decisions beyond the design doc: cap private query readers at 8; retry missing User migrations on a 60-second Jobs sweep; retain Shared-root Search on the legacy generation to avoid dropping existing Shared content results. These choices need owner review. The issue remains open.

# Per-User Index job report Head: 6e317707356d4fc2bd95ae959763acf2d2e62aac (`job/per-user-index`; merged current `dev` once, no push or merge into `dev`). Working tree clean. Built: - Per-User Tantivy active and staging directories, durable readiness marker, validated directory-handle access, generation exchange, and deletion purge for purge/archive/transfer. - Search actor rebuilds one Home per User; ready Users route to a private Index with an 8-reader LRU cache. Shared migration generation continues to serve not-ready Users and views with Shared roots. Ready private generations receive owner-scoped upserts/deletes. - Durable `system.search.user-rebuild` Jobs are enqueued after startup reconciliation and on a 60-second sweep; admin rebuild enqueues forced User jobs. - Cross-User Search/Tags/Photos matrix extensions, an optional Search p50/p95 and RSS probe, direct forced-filter regression test, private rebuild/update/delete test, CLIP owner-partition test, and `docs/audits/per-user-index-400.md`. Files: `crates/calternal-fs/src/{path,root,user_homes}.rs`, `crates/calternal-search/src/{index,indexer,plugin}.rs`, `crates/calternal-server/src/wire.rs`, `crates/calternal-embed/src/clip_store.rs`, `tests/adversarial/{run.sh,xuser_matrix.py}`, `docs/audits/per-user-index-400.md`. Validation (verbatim gate excerpts): ``` $ cargo fmt --check (no output; exit 0) $ bun run check # apps/web Text sizes use shared role tokens. svelte-check found 0 errors and 0 warnings $ bun run test # apps/web Test Files 123 passed (123) Tests 787 passed (787) ``` Focused `calternal-fs` library test: `test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.52s`. Focused Search index test: `test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 34 filtered out; finished in 2.07s`. Focused private rebuild test: `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 3.90s`. Focused CLIP test: `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 24 filtered out; finished in 0.11s`. Incomplete: full `cargo clippy --all-targets -- -D warnings` and `cargo test` remained in dependency compilation and were stopped near the four-hour job limit (exit 130). `cargo build -p calternal-server` was also stopped before a binary existed (exit 130). Therefore the required live two-User adversarial round and Search latency/RSS measurements did not run. `cargo clean` completed: `Removed 11135 files, 6.2GiB total`. Known gaps: the old shared Tantivy generation still exists and handles Shared-root views; Shared content has no recipient-side private projection. Text and CLIP embeddings and Tags/Analytics/Notifications/Photos projections remain in shared SQLite tables with User-keyed rows. This does not yet meet the issue's full physical isolation target. Server Jobs wiring did not finish compiling in a combined server build, so integration remains unverified. Decisions beyond the design doc: cap private query readers at 8; retry missing User migrations on a 60-second Jobs sweep; retain Shared-root Search on the legacy generation to avoid dropping existing Shared content results. These choices need owner review. The issue remains open.

kayg commented

2026-09-29 08:31:33 +00:00

Author

Owner

Continuing on job/per-user-index at 6e31770735 (base c465150468). Building the production web app and server first, then replacing the remaining Shared-root migration read and auditing User-keyed projections. I will report measured gates and the live two-User probe before finishing.

Continuing on job/per-user-index at 6e317707356d4fc2bd95ae959763acf2d2e62aac (base c46515046871936ad681c4d00e8b76de69dde3b7). Building the production web app and server first, then replacing the remaining Shared-root migration read and auditing User-keyed projections. I will report measured gates and the live two-User probe before finishing.

kayg commented

2026-09-29 08:40:56 +00:00

Author

Owner

Finding: real Search requests carried only the caller Home in allowed_roots; the existing Shared-root fallback was exercised by direct contexts but did not project validated incoming Shares. The new slice uses Files immutable-item validation for request roots, stages each recipient Share in that recipient private Tantivy generation, and mirrors owner file changes into ready recipient generations. Share create/revoke queues a recipient rebuild; query roots come from live grants, so stale copies are not authorization. Compilation and the live probe are pending.

kayg commented

2026-09-29 08:43:38 +00:00

Author

Owner

Audit decision for the issue’s explicit table-safety option: text vectors, CLIP vectors, Tags, Analytics, Notifications and Photos projections stay in shared SQLite tables. Their queries bind owner/user/viewer keys; existing equal-vector unit probes cover text and CLIP, and the live matrix now checks B/C Search, Tags, Photos, Notifications and Analytics. This preserves their current access path but is weaker than physical per-User separation if a future query omits a predicate. docs/audits/per-user-index-400.md records the boundary and remaining hardening gap.

kayg commented

2026-09-29 08:53:43 +00:00

Author

Owner

Committed ed70e73e: the two-User matrix now checks recipient Share Search, immediate revocation, Notifications and Analytics; the audit records keyed SQLite table boundaries and the physical-isolation limitation. Python syntax check and git diff --check passed. The server build is still compiling dependencies on the shared host; Rust verification of the code slice remains pending.

Committed ed70e73e: the two-User matrix now checks recipient Share Search, immediate revocation, Notifications and Analytics; the audit records keyed SQLite table boundaries and the physical-isolation limitation. Python syntax check and git diff --check passed. The server build is still compiling dependencies on the shared host; Rust verification of the code slice remains pending.

kayg commented

2026-09-29 08:57:14 +00:00

Author

Owner

Finding: bulk Share removal during User deletion bypassed the normal Share route hook. A recipient private Search generation could keep stale derived bytes after the owner Home was deleted, even though live query authority denied the result. I added a Files migration with durable per-recipient invalidation triggers for Share insert/update/delete, seeded existing Shares, and wired the Search Job sweep to rebuild invalidated recipients. The Job clears a revision only if no newer change arrived during its scan; an existing deletion test now checks the durable row.

kayg commented

2026-09-29 09:00:35 +00:00

Author

Owner

The durable invalidation migration also records Files item-ID replacement and deletion, so a stale Share cannot leave recipient-derived bytes indefinitely. A direct in-memory SQLite execution of the migration, followed by Share insert, item replacement and Share delete, returned revision 3 for the recipient. Search additionally joins Files item IDs before staging or mirroring a Share. Rust build and focused tests remain in progress.

kayg commented

2026-09-29 09:16:29 +00:00

Author

Owner

Review finding: a ready User still fell back to the shared migration Index when its private reader could not open, and private write errors cleared readiness. I removed that fallback. A damaged ready generation now fails the request closed and queues a bounded repair; write failures keep readiness and record a durable rebuild request. The focused private Search test now clears Bob’s generation and asserts that no shared-index result is returned. Compilation remains active.

kayg commented

2026-09-29 09:25:07 +00:00

Author

Owner

Committed f8e1da63 (text-vector owner join regression) and e2ac6df4 (recipient private Search projection, live Share authority, durable invalidations, fail-closed repair, matrix and forced-filter coverage). Formatting, diff checks, Python syntax and direct SQLite trigger execution passed. The cold server build is still running through native XMP dependencies; crate tests and live results are pending, and any compiler/test findings will get follow-up fix commits.

Committed f8e1da63 (text-vector owner join regression) and e2ac6df4 (recipient private Search projection, live Share authority, durable invalidations, fail-closed repair, matrix and forced-filter coverage). Formatting, diff checks, Python syntax and direct SQLite trigger execution passed. The cold server build is still running through native XMP dependencies; crate tests and live results are pending, and any compiler/test findings will get follow-up fix commits.

kayg commented

2026-09-29 09:27:38 +00:00

Author

Owner

Merged current dev once into job/per-user-index; ort resolved overlap without conflicts. Final gates will run against this merge.

kayg commented

2026-09-29 09:30:34 +00:00

Author

Owner

Merged dev web gate finding: bun run check reports 20 TypeScript errors in app-sidebar.svelte, PhotoTimeline.svelte and settings routes after the merge (outside #400 owned files). Search Home grant comparison now uses calternal-fs RelPath::user_home; commit 35060cde. Rust and live gates continue.

Merged dev web gate finding: bun run check reports 20 TypeScript errors in app-sidebar.svelte, PhotoTimeline.svelte and settings routes after the merge (outside #400 owned files). Search Home grant comparison now uses calternal-fs RelPath::user_home; commit 35060cde. Rust and live gates continue.

kayg commented

2026-09-29 09:33:43 +00:00

Author

Owner

Correction to the earlier gate observation: the first web check overlapped the dev merge and read a mixed tree. The final post-merge bun run check passed: svelte-check found 0 errors and 0 warnings. The server build also restarted after its pre-merge Notes Core artifact caused a stale import error.

kayg commented

2026-09-29 09:37:46 +00:00

Author

Owner

Audit found readiness-read failure could select the legacy shared Search generation. Commit 74e993d2 now returns the calternal-fs error instead, with a regression assertion using an invalid User ID while the shared generation contains Alice data.

Audit found readiness-read failure could select the legacy shared Search generation. Commit 74e993d2 now returns the calternal-fs error instead, with a regression assertion using an invalid User ID while the shared generation contains Alice data.

kayg commented

2026-09-29 09:45:58 +00:00

Author

Owner

CLIP matrix finding: a mismatched LSH owner/document owner row could consume a recipient candidate slot even though the final result filter removed it. Commit 65495177 binds both owners in the candidate query and tests equal A/B vectors plus the mismatched row; audit updated in ef6d76f4.

CLIP matrix finding: a mismatched LSH owner/document owner row could consume a recipient candidate slot even though the final result filter removed it. Commit 65495177 binds both owners in the candidate query and tests equal A/B vectors plus the mismatched row; audit updated in ef6d76f4.

kayg commented

2026-09-29 10:28:19 +00:00

Author

Owner

Live matrix finding: the D Share assertion used contract.OWNER, which authenticates a different User than user_d.json. Files correctly returned no incoming Share; A Search found the fixture and the Share/File item IDs matched. Commit dad018a5 uses bearer_d for D. Focused matrix validation is rerunning.

Live matrix finding: the D Share assertion used contract.OWNER, which authenticates a different User than user_d.json. Files correctly returned no incoming Share; A Search found the fixture and the Share/File item IDs matched. Commit dad018a5 uses bearer_d for D. Focused matrix validation is rerunning.

kayg commented

2026-09-29 10:39:18 +00:00

Author

Owner

Final focused two-User derived matrix passed on the merged production server (Search keyword/hybrid, Tags, Photos, Share recipient and revocation). Latency/RSS under concurrent Rust compilation: A p50/p95 20.88/43.98 ms, RSS +978944 B; B 58.69/104.03 ms, +21278720 B; D 74.28/135.50 ms, +20672512 B. Full cross-API xuser replay is blocked by new dev OpenAPI auth policy for app_surfaces_get; #400 derived-only mode passes.

kayg commented

2026-09-29 11:04:10 +00:00

Author

Owner

Structural isolation proof passed: cargo test --profile dev -p calternal-search --lib --features unsafe-test-no-root-filter private_rebuild_routes_queries_and_updates_only_to_the_owner returned 1 passed, 0 failed. The test build omits the query root clause, so Bob still cannot read Alice from his private Tantivy directory.

kayg commented

2026-09-29 12:22:05 +00:00

Author

Owner

Forgejo #400 report

Branch: job/per-user-index; head: a9b840199dac7ee5b1065bbc5f3696ab675b761a. Merged dev once into the job branch. Pushed the job branch; did not merge into dev.

Built

Per-User Tantivy Search generations and bounded private reader cache. A ready User never opens the legacy shared Search generation. A pending Shared-root view waits for its recipient generation. An unreadable private generation fails closed and queues repair.
Incoming Shares are copied into the recipient's private Search generation. Immutable item IDs gate each Share. Files Share and item changes record durable recipient invalidations; immediate hooks and the Search Job sweep rebuild recipients. Owner file changes update ready recipient generations.
Text and CLIP embedding candidate joins require matching document and vector/LSH owners. The table-safety matrix is in docs/audits/per-user-index-400.md.
tests/adversarial/xuser_matrix.py probes Search keyword and hybrid, Tags, Photos, Share grant/revocation, and private latency/RSS. Its recipient persona now uses Dora's bearer token.

Main files: crates/calternal-search/src/{indexer,index,query,plugin}.rs, crates/calternal-embed/src/{store,clip_store}.rs, crates/calternal-fs/src/{root,path,user_homes}.rs, crates/calternal-server/src/{main,wire}.rs, crates/plugins/files/src/{lib,shares}.rs, crates/plugins/files/migrations/0016_share_search_invalidations.sql, tests/adversarial/xuser_matrix.py, docs/audits/per-user-index-400.md.

Gates and live evidence

cargo fmt --check: exit 0, no output.
Final cargo build -p calternal-server:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 18s

bun run check:

svelte-check found 0 errors and 0 warnings

bun run test:

 Test Files  125 passed (125)
      Tests  800 passed (800)

cargo test --profile dev -p calternal-embed:

test result: ok. 21 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.45s

cargo test --profile dev -p calternal-fs:

test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.89s
test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 18.69s

cargo test --profile dev -p calternal-tags:

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.32s

cargo test --profile dev -p calternal-server:

test result: ok. 82 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 42.01s

cargo test --profile dev -p calternal-search: unit and operator suites passed. Its 20-test integration suite had one timing failure under host load:

test result: FAILED. 19 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 21.84s

The unchanged watcher test passed on isolated retry:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 19 filtered out; finished in 7.89s

The pending-Share regression added after that suite passed:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 6.47s

With the query root clause removed in the test build (--features unsafe-test-no-root-filter), the private-directory isolation test passed:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 32.90s

Live derived two-User matrix on the production server: exit 0. It covered Search keyword/hybrid, Tags, Photos, Share recipient and revocation. Under concurrent Cargo compilation:

PER_USER_SEARCH A p50_ms=20.88 p95_ms=43.98 rss_delta_bytes=978944
PER_USER_SEARCH B p50_ms=58.69 p95_ms=104.03 rss_delta_bytes=21278720
PER_USER_SEARCH D p50_ms=74.28 p95_ms=135.50 rss_delta_bytes=20672512

Decisions and gaps

Text and CLIP embeddings, Tags, Analytics, Notifications, and Photos projections remain User-keyed shared SQLite tables. The audit names every owner/viewer predicate and the matrix probes the API boundary. This is the explicit table-safety option in the job prompt; physical per-User SQLite files remain a later hardening option.
A Home-only request from a User not yet privately rebuilt may still use the legacy shared migration generation. Shared-root requests wait for the recipient generation. The old generation can be removed after migration completes.
The full cross-API adversarial replay is blocked by a new dev OpenAPI authorization-policy entry (app_surfaces_get) outside #400. The #400 derived-only live matrix passed.
cargo clippy --all-targets -- -D warnings started after the merge but was interrupted at dependency check compilation when the pending-Share fix required a source change. It has no completed gate result. cargo test for the full workspace was not run in the four-hour job window. cargo test --profile dev -p calternal-plugin-files started and was interrupted during dependency compilation at the same time limit, before tests ran. These gates remain for the orchestrator or a follow-up job. No test expectation was changed.
The source-level pending-Share fix has its own passing regression test and a passing final server build. The live matrix ran against the immediately preceding build, so the final build has not had a second full live round.
Production web build output and retained adversarial data were removed; cargo clean was run at the end.

## Forgejo #400 report Branch: `job/per-user-index`; head: `a9b840199dac7ee5b1065bbc5f3696ab675b761a`. Merged `dev` once into the job branch. Pushed the job branch; did not merge into `dev`. ### Built - Per-User Tantivy Search generations and bounded private reader cache. A ready User never opens the legacy shared Search generation. A pending Shared-root view waits for its recipient generation. An unreadable private generation fails closed and queues repair. - Incoming Shares are copied into the recipient's private Search generation. Immutable item IDs gate each Share. Files Share and item changes record durable recipient invalidations; immediate hooks and the Search Job sweep rebuild recipients. Owner file changes update ready recipient generations. - Text and CLIP embedding candidate joins require matching document and vector/LSH owners. The table-safety matrix is in `docs/audits/per-user-index-400.md`. - `tests/adversarial/xuser_matrix.py` probes Search keyword and hybrid, Tags, Photos, Share grant/revocation, and private latency/RSS. Its recipient persona now uses Dora's bearer token. Main files: `crates/calternal-search/src/{indexer,index,query,plugin}.rs`, `crates/calternal-embed/src/{store,clip_store}.rs`, `crates/calternal-fs/src/{root,path,user_homes}.rs`, `crates/calternal-server/src/{main,wire}.rs`, `crates/plugins/files/src/{lib,shares}.rs`, `crates/plugins/files/migrations/0016_share_search_invalidations.sql`, `tests/adversarial/xuser_matrix.py`, `docs/audits/per-user-index-400.md`. ### Gates and live evidence - `cargo fmt --check`: exit 0, no output. - Final `cargo build -p calternal-server`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 18s ``` - `bun run check`: ``` svelte-check found 0 errors and 0 warnings ``` - `bun run test`: ``` Test Files 125 passed (125) Tests 800 passed (800) ``` - `cargo test --profile dev -p calternal-embed`: ``` test result: ok. 21 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.45s ``` - `cargo test --profile dev -p calternal-fs`: ``` test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.89s test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 18.69s ``` - `cargo test --profile dev -p calternal-tags`: ``` test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.32s ``` - `cargo test --profile dev -p calternal-server`: ``` test result: ok. 82 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 42.01s ``` - `cargo test --profile dev -p calternal-search`: unit and operator suites passed. Its 20-test integration suite had one timing failure under host load: ``` test result: FAILED. 19 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 21.84s ``` The unchanged watcher test passed on isolated retry: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 19 filtered out; finished in 7.89s ``` The pending-Share regression added after that suite passed: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 6.47s ``` - With the query root clause removed in the test build (`--features unsafe-test-no-root-filter`), the private-directory isolation test passed: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 32.90s ``` - Live derived two-User matrix on the production server: exit 0. It covered Search keyword/hybrid, Tags, Photos, Share recipient and revocation. Under concurrent Cargo compilation: ``` PER_USER_SEARCH A p50_ms=20.88 p95_ms=43.98 rss_delta_bytes=978944 PER_USER_SEARCH B p50_ms=58.69 p95_ms=104.03 rss_delta_bytes=21278720 PER_USER_SEARCH D p50_ms=74.28 p95_ms=135.50 rss_delta_bytes=20672512 ``` ### Decisions and gaps - Text and CLIP embeddings, Tags, Analytics, Notifications, and Photos projections remain User-keyed shared SQLite tables. The audit names every owner/viewer predicate and the matrix probes the API boundary. This is the explicit table-safety option in the job prompt; physical per-User SQLite files remain a later hardening option. - A Home-only request from a User not yet privately rebuilt may still use the legacy shared migration generation. Shared-root requests wait for the recipient generation. The old generation can be removed after migration completes. - The full cross-API adversarial replay is blocked by a new `dev` OpenAPI authorization-policy entry (`app_surfaces_get`) outside #400. The #400 derived-only live matrix passed. - `cargo clippy --all-targets -- -D warnings` started after the merge but was interrupted at dependency check compilation when the pending-Share fix required a source change. It has no completed gate result. `cargo test` for the full workspace was not run in the four-hour job window. `cargo test --profile dev -p calternal-plugin-files` started and was interrupted during dependency compilation at the same time limit, before tests ran. These gates remain for the orchestrator or a follow-up job. No test expectation was changed. - The source-level pending-Share fix has its own passing regression test and a passing final server build. The live matrix ran against the immediately preceding build, so the final build has not had a second full live round. - Production web build output and retained adversarial data were removed; `cargo clean` was run at the end.

kayg referenced this issue

2026-09-29 12:22:32 +00:00

Adversarial authz matrix lacks Apps surface policy #432

kayg commented

2026-09-29 12:22:53 +00:00

Author

Owner

Filed the unrelated full adversarial replay policy gap as kayg/calternal#432. The #400 derived-only live matrix passed; this issue stays open for orchestrator review.

kayg referenced this issue

2026-09-29 12:45:21 +00:00

ISOLATION: physically separate per-User derived stores (vectors) + full cross-user inventory #435

kayg referenced this issue

2026-09-29 12:46:25 +00:00

ISOLATION: physically separate per-User derived stores (vectors) + full cross-user inventory #435

kayg referenced this issue from a commit

2026-09-29 13:27:07 +00:00

docs(isolation): mark #400 shared-table exception historical

kayg referenced this issue from a commit

2026-09-29 13:58:32 +00:00

Merge job/per-user-index (#400: per-User search index, owner-checked vector queries, share invalidation, two-User matrix)