#72 - Files: keyset-paginated folder listing from the Index (10k folder first rows < 150 ms) - kayg/calternal

kayg commented

2026-09-24 19:02:23 +00:00

Owner

Found by the Files UI build (#35/#36/#37/#66). A 10,000-file folder takes about 2.1 s to show its first rows and about 13 s to finish loading (debug server, loaded machine). The budget is < 150 ms to first rows. Cause: GET /api/v1/files/entries lists and sorts the whole directory again for every 500-item page.

Fix in crates/plugins/files:

Serve folder listings from the Index (the files table already has path, name, kind, size and mtime) with keyset pagination: cursor = the last (sort key, name, item id), plus a stable sort for name/modified/size/kind in both directions. The first page must not need the full directory.
Keep correctness vs. the disk: the Index is derived; if the folder's reconcile generation is stale, serve from the Index and trigger a background reconcile of that folder, then push changes over the existing files event stream (the UI already live-refreshes).
Return total from a cheap count, not by materializing the list.
Budgets (release build, idle machine): first page of a 10k folder < 50 ms server time; full 10k folder via pages < 1 s; 100k folder first page < 80 ms.
Tests: pagination stability while files are added/renamed/deleted between pages (no duplicates, no skips for untouched items), Unicode/NFC sort order, and all sort modes. Extend tests/adversarial/attack.py with cursor tampering (a forged cursor must be rejected with 400 and must never leak another folder's or another user's entries).

Found by the Files UI build (#35/#36/#37/#66). A 10,000-file folder takes about 2.1 s to show its first rows and about 13 s to finish loading (debug server, loaded machine). The budget is < 150 ms to first rows. Cause: `GET /api/v1/files/entries` lists and sorts the whole directory again for every 500-item page. Fix in `crates/plugins/files`: - Serve folder listings from the Index (the files table already has path, name, kind, size and mtime) with **keyset pagination**: `cursor` = the last (sort key, name, item id), plus a stable sort for name/modified/size/kind in both directions. The first page must not need the full directory. - Keep correctness vs. the disk: the Index is derived; if the folder's reconcile generation is stale, serve from the Index and trigger a background reconcile of that folder, then push changes over the existing files event stream (the UI already live-refreshes). - Return `total` from a cheap count, not by materializing the list. - Budgets (release build, idle machine): first page of a 10k folder < 50 ms server time; full 10k folder via pages < 1 s; 100k folder first page < 80 ms. - Tests: pagination stability while files are added/renamed/deleted between pages (no duplicates, no skips for untouched items), Unicode/NFC sort order, and all sort modes. Extend `tests/adversarial/attack.py` with cursor tampering (a forged cursor must be rejected with 400 and must never leak another folder's or another user's entries).

kayg commented

2026-09-24 19:41:29 +00:00

Author

Owner

Starting files listing implementation on branch job/files-listing, based on 8c216759f0. I read issue #72 and its comments (none) and am implementing the indexed keyset listing, stale-folder reconciliation, regression coverage, generated contract, and adversarial cursor checks.

Starting files listing implementation on branch job/files-listing, based on 8c216759f01d704e69ecf37ef8293f422abf5552. I read issue #72 and its comments (none) and am implementing the indexed keyset listing, stale-folder reconciliation, regression coverage, generated contract, and adversarial cursor checks.

kayg commented

2026-09-24 20:00:15 +00:00

Author

Owner

Finding: the new API regression fails against the base handler because GET /entries?limit=2&sort=name returns total: null where the five indexed folders require total: 5 (targeted test: folder_entries_use_stable_keyset_pages_and_return_total, assertion at crates/plugins/files/src/lib.rs:1830). The current handler calls root.list, loads every files_index row for metadata, sorts the full vector, and uses a decimal offset cursor. I am replacing this path with indexed keyset queries and a cheap count.

Finding: the new API regression fails against the base handler because `GET /entries?limit=2&sort=name` returns `total: null` where the five indexed folders require `total: 5` (targeted test: `folder_entries_use_stable_keyset_pages_and_return_total`, assertion at `crates/plugins/files/src/lib.rs:1830`). The current handler calls `root.list`, loads every `files_index` row for metadata, sorts the full vector, and uses a decimal offset cursor. I am replacing this path with indexed keyset queries and a cheap count.

kayg commented

2026-09-24 20:20:11 +00:00

Author

Owner

Finding: stale_folder_listing_serves_index_then_reconciles_and_publishes holds the Home mutation lock after adding a disk-only child. The first page correctly returns only the indexed known directory (total 1), but after releasing the lock the second child is still absent after 5 seconds (folder did not reconcile). This confirms the missing piece is stale-folder reconciliation and event publication, not page materialization.

Finding: `stale_folder_listing_serves_index_then_reconciles_and_publishes` holds the Home mutation lock after adding a disk-only child. The first page correctly returns only the indexed `known` directory (total 1), but after releasing the lock the second child is still absent after 5 seconds (`folder did not reconcile`). This confirms the missing piece is stale-folder reconciliation and event publication, not page materialization.

kayg commented

2026-09-24 20:26:39 +00:00

Author

Owner

Implementation note: the production migration registry in crates/calternal-server/src/wire.rs had a duplicate hard-coded Files migration list. I added the small public calternal_plugin_files::migrations() helper and switched the registry to call it, so migration 0008 (and now 0009) is applied from the Files crate's ordered set. This keeps the namespace and existing migration descriptions unchanged.

Implementation note: the production migration registry in `crates/calternal-server/src/wire.rs` had a duplicate hard-coded Files migration list. I added the small public `calternal_plugin_files::migrations()` helper and switched the registry to call it, so migration 0008 (and now 0009) is applied from the Files crate's ordered set. This keeps the namespace and existing migration descriptions unchanged.

kayg commented

2026-09-24 20:32:21 +00:00

Author

Owner

Finding: while reviewing the full /entries route, Shared/<owner>/<folder> still calls shares::list_virtual, which uses root.list and materializes the directory before paging. The owner Home route now pages from files_index, but shared folder pages would retain the same full-directory bottleneck from the issue. I am routing authorized shared folders through the same Index keyset query, while leaving grant-derived virtual grouping pages on their small grant set.

Finding: while reviewing the full `/entries` route, `Shared/<owner>/<folder>` still calls `shares::list_virtual`, which uses `root.list` and materializes the directory before paging. The owner Home route now pages from `files_index`, but shared folder pages would retain the same full-directory bottleneck from the issue. I am routing authorized shared folders through the same Index keyset query, while leaving grant-derived virtual grouping pages on their small grant set.

kayg commented

2026-09-24 20:45:17 +00:00

Author

Owner

Checkpoint: backend implementation and regression coverage are committed as 60091162fee7900c1b85ca7b9d91bcb5ba859a45 on job/files-listing. cargo fmt --all --check and git diff --check passed before the commit. The API generation command is building the server's vendored OpenSSL dependency; full package and workspace gates remain pending.

Checkpoint: backend implementation and regression coverage are committed as `60091162fee7900c1b85ca7b9d91bcb5ba859a45` on `job/files-listing`. `cargo fmt --all --check` and `git diff --check` passed before the commit. The API generation command is building the server's vendored OpenSSL dependency; full package and workspace gates remain pending.

kayg commented

2026-09-24 21:02:43 +00:00

Author

Owner

Finding: the first workspace Clippy run reported two -D warnings failures in the new Files code: index_page_scoped had 9 parameters (clippy::too_many_arguments at listing.rs:221), and FilesState.reconcile_folders exposed a nested map/set type (clippy::type_complexity at lib.rs:152). I am grouping pager identity into a named scope and naming the pending reconciliation types before rerunning the gates.

Finding: the first workspace Clippy run reported two `-D warnings` failures in the new Files code: `index_page_scoped` had 9 parameters (`clippy::too_many_arguments` at `listing.rs:221`), and `FilesState.reconcile_folders` exposed a nested map/set type (`clippy::type_complexity` at `lib.rs:152`). I am grouping pager identity into a named scope and naming the pending reconciliation types before rerunning the gates.

kayg commented

2026-09-24 21:07:28 +00:00

Author

Owner

Test investigation: the first post-change Files suite had 6 upload-related failures with HTTP 413. uploads::create_impl reserves 1 GiB by rejecting when available space is below that floor; df -h /tmp showed only 848 MiB free on the shared tmpfs. Re-running with TMPDIR under this worktree (the /home filesystem had over 60 GiB free) made the representative upload test pass and the full Files suite pass: 46 passed; 0 failed. No upload behavior change was needed.

Test investigation: the first post-change Files suite had 6 upload-related failures with HTTP 413. `uploads::create_impl` reserves 1 GiB by rejecting when available space is below that floor; `df -h /tmp` showed only 848 MiB free on the shared tmpfs. Re-running with `TMPDIR` under this worktree (the `/home` filesystem had over 60 GiB free) made the representative upload test pass and the full Files suite pass: `46 passed; 0 failed`. No upload behavior change was needed.

kayg commented

2026-09-24 21:33:37 +00:00

Author

Owner

Finding: the full workspace test run reached calternal-collab/tests/shared_notes.rs and failed in FilesState::reconcile_all with files Index update failed. Its test database applies only Files migrations 1–5, while the new index::record_impl inserts name_key from migration 0008. crates/plugins/calendar/src/view.rs also has a duplicate test migration list ending at 0007. Both crates already depend on the Files plugin. I am switching these test fixtures to the Files crate's exported migrations() set so their schemas track the production registry; this changes test setup only.

Finding: the full workspace test run reached `calternal-collab/tests/shared_notes.rs` and failed in `FilesState::reconcile_all` with `files Index update failed`. Its test database applies only Files migrations 1–5, while the new `index::record_impl` inserts `name_key` from migration 0008. `crates/plugins/calendar/src/view.rs` also has a duplicate test migration list ending at 0007. Both crates already depend on the Files plugin. I am switching these test fixtures to the Files crate's exported `migrations()` set so their schemas track the production registry; this changes test setup only.

kayg commented

2026-09-24 22:26:04 +00:00

Author

Owner

Release measurements from an ignored temporary harness against the Files Axum router with generated Index rows (response body collection included):

10k rows: median first page 4.408 ms; full 20-page walk 117.192 ms.
100k rows: median first page 10.632 ms.
Release compilation completed in 25m32s; measurement test ran in 5.43s.

All three issue budgets were met. The host was not idle during measurement (8 CPUs, observed load average near 20), so these results are indicative rather than an idle-machine certification. The harness is temporary and will be removed before final gates.

Release measurements from an ignored temporary harness against the Files Axum router with generated Index rows (response body collection included): - 10k rows: median first page 4.408 ms; full 20-page walk 117.192 ms. - 100k rows: median first page 10.632 ms. - Release compilation completed in 25m32s; measurement test ran in 5.43s. All three issue budgets were met. The host was not idle during measurement (8 CPUs, observed load average near 20), so these results are indicative rather than an idle-machine certification. The harness is temporary and will be removed before final gates.

kayg commented

2026-09-24 22:43:28 +00:00

Author

Owner

Adversarial run evidence: round 2 reported 0 findings. Round 1 reported 19 findings, all Task storm latency checks: HTTP 201 responses taking 5.4–8.1 s (request indices 5–23, 24-request concurrency). The server stayed alive, and the other printed probes had expected status/results. The host had 8 CPUs and load average near 20 while unrelated Cargo builds were active. This is outside the Files listing path; I am recording it as a host-contention performance finding and will not change another plugin's behavior in this job.

Adversarial run evidence: round 2 reported 0 findings. Round 1 reported 19 findings, all `Task storm` latency checks: HTTP 201 responses taking 5.4–8.1 s (request indices 5–23, 24-request concurrency). The server stayed alive, and the other printed probes had expected status/results. The host had 8 CPUs and load average near 20 while unrelated Cargo builds were active. This is outside the Files listing path; I am recording it as a host-contention performance finding and will not change another plugin's behavior in this job.

kayg commented

2026-09-24 22:46:17 +00:00

Author

Owner

Finished issue #72.

Head SHA: 5b080d51dc2416c75969c3aa97048cf2ede4989e
Commits:

6009116 Serve Files folder pages from the Index
34f6ffc Cover equivalent Unicode listing names
d783193 Generate Files listing page contract
66f12bf Group Files listing scope parameters
5b080d5 Use canonical Files migrations in test fixtures

Gate output:

cargo fmt --all --check exited 0 with empty output.

cargo clippy --workspace --all-targets -- -D warnings:

    Checking calternal-plugin-files v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/files)
   Compiling calternal-server v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-server)
    Checking calternal-plugin-calendar v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/calendar)
    Checking calternal-collab v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.69s

cargo test --workspace completed with passing test and doc-test suites. The Files suite reported:

test result: ok. 46 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.46s

The shared-notes integration reported test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.55s.

bash packages/api-client/check-generated.sh:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 22s
     Running `target/debug/calternal-server openapi`
$ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts
✨ openapi-typescript 7.13.0
🚀 ../../contracts/openapi.json → src/generated.ts [295.2ms]

bash tests/adversarial/run.sh: round 1 reported ==== FINDINGS 19, all Task storm latency warnings on HTTP 201 responses, 5.4–8.1 s for requests 5–23. Round 2 reported ==== ROUND 2 FINDINGS 0. The server stayed alive; the other observed probe results had expected statuses. The shared host had 8 CPUs and a load average near 20 with unrelated Cargo builds active. This leaves the requested zero-findings condition unmet for round 1; no change was made to Notes Task behavior outside this job's ownership.

Release timing harness output:

BENCH folder_rows=10000 first_page_median_ms=4.408 full_rows_ms=117.192 pages=20 entries=10000
BENCH folder_rows=100000 first_page_median_ms=10.632 full_rows_ms=2603.120 pages=200 entries=100000
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.43s

The release profile build took 25m 32s. All three issue timing budgets were met. The timings used the Files Axum router and body collection; the machine was not idle, so they are indicative rather than an idle-host certification.

cargo clean:

     Removed 20215 files, 12.4GiB total

The working tree is clean at the head SHA above.

Finished issue #72. Head SHA: `5b080d51dc2416c75969c3aa97048cf2ede4989e` Commits: - `6009116` Serve Files folder pages from the Index - `34f6ffc` Cover equivalent Unicode listing names - `d783193` Generate Files listing page contract - `66f12bf` Group Files listing scope parameters - `5b080d5` Use canonical Files migrations in test fixtures Gate output: `cargo fmt --all --check` exited 0 with empty output. `cargo clippy --workspace --all-targets -- -D warnings`: ```text Checking calternal-plugin-files v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/files) Compiling calternal-server v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-server) Checking calternal-plugin-calendar v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/calendar) Checking calternal-collab v0.1.0 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.69s ``` `cargo test --workspace` completed with passing test and doc-test suites. The Files suite reported: ```text test result: ok. 46 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.46s ``` The shared-notes integration reported `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.55s`. `bash packages/api-client/check-generated.sh`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 22s Running `target/debug/calternal-server openapi` $ bunx --package openapi-typescript@7.13.0 openapi-typescript ../../contracts/openapi.json -o src/generated.ts ✨ openapi-typescript 7.13.0 🚀 ../../contracts/openapi.json → src/generated.ts [295.2ms] ``` `bash tests/adversarial/run.sh`: round 1 reported `==== FINDINGS 19`, all `Task storm` latency warnings on HTTP 201 responses, 5.4–8.1 s for requests 5–23. Round 2 reported `==== ROUND 2 FINDINGS 0`. The server stayed alive; the other observed probe results had expected statuses. The shared host had 8 CPUs and a load average near 20 with unrelated Cargo builds active. This leaves the requested zero-findings condition unmet for round 1; no change was made to Notes Task behavior outside this job's ownership. Release timing harness output: ```text BENCH folder_rows=10000 first_page_median_ms=4.408 full_rows_ms=117.192 pages=20 entries=10000 BENCH folder_rows=100000 first_page_median_ms=10.632 full_rows_ms=2603.120 pages=200 entries=100000 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.43s ``` The release profile build took `25m 32s`. All three issue timing budgets were met. The timings used the Files Axum router and body collection; the machine was not idle, so they are indicative rather than an idle-host certification. `cargo clean`: ```text Removed 20215 files, 12.4GiB total ``` The working tree is clean at the head SHA above.

kayg commented

2026-09-24 22:46:37 +00:00

Author

Owner

Correction to the finished gate comment: the Clippy output package versions were v0.0.1, not v0.1.0. The exact first four lines were:

    Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/files)
   Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-server)
    Checking calternal-plugin-calendar v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/calendar)
    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-collab)

The command finished successfully in 11.69s as reported.

Correction to the finished gate comment: the Clippy output package versions were `v0.0.1`, not `v0.1.0`. The exact first four lines were: ```text Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/files) Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-server) Checking calternal-plugin-calendar v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/plugins/calendar) Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/files-listing/crates/calternal-collab) ``` The command finished successfully in 11.69s as reported.

kayg commented

2026-09-24 22:47:18 +00:00

Author

Owner

Decisions where the design did not define a detail:

Descending order uses sort=-<key> for name, kind, size, and modified.
The Name sort key is NFC; the raw name breaks NFC-equivalent ties. Other sort modes break equal primary keys by NFC name, then item ID.
The cursor key is process-local and random. Cursors from before a server restart are rejected; no durable secret contract exists for persistent cursors.
A Shared physical folder reads the owner's Index, while the cursor binds to the recipient and the visible Shared/... path.
The canonical Files migration set is exposed as migrations() and used by server wiring and the affected test fixtures.

Decisions where the design did not define a detail: - Descending order uses `sort=-<key>` for name, kind, size, and modified. - The Name sort key is NFC; the raw name breaks NFC-equivalent ties. Other sort modes break equal primary keys by NFC name, then item ID. - The cursor key is process-local and random. Cursors from before a server restart are rejected; no durable secret contract exists for persistent cursors. - A Shared physical folder reads the owner's Index, while the cursor binds to the recipient and the visible `Shared/...` path. - The canonical Files migration set is exposed as `migrations()` and used by server wiring and the affected test fixtures.

kayg referenced this issue

2026-09-24 23:08:28 +00:00

Adversarial harness: refresh passkey freshness before admin probes; update shared-listing probe for Index listings; load-aware SLOW #76

kayg referenced this issue from a commit

2026-09-24 23:13:21 +00:00

Merge job/files-listing: keyset-paginated folder listing from the Index (#72)

kayg referenced this issue from a commit

2026-09-24 23:28:21 +00:00

Files tests: align round 2 shared-listing tests with Index-backed listing

kayg referenced this issue from a commit

2026-09-25 01:32:37 +00:00

Files tests: undo test uses the real Files schema; shared-listing test tolerates owner reconcile