kayg/calternal

Watch

Fork 0

notes: paged identity projection and bounded child lookups (#761 C) #792

Closed

kayg wants to merge 16 commits from feat/tiering-761-c-index into feat/tiering-761-b-reader

kayg commented

2026-10-31 16:18:34 +00:00

Owner

SQLite temp-storage follow-up plan (2026-11-01)

Cause under investigation. A path-annotated C trace recorded 236,342 writes to five SQLite etilqs_* temporary files while the scan had staged 20,000 identities. The writes were 4-byte markers and 4 KiB pages; they came from the same identity-writer thread as ids.db WAL writes and identity-log fsyncs. The main Index::load connection already uses in-memory temp storage. The traced run then hit the existing 60-second writer acknowledgement timeout under instrumentation, so this is attribution evidence, not a completed 100k trace or timing result.

Change. Apply temp_store=MEMORY and temp.cache_size=-128 to the identity-writer and compaction-reader SQLite connections. Keep synchronous=FULL, the identity-log fsync, directory sync, and durable acknowledgements unchanged. The per-connection temporary cache remains 128 KiB; the existing projection lease continues to bound aggregate SQLite caches to 64 MiB across 128 Homes. Runtime and storage remain O(N) in indexed notes, with O(128 KiB) temp-cache budget per connection.

failure	effect	detection	recovery	test
A writer/compaction connection spills temp pages	extra SQLite temp-file writes remain	path-annotated `strace` still shows `etilqs_*` writes	identify the remaining connection and keep its temp cache bounded	writer and snapshot connection regression checks
projection acknowledgement is delayed	scan remains incomplete; no success is reported	existing scan timeout and incomplete-list state	retry the scan after the writer recovers	existing scan retry and durable batch tests
process stops during compaction	committed identity log remains authoritative	reopen/replay sees the old log or published snapshot	remove scratch and rebuild projection	existing compaction recovery and replay tests
durability setting regresses	an identity may be acknowledged before the durable projection	pragma assertion and crash tests	restore FULL sync and replay from the log	FULL-sync regression assertion and real-binary crash tests

Gates. On dev-01 through Nix and ~/bin/slot: full workspace tests, format, strict clippy, and release build. Build and SHA-256-check C/dev/perf binaries on dev-01; run three alternating cold pairs and one warm pair on perf-01 under ~/bin/perf-lock with RuntimeMaxSec >= 6h. Cold p95 must be at most 1.25x dev; open.note p95 must be at most 15 ms and idle.rss p95 at most 100 MB. No durability relaxation is allowed.

scan batch cache experiment: rejected

The 400-row experiment (df742b61) kept log fsync and FULL projection commits but did not reduce syscall writes: 1,876,854 pwrite64 calls versus 1,868,042 at 4,000 rows. Its untraced cold run took 47.447 s and wrote 1,306.8 MB versus the prior diagnostic’s 26.068 s / 983.0 MB (shared dev-01, so timing is diagnostic). Reverted the experiment; the retained implementation and durability protocol are identical to 92a28435. The fresh retained-binary comparison measured 25.500 s / 962.302 MB. Final head 558e78e93ae4fd5aa3d0400216573509f1ecbde5 passes targeted tests, crash tests, fmt, strict clippy and release build; isolated qualification remains blocked by both shared perf locks. No job of ours remains running.

Bounded discovery reader follow-up

The latest shared dev-01 pair still measures C 29.3 s / dev 16.2 s. Metadata description is 14.3 s; the reader performs repeated absolute-root checks for each file. Add a background-only reader group capped at 128 opens: validate the original root before and after the group, retain per-file plain-name/no-symlink/no-cross-mount opens, and return derived results only after final validation. Metadata still validates again before its transaction. Ordinary request reads keep their existing checks. Each worker describes one body at a time; no group retains 128 bodies. Root validation becomes O(groups), with O(128) derived results per worker and unchanged 64-worker admission.

Failure: root replacement during a group discards the entire result and fails discovery for retry; no note file is changed. Gate this with deterministic replacement and group-limit regressions, the existing retained-reader tests, notes and crash suites. Both perf VMs are currently occupied by the separate 759b matrix; isolated qualification is still required.

Resumed cold-index plan (2026-11-01)

The new dev-01 phase profile shows repeated exact SQL preparation and repeated absolute-root ancestor opens, in addition to the pending-overlay exclusion scan. Cache the two point-lookup statements and replace Linux root-validation component walks with one openat2 rooted at /, retaining no-symlink resolution and all before/after root mount/device/inode/owner checks. Root path parsing stays O(depth), descriptor resolution becomes O(1) syscalls; each validation holds at most two transient descriptors. No bodies are added to tier storage and no log sync or FULL projection commit is removed.

failure	effect / detection	recovery	gate
ancestor becomes a symlink	no descriptor returned; no-symlink resolution fails	restore the path and restart	existing ancestor-symlink reader regression
root is replaced or remounted	existing identity comparison refuses the read	offline repair and restart	pinned-reader replacement regression
parent component supplied	reject before resolution, preserving current input rules	use a plain root path	root-path regression
interrupted identity append / projection	durable log and FULL commit rules unchanged	existing replay and scan recovery	notes suite and real-binary crash tests

Re-run fsio, notes, crash and fmt/clippy gates on dev-01; build there, verify SHA-256 through control and perf hosts, then collect three alternating cold pairs on perf-01 and one designated warm pair on perf-02 under the shared locks. Nothing merged.

Paged identity projection and bounded background I/O (#761 C)

Stacked on B #791. Identity, folded-path, child, and descendant lookups use SQLite B-trees. ids.jsonl remains the committed rollback/recovery log; its identity and length validate the projection on reopen.

Cold-index follow-up plan and dev-01 profile

Cause. On C head 9c8c2873, identify_many awaited a durable write after each 50-note batch. At 100k notes that made about 2,000 sequential writer transactions, preventing the writer's eight-command queue from grouping scan commits. A server-only dev-01 strace profile counted 6,569 fsync/fdatasync calls: 2,000 on ids.jsonl, 2,004 on its directory, 2,295 on ids.db-wal, 144 on ids.db, 98 on meta.db, 16 on history.db, and 12 other syncs. The perf harness reported 2,253 MB physically written for that traced cold run; the instrumented cold.index time was 920 s, so its elapsed time is diagnostic only. The final files were 8.8 MB ids.jsonl, 35.2 MB ids.db, and 4.5 MB ids.db-wal. The scan reads note bodies but its write paths are the identity log and SQLite index/metadata/history stores.

Fix. Cold discovery now uses a scan-only limit of 400 identities per durable append and synchronous=FULL projection transaction. Ordinary put_many remains capped at 50. Staging clears up to the same group size of stale overlay rows, so the disk-backed pending table remains O(400). At 100k notes this reduces scan groups from about 2,000 to 250; discovery and log replay remain O(N). The identity log sync, directory sync, and FULL projection commit are retained.

Failure point	Effect	Detection	Recovery	Regression evidence
Crash before the current group's log sync	Up to 400 derived identities may be absent or partly appended; note files are untouched	Startup scan finds source notes missing from the identity index; the loader skips incomplete JSONL records	Append repairs a partial trailing line; the scan rebuilds missing identities from files	`r2_scan_recovers_after_restart_between_identity_groups`
Crash after log sync but before projection commit	The complete group is in `ids.jsonl`; SQLite projection may lag	Projection length or identity does not match the log on open	Replay the durable log into SQLite	`scan_batch_of_400_is_durable_and_replayed`
Crash during projection commit	SQLite WAL recovers; the durable log remains authoritative	Projection metadata validation on open	Rebuild the projection from `ids.jsonl` if needed	`scan_batch_of_400_is_durable_and_replayed`
Repeated scan groups	Stale pending rows could accumulate in TEMP storage	Overlay row count exceeds one group	Each scan staging transaction clears up to 400 stale rows; normal staging remains capped at 50	`scan_batches_clear_the_previous_staged_group`

The scan still reports incomplete until discovery finishes. No note file is modified. The event batch remains 1,000 records. The dev-01 trace is diagnostic, not a performance qualification.

Gates. Run the notes crate suite on dev-01. On perf VMs, alternate C/dev samples under ~/bin/perf-lock using the #issuecomment-72012 harness options; collect three cold and five warm 100k runs and report cold.index, open.note, and idle.rss. Require cold.index p95 at most 60 s and at most 1.25x dev. Report page-cache drop availability. Keep C idle RSS at or below the prior C p95 of 107.668 MB; the existing 100 MB RSS budget remains tracked separately.

Round-2 design and bounds

Requests reserve one of 64 semaphore permits with try_acquire_owned() before staging identities or changing create/rename/delete files and history. The transport cannot accumulate more than 64 admitted commands, including in-flight commands. Admission never waits. Saturation/closed writers return NotSavedYet (HTTP 503, retry).
A dedicated writer connection performs log append/fsync and projection commit before acknowledging. Callers await outside the Home index guard with a five-second deadline. Up to eight queued commands share a commit, with at most 50 rows per command. Reservations do not retain a channel sender, so a paused, unsent request cannot keep shutdown alive.
A same-ID create retry checks the pending sequence against the committed projection watermark. It returns 503 while the earlier identity is uncommitted, including after the first request timed out.
Compaction runs only on that background writer. A pinned read transaction streams 50 snapshot rows per step, then copies at most 64 KiB of the committed log tail per step. It serves at most eight queued commands between steps. Publication atomically replaces ids.jsonl; it never rebuilds the request connection's projection.
Cold discovery stages and acknowledges identities in groups of at most 400 through a scan-only API; ordinary identity writes remain at 50. A bounded pending page resolves duplicate embedded IDs before commit. The existing 10k task cold-fill test, r2_scan_batches_keep_duplicate_embedded_ids_distinct_across_pages, and the new restart regression cover completion, identity preservation, and interrupted discovery.
Descendant queries use keyset pages of at most 50 rows. Watcher removals and rename matching retain one page. Contact linking also pages immediate children and retains at most two candidates per requested name.
Scratch is never committed state: only ids.jsonl plus matching projection metadata is authoritative. Startup takes the exclusive index ownership lock, then removes reserved UUID scratch names, including interrupted legacy atomic-write scratch. Unrelated files remain untouched. A live owner prevents cleanup of active snapshots.
SQLite cache allowances remain capped at 128 Homes × (128 KiB reader + 128 KiB TEMP + 128 KiB writer + 128 KiB snapshot) = 64 MiB. Pending rows use a disk-backed TEMP overlay; only sequences newer than the committed watermark are visible. Cleanup is capped at 50 stale rows for ordinary staging and 400 for one cold-scan group.

Exact lookups are O(log n); descendant/child pages are O(log n + 50). Replay and complete compaction remain O(entries), with bounded compaction steps outside request locks. Initial projection replay occurs in the existing blocking Home-open worker.

Review findings and failure recovery

Finding / failure	Result and recovery	Regression evidence
P1: blocking full queue	Immediate 503/retry before rejected identities or create files are staged; another Home saves on the same runtime	`r2_full_queue_never_waits_for_the_writer_or_stages_rejected_ids`, `r2_saturated_identity_queue_refuses_create_without_blocking_another_home`
P1: inline compaction	Background snapshot creation failure cannot fail a successfully admitted append; bounded steps preserve concurrent changes/deletions	`r2_compaction_failure_does_not_fail_an_admitted_append`, `r2_compaction_steps_keep_concurrent_updates_and_deletions_on_reopen` (50k rows)
P2: unbounded descendants	Pages contain at most 50 rows; watcher processes all pages	`r2_descendant_queries_never_materialize_a_wide_subtree` (50k), `r2_watcher_removes_every_descendant_across_pages`, `r2_contact_candidates_page_beyond_fifty_and_bound_ambiguous_names`
P2: orphan snapshots	Drop removes scratch; startup removes interrupted/replacement-failure orphans only after exclusive ownership	`r2_startup_removes_only_uncommitted_compaction_scratch`, `r2_replacement_error_orphans_are_recovered_only_after_owner_exits`
Writer stalls / exits	Async acknowledgement expires after five seconds or reports disconnect as 503/retry; fix disk and restart if writer exited	`r2_acknowledgement_has_a_deadline_and_another_writer_keeps_saving`, existing failed-log-write regression
Create retries before an earlier identity commit	503 until the committed watermark advances; then the same identity succeeds	`r2_create_retry_cannot_acknowledge_an_uncommitted_identity` failed before the guard, passed after
Shutdown with an unsent reservation	Join buffered writer work without waiting for the paused request; late staging is refused	`r2_shutdown_does_not_wait_for_an_unsent_admission`, `a_create_that_outlives_shutdown_writes_nothing`

The four original regressions (queue, compaction failure, 50k descendant cap, orphan startup cleanup) all failed on the pre-fix C head cb8ef0b0. The map functional model waits for actual disk acknowledgement rather than applying the request deadline. The anchor refusal fixture waits for save-time title normalization before its whole-note equality snapshot; the previous full run captured only that title property racing the snapshot, with unchanged block text. Both original assertions remain. Restart fixtures now release the old index owner without settling unsaved rooms; the crash-recovery assertions remain in place.

Validation

Head: 9c8c287357240912daf895c4166bf00f2212ce4a.

PASS at 9c8c287357240912daf895c4166bf00f2212ce4a: fmt, strict clippy, and the full workspace suite; 1,205 passed, zero failed, 20 ignored, exit 0. Run sequentially through Nix/slot on dev-01 with RUST_TEST_THREADS=2. Evidence: ~/tier-c-r2-final6.{log,exit}. D's exact-head gate also passed: 1,208 passed, zero failed, 20 ignored (~/tier-d-r2-final6.{log,exit}). The final-head release rebuild passed; runtime hashes match the prepared perf binaries.

cargo fmt --all --check
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings
RUST_TEST_THREADS=2 cargo test -j4 --workspace --locked --no-fail-fast -- --nocapture

Final head 9c8c287357240912daf895c4166bf00f2212ce4a built successfully on dev-01. The final-head rebuild produced the same runtime binaries as 9e15ece7; the only later change is a test fixture barrier. SHA-256 matched on dev-01, control, and perf-01:

server: 0a541a0ba472631fe53872fbdce85f6e93663188144434c1b2e24f2bf15c7ea5
harness: 081e036a464cb95bc13f67732e87079437874155f3c5bf9933ebc7f66de5b297

Final-head performance qualification is blocked. Both perf VMs are occupied by another campaign using /var/lock/calternal-perf.lock, which claude cannot open. The attempted final-head run under the documented ~/bin/perf-lock was stopped after detecting the other campaign. No results from that attempt are treated as qualification. No other processes were stopped. Earlier 48ae327e measurements (3.218 ms open.note p95; 89.293 MB idle RSS) remain historical evidence, not isolated final-head qualification.

The outstanding command, on an idle perf VM under its exclusive measurement lock:

calternal-perf --bin "$PWD/bin/calternal" --work "$PWD/work" --sizes 100k --runs 3 --cold-runs 1 --idle-secs 60 --budgets "$PWD/budgets.json" --gate --json "$PWD/report.json" --md "$PWD/report.md"

The budget file contains only open.note@100000.p95=15 and idle.rss@100000.p95=100. Prepared binaries/fixture/run script: perf-01 ~/tier-c-perf-9e15/. Aborted measurement log: ~/tier-c-r2-perf-final4.log. The attempted shared-lock acquisition returned permission denied (~/tier-c-r2-perf-final5.{log,exit}, exit 66). No measurement remains running.

Remaining limits

A five-second timeout does not cancel an admitted disk write; the client retries through existing idempotent operations. A stopped writer requires disk repair/restart. Cache-lease exhaustion refuses a new Home; fair eviction is future work. These changes do not qualify FO1, online tier movement, history-heavy 100k-edit opens, or power-loss protocols.

Stack: A #790 → B #791 → C #792 → D #793. Nothing merged.

### SQLite temp-storage follow-up plan (2026-11-01) **Cause under investigation.** A path-annotated C trace recorded 236,342 writes to five SQLite `etilqs_*` temporary files while the scan had staged 20,000 identities. The writes were 4-byte markers and 4 KiB pages; they came from the same identity-writer thread as `ids.db` WAL writes and identity-log fsyncs. The main `Index::load` connection already uses in-memory temp storage. The traced run then hit the existing 60-second writer acknowledgement timeout under instrumentation, so this is attribution evidence, not a completed 100k trace or timing result. **Change.** Apply `temp_store=MEMORY` and `temp.cache_size=-128` to the identity-writer and compaction-reader SQLite connections. Keep `synchronous=FULL`, the identity-log fsync, directory sync, and durable acknowledgements unchanged. The per-connection temporary cache remains 128 KiB; the existing projection lease continues to bound aggregate SQLite caches to 64 MiB across 128 Homes. Runtime and storage remain O(N) in indexed notes, with O(128 KiB) temp-cache budget per connection. | failure | effect | detection | recovery | test | |---|---|---|---|---| | A writer/compaction connection spills temp pages | extra SQLite temp-file writes remain | path-annotated `strace` still shows `etilqs_*` writes | identify the remaining connection and keep its temp cache bounded | writer and snapshot connection regression checks | | projection acknowledgement is delayed | scan remains incomplete; no success is reported | existing scan timeout and incomplete-list state | retry the scan after the writer recovers | existing scan retry and durable batch tests | | process stops during compaction | committed identity log remains authoritative | reopen/replay sees the old log or published snapshot | remove scratch and rebuild projection | existing compaction recovery and replay tests | | durability setting regresses | an identity may be acknowledged before the durable projection | pragma assertion and crash tests | restore FULL sync and replay from the log | FULL-sync regression assertion and real-binary crash tests | **Gates.** On dev-01 through Nix and `~/bin/slot`: full workspace tests, format, strict clippy, and release build. Build and SHA-256-check C/dev/perf binaries on dev-01; run three alternating cold pairs and one warm pair on perf-01 under `~/bin/perf-lock` with `RuntimeMaxSec >= 6h`. Cold p95 must be at most 1.25x dev; open.note p95 must be at most 15 ms and idle.rss p95 at most 100 MB. No durability relaxation is allowed. --- ### scan batch cache experiment: rejected The 400-row experiment (`df742b61`) kept log fsync and FULL projection commits but did not reduce syscall writes: 1,876,854 pwrite64 calls versus 1,868,042 at 4,000 rows. Its untraced cold run took 47.447 s and wrote 1,306.8 MB versus the prior diagnostic’s 26.068 s / 983.0 MB (shared dev-01, so timing is diagnostic). Reverted the experiment; the retained implementation and durability protocol are identical to `92a28435`. The fresh retained-binary comparison measured 25.500 s / 962.302 MB. Final head `558e78e93ae4fd5aa3d0400216573509f1ecbde5` passes targeted tests, crash tests, fmt, strict clippy and release build; isolated qualification remains blocked by both shared perf locks. No job of ours remains running. ### Bounded discovery reader follow-up The latest shared dev-01 pair still measures C 29.3 s / dev 16.2 s. Metadata description is 14.3 s; the reader performs repeated absolute-root checks for each file. Add a background-only reader group capped at 128 opens: validate the original root before and after the group, retain per-file plain-name/no-symlink/no-cross-mount opens, and return derived results only after final validation. Metadata still validates again before its transaction. Ordinary request reads keep their existing checks. Each worker describes one body at a time; no group retains 128 bodies. Root validation becomes O(groups), with O(128) derived results per worker and unchanged 64-worker admission. Failure: root replacement during a group discards the entire result and fails discovery for retry; no note file is changed. Gate this with deterministic replacement and group-limit regressions, the existing retained-reader tests, notes and crash suites. Both perf VMs are currently occupied by the separate 759b matrix; isolated qualification is still required. ### Resumed cold-index plan (2026-11-01) The new dev-01 phase profile shows repeated exact SQL preparation and repeated absolute-root ancestor opens, in addition to the pending-overlay exclusion scan. Cache the two point-lookup statements and replace Linux root-validation component walks with one `openat2` rooted at `/`, retaining no-symlink resolution and all before/after root mount/device/inode/owner checks. Root path parsing stays O(depth), descriptor resolution becomes O(1) syscalls; each validation holds at most two transient descriptors. No bodies are added to tier storage and no log sync or FULL projection commit is removed. | failure | effect / detection | recovery | gate | |---|---|---|---| | ancestor becomes a symlink | no descriptor returned; no-symlink resolution fails | restore the path and restart | existing ancestor-symlink reader regression | | root is replaced or remounted | existing identity comparison refuses the read | offline repair and restart | pinned-reader replacement regression | | parent component supplied | reject before resolution, preserving current input rules | use a plain root path | root-path regression | | interrupted identity append / projection | durable log and FULL commit rules unchanged | existing replay and scan recovery | notes suite and real-binary crash tests | Re-run fsio, notes, crash and fmt/clippy gates on dev-01; build there, verify SHA-256 through control and perf hosts, then collect three alternating cold pairs on perf-01 and one designated warm pair on perf-02 under the shared locks. Nothing merged. ## Paged identity projection and bounded background I/O (#761 C) Stacked on B #791. Identity, folded-path, child, and descendant lookups use SQLite B-trees. `ids.jsonl` remains the committed rollback/recovery log; its identity and length validate the projection on reopen. ### Cold-index follow-up plan and dev-01 profile **Cause.** On C head `9c8c2873`, `identify_many` awaited a durable write after each 50-note batch. At 100k notes that made about 2,000 sequential writer transactions, preventing the writer's eight-command queue from grouping scan commits. A server-only dev-01 `strace` profile counted 6,569 `fsync`/`fdatasync` calls: 2,000 on `ids.jsonl`, 2,004 on its directory, 2,295 on `ids.db-wal`, 144 on `ids.db`, 98 on `meta.db`, 16 on `history.db`, and 12 other syncs. The perf harness reported 2,253 MB physically written for that traced cold run; the instrumented `cold.index` time was 920 s, so its elapsed time is diagnostic only. The final files were 8.8 MB `ids.jsonl`, 35.2 MB `ids.db`, and 4.5 MB `ids.db-wal`. The scan reads note bodies but its write paths are the identity log and SQLite index/metadata/history stores. **Fix.** Cold discovery now uses a scan-only limit of 400 identities per durable append and `synchronous=FULL` projection transaction. Ordinary `put_many` remains capped at 50. Staging clears up to the same group size of stale overlay rows, so the disk-backed pending table remains O(400). At 100k notes this reduces scan groups from about 2,000 to 250; discovery and log replay remain O(N). The identity log sync, directory sync, and FULL projection commit are retained. | Failure point | Effect | Detection | Recovery | Regression evidence | |---|---|---|---|---| | Crash before the current group's log sync | Up to 400 derived identities may be absent or partly appended; note files are untouched | Startup scan finds source notes missing from the identity index; the loader skips incomplete JSONL records | Append repairs a partial trailing line; the scan rebuilds missing identities from files | `r2_scan_recovers_after_restart_between_identity_groups` | | Crash after log sync but before projection commit | The complete group is in `ids.jsonl`; SQLite projection may lag | Projection length or identity does not match the log on open | Replay the durable log into SQLite | `scan_batch_of_400_is_durable_and_replayed` | | Crash during projection commit | SQLite WAL recovers; the durable log remains authoritative | Projection metadata validation on open | Rebuild the projection from `ids.jsonl` if needed | `scan_batch_of_400_is_durable_and_replayed` | | Repeated scan groups | Stale pending rows could accumulate in TEMP storage | Overlay row count exceeds one group | Each scan staging transaction clears up to 400 stale rows; normal staging remains capped at 50 | `scan_batches_clear_the_previous_staged_group` | The scan still reports incomplete until discovery finishes. No note file is modified. The event batch remains 1,000 records. The dev-01 trace is diagnostic, not a performance qualification. **Gates.** Run the notes crate suite on dev-01. On perf VMs, alternate C/dev samples under `~/bin/perf-lock` using the #issuecomment-72012 harness options; collect three cold and five warm 100k runs and report `cold.index`, `open.note`, and `idle.rss`. Require `cold.index` p95 at most 60 s and at most 1.25x dev. Report page-cache drop availability. Keep C idle RSS at or below the prior C p95 of 107.668 MB; the existing 100 MB RSS budget remains tracked separately. ### Round-2 design and bounds - Requests reserve one of 64 semaphore permits with `try_acquire_owned()` before staging identities or changing create/rename/delete files and history. The transport cannot accumulate more than 64 admitted commands, including in-flight commands. Admission never waits. Saturation/closed writers return `NotSavedYet` (HTTP 503, retry). - A dedicated writer connection performs log append/fsync and projection commit before acknowledging. Callers await outside the Home index guard with a five-second deadline. Up to eight queued commands share a commit, with at most 50 rows per command. Reservations do not retain a channel sender, so a paused, unsent request cannot keep shutdown alive. - A same-ID create retry checks the pending sequence against the committed projection watermark. It returns 503 while the earlier identity is uncommitted, including after the first request timed out. - Compaction runs only on that background writer. A pinned read transaction streams 50 snapshot rows per step, then copies at most 64 KiB of the committed log tail per step. It serves at most eight queued commands between steps. Publication atomically replaces `ids.jsonl`; it never rebuilds the request connection's projection. - Cold discovery stages and acknowledges identities in groups of at most 400 through a scan-only API; ordinary identity writes remain at 50. A bounded pending page resolves duplicate embedded IDs before commit. The existing 10k task cold-fill test, `r2_scan_batches_keep_duplicate_embedded_ids_distinct_across_pages`, and the new restart regression cover completion, identity preservation, and interrupted discovery. - Descendant queries use keyset pages of at most 50 rows. Watcher removals and rename matching retain one page. Contact linking also pages immediate children and retains at most two candidates per requested name. - Scratch is never committed state: only `ids.jsonl` plus matching projection metadata is authoritative. Startup takes the exclusive index ownership lock, then removes reserved UUID scratch names, including interrupted legacy atomic-write scratch. Unrelated files remain untouched. A live owner prevents cleanup of active snapshots. - SQLite cache allowances remain capped at 128 Homes × (128 KiB reader + 128 KiB TEMP + 128 KiB writer + 128 KiB snapshot) = 64 MiB. Pending rows use a disk-backed TEMP overlay; only sequences newer than the committed watermark are visible. Cleanup is capped at 50 stale rows for ordinary staging and 400 for one cold-scan group. Exact lookups are O(log n); descendant/child pages are O(log n + 50). Replay and complete compaction remain O(entries), with bounded compaction steps outside request locks. Initial projection replay occurs in the existing blocking Home-open worker. ### Review findings and failure recovery | Finding / failure | Result and recovery | Regression evidence | |---|---|---| | P1: blocking full queue | Immediate 503/retry before rejected identities or create files are staged; another Home saves on the same runtime | `r2_full_queue_never_waits_for_the_writer_or_stages_rejected_ids`, `r2_saturated_identity_queue_refuses_create_without_blocking_another_home` | | P1: inline compaction | Background snapshot creation failure cannot fail a successfully admitted append; bounded steps preserve concurrent changes/deletions | `r2_compaction_failure_does_not_fail_an_admitted_append`, `r2_compaction_steps_keep_concurrent_updates_and_deletions_on_reopen` (50k rows) | | P2: unbounded descendants | Pages contain at most 50 rows; watcher processes all pages | `r2_descendant_queries_never_materialize_a_wide_subtree` (50k), `r2_watcher_removes_every_descendant_across_pages`, `r2_contact_candidates_page_beyond_fifty_and_bound_ambiguous_names` | | P2: orphan snapshots | Drop removes scratch; startup removes interrupted/replacement-failure orphans only after exclusive ownership | `r2_startup_removes_only_uncommitted_compaction_scratch`, `r2_replacement_error_orphans_are_recovered_only_after_owner_exits` | | Writer stalls / exits | Async acknowledgement expires after five seconds or reports disconnect as 503/retry; fix disk and restart if writer exited | `r2_acknowledgement_has_a_deadline_and_another_writer_keeps_saving`, existing failed-log-write regression | | Create retries before an earlier identity commit | 503 until the committed watermark advances; then the same identity succeeds | `r2_create_retry_cannot_acknowledge_an_uncommitted_identity` failed before the guard, passed after | | Shutdown with an unsent reservation | Join buffered writer work without waiting for the paused request; late staging is refused | `r2_shutdown_does_not_wait_for_an_unsent_admission`, `a_create_that_outlives_shutdown_writes_nothing` | The four original regressions (queue, compaction failure, 50k descendant cap, orphan startup cleanup) all failed on the pre-fix C head `cb8ef0b0`. The map functional model waits for actual disk acknowledgement rather than applying the request deadline. The anchor refusal fixture waits for save-time title normalization before its whole-note equality snapshot; the previous full run captured only that title property racing the snapshot, with unchanged block text. Both original assertions remain. Restart fixtures now release the old index owner without settling unsaved rooms; the crash-recovery assertions remain in place. ### Validation Head: `9c8c287357240912daf895c4166bf00f2212ce4a`. **PASS at `9c8c287357240912daf895c4166bf00f2212ce4a`**: fmt, strict clippy, and the full workspace suite; **1,205 passed, zero failed, 20 ignored**, exit 0. Run sequentially through Nix/slot on dev-01 with `RUST_TEST_THREADS=2`. Evidence: `~/tier-c-r2-final6.{log,exit}`. D's exact-head gate also passed: 1,208 passed, zero failed, 20 ignored (`~/tier-d-r2-final6.{log,exit}`). The final-head release rebuild passed; runtime hashes match the prepared perf binaries. ```sh cargo fmt --all --check cargo clippy -j4 --workspace --all-targets --locked -- -D warnings RUST_TEST_THREADS=2 cargo test -j4 --workspace --locked --no-fail-fast -- --nocapture ``` Final head `9c8c287357240912daf895c4166bf00f2212ce4a` built successfully on dev-01. The final-head rebuild produced the same runtime binaries as `9e15ece7`; the only later change is a test fixture barrier. SHA-256 matched on dev-01, control, and perf-01: - server: `0a541a0ba472631fe53872fbdce85f6e93663188144434c1b2e24f2bf15c7ea5` - harness: `081e036a464cb95bc13f67732e87079437874155f3c5bf9933ebc7f66de5b297` **Final-head performance qualification is blocked.** Both perf VMs are occupied by another campaign using `/var/lock/calternal-perf.lock`, which `claude` cannot open. The attempted final-head run under the documented `~/bin/perf-lock` was stopped after detecting the other campaign. No results from that attempt are treated as qualification. No other processes were stopped. Earlier `48ae327e` measurements (3.218 ms open.note p95; 89.293 MB idle RSS) remain historical evidence, not isolated final-head qualification. The outstanding command, on an idle perf VM under its exclusive measurement lock: ```sh calternal-perf --bin "$PWD/bin/calternal" --work "$PWD/work" --sizes 100k --runs 3 --cold-runs 1 --idle-secs 60 --budgets "$PWD/budgets.json" --gate --json "$PWD/report.json" --md "$PWD/report.md" ``` The budget file contains only `open.note@100000.p95=15` and `idle.rss@100000.p95=100`. Prepared binaries/fixture/run script: perf-01 `~/tier-c-perf-9e15/`. Aborted measurement log: `~/tier-c-r2-perf-final4.log`. The attempted shared-lock acquisition returned permission denied (`~/tier-c-r2-perf-final5.{log,exit}`, exit 66). No measurement remains running. ### Remaining limits A five-second timeout does not cancel an admitted disk write; the client retries through existing idempotent operations. A stopped writer requires disk repair/restart. Cache-lease exhaustion refuses a new Home; fair eviction is future work. These changes do not qualify FO1, online tier movement, history-heavy 100k-edit opens, or power-loss protocols. Stack: [A #790](https://git.kayg.org/kayg/calternal/pulls/790) → [B #791](https://git.kayg.org/kayg/calternal/pulls/791) → [C #792](https://git.kayg.org/kayg/calternal/pulls/792) → [D #793](https://git.kayg.org/kayg/calternal/pulls/793). Nothing merged.

kayg added 4 commits

2026-10-31 16:18:34 +00:00

WIP paged index from s1 29a8c76c7b

C: tiering model goes to D 7e2dae82e5

C: report a stopped id index writer instead of dropping changes silently 2e70d5176f

Persist actual identity-log record counts across reopen

CI / server (pull_request) Has been cancelled

Details

CI / rust (pull_request) Has been cancelled

Details

CI / web (pull_request) Has been cancelled

Details

618e2c4e9c

Failure	Effect / detection	Recovery	Regression
Full queue / stalled writer	Immediate admission refusal or bounded async wait; 503/retry, no successful durability ack	Retry after writer catches up	deterministic full queue plus unrelated-save progress
Compaction overlaps writes	Stable snapshot plus streamed committed tail	Continue bounded steps; restart rebuilds from committed log	writes/reopen/deletions during incremental compaction
Large removed subtree	At most 50 descendant rows per page	Reconcile batches and yield	50k descendant pagination and watcher coverage
Kill/error after scratch creation or replacement	Scratch is never authoritative; log identity detects incomplete projection update	Ownership-locked startup deletes orphans and replays if needed	interruption and replacement-error scratch recovery

Finding / gate	Status	Evidence
P1: request-thread queue backpressure	Fixed: 64 bounded permits, immediate admission, async five-second acknowledgement, 503/retry	full-queue, saturated-create, independent-writer and shutdown regressions PASS
P1: inline O(entries) compaction	Fixed: background connection; 50 rows / 64 KiB per step; at most eight commands between steps	failure isolation and concurrent 50k snapshot/reopen regressions PASS
P2: unbounded descendants	Fixed: keyset pages at most 50, watcher batches and bounded contact candidates	50k traversal, multi-page deletion and contact paging regressions PASS
P2: orphan snapshots	Fixed: exclusive ownership before reserved scratch cleanup	interruption, legacy scratch and replacement-error recovery regressions PASS
Timed-out create retry	Fixed: return 503 until the committed watermark covers the identity	deterministic stalled-writer test failed before the guard, PASS after; same-ID retry succeeds after commit
Cold-fill batching	Fixed: scan identities staged/acknowledged in pages of 50	10k task cold-fill, duplicate IDs across pages and mid-scan regressions PASS
fmt / clippy `-D warnings`	PASS on both final heads	Nix + slot, dev-01
`cargo test --workspace`	C: 1,205 passed; D: 1,208 passed; zero failed, 20 ignored each	`~/tier-{c,d}-r2-final6.{log,exit}`, exit 0; two test threads, sequential suites
100k open.note p95 at most 15 ms / idle RSS at most 100 MB	BLOCKED: final-head isolated run pending	both perf hosts occupied by another campaign; its shared lock is inaccessible to `claude`; overlapping attempt stopped

Round-2 item	Final status	Does the test demonstrate it?
P1: synchronous queue backpressure	Resolved; no regression found. Admission uses an immediate permit attempt before staging, with 64 permits retained through writer processing. Acknowledgement is asynchronous, with a five-second deadline and retryable 503.	Yes. `r2_full_queue_never_waits_for_the_writer_or_stages_rejected_ids` stalls the writer, exceeds capacity and checks prompt return without rejected identities. `r2_saturated_identity_queue_refuses_create_without_blocking_another_home` checks refusal before file creation and progress in another Home. The deadline/independent-writer test also passes.
P1: O(entries) compaction on the request path	Resolved; no regression found. Snapshot creation, log writes/fsync and compaction run on the writer's connections. Snapshot steps contain at most 50 rows or 64 KiB; the writer services at most eight commands between steps. No request builds the snapshot or rebuilds the committed projection during publication.	Yes for the functional/bounded-step claim. `r2_compaction_failure_does_not_fail_an_admitted_append` isolates snapshot creation failure. `r2_compaction_steps_keep_concurrent_updates_and_deletions_on_reopen` uses 50k rows, requires at least 1,000 steps and checks updated/deleted/new identities after reopen. These tests do not establish the latency/RSS budgets.
P2: unbounded descendant query/removal	Resolved; no regression found. Keyset pages contain at most 50 rows; watcher deletion/rename matching consumes pages. Contact candidate lookup pages children and retains at most two candidates per requested name.	Yes. `r2_descendant_queries_never_materialize_a_wide_subtree` checks all 50k descendants with a per-page cap. `r2_watcher_removes_every_descendant_across_pages` verifies 125 removals and the unrelated survivor. The contact regression reaches beyond the first 50 children and checks bounded ambiguity.
P2: orphan compaction snapshots	Resolved; no regression found. Startup takes exclusive ownership before removing reserved UUID scratch names. Scratch is not authoritative; the published log identity/length guards projection reuse.	Yes. `r2_startup_removes_only_uncommitted_compaction_scratch` covers current/legacy scratch and preserves unrelated files. `r2_replacement_error_orphans_are_recovered_only_after_owner_exits` leaves a real replacement/removal-error orphan, refuses a second live owner, then cleans it on reopen while preserving committed state.
Inherited A/B findings	Resolved, as assessed in the final #790/#791 reviews.	Their nine `r2_` regressions pass in the independent downstream run.

Metric	Budget	Dev p50 / p95	C p50 / p95	Result
`open.note@100000`	p95 ≤ 15 ms	0.737 / 3.165 ms	1.078 / 4.029 ms	Both pass
`idle.rss@100000`	p95 ≤ 100 MB	116.020 / 117.055 MB	104.539 / 107.668 MB	Both fail

metric / cohort	dev p50 / p95	C p50 / p95	gate	result
cold.index, 3 cold pairs, perf-01	11.305 / 11.452 s	40.185 / 41.254 s	p95 ≤60 s and ≤1.25× dev (14.315 s)	fail: 3.602× dev; absolute limit passes
open.note, 5 warm pairs, combined	0.743 / 2.989 ms	1.059 / 3.407 ms	p95 ≤15 ms	pass
idle.rss, 5 warm pairs, combined	99.129 / 103.914 MB	90.055 / 94.379 MB	p95 no worse than dev	pass; C 9.2% lower
open.note, perf-01, 3 warm pairs	0.743 / 2.989 ms	1.059 / 3.578 ms	p95 ≤15 ms	pass
idle.rss, perf-01, 3 warm pairs	100.848 / 103.914 MB	92.617 / 94.379 MB	p95 no worse than dev	pass
open.note, perf-02, 2 warm pairs	0.588 / 2.804 ms	1.017 / 3.332 ms	p95 ≤15 ms	pass
idle.rss, perf-02, 2 warm pairs	98.918 / 99.129 MB	85.867 / 86.707 MB	p95 no worse than dev	pass

cohort	C cold.index	dev cold.index	result
historical isolated 3-pair p95, `02491f79` (#issuecomment-72561)	41.254 s	11.452 s	fail, 3.602×
inherited `e0af7578`, dev-01 single cold each	56.908 s	12.729 s	diagnostic only
retained final source, latest dev-01 repeat	25.500 s	not contemporaneously paired	diagnostic only; 962.302 MB written
final isolated 3 cold pairs + 1 warm pair	not collected	not collected	blocked by both shared perf locks

cold.index	C (s)	dev (s)
Pair 1	14.120	12.646
Pair 2	15.273	10.807
Pair 3	15.776	11.411
nearest-rank p95 (max of 3)	15.776	12.646

sample 4 warm metric	C	dev	result
open.note p95	5.036 ms	3.397 ms	both under 15 ms
idle.rss	96.609 MB	104.543 MB	C under 100 MB and below dev

Gate (dev-01, Nix, `~/bin/slot`)	A `91c034eb`	B `efcaaaf7`	C `487e0589`	D `d422d0ca`
`cargo fmt --all --check`	pass	pass	pass	pass
`cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`	pass	pass	pass	pass ¹
`cargo test -j4 --workspace --locked -- --test-threads=2`	1,495 passed, 0 failed, 22 ignored	1,504 / 0 / 22	1,537 / 0 / 22 ²	1,540 / 0 / 22 ¹

Gate	Head (C/D)	dev `41f2efa3`	Limit	Result
cold.index, 3 cold pairs, perf-01, p95 = max of 3	25.0 s (22.6 / 25.0 / 24.3)	20.9 s (20.9 / 18.6 / 20.2)	≤ 1.25× dev and ≤ 60 s	pass, 1.196× (was 1.248×)
cold.mem_peak (harness), 3 cold runs, max	227 MB	291 MB	not worse than dev	pass, 0.78× dev (was 175 vs 186 MiB = 0.94×)
server VmHWM, sampled every 200 ms, max over 4 samples	227.5 MiB	291.5 MiB	—	head lower
open.note@100000 p95, warm pair	12.4 ms	10.6 ms	≤ 15 ms	pass
idle.rss@100000, warm pair	60.6 MB	71.8 MB	≤ 100 MB	pass
200 sequential `notes.append`, 30 alternating pairs, perf-02, median total	1.578 s	1.595 s	within 1% of dev	pass, −1.09% (pair-ratio median 0.992)

pair (time)	dev `41f2efa3` first page / cold index / open.note p50 / p95	`d422d0ca` (before)	fixed tree
15:03 rep 1	1.67 s / 19.7 s / 3.48 / 9.0 ms	5.73 s / 25.5 s / 8.4 / 22.2	`h4` 4.83 s / 23.6 s / 9.0 / 22.6
15:05 rep 2	1.73 / 20.3 / 7.98 / 14.7	6.08 / 26.2 / 9.1 / 44.6	`h4` 4.81 / 23.4 / 8.9 / 28.8
17:16 rep 1	1.92 / 20.4 / 4.71 / 12.2	6.36 / 25.2 / 4.68 / 16.3	`h5` 4.81 / 22.3 / 2.89 / 12.5
17:21 rep 2	1.88 / 19.4 / 5.20 / 15.2	6.34 / 25.4 / 4.71 / 19.2	`h5` 4.65 / 22.3 / 2.96 / 8.2
18:00 rep 1 (noisy, others on the VM)	2.33 / 26.6 / 6.19 / 17.7	7.39 / 32.4 / 3.46 / 19.7	`h6` 5.59 / 30.5 / 6.0 / 43.5
18:07 rep 2	2.04 / 21.5 / 2.78 / 9.9	5.99 / 26.3 / 3.88 / 16.1	`h6` 5.47 / 24.0 / 2.98 / 21.4

metric	head	dev	dev copy	head / dev	control (devB / dev)
cold.first_page (ms)	4048	1624	1619	2.49×	1.00
cold.index (s)	17.2	14.9	15.4	1.16×	1.04
open.note p50 / p95 (ms)	0.92 / 3.80	0.75 / 3.43	0.77 / 3.50	1.22× / 1.11×	1.03 / 1.02
open.api_get p50 / p95 (ms)	6.5 / 27.2	6.1 / 9.2	6.5 / 10.3	1.08× / 2.95×	1.07 / 1.12
home.first_page (ms)	6.18	3.44	3.26	1.80×	0.95
home.page / page_next / page_title (ms)	0.98 / 0.77 / 0.80	1.06 / 0.78 / 0.84	0.95 / 0.74 / 0.81	0.93 / 0.99 / 0.94
home.page_tag (ms)	70.4	52.2	53.1	1.35×	1.02
home.page_big (ms)	23.6	18.6	18.3	1.27×	0.98
journal.page (ms)	2.99	2.49	2.51	1.20×	1.00
search.*	not in this harness output (see note)
cold.mem_peak (MB)	240	291	291	0.82×	1.00
server VmHWM, 200 ms sampler (MB, median / max of 6)	240 / 243	291 / 294	291 / 293	0.83×
200 sequential `notes.append`, 30 pairs (`CALTERNAL_NOTES_SAVE_MS=15`)	1.490 s	1.480 s		+0.62% (pair-ratio spread 0.94–1.02)	control 15 pairs dev vs devB: +0.22%

metric	head	dev	dev copy	head/dev	control
cold.first_page (ms)	4025	1812	1763	2.22×	0.97
cold.index (s)	17.31	14.67	15.10	1.18×	1.03
cold.mem_peak (MB)	239	290	287	0.82×	0.99
cold.disk_written (MB)	1973	1918	1919	1.03×	1.00
home.page_tag p50 / p95 (ms)	68.3 / 74.2	52.4 / 56.4	52.3 / 54.7	1.30× / 1.31×	1.00 / 0.97
home.page_big p50 / p95 (ms)	22.4 / 26.0	18.4 / 20.9	19.1 / 20.7	1.22× / 1.25×	1.04 / 0.99
journal.page p50 / p95 (ms)	3.03 / 4.52	2.39 / 2.69	2.49 / 3.12	1.26× / 1.68×	1.04 / 1.16
open.note p50 / p95 (ms)	0.98 / 3.65	0.77 / 3.29	0.76 / 3.42	1.27× / 1.11×	0.99 / 1.04

piece	head	dev	note
`scan::identify_many`	1,008 M	213 M	+795 M
↳ `flush_identities` → `Index::enqueue` (stage into the TEMP overlay with 3 indexes + `known_ids`)	537 M	–	on the scan's critical path
↳ `paths_of` / `ids_at_folds`	112 M / 64 M	(map)
identity writer thread (`index::apply`, own connection)	543 M	~0	dev only appends the log
classify extra (`page_after` + `ids_at_folds`)	200 M	(map)
JSON/markdown description (`meta::described`)	3.35 G	3.34 G	not a cause
reading heads (`read_ids`)	486 M	488 M	not a cause (the fs walk and reader checks cost the same)
metadata store (`MetaDb::apply` / `end_fresh`)	2.19 G / 0.75 G	2.18 G / 0.63 G	about flat

Gate	Result
`cargo fmt --all --check`	pass (`3a65e27c`)
`cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`	pass (at `2473b998`)
`cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2`	1,540 passed, 0 failed, 22 ignored (at `2473b998`; `3a65e27c` is rustfmt only)

variant	wall	vs head
dev pattern: log append + fsync only	0.42 s
head pattern (TEXT PK + fold index + per-row totals trigger + TEMP staging + commit per group, 4 MiB cache)	7.44 s
− TEMP staging	6.84 s	−0.6 s
− staging − per-row totals trigger (count once after)	4.86 s	−2.0 s for the trigger
… + 64 MiB cache, still a commit per group	4.56 s
… + one commit per fill, 4 MiB cache	4.32 s
… + one commit per fill and 64 MiB fill cache	1.68 s	−5.8 s (the page churn of rewriting the B-tree each group)
… same + indexes built after the fill ("best")	1.50 s
integer rowid key + unique id index (instead of TEXT PK)	5.11 s	no gain (+0.25 s)
WITHOUT ROWID	13.08 s	much worse
inserts in id order instead of path order	4.11 s	−0.75 s
`synchronous=NORMAL` instead of FULL	7.42 s	none

median of 3 (ms)	during scan: head / dev / devB	head/dev	after settle: head / dev / devB	head/dev
page_tag	67.9 / 51.9 / 52.0	1.31×	50.6 / 50.0 / 50.5	1.01×
page_big	19.2 / 14.6 / 14.4	1.31×	14.6 / 14.4 / 14.4	1.01×
journal_page	2.15 / 1.69 / 1.68	1.27×	1.73 / 1.55 / 1.58	1.12× (0.18 ms, not attributed)
page	1.01 / 1.00 / 0.95	1.01×	0.66 / 0.71 / 0.61	0.93×

	design	what changes	predicted at 100k on perf-01 (from the shares above)	keeps the paging goal?
C	keep dev's identity layout; slice 1 ships the reader/root seam and the state split only	drop C's projection (revert `index.rs`/`scan.rs` to dev's map + log)	every hot path = dev (it is dev's code)	no: idle RSS stays dev's (~+10%) and ten active 100k Homes hold ten maps; the paging goal moves to a later slice
A	trim the writer (kayg's option 2)	ack a fill group after the log fsync; one projection transaction per fill with a 64 MiB fill-only cache; totals counted once at fill end (no per-row trigger); TEMP staging kept for reads during the fill	writer cost −70 to −80% (7.44 → 1.68 s in the replay), but staging (537 M of the scan's +795 M) stays on the critical path: first page ≈ 2.6–3.0 s (≈1.45–1.65×), cold index ≈ 1.05–1.09×. Not flat.	yes
B	fill with dev's map, write the projection once	During a Home's first fill, lookups use dev's in-memory map; groups are durable at the log fsync (the log already is the authority). After the scan reports complete, the writer bulk-loads the projection in one FULL transaction (sorted, indexes after, totals once), records the log identity/length, then drops the map. Warm starts never build the map. Warm classify becomes one ordered merge over `entries_fold`, not 500 `IN` queries + `page_after`.	first page ≈ dev (the identity path is dev's); cold index ≈ dev (the ~1.5 s projection build runs after complete); cold.mem_peak ≈ dev (~290 MB, losing the 0.82× win); idle RSS = head's (paged); after-settle reads are already 1.01×, and the merge should bring the warm scan near dev, removing the 1.2–1.3× overlap	yes, in steady state

notes: paged identity projection and bounded child lookups (#761 C) #792

SQLite temp-storage follow-up plan (2026-11-01)

scan batch cache experiment: rejected

Bounded discovery reader follow-up

Resumed cold-index plan (2026-11-01)

Paged identity projection and bounded background I/O (#761 C)

Cold-index follow-up plan and dev-01 profile

Round-2 design and bounds

Review findings and failure recovery

Validation

Remaining limits

Independent review, round 2

P1 — synchronous queue backpressure and compaction block request threads

P2 — descendant removal materializes every path

P2 — interrupted compaction leaves orphan snapshots

Final round-2 fix plan

Independent review, round 3 (final)

100k qualification results

100k qualification: fail on relative cold.index

execution and evidence

cold-index resume: fixes checked, isolated qualification blocked

profile and retained fixes

before / after

exact-head checks

qualification blocker and prepared inputs

100k cold-index attribution and qualification — fab01191

Write attribution and fix

Perf qualification

Independent review: cold-index perf fixes

Findings

Checks

Verdict

Rebase and conflict resolutions

C on dev 41f2efa3: restack, resolutions

Resolutions (dev's side kept; the paged traversal goes through it)

Gate table — stack on dev 41f2efa3 (takeover after the Codex usage limit)

Perf pass on the stack: causes, fixes, and where it still regresses (not mergeable yet)

Causes found (callgrind on a 10k cold start, same fixture, dev vs head)

Commits (all pushed, one change each)

Measured (perf-01, calternal-perf --sizes 100k --runs 3 --cold-runs 1 --idle-secs 0, scenarios list,query,journal,open, alternating, same VM, no page-cache drop)

Options for the remaining gap (need a decision, not a patch)

Sorter memory budget (item 4)

Gates (dev-01, ~/bin/slot)

Option A (secondary indexes after the first fill): implemented, target not met — first page 2.5× dev, cold index 1.16× dev

What changed

Tests (index/deferred_tests.rs, 7 new)

Perf-01, 100k Home, 6 reps, rotated order (head / dev / dev copy "devB" as control), no page-cache drop, both perf locks held

Reading

Gates (dev-01, ~/bin/slot, tree 43d81bc1)

Rethink after repeated misses: where the time goes, and three designs

What tiering slice 1 has to achieve

Measured (perf-01, 100k Home, seed 1, both perf locks held, three rotated reps head/dev/devB, no page-cache drop)

Cost breakdown: cold first page and cold index

Steady-state regressions: query shape is unchanged, they overlap the warm start-up scan

Three designs, simplest first

Pull request closed

100k cold-index attribution and qualification — `fab01191`

C on dev `41f2efa3`: restack, resolutions

Gate table — stack on dev `41f2efa3` (takeover after the Codex usage limit)

Measured (perf-01, `calternal-perf --sizes 100k --runs 3 --cold-runs 1 --idle-secs 0`, scenarios list,query,journal,open, alternating, same VM, no page-cache drop)

Gates (dev-01, `~/bin/slot`)

Tests (`index/deferred_tests.rs`, 7 new)

Gates (dev-01, `~/bin/slot`, tree `43d81bc1`)