kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

storage: tiering qualification fixtures and specification models (#761 D) #793

Closed
kayg wants to merge 18 commits from feat/tiering-761-d-scale into feat/tiering-761-c-index
pull from: feat/tiering-761-d-scale
merge into: kayg:feat/tiering-761-c-index
Conversation 12 Commits 18 Files changed 73 +454
kayg commented 2026-10-31 16:18:57 +00:00
Owner
Copy link

Plan — qualification fixtures and independent tiering models (#761), D of A → B → C → D

Stacked on C, ultimately into dev. Move tiering-scale, independent specification models, and the validation ledger out of #784's runtime migration/read/index code.

  • Generate deterministic identity, wide-folder, inventory, repeated-block and serialized-history workloads. Each run takes an absent fixture directory and refuses reuse.
  • Keep metadata-only generation explicitly distinct from real byte/CRDT workloads; decode/apply history and compare complete replayed text.
  • Exercise independent persistence/pair, retained-inode and mixed-root quota specifications. They state requirements for later tiering/FO1 code; they do not validate unimplemented production protocols.
  • Preserve the full qualification ledger, including unrun disk fault, 100k-edit history, successful-open and FO1 latency gates. No runtime tier movement or dedupe is enabled.

Cost: streaming O(entries + generated bytes) fixture work; history replay includes the full expanded CRDT cost. Large aggregate and real-disk campaigns belong on the locked perf VMs, not this shared build host.

Failure → effect → detection → recovery → test

Failure Effect / detection Recovery Test
Either disk persists independent subset Specification retains pair bytes once prepared, never invents HEAD Model replay pair_bytes_and_auth_survive_independent_disk_orders
Late fd/mmap/alias write after displacement Specification retains old inode's bytes Retained names remain reachable displaced_inode_retention_covers_late_fd_mmap_and_alias_writes
Mixed-generation quota walk Reject as admission evidence Repeat one complete generation mixed_root_quota_observations_never_authorize_admission
Fixture truncates or history replay differs Generator verification fails Regenerate in absent directory tiering-scale workload checks
Metadata-only run mistaken for qualification Explicit mode and validation ledger Run real workload on perf VM docs/plans/0761-slice-1-validation.md

Validation

Rebased unchanged onto C 9c8c287357240912daf895c4166bf00f2212ce4a. Current D head 659b0f4e420f6272e801ea78eba8fe10d66c1ca5. Range-diff confirms both D-only patches are unchanged.

PASS: fmt, strict clippy, and full workspace tests on dev-01 through Nix/slot. 1,208 passed, zero failed, 20 ignored, exit 0. RUST_TEST_THREADS=2; evidence ~/tier-d-r2-final6.{log,exit}. C and D ran sequentially.

cargo fmt --all --check
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings
RUST_TEST_THREADS=2 cargo test -j4 --workspace --locked --no-fail-fast -- --nocapture

Decisions and remaining risks

These are functional fixtures and independent models, not certification of real paired replay, auth/HEAD protocols, retained-inode mutation recipes or quota admission. History-heavy open, 15 ms successful opens, RSS/FD/watch limits, XFS/ext4 comparisons, SIGKILL and VM power cuts remain unqualified. No production data, merge, deployment, provisioning or cutover was performed. The prior shared-host history fixture attempt is retained as incomplete evidence, not relabeled as passing.

Stack: A #790 → B #791 → C #792 → D #793. Final round-3 review follows completion of the exact-head gates; nothing merged.

## Plan — qualification fixtures and independent tiering models (#761), D of A → B → C → D Stacked on C, ultimately into `dev`. Move `tiering-scale`, independent specification models, and the validation ledger out of #784's runtime migration/read/index code. - Generate deterministic identity, wide-folder, inventory, repeated-block and serialized-history workloads. Each run takes an absent fixture directory and refuses reuse. - Keep metadata-only generation explicitly distinct from real byte/CRDT workloads; decode/apply history and compare complete replayed text. - Exercise independent persistence/pair, retained-inode and mixed-root quota specifications. They state requirements for later tiering/FO1 code; they do not validate unimplemented production protocols. - Preserve the full qualification ledger, including unrun disk fault, 100k-edit history, successful-open and FO1 latency gates. No runtime tier movement or dedupe is enabled. Cost: streaming O(entries + generated bytes) fixture work; history replay includes the full expanded CRDT cost. Large aggregate and real-disk campaigns belong on the locked perf VMs, not this shared build host. ## Failure → effect → detection → recovery → test | Failure | Effect / detection | Recovery | Test | |---|---|---|---| | Either disk persists independent subset | Specification retains pair bytes once prepared, never invents HEAD | Model replay | `pair_bytes_and_auth_survive_independent_disk_orders` | | Late fd/mmap/alias write after displacement | Specification retains old inode's bytes | Retained names remain reachable | `displaced_inode_retention_covers_late_fd_mmap_and_alias_writes` | | Mixed-generation quota walk | Reject as admission evidence | Repeat one complete generation | `mixed_root_quota_observations_never_authorize_admission` | | Fixture truncates or history replay differs | Generator verification fails | Regenerate in absent directory | `tiering-scale` workload checks | | Metadata-only run mistaken for qualification | Explicit mode and validation ledger | Run real workload on perf VM | docs/plans/0761-slice-1-validation.md | ## Validation Rebased unchanged onto C `9c8c287357240912daf895c4166bf00f2212ce4a`. Current D head `659b0f4e420f6272e801ea78eba8fe10d66c1ca5`. Range-diff confirms both D-only patches are unchanged. **PASS**: fmt, strict clippy, and full workspace tests on dev-01 through Nix/slot. **1,208 passed, zero failed, 20 ignored**, exit 0. `RUST_TEST_THREADS=2`; evidence `~/tier-d-r2-final6.{log,exit}`. C and D ran sequentially. ```sh cargo fmt --all --check cargo clippy -j4 --workspace --all-targets --locked -- -D warnings RUST_TEST_THREADS=2 cargo test -j4 --workspace --locked --no-fail-fast -- --nocapture ``` ## Decisions and remaining risks These are functional fixtures and independent models, not certification of real paired replay, auth/HEAD protocols, retained-inode mutation recipes or quota admission. History-heavy open, 15 ms successful opens, RSS/FD/watch limits, XFS/ext4 comparisons, SIGKILL and VM power cuts remain unqualified. No production data, merge, deployment, provisioning or cutover was performed. The prior shared-host history fixture attempt is retained as incomplete evidence, not relabeled as passing. Stack: [A #790](https://git.kayg.org/kayg/calternal/pulls/790) → [B #791](https://git.kayg.org/kayg/calternal/pulls/791) → [C #792](https://git.kayg.org/kayg/calternal/pulls/792) → [D #793](https://git.kayg.org/kayg/calternal/pulls/793). Final round-3 review follows completion of the exact-head gates; nothing merged.
D: point the validation record at the split PRs
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
ecf80b3b82
kayg commented 2026-10-31 17:40:25 +00:00
Author
Owner
Copy link

Independent review, round 2

Verdict: ready

No actionable P0–P3 findings in D's added qualification fixture, model, and validation ledger. The code and docs distinguish specification models from production protocols, and explicitly leave crash/power-cut, successful-open, RSS, and FO1 latency gates unqualified. The tiering-scale fixtures check replay and structural counts without claiming production latency qualification. D is ready in its stated scope; the stack remains downstream of C.

## Independent review, round 2 **Verdict: ready** No actionable P0–P3 findings in D's added qualification fixture, model, and validation ledger. The code and docs distinguish specification models from production protocols, and explicitly leave crash/power-cut, successful-open, RSS, and FO1 latency gates unqualified. The `tiering-scale` fixtures check replay and structural counts without claiming production latency qualification. D is ready in its stated scope; the stack remains downstream of C.
kayg commented 2026-10-31 21:32:33 +00:00
Author
Owner
Copy link

Independent review, round 3 (final)

Reviewed D at 659b0f4e420f6272e801ea78eba8fe10d66c1ca5, round-2 comment 71075 and the final status table. Verdict: ready in its qualification-fixture/specification-model scope.

Round-2 item Final status Does the evidence demonstrate it?
D ready for scope, no round-2 findings Unchanged; no regression found. D adds the fixture tool, independent models, test-module registration and validation ledger over final C. Source review confirms fixture reuse is refused, metadata-only work is labelled, history is decoded/applied and compared, and the ledger distinguishes models from implemented protocols. The recorded exact-head workspace gate passes.
Inherited A/B/C findings Resolved, as assessed in the final #790/#791/#792 reviews. Independent downstream r2_ run: 22 passed, zero failed.

New findings, P0–P3: none. The models remain specifications for later protocols; they do not certify production paired recovery, auth/HEAD, retained-inode mutation or quota admission.

Independent command on dev-01, in this exact D checkout inside Nix through ~/bin/slot:

cargo test -j4 --workspace --locked r2_ -- --test-threads=2

Evidence: ~/tier-r3-independent.{log,exit}, exit 0. I inspected the exact-head ~/tier-d-r2-final6.log and exit 0; I did not independently rerun the full workspace suite or large fixture workloads. No code pushed or production access.

Open gates remain open: C's final isolated 100k latency/RSS run, history-heavy production opens, real disk/crash/power-cut campaigns and FO1 qualification. C's pending performance run is a gate, not a new finding in D. This verdict does not authorize cutover or certify those measurements.

# Independent review, round 3 (final) Reviewed D at `659b0f4e420f6272e801ea78eba8fe10d66c1ca5`, round-2 comment 71075 and the final status table. **Verdict: ready** in its qualification-fixture/specification-model scope. | Round-2 item | Final status | Does the evidence demonstrate it? | |---|---|---| | D ready for scope, no round-2 findings | **Unchanged; no regression found.** D adds the fixture tool, independent models, test-module registration and validation ledger over final C. | Source review confirms fixture reuse is refused, metadata-only work is labelled, history is decoded/applied and compared, and the ledger distinguishes models from implemented protocols. The recorded exact-head workspace gate passes. | | Inherited A/B/C findings | **Resolved**, as assessed in the final #790/#791/#792 reviews. | Independent downstream `r2_` run: 22 passed, zero failed. | New findings, P0–P3: **none**. The models remain specifications for later protocols; they do not certify production paired recovery, auth/HEAD, retained-inode mutation or quota admission. Independent command on dev-01, in this exact D checkout inside Nix through `~/bin/slot`: ```sh cargo test -j4 --workspace --locked r2_ -- --test-threads=2 ``` Evidence: `~/tier-r3-independent.{log,exit}`, exit 0. I inspected the exact-head `~/tier-d-r2-final6.log` and exit 0; I did not independently rerun the full workspace suite or large fixture workloads. No code pushed or production access. Open gates remain open: C's final isolated 100k latency/RSS run, history-heavy production opens, real disk/crash/power-cut campaigns and FO1 qualification. C's pending performance run is a gate, not a new finding in D. This verdict does not authorize cutover or certify those measurements.
kayg commented 2026-11-03 12:18:52 +00:00
Author
Owner
Copy link

D on dev 41f2efa3: restack, resolutions, qualification fixtures

New head d422d0ca95499937ef5bbc30fc4ba72cbf91aada (was 89989e62). D's own commits were replayed onto C's new head 487e0589. 0200eb7a (rustfmt module order) was dropped as already upstream, since C now carries it. git diff 89989e62 d422d0ca is empty: same tree, so the earlier gates at 89989e62 apply unchanged.

Resolution: crates/notes/src/lib.rs keeps dev's capture_tests and this PR's tiering_model test modules (one conflict, resolved to the old D head's exact text). D touches no read, executor, watcher or restart code.

Qualification fixtures (tiering-scale, dev-01, ~/bin/slot, fixture dirs on disk under $HOME, TMPDIR=~/tiering-tmp)

Workload Build Result
ids debug pass: 1k/10k/100k identities, 20/200/2,000 completed pages, 0 pending, 3.4 s
inventory --metadata-only debug pass: 1,000,000 entries, 10,000 successful opens, 0 pending, 67.5 s
aggregate --metadata-only debug pass: 100 people, 10,000,000 notes, 10 active (1,000,000 active notes), 1,242 s
history 100k × 64 B debug and release pass: serialized 7,833,947 B, checkpoint 6,400,014 B, expanded text 6,400,000 B, full replay equal; release ≈ 6 min
history 100k × 4 KiB debug and release not completed: stopped by me (see below)
blocks release pass: 10,000 repeated-block lines, 1,020,000 B, 1 completed open
moves --metadata-only release pass: 10,000 entries, physical folder move completed (app folder intents remain slice 2)

The release tiering-scale came from the same tree (cargo build --release -p calternal-perf --bins at 89989e62, identical to d422d0ca).

history 100k × 4 KiB is unqualified. The generation is O(n²): each append names the previous clock as its origin, and yrs merges the appends into one growing block, so every update costs more than the last. Release had written 45,237 of 100,000 updates after 33 min, at ~13 updates/s and falling. Projection: ~1.9 h more to generate, plus a replay of the same cost (~2.4 h), ~5 h in total on one core. The debug run would hit its 3 h unit limit before replay. Neither run could finish, so I stopped both (own units only). No earlier run on dev-01 has a history edits=… edit_bytes=4096 line either.

This is the fixture showing what this PR already lists as unqualified: history-heavy open over 100k-edit histories. It is not a rebase effect, because the D tree is unchanged. Suggested follow-up: make the 4 KiB case a perf-VM job with a measured time budget, or check whether production rooms hit the same per-append cost. I did not change the fixture or any limit.

Stack gate table, perf and the 200-append result: see the gate comment below; the prod-copy dry run is on #790.

## D on dev `41f2efa3`: restack, resolutions, qualification fixtures New head **`d422d0ca95499937ef5bbc30fc4ba72cbf91aada`** (was `89989e62`). D's own commits were replayed onto C's new head `487e0589`. `0200eb7a` (rustfmt module order) was dropped as already upstream, since C now carries it. `git diff 89989e62 d422d0ca` is **empty**: same tree, so the earlier gates at `89989e62` apply unchanged. **Resolution:** `crates/notes/src/lib.rs` keeps dev's `capture_tests` and this PR's `tiering_model` test modules (one conflict, resolved to the old D head's exact text). D touches no read, executor, watcher or restart code. ### Qualification fixtures (`tiering-scale`, dev-01, `~/bin/slot`, fixture dirs on disk under `$HOME`, `TMPDIR=~/tiering-tmp`) | Workload | Build | Result | |---|---|---| | `ids` | debug | **pass**: 1k/10k/100k identities, 20/200/2,000 completed pages, 0 pending, 3.4 s | | `inventory --metadata-only` | debug | **pass**: 1,000,000 entries, 10,000 successful opens, 0 pending, 67.5 s | | `aggregate --metadata-only` | debug | **pass**: 100 people, 10,000,000 notes, 10 active (1,000,000 active notes), 1,242 s | | `history` 100k × 64 B | debug and release | **pass**: serialized 7,833,947 B, checkpoint 6,400,014 B, expanded text 6,400,000 B, full replay equal; release ≈ 6 min | | `history` 100k × 4 KiB | debug and release | **not completed**: stopped by me (see below) | | `blocks` | release | **pass**: 10,000 repeated-block lines, 1,020,000 B, 1 completed open | | `moves --metadata-only` | release | **pass**: 10,000 entries, physical folder move completed (app folder intents remain slice 2) | The release `tiering-scale` came from the same tree (`cargo build --release -p calternal-perf --bins` at `89989e62`, identical to `d422d0ca`). **`history` 100k × 4 KiB is unqualified.** The generation is O(n²): each append names the previous clock as its origin, and yrs merges the appends into one growing block, so every update costs more than the last. Release had written 45,237 of 100,000 updates after 33 min, at ~13 updates/s and falling. Projection: ~1.9 h more to generate, plus a replay of the same cost (~2.4 h), ~5 h in total on one core. The debug run would hit its 3 h unit limit before replay. Neither run could finish, so I stopped both (own units only). No earlier run on dev-01 has a `history edits=… edit_bytes=4096` line either. This is the fixture showing what this PR already lists as unqualified: history-heavy open over 100k-edit histories. It is not a rebase effect, because the D tree is unchanged. Suggested follow-up: make the 4 KiB case a perf-VM job with a measured time budget, or check whether production rooms hit the same per-append cost. I did not change the fixture or any limit. Stack gate table, perf and the 200-append result: see the gate comment below; the prod-copy dry run is on #790.
kayg commented 2026-11-03 12:18:53 +00:00
Author
Owner
Copy link

Gate table — stack on dev 41f2efa3 (takeover after the Codex usage limit)

Heads: #790 A 91c034eb → #791 B efcaaaf7 → #792 C 487e0589 → #793 D d422d0ca. Base: dev 41f2efa387b74175971a838f602535089d0d5fb4 (what prod runs). Nothing merged.

Gate (dev-01, Nix, ~/bin/slot) A 91c034eb B efcaaaf7 C 487e0589 D d422d0ca
cargo fmt --all --check pass pass pass pass
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings pass pass pass pass ¹
cargo test -j4 --workspace --locked -- --test-threads=2 1,495 passed, 0 failed, 22 ignored 1,504 / 0 / 22 1,537 / 0 / 22 ² 1,540 / 0 / 22 ¹

¹ Run by the earlier agent at 89989e62; git diff 89989e62 d422d0ca is empty (same tree).
² Run at 896babb4; 487e0589 differs only by the order of two #[cfg(test)] mod lines (cargo fmt).

Perf VMs (binaries built on dev-01 in release at the D tree, relayed through the control host, SHA-256 checked on both perf VMs: C 2006ab8e…, dev 724535bd…, harness 980045d9…):

Gate Head (C/D) dev 41f2efa3 Limit Result
cold.index, 3 cold pairs, perf-01, p95 = max of 3 25.0 s (22.6 / 25.0 / 24.3) 20.9 s (20.9 / 18.6 / 20.2) ≤ 1.25× dev and ≤ 60 s pass, 1.196× (was 1.248×)
cold.mem_peak (harness), 3 cold runs, max 227 MB 291 MB not worse than dev pass, 0.78× dev (was 175 vs 186 MiB = 0.94×)
server VmHWM, sampled every 200 ms, max over 4 samples 227.5 MiB 291.5 MiB — head lower
open.note@100000 p95, warm pair 12.4 ms 10.6 ms ≤ 15 ms pass
idle.rss@100000, warm pair 60.6 MB 71.8 MB ≤ 100 MB pass
200 sequential notes.append, 30 alternating pairs, perf-02, median total 1.578 s 1.595 s within 1% of dev pass, −1.09% (pair-ratio median 0.992)

Protocol unchanged from the fab01191 run: ~/bin/perf-lock ~/bin/slot run-perf.sh, per sample calternal-perf --sizes 100k --runs 1 --cold-runs {1,1,1,0} --seed 1 --idle-secs 60 --budgets {"open.note@100000":{"p95":15},"idle.rss@100000":{"p95":100}}, C then dev, same VM, no page-cache drop; all 8 reports have no scenario failures. No limit was changed.

Notes:

  • Absolute cold times rose for both sides against the fab01191 run (dev a55521f9 12.6 s → dev 41f2efa3 20.9 s on the same harness and VM), so only the ratio is comparable. The absolute memory numbers are from a different source (harness cold.mem_peak on perf-01 vs one dev-01 sample earlier); the ratio to dev improved.
  • The append gate is noisy: single pairs range 0.82–1.48×. A first 5-pair run read +1.15%; with 30 pairs the median is −1.09%. A 1% gate needs ≥ 30 pairs.
  • Reports, logs and scripts: control host ~/Developer/perf-results/pr792-41f2efa3/. Perf VM binaries and work dirs deleted.
## Gate table — stack on dev `41f2efa3` (takeover after the Codex usage limit) Heads: #790 A `91c034eb` → #791 B `efcaaaf7` → #792 C `487e0589` → #793 D `d422d0ca`. Base: dev `41f2efa387b74175971a838f602535089d0d5fb4` (what prod runs). Nothing merged. | Gate (dev-01, Nix, `~/bin/slot`) | A `91c034eb` | B `efcaaaf7` | C `487e0589` | D `d422d0ca` | |---|---|---|---|---| | `cargo fmt --all --check` | pass | pass | pass | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | pass | pass | pass ¹ | | `cargo test -j4 --workspace --locked -- --test-threads=2` | 1,495 passed, 0 failed, 22 ignored | 1,504 / 0 / 22 | 1,537 / 0 / 22 ² | 1,540 / 0 / 22 ¹ | ¹ Run by the earlier agent at `89989e62`; `git diff 89989e62 d422d0ca` is empty (same tree). ² Run at `896babb4`; `487e0589` differs only by the order of two `#[cfg(test)] mod` lines (`cargo fmt`). **Perf VMs** (binaries built on dev-01 in release at the D tree, relayed through the control host, SHA-256 checked on both perf VMs: C `2006ab8e…`, dev `724535bd…`, harness `980045d9…`): | Gate | Head (C/D) | dev `41f2efa3` | Limit | Result | |---|---:|---:|---|---| | cold.index, 3 cold pairs, perf-01, p95 = max of 3 | 25.0 s (22.6 / 25.0 / 24.3) | 20.9 s (20.9 / 18.6 / 20.2) | ≤ 1.25× dev and ≤ 60 s | **pass, 1.196×** (was 1.248×) | | cold.mem_peak (harness), 3 cold runs, max | 227 MB | 291 MB | not worse than dev | **pass**, 0.78× dev (was 175 vs 186 MiB = 0.94×) | | server VmHWM, sampled every 200 ms, max over 4 samples | 227.5 MiB | 291.5 MiB | — | head lower | | open.note@100000 p95, warm pair | 12.4 ms | 10.6 ms | ≤ 15 ms | pass | | idle.rss@100000, warm pair | 60.6 MB | 71.8 MB | ≤ 100 MB | pass | | 200 sequential `notes.append`, 30 alternating pairs, perf-02, median total | 1.578 s | 1.595 s | within 1% of dev | **pass, −1.09%** (pair-ratio median 0.992) | Protocol unchanged from the `fab01191` run: `~/bin/perf-lock ~/bin/slot run-perf.sh`, per sample `calternal-perf --sizes 100k --runs 1 --cold-runs {1,1,1,0} --seed 1 --idle-secs 60 --budgets {"open.note@100000":{"p95":15},"idle.rss@100000":{"p95":100}}`, C then dev, same VM, no page-cache drop; all 8 reports have no scenario failures. No limit was changed. Notes: - Absolute cold times rose for **both** sides against the `fab01191` run (dev a55521f9 12.6 s → dev 41f2efa3 20.9 s on the same harness and VM), so only the ratio is comparable. The absolute memory numbers are from a different source (harness `cold.mem_peak` on perf-01 vs one dev-01 sample earlier); the ratio to dev improved. - The append gate is noisy: single pairs range 0.82–1.48×. A first 5-pair run read +1.15%; with 30 pairs the median is −1.09%. A 1% gate needs ≥ 30 pairs. - Reports, logs and scripts: control host `~/Developer/perf-results/pr792-41f2efa3/`. Perf VM binaries and work dirs deleted.
kayg commented 2026-11-03 16:00:21 +00:00
Author
Owner
Copy link

yrs squash fix: already fixed upstream, nothing to report.

  • yrs 0.28.0 (latest release, 2026-09-17) still has the bug: ItemPtr::try_squash recounts the merged string (self.len = self.content.len(OffsetKind::Utf16)).
  • y-crdt/y-crdt main already has our one-line fix (self.len += other.len), from PR https://github.com/y-crdt/y-crdt/pull/737 ("an item's UTF-16 length without counting its string", fixes https://github.com/y-crdt/y-crdt/issues/668), merged 2026-11-03 07:54 UTC, merge commit 26cc6ae3b8722682ee477840ac0f3e976db521c6.
  • No issue or PR opened on our side. No release contains the fix yet, so the vendored copy (af4f100b) stays until a yrs release after 0.28.0 ships; then drop vendor/yrs.
  • I did not run a reproduction: the upstream code and PR already cover it, so there was nothing new to report.
**yrs squash fix: already fixed upstream, nothing to report.** - yrs 0.28.0 (latest release, 2026-09-17) still has the bug: `ItemPtr::try_squash` recounts the merged string (`self.len = self.content.len(OffsetKind::Utf16)`). - y-crdt/y-crdt `main` already has our one-line fix (`self.len += other.len`), from PR https://github.com/y-crdt/y-crdt/pull/737 ("an item's UTF-16 length without counting its string", fixes https://github.com/y-crdt/y-crdt/issues/668), merged 2026-11-03 07:54 UTC, merge commit 26cc6ae3b8722682ee477840ac0f3e976db521c6. - No issue or PR opened on our side. No release contains the fix yet, so the vendored copy (af4f100b) stays until a yrs release after 0.28.0 ships; then drop `vendor/yrs`. - I did not run a reproduction: the upstream code and PR already cover it, so there was nothing new to report.
kayg commented 2026-11-03 21:38:05 +00:00
Author
Owner
Copy link

#793 fixtures and appends after the fixes (branch perf/tiering-761-fix)

History is O(1) per append now

Cause: not this stack. yrs 0.28.0 try_squash counts the whole merged string again after every append, so a run of appends costs the block so far each time (O(n²)). Fix is upstream (y-crdt#737, merged 2026-11-03, unreleased) and is in its own PR #899 ([patch.crates-io] to the merged commit). This branch carries a vendored one-line copy of the same fix until #899 lands (af4f100b).

tiering-scale history (release, dev-01) before (d422d0ca) after
100k × 64 B ≈ 6–11 min passes
100k × 4 KiB 45k of 100k in 33 min; projected ≈ 5 h passes in 7.4 s (whole command incl. both sizes 8.5 s): serialized 411,267,893 B, checkpoint 409,600,015 B, expanded text 409,600,000 B, completed_replays=1

(TIERING_HISTORY_EDITS and progress lines added to the tool in 7e196167, for experiments only.)

Every #793 fixture, release build of this branch (tree 2473b998), dev-01

Workload Result
ids pass: 1k/10k/100k identities, 20/200/2,000 pages, 0 pending, 2.3 s
inventory --metadata-only pass: 1,000,000 entries, 10,000 opens, 0 pending, 42.2 s, peak RSS 11.4 MiB
aggregate --metadata-only pass: 100 people, 10,000,000 notes, 10 active (1,000,000 active notes), 551 s, peak RSS 11.4 MiB
history pass (above), peak RSS 2.0 GiB (two full documents held by the tool)
blocks pass: 10,000 lines, checkpoint 1,020,020 B
moves --metadata-only pass: 10,000 entries, 0.4 s

200 sequential notes.append, 30 alternating pairs, perf-02, CALTERNAL_NOTES_SAVE_MS=15

dev 41f2efa3 this branch (h6) delta
median total 1.510 s 1.515 s +0.31% (limit 1%)

Not measured / open

Peak RSS at 1M notes, and a 5-pair cold/open.note re-run, need the perf lock (held by other jobs for hours). The cold-index, first-page and open.note gaps are described on #792; they remain.

## #793 fixtures and appends after the fixes (branch `perf/tiering-761-fix`) ### History is O(1) per append now Cause: not this stack. yrs 0.28.0 `try_squash` counts the whole merged string again after every append, so a run of appends costs the block so far each time (O(n²)). Fix is upstream (y-crdt#737, merged 2026-11-03, unreleased) and is in its own PR #899 (`[patch.crates-io]` to the merged commit). This branch carries a vendored one-line copy of the same fix until #899 lands (`af4f100b`). | `tiering-scale history` (release, dev-01) | before (`d422d0ca`) | after | |---|---:|---:| | 100k × 64 B | ≈ 6–11 min | passes | | 100k × 4 KiB | 45k of 100k in 33 min; projected ≈ 5 h | **passes in 7.4 s** (whole command incl. both sizes 8.5 s): serialized 411,267,893 B, checkpoint 409,600,015 B, expanded text 409,600,000 B, `completed_replays=1` | (`TIERING_HISTORY_EDITS` and progress lines added to the tool in `7e196167`, for experiments only.) ### Every #793 fixture, release build of this branch (tree `2473b998`), dev-01 | Workload | Result | |---|---| | `ids` | pass: 1k/10k/100k identities, 20/200/2,000 pages, 0 pending, 2.3 s | | `inventory --metadata-only` | pass: 1,000,000 entries, 10,000 opens, 0 pending, 42.2 s, peak RSS 11.4 MiB | | `aggregate --metadata-only` | pass: 100 people, 10,000,000 notes, 10 active (1,000,000 active notes), 551 s, peak RSS 11.4 MiB | | `history` | pass (above), peak RSS 2.0 GiB (two full documents held by the tool) | | `blocks` | pass: 10,000 lines, checkpoint 1,020,020 B | | `moves --metadata-only` | pass: 10,000 entries, 0.4 s | ### 200 sequential `notes.append`, 30 alternating pairs, perf-02, `CALTERNAL_NOTES_SAVE_MS=15` | | dev `41f2efa3` | this branch (`h6`) | delta | |---|---:|---:|---:| | median total | 1.510 s | 1.515 s | **+0.31%** (limit 1%) | ### Not measured / open Peak RSS at 1M notes, and a 5-pair cold/open.note re-run, need the perf lock (held by other jobs for hours). The cold-index, first-page and open.note gaps are described on #792; they remain.
kayg commented 2026-11-04 10:26:03 +00:00
Author
Owner
Copy link

Option A (secondary indexes after the first fill): implemented, target not met — first page 2.5× dev, cold index 1.16× dev

Branch perf/tiering-761-fix. New commits (production change is fe04455e; the rest are test/format fixes): fe04455e build the projection's secondary indexes after the first fill · 4a5e2dc6 rustfmt · 362e6d1b, c0b06fb1 clippy fixes in the new tests · 43d81bc1 rustfmt. Head 43d81bc1. Nothing merged.

What changed

  • A fresh projection starts with the id key and entries_fold only. entries_fold is what the scan's ids_at_folds/id_at read while filling (scan.rs classify and identify), so it stays. entries_child (children by name) and entries_descendants (watcher, under_after) are left out.
  • When the scan finishes (scan::run end → Index::fill_done), the writer thread builds each missing index with one CREATE INDEX IF NOT EXISTS, one per idle slot (only when no append is waiting). A restart finds what is missing and builds it again (index/deferred.rs).
  • A legacy log import builds all indexes in the same transaction as the import (one pass each). Every later write keeps all existing indexes current as before.
  • Queries are not switched: with an index missing SQLite scans and sorts, with the same rows (the planner, not new code, decides). Crash safety: each build is one transaction; a kill leaves the index absent and the database whole (PRAGMA integrity_check = ok in the test).

Tests (index/deferred_tests.rs, 7 new)

fresh fill keeps only the fill indexes · scan signal builds both, children/descendants answers identical before and after, later put/remove keep them current, EXPLAIN QUERY PLAN uses entries_child · populated projection without the signal builds on restart · empty projection does not build before its first fill · imported log builds all at once · kill mid-build: a child process stops between CREATE INDEX and COMMIT, the parent kills it, checks the index is absent and integrity_check ok, restarts and checks both indexes plus 3,000 children/descendants/path_of answers.

Perf-01, 100k Home, 6 reps, rotated order (head / dev / dev copy "devB" as control), no page-cache drop, both perf locks held

Binaries built on dev-01 (release): head tree 4a5e2dc6 (later commits change only #[cfg(test)] code and formatting) sha256 f6ac38e1…, dev 41f2efa3 01522f72… (devB is a byte copy), harness 352d448a…. PERF_ONLY=list,query,journal,open, --runs 3 --cold-runs 1 --seed 1 --idle-secs 0. Medians over the 6 reps.

metric head dev dev copy head / dev control (devB / dev)
cold.first_page (ms) 4048 1624 1619 2.49× 1.00
cold.index (s) 17.2 14.9 15.4 1.16× 1.04
open.note p50 / p95 (ms) 0.92 / 3.80 0.75 / 3.43 0.77 / 3.50 1.22× / 1.11× 1.03 / 1.02
open.api_get p50 / p95 (ms) 6.5 / 27.2 6.1 / 9.2 6.5 / 10.3 1.08× / 2.95× 1.07 / 1.12
home.first_page (ms) 6.18 3.44 3.26 1.80× 0.95
home.page / page_next / page_title (ms) 0.98 / 0.77 / 0.80 1.06 / 0.78 / 0.84 0.95 / 0.74 / 0.81 0.93 / 0.99 / 0.94
home.page_tag (ms) 70.4 52.2 53.1 1.35× 1.02
home.page_big (ms) 23.6 18.6 18.3 1.27× 0.98
journal.page (ms) 2.99 2.49 2.51 1.20× 1.00
search.* not in this harness output (see note)
cold.mem_peak (MB) 240 291 291 0.82× 1.00
server VmHWM, 200 ms sampler (MB, median / max of 6) 240 / 243 291 / 294 291 / 293 0.83×
200 sequential notes.append, 30 pairs (CALTERNAL_NOTES_SAVE_MS=15) 1.490 s 1.480 s +0.62% (pair-ratio spread 0.94–1.02) control 15 pairs dev vs devB: +0.22%

Per-rep cold.index / first page: head 17.2 s/3.9 s, 17.5/4.3, 18.7/5.5, 16.8/3.9, 17.1/3.9, 17.3/4.2; dev 14.9/1.63, 14.8/1.62, 18.8/1.69, 15.1/1.66, 14.8/1.59, 14.7/1.59; control within ±3% of dev on every rep except one (cold.index 18.8 s).

Note on search: the scenario filter PERF_ONLY=list,query,journal,open did not emit search.first_query / search.query rows in these reports; I did not run a separate search pass.

Reading

  • Leaving two of the four indexes out of the first fill moved the first page only from 4.7–5.5 s to 4.0 s (4.0–5.5 over reps); dev is 1.6 s. Cold index is 1.16× dev (the previous pass measured 1.09–1.15×, so no gain either). The index maintenance was not the main cost of the first page.
  • Everything that reads the finished projection is still 1.2–1.35× dev (tag page, big page, journal page, open.note p50, first page of the list) in the same session where the control reads 0.95–1.04×. These come from the stack's design (paged reads), not from this change, and are regressions against dev by the 11-03 rule. Memory is the one win (0.82×).
  • Appends: +0.62% (limit 1%, control noise ±0.2%).
  • Not tried, my guesses for where the first-page time is: id TEXT PRIMARY KEY random-order inserts (two B-trees per row), the totals trigger running an UPDATE per inserted row, and one synchronous=FULL commit per 4,000-row group. I have not measured any of them (callgrind on a 10k cold start would split it in one pass). Option 2 from the earlier comment (acknowledge after the log fsync) is still the other route.
  • The stack is not mergeable on this evidence.

Gates (dev-01, ~/bin/slot, tree 43d81bc1)

Gate Result
cargo fmt --all --check pass
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings pass
cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2 1,547 passed, 0 failed, 22 ignored (was 1,540; +7 new)
#793 fixtures (tiering-scale release: ids, inventory, aggregate, history, blocks, moves) not run yet — dev-01 build slots have been held back by train bm14e since 09:30 CEST; my job is queued (tpf-fix). Result follows in a new comment.

Perf VM locks on perf-01 and perf-02 released (tpf4-hold stopped on both). Reports and scripts on the control host: /tmp/tpf/ (JSON per run in /tmp/tpf/reports).

## Option A (secondary indexes after the first fill): implemented, **target not met** — first page 2.5× dev, cold index 1.16× dev Branch `perf/tiering-761-fix`. New commits (production change is `fe04455e`; the rest are test/format fixes): `fe04455e` build the projection's secondary indexes after the first fill · `4a5e2dc6` rustfmt · `362e6d1b`, `c0b06fb1` clippy fixes in the new tests · `43d81bc1` rustfmt. Head `43d81bc1`. Nothing merged. ### What changed - A fresh projection starts with the id key and `entries_fold` only. `entries_fold` is what the scan's `ids_at_folds`/`id_at` read while filling (`scan.rs` classify and identify), so it stays. `entries_child` (children by name) and `entries_descendants` (watcher, `under_after`) are left out. - When the scan finishes (`scan::run` end → `Index::fill_done`), the writer thread builds each missing index with one `CREATE INDEX IF NOT EXISTS`, one per idle slot (only when no append is waiting). A restart finds what is missing and builds it again (`index/deferred.rs`). - A legacy log import builds all indexes in the same transaction as the import (one pass each). Every later write keeps all existing indexes current as before. - Queries are not switched: with an index missing SQLite scans and sorts, with the same rows (the planner, not new code, decides). Crash safety: each build is one transaction; a kill leaves the index absent and the database whole (`PRAGMA integrity_check` = ok in the test). ### Tests (`index/deferred_tests.rs`, 7 new) fresh fill keeps only the fill indexes · scan signal builds both, children/descendants answers identical before and after, later `put`/`remove` keep them current, `EXPLAIN QUERY PLAN` uses `entries_child` · populated projection without the signal builds on restart · empty projection does not build before its first fill · imported log builds all at once · **kill mid-build**: a child process stops between `CREATE INDEX` and `COMMIT`, the parent kills it, checks the index is absent and `integrity_check` ok, restarts and checks both indexes plus 3,000 children/descendants/`path_of` answers. ### Perf-01, 100k Home, 6 reps, rotated order (head / dev / dev copy "devB" as control), no page-cache drop, both perf locks held Binaries built on dev-01 (release): head tree `4a5e2dc6` (later commits change only `#[cfg(test)]` code and formatting) sha256 `f6ac38e1…`, dev `41f2efa3` `01522f72…` (devB is a byte copy), harness `352d448a…`. `PERF_ONLY=list,query,journal,open`, `--runs 3 --cold-runs 1 --seed 1 --idle-secs 0`. Medians over the 6 reps. | metric | head | dev | dev copy | head / dev | control (devB / dev) | |---|---:|---:|---:|---:|---:| | **cold.first_page** (ms) | **4048** | **1624** | 1619 | **2.49×** | 1.00 | | **cold.index** (s) | **17.2** | **14.9** | 15.4 | **1.16×** | 1.04 | | open.note p50 / p95 (ms) | 0.92 / 3.80 | 0.75 / 3.43 | 0.77 / 3.50 | 1.22× / 1.11× | 1.03 / 1.02 | | open.api_get p50 / p95 (ms) | 6.5 / 27.2 | 6.1 / 9.2 | 6.5 / 10.3 | 1.08× / 2.95× | 1.07 / 1.12 | | home.first_page (ms) | 6.18 | 3.44 | 3.26 | 1.80× | 0.95 | | home.page / page_next / page_title (ms) | 0.98 / 0.77 / 0.80 | 1.06 / 0.78 / 0.84 | 0.95 / 0.74 / 0.81 | 0.93 / 0.99 / 0.94 | | | home.page_tag (ms) | 70.4 | 52.2 | 53.1 | 1.35× | 1.02 | | home.page_big (ms) | 23.6 | 18.6 | 18.3 | 1.27× | 0.98 | | journal.page (ms) | 2.99 | 2.49 | 2.51 | 1.20× | 1.00 | | search.* | not in this harness output (see note) | | | | | | cold.mem_peak (MB) | 240 | 291 | 291 | 0.82× | 1.00 | | server VmHWM, 200 ms sampler (MB, median / max of 6) | 240 / 243 | 291 / 294 | 291 / 293 | 0.83× | | | 200 sequential `notes.append`, 30 pairs (`CALTERNAL_NOTES_SAVE_MS=15`) | 1.490 s | 1.480 s | | **+0.62%** (pair-ratio spread 0.94–1.02) | control 15 pairs dev vs devB: +0.22% | Per-rep cold.index / first page: head 17.2 s/3.9 s, 17.5/4.3, 18.7/5.5, 16.8/3.9, 17.1/3.9, 17.3/4.2; dev 14.9/1.63, 14.8/1.62, 18.8/1.69, 15.1/1.66, 14.8/1.59, 14.7/1.59; control within ±3% of dev on every rep except one (cold.index 18.8 s). Note on search: the scenario filter `PERF_ONLY=list,query,journal,open` did not emit `search.first_query` / `search.query` rows in these reports; I did not run a separate search pass. ### Reading - Leaving two of the four indexes out of the first fill moved the first page only from 4.7–5.5 s to **4.0 s** (4.0–5.5 over reps); dev is 1.6 s. Cold index is **1.16×** dev (the previous pass measured 1.09–1.15×, so no gain either). The index maintenance was not the main cost of the first page. - Everything that reads the finished projection is still 1.2–1.35× dev (tag page, big page, journal page, open.note p50, first page of the list) in the same session where the control reads 0.95–1.04×. These come from the stack's design (paged reads), not from this change, and are **regressions against dev by the 11-03 rule**. Memory is the one win (0.82×). - Appends: +0.62% (limit 1%, control noise ±0.2%). - Not tried, my guesses for where the first-page time is: `id TEXT PRIMARY KEY` random-order inserts (two B-trees per row), the `totals` trigger running an `UPDATE` per inserted row, and one `synchronous=FULL` commit per 4,000-row group. I have not measured any of them (callgrind on a 10k cold start would split it in one pass). Option 2 from the earlier comment (acknowledge after the log fsync) is still the other route. - **The stack is not mergeable on this evidence.** ### Gates (dev-01, `~/bin/slot`, tree `43d81bc1`) | Gate | Result | |---|---| | `cargo fmt --all --check` | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | | `cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2` | **1,547 passed, 0 failed, 22 ignored** (was 1,540; +7 new) | | #793 fixtures (`tiering-scale` release: ids, inventory, aggregate, history, blocks, moves) | **not run yet** — dev-01 build slots have been held back by train `bm14e` since 09:30 CEST; my job is queued (`tpf-fix`). Result follows in a new comment. | Perf VM locks on perf-01 and perf-02 released (`tpf4-hold` stopped on both). Reports and scripts on the control host: `/tmp/tpf/` (JSON per run in `/tmp/tpf/reports`).
kayg commented 2026-11-04 10:26:19 +00:00
Author
Owner
Copy link

Correction to the table above:

  • Search was not measured. The scenario filter I used (PERF_ONLY=list,query,journal,open) produced no search.* rows (checked in the JSON). The brief asked for search; that is still open. It needs another lock window with the scenario name the harness uses for search.
  • I wrote that the 1.2–1.35× deltas (tag page, big page, journal page, open.note p50, home first page) come "from the stack's design". I did not measure the cause. They are regressions against dev in this run, cause unknown.
Correction to the table above: - **Search was not measured.** The scenario filter I used (`PERF_ONLY=list,query,journal,open`) produced no `search.*` rows (checked in the JSON). The brief asked for search; that is still open. It needs another lock window with the scenario name the harness uses for search. - I wrote that the 1.2–1.35× deltas (tag page, big page, journal page, open.note p50, home first page) come "from the stack's design". I did not measure the cause. They are regressions against dev in this run, cause unknown.
kayg commented 2026-11-04 10:49:48 +00:00
Author
Owner
Copy link

#793 fixtures at 43d81bc1: not run. Train bm14e held dev-01's build slots for 2.5 h+ (~/.train.lock), so my tiering-scale release build never started; I stopped my queued job (tpf-fix) rather than leave it running. The previous fixture results (tree 2473b998, comment above) are the latest. The change here touches index.rs/scan.rs only through the deferred index build; the full workspace suite passed (1,547).

#793 fixtures at `43d81bc1`: **not run**. Train `bm14e` held dev-01's build slots for 2.5 h+ (`~/.train.lock`), so my `tiering-scale` release build never started; I stopped my queued job (`tpf-fix`) rather than leave it running. The previous fixture results (tree `2473b998`, comment above) are the latest. The change here touches `index.rs`/`scan.rs` only through the deferred index build; the full workspace suite passed (1,547).
kayg commented 2026-11-04 11:48:31 +00:00
Author
Owner
Copy link

Update on the #793 fixtures at 43d81bc1: still not run. My first fixture job (tpf-fix, started 09:21 CEST) ended without output and I do not know why; the second (tpf-fix2, 11:50) waited about 35 min with no ~/bin/slot slot while train bm14e held ~/.train.lock (it had run 2.5 h+), and I stopped it. Run tiering-scale DIR ids|inventory|aggregate|history|blocks|moves (release, --metadata-only where noted in #793) once the train ends; I did not do it, so the last fixture results are still those at 2473b998.

Update on the #793 fixtures at `43d81bc1`: still **not run**. My first fixture job (`tpf-fix`, started 09:21 CEST) ended without output and I do not know why; the second (`tpf-fix2`, 11:50) waited about 35 min with no `~/bin/slot` slot while train `bm14e` held `~/.train.lock` (it had run 2.5 h+), and I stopped it. Run `tiering-scale DIR ids|inventory|aggregate|history|blocks|moves` (release, `--metadata-only` where noted in #793) once the train ends; I did not do it, so the last fixture results are still those at `2473b998`.
kayg commented 2026-11-07 19:10:08 +00:00
Author
Owner
Copy link

housekeeping #1009: needs-decision.

Qualification fixtures depend on the superseded #790–#792 representation. Preserve fixture/model code for the v3 qualification lane; the latest comment also records its fixture rerun as incomplete.

original head: d422d0ca95499937ef5bbc30fc4ba72cbf91aada. against dev cd7f84432febfc63b788fbb9b9f9950452982d74, git cherry reports 71 unmatched and 0 equivalent patches. the landing audit lists each candidate missing commit and stable patch match in pr-793.md; unmatched patches can include superseded behavior.

left open, source branch unchanged. no rebase or new test claim for this decision lane.

housekeeping #1009: needs-decision. Qualification fixtures depend on the superseded #790–#792 representation. Preserve fixture/model code for the v3 qualification lane; the latest comment also records its fixture rerun as incomplete. original head: `d422d0ca95499937ef5bbc30fc4ba72cbf91aada`. against dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, git cherry reports 71 unmatched and 0 equivalent patches. the [landing audit](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5) lists each candidate missing commit and stable patch match in `pr-793.md`; unmatched patches can include superseded behavior. left open, source branch unchanged. no rebase or new test claim for this decision lane.
kayg commented 2026-11-07 21:21:11 +00:00
Author
Owner
Copy link

Superseded by tiering v3 (#968/#970/#1008). The branch and original tips are archived under archive/ (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.

Superseded by tiering v3 (#968/#970/#1008). The branch and original tips are archived under `archive/` (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.
kayg closed this pull request 2026-11-07 21:21:11 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!793
No description provided.