fix(notes): take the O(n^2) yrs append fix from upstream (y-crdt#737)

kayg commented

2026-11-03 15:18:42 +00:00

Owner

yrs 0.28.0 ItemPtr::try_squash recounts the whole merged string after every squash, so sequential appends cost the block so far each (O(n^2)). Found by the 100k x 4 KiB history fixture in #793 (45k of 100k in 33 min).

Fix is merged upstream (y-crdt#737, 26cc6ae3) but unreleased: [patch.crates-io] yrs = { git, rev } with a comment to drop it once a release after 0.28.0 ships. See the comment for the extra upstream changes in that range, gates and perf-VM numbers.

Not reported upstream; kayg decides.

🤖 Generated with Claude Code

yrs 0.28.0 `ItemPtr::try_squash` recounts the whole merged string after every squash, so sequential appends cost the block so far each (O(n^2)). Found by the 100k x 4 KiB history fixture in #793 (45k of 100k in 33 min). Fix is merged upstream (y-crdt#737, `26cc6ae3`) but unreleased: `[patch.crates-io] yrs = { git, rev }` with a comment to drop it once a release after 0.28.0 ships. See the comment for the extra upstream changes in that range, gates and perf-VM numbers. Not reported upstream; kayg decides. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kayg added 2 commits

2026-11-03 15:18:42 +00:00

fix(notes): vendor yrs with the append-length fix 1372b812dc

yrs 0.28.0 counts the whole merged string again after every squash of two
string items, so a run of appends costs the block so far each time (O(n^2)).
Vendored copy with the one-line fix: add the two lengths.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

test(notes): appends cost the same early and late, UTF-16 length stays exact

CI / server (pull_request) Has been cancelled

Details

CI / rust (pull_request) Has been cancelled

Details

CI / web (pull_request) Has been cancelled

Details

Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled

Details

Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled

Details

76759d9da5

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

kayg commented

2026-11-03 19:11:09 +00:00

Author

Owner

yrs append fix: gates and perf-VM before/after

Head f04165dc (patch = upstream y-crdt 26cc6ae3, PR #737, merged 2026-11-03, no release yet). Cargo.toml carries the comment "drop this patch once a yrs release after 0.28.0 ships".

Upstream diff v0.28.0..26cc6ae3 is more than the one-liner (18 commits, yrs/ only: block.rs, text.rs, undo.rs, update.rs, updates/decoder.rs, encoding/read.rs, store/doc/transaction). Besides the length fix it also: splits surrogate pairs into U+FFFD like Yjs (#714); rejects malformed v2/v1 updates with a new Error::Malformed (empty block lists, empty GC/Skip blocks); fixes selective undo/redo clearing and sticky-index follow-redone. No new dependencies (only a bench entry in Cargo.toml). Our workspace tests pass with all of it (below), but the malformed-update rejection and surrogate handling change behaviour on odd input. If kayg would rather take only the length fix, fix(notes): vendor yrs… (1372b812 in this branch's history) has the vendored one-line version.

Gates (dev-01, Nix, `~/bin/slot`)

Gate	Result
`cargo fmt --all --check`	pass
`cargo clippy -j4 --workspace --all-targets --locked -- -D warnings`	pass
`cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2` (at `6862a239`, same code as head bar rustfmt)	1,473 passed, 0 failed, 22 ignored
New test `long_runs_of_appends_do_not_go_quadratic` (6,000 × 4 KiB appends, 5 s hang guard)	stock yrs 0.28.0 at dev `41f2efa3`: FAILS, 33.0 s; patched: passes in 0.09 s
New test `merged_appends_keep_their_utf16_length` (surrogate pairs, combining marks, later edit offsets)	passes on both

(An earlier version compared the two halves of the run; it failed once at 3.2× on a loaded full-suite run, so I replaced it with the whole-run bound above. No wall-clock speed is asserted; speed is below.)

Perf VMs (release binaries built on dev-01 and relayed, sha256 checked on both perf VMs: yrs `calternal`, dev `calternal`)

200 sequential notes.append, 30 alternating pairs, perf-02 (append.py from the #792 gate, CALTERNAL_NOTES_SAVE_MS=15)

	dev `41f2efa3`	this PR	delta
median total	2.100 s	2.092 s	−0.40% (flat; pair-ratio spread 0.75–1.06)
median per append / p95	9.13 / 17.91 ms	9.18 / 17.77 ms	flat

Long-note and typing cases, perf-01, calternal-perf --sizes 100k --runs 3 --cold-runs 1 --idle-secs 0, default scenarios (rep 1 clean; rep 2 below)

Metric (p50 / p95)	dev rep 1	PR rep 1	dev rep 2	PR rep 2
`open.long_note` (50k-word note rebuilt from history), ms	105 / 109	83.9 / 87.9	127 / 295	77.3 / 80.0
`storm.typing` (10,000 edits, 4 clients), s	3.34 / 3.84	2.93 / 3.25	9.81 / 18.3	3.57 / 3.90
`storm.save_ack`, ms	8.39 / 19.5	7.51 / 17.4	20.6 / 56.8	10.0 / 22.3
`storm.edits_per_s`	2,998	3,412	1,019	2,803
`open.note`, ms	3.24 / 9.77	2.27 / 8.69	4.21 / 11.0	1.40 / 5.72
`cold.index`, s	22.0	21.0	28.4	16.3

Rep 2's dev run was disturbed (storm save_ack max 11.1 s, cold index 28 s against 22 s in rep 1), so only rep 1 is a clean pair; rep 2 is shown for completeness and points the same way. No metric regresses; long-note rebuild and typing are 10–20% faster. Not reported upstream; kayg decides.

## yrs append fix: gates and perf-VM before/after Head `f04165dc` (patch = upstream y-crdt `26cc6ae3`, PR #737, merged 2026-11-03, no release yet). `Cargo.toml` carries the comment "drop this patch once a yrs release after 0.28.0 ships". **Upstream diff v0.28.0..26cc6ae3 is more than the one-liner** (18 commits, `yrs/` only: block.rs, text.rs, undo.rs, update.rs, updates/decoder.rs, encoding/read.rs, store/doc/transaction). Besides the length fix it also: splits surrogate pairs into U+FFFD like Yjs (#714); rejects malformed v2/v1 updates with a new `Error::Malformed` (empty block lists, empty GC/Skip blocks); fixes selective undo/redo clearing and sticky-index follow-redone. No new dependencies (only a bench entry in `Cargo.toml`). Our workspace tests pass with all of it (below), but the malformed-update rejection and surrogate handling change behaviour on odd input. If kayg would rather take only the length fix, `fix(notes): vendor yrs…` (`1372b812` in this branch's history) has the vendored one-line version. ### Gates (dev-01, Nix, `~/bin/slot`) | Gate | Result | |---|---| | `cargo fmt --all --check` | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | | `cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2` (at `6862a239`, same code as head bar rustfmt) | **1,473 passed, 0 failed, 22 ignored** | | New test `long_runs_of_appends_do_not_go_quadratic` (6,000 × 4 KiB appends, 5 s hang guard) | stock yrs 0.28.0 at dev `41f2efa3`: **FAILS, 33.0 s**; patched: passes in 0.09 s | | New test `merged_appends_keep_their_utf16_length` (surrogate pairs, combining marks, later edit offsets) | passes on both | (An earlier version compared the two halves of the run; it failed once at 3.2× on a loaded full-suite run, so I replaced it with the whole-run bound above. No wall-clock speed is asserted; speed is below.) ### Perf VMs (release binaries built on dev-01 and relayed, sha256 checked on both perf VMs: yrs `calternal`, dev `calternal`) **200 sequential `notes.append`, 30 alternating pairs, perf-02** (`append.py` from the #792 gate, `CALTERNAL_NOTES_SAVE_MS=15`) | | dev `41f2efa3` | this PR | delta | |---|---:|---:|---:| | median total | 2.100 s | 2.092 s | **−0.40%** (flat; pair-ratio spread 0.75–1.06) | | median per append / p95 | 9.13 / 17.91 ms | 9.18 / 17.77 ms | flat | **Long-note and typing cases, perf-01, `calternal-perf --sizes 100k --runs 3 --cold-runs 1 --idle-secs 0`, default scenarios** (rep 1 clean; rep 2 below) | Metric (p50 / p95) | dev rep 1 | PR rep 1 | dev rep 2 | PR rep 2 | |---|---:|---:|---:|---:| | `open.long_note` (50k-word note rebuilt from history), ms | 105 / 109 | **83.9 / 87.9** | 127 / 295 | 77.3 / 80.0 | | `storm.typing` (10,000 edits, 4 clients), s | 3.34 / 3.84 | **2.93 / 3.25** | 9.81 / 18.3 | 3.57 / 3.90 | | `storm.save_ack`, ms | 8.39 / 19.5 | **7.51 / 17.4** | 20.6 / 56.8 | 10.0 / 22.3 | | `storm.edits_per_s` | 2,998 | 3,412 | 1,019 | 2,803 | | `open.note`, ms | 3.24 / 9.77 | 2.27 / 8.69 | 4.21 / 11.0 | 1.40 / 5.72 | | `cold.index`, s | 22.0 | 21.0 | 28.4 | 16.3 | Rep 2's dev run was disturbed (storm `save_ack` max 11.1 s, cold index 28 s against 22 s in rep 1), so only rep 1 is a clean pair; rep 2 is shown for completeness and points the same way. No metric regresses; long-note rebuild and typing are 10–20% faster. Not reported upstream; kayg decides.

kayg changed title from ~~fix(notes): vendor yrs with the O(n^2) append fix~~ to fix(notes): take the O(n^2) yrs append fix from upstream (y-crdt#737)

2026-11-03 19:11:10 +00:00

kayg referenced this pull request

2026-11-03 21:38:05 +00:00

storage: tiering qualification fixtures and specification models (#761 D) #793

kayg commented

2026-11-03 21:52:49 +00:00

Author

Owner

Switched the yrs fix from the upstream pin to the vendored copy. Head: 8484849c.

Why: the pin (26cc6ae3) is 18 commits past 0.28.0. Some change how unusual input is read, e.g. one rejects malformed updates with Error::Malformed. That could make an already stored document unopenable. The vendored copy is the 0.28.0 release plus only self.len += other.len in try_squash (same tree as 1372b812, see vendor/yrs/CALTERNAL.md).

Change (one commit, history not rewritten): [patch.crates-io] yrs = { path = "vendor/yrs" }, vendor/yrs excluded from the workspace, Cargo.lock back to a path source. The "drop once a release after 0.28.0 ships" comment and the regression tests stay.

Gates (dev-01, via ~/bin/slot, on 8484849c):

cargo fmt --check: pass
cargo clippy --workspace --all-targets -j4 -- -D warnings: pass
cargo test --workspace -j4: pass (124 test binaries ok, 0 failed)

Regression test timing (ydoc::squash_tests::long_runs_of_appends_do_not_go_quadratic, 5999 appends of 4 KiB, 3 runs each):

stock yrs 0.28.0: FAILED, 28.3 s / 28.6 s / 28.7 s (limit 5 s)
patched (vendored): passes, whole test 0.09 s / 0.19 s / 0.07 s
merged_appends_keep_their_utf16_length passes on both.

Not merged.

Switched the yrs fix from the upstream pin to the vendored copy. Head: 8484849c. **Why:** the pin (26cc6ae3) is 18 commits past 0.28.0. Some change how unusual input is read, e.g. one rejects malformed updates with `Error::Malformed`. That could make an already stored document unopenable. The vendored copy is the 0.28.0 release plus only `self.len += other.len` in `try_squash` (same tree as 1372b812, see `vendor/yrs/CALTERNAL.md`). **Change (one commit, history not rewritten):** `[patch.crates-io] yrs = { path = "vendor/yrs" }`, `vendor/yrs` excluded from the workspace, Cargo.lock back to a path source. The "drop once a release after 0.28.0 ships" comment and the regression tests stay. **Gates (dev-01, via `~/bin/slot`, on 8484849c):** - `cargo fmt --check`: pass - `cargo clippy --workspace --all-targets -j4 -- -D warnings`: pass - `cargo test --workspace -j4`: pass (124 test binaries ok, 0 failed) **Regression test timing** (`ydoc::squash_tests::long_runs_of_appends_do_not_go_quadratic`, 5999 appends of 4 KiB, 3 runs each): - stock yrs 0.28.0: FAILED, 28.3 s / 28.6 s / 28.7 s (limit 5 s) - patched (vendored): passes, whole test 0.09 s / 0.19 s / 0.07 s - `merged_appends_keep_their_utf16_length` passes on both. Not merged.

kayg referenced this pull request

2026-11-04 00:44:15 +00:00

Merge train BM14d #907

kayg referenced this pull request

2026-11-04 04:35:52 +00:00

Merge train BM14d #907

kayg referenced this pull request

2026-11-04 04:36:42 +00:00

Merge train BM14d #907

kayg referenced this pull request

2026-11-04 08:19:16 +00:00

Merge train BM14e #914