kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 803 Pull requests Projects Releases Packages Wiki Activity Actions 434

COLLAB: live document history — benchmark the cheapest footprint, then build (§61) #975

New issue
Open
opened 2026-10-03 06:38:49 +00:00 by kayg · 66 comments
kayg commented 2026-10-03 06:38:49 +00:00
Owner
Copy link

Owner decision (2026-10-03, #509 grill C1)

"Agreed but this needs to be benchmarked for the utmost cheapest footprint." Contract: docs/DESIGN.md §61. Context: §60 (Canvas), §10 (agents).

Goal

One history primitive for live (collaborative) documents. Canvas uses it first, then Notes. It must have the smallest disk, CPU and memory cost we can measure.

  • An append-only log of compressed collaboration updates. Each update records its author: a User, a guest, or an agent turn ID.
  • A compact snapshot on idle or every N updates. A retention rule folds old updates into snapshots.
  • Restore any point from Version history.
  • Per-author undo: reverse one author's changes (one agent turn, for example) after others kept editing. Skip elements someone else changed later, and report them.

Phase 1: measurement (do this first; it decides the design)

Follow the measurement protocol: perf VM root@10.69.69.63 via netbird ssh, hold flock /root/perf.lock, HDD-backed path, interleaved runs, at least 5 repetitions, median and p95.
Write the decision rule in the issue before any run. Suggested rule: pick the candidate with the lowest disk bytes per hour of editing, unless its p95 restore or undo exceeds 50 ms on the 5k-element canvas or its peak RSS exceeds 2x the baseline.

Candidates (all need a licence compatible with AGPL-3.0-only):

  1. Today's yrs (crates/calternal-collab): an update log plus periodic encode_state_as_update_v2 snapshots. Test v1 versus v2 encoding, zstd on and off, and snapshot interval N ∈ {100, 500, 2000}.
  2. yrs with deleted content dropped from snapshots (gc on) versus kept (gc off, needed for Restore). Measure what Restore costs under each.
  3. At least one alternative CRDT history engine with a compatible licence that has built-in history and shallow snapshots. Check licence, maturity and whether the browser client can stay on Yjs (a change of client CRDT is a large cost; count it).
  4. Plain JSON element diffs (no CRDT history; CRDT for live sync only), as a lower bound.

Workloads (generated, deterministic seeds, in bench/):

  • Canvas: 5k elements; 1 hour of edits (moves, resizes, freehand strokes with 200 points, text edits); 3 authors interleaved, one of them an "agent" that adds 300 elements in one turn.
  • Note: a 50 KB note with 2 hours of typing in bursts.

For each candidate and workload, measure:

  • bytes on disk after 1 h;
  • append CPU per update;
  • snapshot time;
  • Restore to 3 points (start, middle, end): time and peak RSS;
  • per-author undo of the agent turn after 10 minutes of other edits: time and peak RSS, and that it is correct;
  • cold open of the current version.

Post the table and the chosen design on this issue. Then stop, and wait for review before Phase 2.

Phase 2: implementation (after review)

  • A shared module in calternal-collab (reuse gate: extend, do not duplicate) with the chosen encoding. The author tag comes from the collaboration event path. There is one path for the web, API, CLI, MCP and WebMCP (§60).
  • Storage goes through calternal-fs and the Index. Never build paths from strings. Writes are crash-safe: a torn tail is dropped on open, never fatal.
  • API: list history points, Restore, per-author undo (preflight report first). Owner-only, as for Version history (§ sharing rules: shares never expose versions).
  • Bench profile in bench/ with the Phase 1 workload. Regression threshold in docs/perf/.
  • Tests: property tests for "Restore(point) == document state at point" and "undo(author) leaves other authors' changes intact"; a crash-tail test; a cross-user isolation case (#472 matrix).

Gates

cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test for the touched crates. Quote the output verbatim. Comments are docs: the module doc explains the encoding choice and cites this issue's numbers.

## Owner decision (2026-10-03, #509 grill C1) "Agreed but this needs to be benchmarked for the utmost cheapest footprint." Contract: `docs/DESIGN.md` §61. Context: §60 (Canvas), §10 (agents). ## Goal One history primitive for live (collaborative) documents. Canvas uses it first, then Notes. It must have the smallest disk, CPU and memory cost we can measure. - An append-only log of compressed collaboration updates. Each update records its author: a User, a guest, or an agent turn ID. - A compact snapshot on idle or every N updates. A retention rule folds old updates into snapshots. - **Restore** any point from Version history. - **Per-author undo:** reverse one author's changes (one agent turn, for example) after others kept editing. Skip elements someone else changed later, and report them. ## Phase 1: measurement (do this first; it decides the design) Follow the measurement protocol: perf VM `root@10.69.69.63` via `netbird ssh`, hold `flock /root/perf.lock`, HDD-backed path, interleaved runs, at least 5 repetitions, median and p95. **Write the decision rule in the issue before any run.** Suggested rule: pick the candidate with the lowest disk bytes per hour of editing, unless its p95 restore or undo exceeds 50 ms on the 5k-element canvas or its peak RSS exceeds 2x the baseline. Candidates (all need a licence compatible with AGPL-3.0-only): 1. Today's `yrs` (crates/calternal-collab): an update log plus periodic `encode_state_as_update_v2` snapshots. Test v1 versus v2 encoding, zstd on and off, and snapshot interval N ∈ {100, 500, 2000}. 2. `yrs` with deleted content dropped from snapshots (gc on) versus kept (gc off, needed for Restore). Measure what Restore costs under each. 3. At least one alternative CRDT history engine with a compatible licence that has built-in history and shallow snapshots. Check licence, maturity and whether the browser client can stay on Yjs (a change of client CRDT is a large cost; count it). 4. Plain JSON element diffs (no CRDT history; CRDT for live sync only), as a lower bound. Workloads (generated, deterministic seeds, in `bench/`): - Canvas: 5k elements; 1 hour of edits (moves, resizes, freehand strokes with 200 points, text edits); 3 authors interleaved, one of them an "agent" that adds 300 elements in one turn. - Note: a 50 KB note with 2 hours of typing in bursts. For each candidate and workload, measure: - bytes on disk after 1 h; - append CPU per update; - snapshot time; - Restore to 3 points (start, middle, end): time and peak RSS; - per-author undo of the agent turn after 10 minutes of other edits: time and peak RSS, and that it is correct; - cold open of the current version. Post the table and the chosen design on this issue. Then stop, and wait for review before Phase 2. ## Phase 2: implementation (after review) - A shared module in `calternal-collab` (reuse gate: extend, do not duplicate) with the chosen encoding. The author tag comes from the collaboration event path. There is one path for the web, API, CLI, MCP and WebMCP (§60). - Storage goes through `calternal-fs` and the Index. Never build paths from strings. Writes are crash-safe: a torn tail is dropped on open, never fatal. - API: list history points, Restore, per-author undo (preflight report first). Owner-only, as for Version history (§ sharing rules: shares never expose versions). - Bench profile in `bench/` with the Phase 1 workload. Regression threshold in `docs/perf/`. - Tests: property tests for "Restore(point) == document state at point" and "undo(author) leaves other authors' changes intact"; a crash-tail test; a cross-user isolation case (#472 matrix). ## Gates `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test` for the touched crates. Quote the output verbatim. Comments are docs: the module doc explains the encoding choice and cites this issue's numbers.
390-light.png
77 KiB
390-dark.png
93 KiB
820-light.png
244 KiB
820-dark.png
275 KiB
1440-light.png
577 KiB
1440-dark.png
598 KiB
history-975-1440-dark.png
460 KiB
history-975-1440-light.png
495 KiB
history-975-390-dark.png
88 KiB
history-975-390-light.png
73 KiB
history-975-820-dark.png
118 KiB
history-975-820-light.png
102 KiB
390-light.png 390-dark.png 820-light.png 820-dark.png 1440-light.png 1440-dark.png history-975-1440-dark.png history-975-1440-light.png history-975-390-dark.png history-975-390-light.png history-975-820-dark.png history-975-820-light.png
kayg commented 2026-10-03 06:45:57 +00:00
Author
Owner
Copy link

Phase 1 prep: research (2026-10-03)

Decision rule and shortlist below are the starting point; the measurement job confirms figures marked unverified before quoting them.

#975 Phase 1 research: history primitive for live documents (DESIGN §61)

Read-only research, 2026-10-03. No builds or runs. Numbers marked "published" come from the cited sources; every other figure is an estimate that Phase 1 must measure.

0. What the code does today (crates/calternal-collab)

  • yrs = "0.28.0" (workspace Cargo.toml:67), feature sync. Client: yjs ^13.6.33 (apps/web, packages/editor).
  • There is no update log and no history. stored.rs keeps one row per Note in the SQLite table
    note_collab_state(user_id, note_id, etag, epoch, seed, state). state is the full
    encode_state_as_update_v1(&StateVector::default()) (v1, default Doc options, so gc is ON).
    The row is a cache keyed by the Markdown etag; a changed file drops it (new epoch). Size cap 16 MiB.
  • session.rs writes v1 everywhere (sync frames, flush, conflict shadow). It already reads Yjs
    UndoManager markers from clients (paste/undo capture tests near session.rs:2013-2160), so
    origin-aware handling exists in the code.
  • Consequence for Phase 2: the log, snapshots, author tags and retention are all new. Extend
    stored.rs (reuse gate), do not add a parallel store. The table must move behind calternal-fs
    • Index, or stay in calternal-db: Phase 2 decides; the benchmark must measure both "SQLite BLOB
      rows" and "append-only segment file" for the winning encoding, because SQLite page overhead
      (4 KiB pages, overflow pages for >~4 KB blobs) can dominate small-update storage.

1. yrs / Yjs facts

1.1 v1 vs v2 encoding

  • v2 is a columnar encoding with run-length and string-table compression. Kevin Jahns: v2 is
    larger for small updates (single keystrokes) and better for large updates (whole document);
    slightly slower to encode. Published figure: v2 has ~50% overhead over the UTF-8 text on the
    text-trace datasets. Extreme published case: one transaction with 100k repetitive ops,
    v2 = 62 B vs v1 = 1,983,505 B (synthetic, not representative).
    Sources: https://discuss.yjs.dev/t/how-efficient-is-updatev2-encoding/1148 ,
    https://discuss.yjs.dev/t/is-ykeyvalue-still-necessary-with-yjs-update-version-2-optimizations/3516
  • Implication: per-update log rows are probably smaller as v1 (+ batch zstd); snapshots are
    probably smaller as v2. Measure both per role; do not assume one encoding for both.
  • zstd on a single keystroke update (~20-40 B) gains nothing or loses (frame header ~9+ B).
    zstd only pays when updates are batched (one compressed frame per N updates or per idle
    flush) or with a trained dictionary. Add both to the matrix.

1.2 gc on/off and Restore

  • Yjs Snapshot = state vector + delete set only; it carries no content. createDocFromSnapshot
    / yrs encode_state_from_snapshot rebuild a past state from the current doc, so the doc must
    keep deleted content: gc must be off on every replica for the whole life of the doc
    (published: https://discuss.yjs.dev/t/how-to-restore-the-document-by-snapshot/3815 ,
    https://discuss.yjs.dev/t/how-to-recover-to-the-specified-version/2301). In yrs this is
    Options { skip_gc: true, .. }.
  • Cost of gc off: every deleted Item keeps its content and its struct instead of collapsing into a
    GC range. For typing traces (automerge-paper: 182k inserts, 77k deletes, 105k final chars) that is
    roughly +30-75% of content bytes kept forever, plus live memory in the server doc and in every
    browser tab    . For Canvas (Excalidraw elements as Y.Map values, LWW per key) each move/resize
    overwrites a map value; with gc off every overwritten value (whole element props, freehand point
    arrays of 200 points) stays in the doc. This is the expensive case: 1 h of moves on 5k elements
    may grow the live doc by orders of magnitude. Measure live-doc bytes and RSS with gc off.
  • Key design observation (cheapest candidate): gc off is only needed for the Snapshot API.
    An update log with periodic full-state checkpoints gives Restore with gc ON:
    Restore(P) = load the newest checkpoint <= P, replay logged updates up to P into a temporary doc.
    The checkpoint is an exact state at its point (gc loses only deleted content, which is not part of
    that state). The live doc and clients stay gc on. Restore is then applied to the live room as a
    forward edit (diff restored content vs current, apply as new ops), so clients never reload and
    there is no epoch change. Cost: Restore time = checkpoint decode + replay of <= N updates, so N
    bounds restore latency. Retention folding = delete log rows between kept checkpoints (coarser
    points, never wrong state).
  • Known benchmark (dmonad/crdt-benchmarks, B4 = automerge-paper trace, Node 20, published):
    lib version encoded doc parse
    Yjs 13.6.11 159,929 B 39 ms
    ywasm (yrs) 0.9.3 159,929 B 16 ms
    Loro 0.10.1 258,228 B 13 ms
    Automerge 2.1.10 129,116 B 1,805 ms
    B4x100 (10.5M chars): Yjs 15,989,244 B, 327 MB JS heap, 2,622 ms parse; Loro 25.8 MB, 1,304 ms;
    Automerge skipped. Source: https://github.com/dmonad/crdt-benchmarks
    (Yjs numbers there are gc ON; no published gc-off figure.)
  • Loro's own native benchmark (Rust, M2 Max, 2024-10-18, automerge-paper): yrs 226,973 B, Automerge
    292,742 B (largest), Loro shallow snapshot 63,352 B (54,517 B compressed). Source:
    https://loro.dev/docs/performance/native , https://cn.loro.dev/docs/performance/docsize
    (site returned 403 to fetch; figures via search excerpts, verify before quoting in the issue).

2. Alternatives with AGPL-compatible licences

engine licence history / time travel maps (Canvas) text (Notes) Yjs client stays?
Loro 1.x MIT full history, checkout(frontiers), shallow snapshots (git shallow clone), built-in UndoManager yes yes (Fugue + Peritext rich text) no CRDT interop; "Loro protocol" only multiplexes Yjs and Loro rooms on one wire
Automerge 3 MIT full history always kept, view/fork at heads; columnar compressed in memory (3.0: >10x less RAM, Moby Dick paste 700 MB -> 1.3 MB; a doc that did not load in 17 h loads in 9 s) yes yes no
diamond-types ISC full history, very fast no (plain text only; JSON types in progress) yes no

Sources: https://www.npmjs.com/package/loro-crdt , https://loro.dev/blog/v1.0 ,
https://automerge.org/blog/automerge-3/ , https://docs.rs/diamond-types .

  • diamond-types is out: no map type, so no Canvas.
  • Server-only alternative engine with a Yjs browser: possible only as a mirror: apply each Yjs
    update to the yrs room, observe events (map key changes, text deltas), replay them into a Loro or
    Automerge doc with the author as peer. Costs: two CRDT docs in memory per room (>= 2x RSS),
    per-update translation CPU, IDs that do not match Yjs IDs (so per-author undo results must be
    translated back to Yjs ops), and a second source of truth that can drift. The live state is
    already in yrs, so the mirror adds cost on every edit to save bytes only in history.
    Recommendation: measure Loro standalone (history bytes, checkout time) as a reference point
    only; reject the mirror unless Loro is >3x smaller AND the yrs log fails the latency rule.
  • Switching the client: y-prosemirror -> loro-prosemirror exists (Loro team); Automerge has
    automerge-prosemirror. No maintained Excalidraw binding for Loro or Automerge found; Canvas would
    need a new binding. Also sync server, awareness, epoch/seed logic in stored.rs, offline tabs,
    the conflict shadow, adversarial tests. Estimate: a multi-week rewrite of calternal-collab plus
    editor bindings. Count it as a fixed cost in the decision; it should need a decisive win.

3. Per-author undo

  • Yjs/yrs UndoManager with tracked_origins is a selective undo: it undoes only
    transactions with tracked origins, and works after other origins edited (removes the tracked
    inserts by ID, re-inserts the tracked deletes, keeps other edits). yrs has the same
    (yrs::undo::UndoManager, Options { tracked_origins, capture_timeout_millis, .. }).
    Docs: https://docs.yjs.dev/api/undo-manager
  • Origins are not encoded in updates. The author tag must be stored per log row (User, guest,
    agent turn ID) by the server, from the collaboration event path.
  • Server-side recipe (Kevin Jahns, published in
    https://discuss.yjs.dev/t/is-there-a-way-to-revert-to-a-specific-version/379):
    1. temp doc (skip_gc on, temp only) <- checkpoint just before the turn;
    2. UndoManager tracking origin = turn ID, capture_timeout = 0;
    3. replay the turn's rows with origin = turn ID, then every later row with origin = its author;
    4. undo(); encode the temp doc diff vs the live state vector; apply to the live room as a normal
      update (no client reload, no epoch change).
      Restoring deleted content works because the temp doc replays from a checkpoint where that content
      still exists; the live doc can stay gc on.
  • "Skip elements someone else changed later, and report them" is not what UndoManager does by
    itself (it would still revert a moved element's other keys). For Canvas, do element-level
    logic: from the log, the set E_turn of element IDs the turn touched; the set E_later of element
    IDs that later rows from other authors touched; revert E_turn \ E_later to their pre-turn value
    (from the checkpoint replay), report E_turn ∩ E_later. Element = Excalidraw element ID (§60). For
    Notes, use UndoManager, then report blocks (block IDs) where later foreign edits overlap.
    Alternative without UndoManager: keep a clientID -> author map per room; walk the turn's update
    (items + delete set by ID) and build the inverse directly. Cheaper in RAM, more code; benchmark
    only if UndoManager misses the 50 ms rule.
  • An agent turn must use its own yrs client ID (or at least its own log rows) so its items are
    separable by ID.

4. Benchmark plan (proposal for the issue; post before any run)

4.1 Decision rule (write on the issue first)

  1. Hard limits (on the perf VM, HDD path, 5k-element Canvas and 50 KB Note):
    p95 Restore <= 50 ms, p95 per-author undo <= 50 ms, p95 cold open <= 50 ms,
    peak RSS <= 2x baseline (baseline = today's yrs, gc on, no history, same workload),
    undo correctness = 100% on the checker, live client doc not grown (gc stays on unless the
    candidate needs gc off, in which case add browser heap growth to the RSS limit).
  2. Among candidates that pass, pick the lowest disk bytes after 1 h (sum of Canvas + Note,
    median of 5).
  3. Tie (within 10%): lower append CPU per update, then lower peak RSS.
  4. Engine switch (Loro/Automerge) wins only if it passes and is >= 3x smaller on disk than the best
    yrs candidate, because the client rewrite is a fixed multi-week cost.

4.2 Workloads (bench/, deterministic seed)

  • C1 Canvas: 5k elements seeded; 1 h edit stream at realistic rate (for example 3 authors x
    ~1 op/s bursts): moves, resizes, freehand strokes of 200 points, text edits; author A3 = agent adds
    300 elements in one turn at t=20 min; then 10 min of edits by A1/A2, some touching agent elements.
  • N1 Note: 50 KB note, 2 h of typing in bursts (keystroke-level updates, as y-prosemirror sends
    them), 2 authors, one agent turn rewriting a section.
  • Updates are produced by real yrs transactions shaped like client traffic (Excalidraw-on-Yjs map
    writes, y-prosemirror XML ops), recorded once to a fixture, then replayed identically into every
    candidate.

4.3 Metrics table (one row per candidate x workload; median and p95 of >= 5 interleaved runs)

metric unit
bytes on disk after 1 h (log + snapshots; also after retention fold) B
bytes per hour of editing B/h
append CPU per update (encode + compress + write) us
snapshot time ms
Restore to start / middle / end: time ms
Restore peak RSS MB
per-author undo of the agent turn after 10 min: time ms
per-author undo peak RSS MB
undo correct (other authors intact, skipped set reported) yes/no
cold open of current version ms
live room RSS at end (server) MB
live doc encoded size sent to a new client B

4.4 Candidate configurations

Y = yrs 0.28 log + checkpoints, author tag per row, gc ON live doc:

  • Y1 v1 rows, no zstd, checkpoint v2, N in {100, 500, 2000}
  • Y2 v1 rows batched per flush (idle 1 s or 64 updates) + zstd-3, checkpoint v2+zstd, N in {100, 500, 2000}
  • Y3 v2 rows batched + zstd, checkpoint v2+zstd, N = best from Y2
  • Y4 Y2 + zstd trained dictionary (per document kind)
  • Y5 Y2 with rows merged (merge_updates_v1) per author per idle window before write (fewer, larger rows; loses intra-window points)
  • each Y in two stores: SQLite BLOB rows vs append-only segment file via calternal-fs
    G = yrs gc OFF + Yjs Snapshot (state vector + delete set) per history point, Restore via encode_state_from_snapshot:
  • G1 v2 full state + snapshots every N; also report browser heap growth (client must run gc off too)
    L = Loro 1.x standalone reference (MIT): full history snapshot, shallow snapshot + update blocks, checkout for Restore, Loro UndoManager per peer
    A = Automerge 3 standalone reference (MIT): save() + incremental save_incremental, view at heads
    J = lower bound: JSON element diffs (Canvas: changed elements per update; Note: text patches) + zstd, CRDT only for live sync
    Optional: Y-inv = Y2 with clientID -> author inverse undo (only if Y2 UndoManager misses 50 ms)

4.5 Expected outcome (hypothesis, to be tested)

Y2 (v1 batched + zstd rows, v2+zstd checkpoints, gc on, N ~500) should land near J on disk, pass
the latency rule because replay is bounded by N, and keep the browser untouched. G1 likely fails the
RSS rule on Canvas (overwritten freehand points kept forever). L may be smallest on disk but loses
on rule 4 (client switch, no Excalidraw binding).

## Phase 1 prep: research (2026-10-03) Decision rule and shortlist below are the starting point; the measurement job confirms figures marked unverified before quoting them. # #975 Phase 1 research: history primitive for live documents (DESIGN §61) Read-only research, 2026-10-03. No builds or runs. Numbers marked "published" come from the cited sources; every other figure is an estimate that Phase 1 must measure. ## 0. What the code does today (crates/calternal-collab) - `yrs = "0.28.0"` (workspace Cargo.toml:67), feature `sync`. Client: `yjs ^13.6.33` (apps/web, packages/editor). - **There is no update log and no history.** `stored.rs` keeps one row per Note in the SQLite table `note_collab_state(user_id, note_id, etag, epoch, seed, state)`. `state` is the full `encode_state_as_update_v1(&StateVector::default())` (v1, default `Doc` options, so gc is ON). The row is a cache keyed by the Markdown etag; a changed file drops it (new epoch). Size cap 16 MiB. - `session.rs` writes v1 everywhere (sync frames, flush, conflict shadow). It already reads Yjs UndoManager markers from clients (paste/undo capture tests near session.rs:2013-2160), so origin-aware handling exists in the code. - Consequence for Phase 2: the log, snapshots, author tags and retention are all new. Extend `stored.rs` (reuse gate), do not add a parallel store. The table must move behind `calternal-fs` + Index, or stay in calternal-db: Phase 2 decides; the benchmark must measure both "SQLite BLOB rows" and "append-only segment file" for the winning encoding, because SQLite page overhead (4 KiB pages, overflow pages for >~4 KB blobs) can dominate small-update storage. ## 1. yrs / Yjs facts ### 1.1 v1 vs v2 encoding - v2 is a columnar encoding with run-length and string-table compression. Kevin Jahns: v2 is **larger for small updates (single keystrokes)** and better for large updates (whole document); slightly slower to encode. Published figure: v2 has ~50% overhead over the UTF-8 text on the text-trace datasets. Extreme published case: one transaction with 100k repetitive ops, v2 = 62 B vs v1 = 1,983,505 B (synthetic, not representative). Sources: https://discuss.yjs.dev/t/how-efficient-is-updatev2-encoding/1148 , https://discuss.yjs.dev/t/is-ykeyvalue-still-necessary-with-yjs-update-version-2-optimizations/3516 - Implication: per-update log rows are probably smaller as **v1 (+ batch zstd)**; snapshots are probably smaller as **v2**. Measure both per role; do not assume one encoding for both. - zstd on a single keystroke update (~20-40 B) gains nothing or loses (frame header ~9+ B). zstd only pays when updates are **batched** (one compressed frame per N updates or per idle flush) or with a **trained dictionary**. Add both to the matrix. ### 1.2 gc on/off and Restore - Yjs `Snapshot` = state vector + delete set only; it carries no content. `createDocFromSnapshot` / yrs `encode_state_from_snapshot` rebuild a past state from the **current** doc, so the doc must keep deleted content: **gc must be off on every replica for the whole life of the doc** (published: https://discuss.yjs.dev/t/how-to-restore-the-document-by-snapshot/3815 , https://discuss.yjs.dev/t/how-to-recover-to-the-specified-version/2301). In yrs this is `Options { skip_gc: true, .. }`. - Cost of gc off: every deleted Item keeps its content and its struct instead of collapsing into a GC range. For typing traces (automerge-paper: 182k inserts, 77k deletes, 105k final chars) that is roughly +30-75% of content bytes kept forever, plus live memory in the server doc and in **every browser tab**. For Canvas (Excalidraw elements as Y.Map values, LWW per key) each move/resize overwrites a map value; with gc off every overwritten value (whole element props, freehand point arrays of 200 points) stays in the doc. This is the expensive case: 1 h of moves on 5k elements may grow the live doc by orders of magnitude. Measure live-doc bytes and RSS with gc off. - **Key design observation (cheapest candidate):** gc off is only needed for the *Snapshot API*. An **update log with periodic full-state checkpoints** gives Restore with gc ON: Restore(P) = load the newest checkpoint <= P, replay logged updates up to P into a temporary doc. The checkpoint is an exact state at its point (gc loses only deleted content, which is not part of that state). The live doc and clients stay gc on. Restore is then applied to the live room as a **forward edit** (diff restored content vs current, apply as new ops), so clients never reload and there is no epoch change. Cost: Restore time = checkpoint decode + replay of <= N updates, so N bounds restore latency. Retention folding = delete log rows between kept checkpoints (coarser points, never wrong state). - Known benchmark (dmonad/crdt-benchmarks, B4 = automerge-paper trace, Node 20, published): | lib | version | encoded doc | parse | |---|---|---|---| | Yjs | 13.6.11 | 159,929 B | 39 ms | | ywasm (yrs) | 0.9.3 | 159,929 B | 16 ms | | Loro | 0.10.1 | 258,228 B | 13 ms | | Automerge | 2.1.10 | 129,116 B | 1,805 ms | B4x100 (10.5M chars): Yjs 15,989,244 B, 327 MB JS heap, 2,622 ms parse; Loro 25.8 MB, 1,304 ms; Automerge skipped. Source: https://github.com/dmonad/crdt-benchmarks (Yjs numbers there are gc ON; no published gc-off figure.) - Loro's own native benchmark (Rust, M2 Max, 2024-10-18, automerge-paper): yrs 226,973 B, Automerge 292,742 B (largest), Loro shallow snapshot 63,352 B (54,517 B compressed). Source: https://loro.dev/docs/performance/native , https://cn.loro.dev/docs/performance/docsize (site returned 403 to fetch; figures via search excerpts, verify before quoting in the issue). ## 2. Alternatives with AGPL-compatible licences | engine | licence | history / time travel | maps (Canvas) | text (Notes) | Yjs client stays? | |---|---|---|---|---|---| | Loro 1.x | MIT | full history, `checkout(frontiers)`, shallow snapshots (git shallow clone), built-in UndoManager | yes | yes (Fugue + Peritext rich text) | no CRDT interop; "Loro protocol" only multiplexes Yjs and Loro rooms on one wire | | Automerge 3 | MIT | full history always kept, view/fork at heads; columnar compressed in memory (3.0: >10x less RAM, Moby Dick paste 700 MB -> 1.3 MB; a doc that did not load in 17 h loads in 9 s) | yes | yes | no | | diamond-types | ISC | full history, very fast | **no** (plain text only; JSON types in progress) | yes | no | Sources: https://www.npmjs.com/package/loro-crdt , https://loro.dev/blog/v1.0 , https://automerge.org/blog/automerge-3/ , https://docs.rs/diamond-types . - diamond-types is out: no map type, so no Canvas. - **Server-only alternative engine with a Yjs browser:** possible only as a mirror: apply each Yjs update to the yrs room, observe events (map key changes, text deltas), replay them into a Loro or Automerge doc with the author as peer. Costs: two CRDT docs in memory per room (>= 2x RSS), per-update translation CPU, IDs that do not match Yjs IDs (so per-author undo results must be translated back to Yjs ops), and a second source of truth that can drift. The live state is already in yrs, so the mirror adds cost on every edit to save bytes only in history. Recommendation: measure Loro **standalone** (history bytes, checkout time) as a reference point only; reject the mirror unless Loro is >3x smaller AND the yrs log fails the latency rule. - **Switching the client:** y-prosemirror -> loro-prosemirror exists (Loro team); Automerge has automerge-prosemirror. No maintained Excalidraw binding for Loro or Automerge found; Canvas would need a new binding. Also sync server, awareness, epoch/seed logic in stored.rs, offline tabs, the conflict shadow, adversarial tests. Estimate: a multi-week rewrite of calternal-collab plus editor bindings. Count it as a fixed cost in the decision; it should need a decisive win. ## 3. Per-author undo - Yjs/yrs `UndoManager` with `tracked_origins` is a **selective** undo: it undoes only transactions with tracked origins, and works after other origins edited (removes the tracked inserts by ID, re-inserts the tracked deletes, keeps other edits). yrs has the same (`yrs::undo::UndoManager`, `Options { tracked_origins, capture_timeout_millis, .. }`). Docs: https://docs.yjs.dev/api/undo-manager - Origins are **not encoded in updates**. The author tag must be stored per log row (User, guest, agent turn ID) by the server, from the collaboration event path. - Server-side recipe (Kevin Jahns, published in https://discuss.yjs.dev/t/is-there-a-way-to-revert-to-a-specific-version/379): 1. temp doc (skip_gc on, temp only) <- checkpoint just before the turn; 2. `UndoManager` tracking origin = turn ID, `capture_timeout = 0`; 3. replay the turn's rows with origin = turn ID, then every later row with origin = its author; 4. `undo()`; encode the temp doc diff vs the live state vector; apply to the live room as a normal update (no client reload, no epoch change). Restoring deleted content works because the temp doc replays from a checkpoint where that content still exists; the live doc can stay gc on. - "Skip elements someone else changed later, and report them" is **not** what UndoManager does by itself (it would still revert a moved element's other keys). For Canvas, do element-level logic: from the log, the set E_turn of element IDs the turn touched; the set E_later of element IDs that later rows from other authors touched; revert E_turn \ E_later to their pre-turn value (from the checkpoint replay), report E_turn ∩ E_later. Element = Excalidraw element ID (§60). For Notes, use UndoManager, then report blocks (block IDs) where later foreign edits overlap. Alternative without UndoManager: keep a clientID -> author map per room; walk the turn's update (items + delete set by ID) and build the inverse directly. Cheaper in RAM, more code; benchmark only if UndoManager misses the 50 ms rule. - An agent turn must use its own yrs client ID (or at least its own log rows) so its items are separable by ID. ## 4. Benchmark plan (proposal for the issue; post before any run) ### 4.1 Decision rule (write on the issue first) 1. Hard limits (on the perf VM, HDD path, 5k-element Canvas and 50 KB Note): p95 Restore <= 50 ms, p95 per-author undo <= 50 ms, p95 cold open <= 50 ms, peak RSS <= 2x baseline (baseline = today's yrs, gc on, no history, same workload), undo correctness = 100% on the checker, live client doc not grown (gc stays on unless the candidate needs gc off, in which case add browser heap growth to the RSS limit). 2. Among candidates that pass, pick the **lowest disk bytes after 1 h** (sum of Canvas + Note, median of 5). 3. Tie (within 10%): lower append CPU per update, then lower peak RSS. 4. Engine switch (Loro/Automerge) wins only if it passes and is >= 3x smaller on disk than the best yrs candidate, because the client rewrite is a fixed multi-week cost. ### 4.2 Workloads (bench/, deterministic seed) - **C1 Canvas:** 5k elements seeded; 1 h edit stream at realistic rate (for example 3 authors x ~1 op/s bursts): moves, resizes, freehand strokes of 200 points, text edits; author A3 = agent adds 300 elements in one turn at t=20 min; then 10 min of edits by A1/A2, some touching agent elements. - **N1 Note:** 50 KB note, 2 h of typing in bursts (keystroke-level updates, as y-prosemirror sends them), 2 authors, one agent turn rewriting a section. - Updates are produced by real yrs transactions shaped like client traffic (Excalidraw-on-Yjs map writes, y-prosemirror XML ops), recorded once to a fixture, then replayed identically into every candidate. ### 4.3 Metrics table (one row per candidate x workload; median and p95 of >= 5 interleaved runs) | metric | unit | |---|---| | bytes on disk after 1 h (log + snapshots; also after retention fold) | B | | bytes per hour of editing | B/h | | append CPU per update (encode + compress + write) | us | | snapshot time | ms | | Restore to start / middle / end: time | ms | | Restore peak RSS | MB | | per-author undo of the agent turn after 10 min: time | ms | | per-author undo peak RSS | MB | | undo correct (other authors intact, skipped set reported) | yes/no | | cold open of current version | ms | | live room RSS at end (server) | MB | | live doc encoded size sent to a new client | B | ### 4.4 Candidate configurations Y = yrs 0.28 log + checkpoints, author tag per row, gc ON live doc: - Y1 v1 rows, no zstd, checkpoint v2, N in {100, 500, 2000} - Y2 v1 rows batched per flush (idle 1 s or 64 updates) + zstd-3, checkpoint v2+zstd, N in {100, 500, 2000} - Y3 v2 rows batched + zstd, checkpoint v2+zstd, N = best from Y2 - Y4 Y2 + zstd trained dictionary (per document kind) - Y5 Y2 with rows merged (`merge_updates_v1`) per author per idle window before write (fewer, larger rows; loses intra-window points) - each Y in two stores: SQLite BLOB rows vs append-only segment file via calternal-fs G = yrs gc OFF + Yjs `Snapshot` (state vector + delete set) per history point, Restore via `encode_state_from_snapshot`: - G1 v2 full state + snapshots every N; also report browser heap growth (client must run gc off too) L = Loro 1.x standalone reference (MIT): full history snapshot, shallow snapshot + update blocks, `checkout` for Restore, Loro UndoManager per peer A = Automerge 3 standalone reference (MIT): `save()` + incremental `save_incremental`, view at heads J = lower bound: JSON element diffs (Canvas: changed elements per update; Note: text patches) + zstd, CRDT only for live sync Optional: Y-inv = Y2 with clientID -> author inverse undo (only if Y2 UndoManager misses 50 ms) ### 4.5 Expected outcome (hypothesis, to be tested) Y2 (v1 batched + zstd rows, v2+zstd checkpoints, gc on, N ~500) should land near J on disk, pass the latency rule because replay is bounded by N, and keep the browser untouched. G1 likely fails the RSS rule on Canvas (overwritten freehand points kept forever). L may be smallest on disk but loses on rule 4 (client switch, no Excalidraw binding).
kayg commented 2026-10-03 07:06:23 +00:00
Author
Owner
Copy link

Phase 1 decision rule (posted before any benchmark run)

I am starting Phase 1 on branch job/hist-975, based on origin/dev at 48c94c9776660cee105be86c5a6ace90dd425367.

The decision rule is:

  1. Run the deterministic C1 Canvas and N1 Note workloads on the perf VM over the HDD-backed path. Interleave candidate order and collect at least five independent repetitions per candidate. Report the median and p95 across repetitions, and record load average inside the perf lock.
  2. A candidate is eligible only if its restore-to-start/middle/end and per-author undo are correct on every run, the p95 restore, undo and cold-open times are each at most 50 ms, and peak process RSS is at most 2x the same-workload baseline (current yrs state, GC on, no history). Live Yjs documents must keep GC on. If an implementation needs GC off in browser clients, it is ineligible unless client heap also stays within the 2x baseline limit. Report skipped later-edited elements and verify other authors' edits remain intact.
  3. Among eligible candidates, choose the lowest median combined history bytes per hour: C1 bytes after one hour plus N1 bytes after two hours divided by two. Also report raw bytes for each workload and after retention folding. A result within 10% is a disk tie; break it by lower median append CPU per update, then lower peak RSS.
  4. Yrs candidates remain preferred while their browser client is Yjs. A standalone Loro result is a reference only; an engine switch can win only if it is correct, meets the same latency and memory limits, and uses at least 3x less disk than the best eligible Yrs candidate. Do not select a server-side Loro mirror unless it also meets that 3x margin and the best Yrs candidate fails a latency limit.
  5. JSON element diffs are a lower-bound comparison, not a candidate for live collaboration. If no implementation candidate passes, report that no winner meets the rule and carry the smallest eligible footprint only as a Phase 2 measurement target; do not waive a limit.

The measured shortlist is Y1–Y4, G1, L and J from the research comment. Measure Y1 and Y2 at N={100,500,2000}; set Y3 and Y4 to the best Y2 interval. Use one deterministic edit trace per workload for all candidates. Report bytes on disk, bytes/hour, append CPU/update, checkpoint/snapshot time, restore latency and peak RSS, agent-turn undo latency and peak RSS plus correctness, cold open, live room RSS, and encoded bytes sent to a new client. The 50 ms and 2x limits apply to the Canvas and Note workloads; all values are summarized with median and p95.

Phase 1 decision rule (posted before any benchmark run) I am starting Phase 1 on branch `job/hist-975`, based on `origin/dev` at `48c94c9776660cee105be86c5a6ace90dd425367`. The decision rule is: 1. Run the deterministic C1 Canvas and N1 Note workloads on the perf VM over the HDD-backed path. Interleave candidate order and collect at least five independent repetitions per candidate. Report the median and p95 across repetitions, and record load average inside the perf lock. 2. A candidate is eligible only if its restore-to-start/middle/end and per-author undo are correct on every run, the p95 restore, undo and cold-open times are each at most 50 ms, and peak process RSS is at most 2x the same-workload baseline (current yrs state, GC on, no history). Live Yjs documents must keep GC on. If an implementation needs GC off in browser clients, it is ineligible unless client heap also stays within the 2x baseline limit. Report skipped later-edited elements and verify other authors' edits remain intact. 3. Among eligible candidates, choose the lowest median combined history bytes per hour: C1 bytes after one hour plus N1 bytes after two hours divided by two. Also report raw bytes for each workload and after retention folding. A result within 10% is a disk tie; break it by lower median append CPU per update, then lower peak RSS. 4. Yrs candidates remain preferred while their browser client is Yjs. A standalone Loro result is a reference only; an engine switch can win only if it is correct, meets the same latency and memory limits, and uses at least 3x less disk than the best eligible Yrs candidate. Do not select a server-side Loro mirror unless it also meets that 3x margin and the best Yrs candidate fails a latency limit. 5. JSON element diffs are a lower-bound comparison, not a candidate for live collaboration. If no implementation candidate passes, report that no winner meets the rule and carry the smallest eligible footprint only as a Phase 2 measurement target; do not waive a limit. The measured shortlist is Y1–Y4, G1, L and J from the research comment. Measure Y1 and Y2 at N={100,500,2000}; set Y3 and Y4 to the best Y2 interval. Use one deterministic edit trace per workload for all candidates. Report bytes on disk, bytes/hour, append CPU/update, checkpoint/snapshot time, restore latency and peak RSS, agent-turn undo latency and peak RSS plus correctness, cold open, live room RSS, and encoded bytes sent to a new client. The 50 ms and 2x limits apply to the Canvas and Note workloads; all values are summarized with median and p95.
kayg commented 2026-10-03 07:07:57 +00:00
Author
Owner
Copy link

Decision-rule amendment before benchmark runs: for every Y1–Y4 configuration, measure both SQLite BLOB rows and append-only segment storage on the HDD path. Count SQLite main, WAL and shared-memory files while the database is open, then also record the compacted size after retention folding. Count segment headers and metadata as well as payload bytes. Keep storage setup, filesystem, sync policy and retention points the same across repetitions. G1 uses its snapshot representation; L and J use their native representation and the same retention points. Report backend separately in the result table. This resolves the SQLite-vs-segment requirement in the research comment; all eligibility and winner rules in the prior comment still apply.

Decision-rule amendment before benchmark runs: for every Y1–Y4 configuration, measure both SQLite BLOB rows and append-only segment storage on the HDD path. Count SQLite main, WAL and shared-memory files while the database is open, then also record the compacted size after retention folding. Count segment headers and metadata as well as payload bytes. Keep storage setup, filesystem, sync policy and retention points the same across repetitions. G1 uses its snapshot representation; L and J use their native representation and the same retention points. Report backend separately in the result table. This resolves the SQLite-vs-segment requirement in the research comment; all eligibility and winner rules in the prior comment still apply.
kayg commented 2026-10-03 07:29:37 +00:00
Author
Owner
Copy link

Phase 1 research update: current registry metadata confirms yrs 0.28.0 (workspace pin), loro 1.16.2 (MIT), and automerge 0.12.0 (MIT). Official Loro docs describe shallow snapshots that discard older history and checkout for prior versions; current Automerge Rust docs expose save_incremental and fork_at. I will benchmark Loro as the one required alternative engine; Automerge remains a documented comparison, not an additional candidate, to keep the shortlist bounded. Sources: https://docs.rs/loro/1.16.2/loro/struct.LoroDoc.html , https://docs.rs/automerge/0.12.0/automerge/struct.AutoCommit.html , https://loro.dev/docs/performance/docsize

Phase 1 research update: current registry metadata confirms `yrs` 0.28.0 (workspace pin), `loro` 1.16.2 (MIT), and `automerge` 0.12.0 (MIT). Official Loro docs describe shallow snapshots that discard older history and `checkout` for prior versions; current Automerge Rust docs expose `save_incremental` and `fork_at`. I will benchmark Loro as the one required alternative engine; Automerge remains a documented comparison, not an additional candidate, to keep the shortlist bounded. Sources: https://docs.rs/loro/1.16.2/loro/struct.LoroDoc.html , https://docs.rs/automerge/0.12.0/automerge/struct.AutoCommit.html , https://loro.dev/docs/performance/docsize
kayg commented 2026-10-03 08:32:21 +00:00
Author
Owner
Copy link

Progress finding: the deterministic traces and smoke checks pass, but a run-path audit found that the runner forced middle/end checkpoints for every candidate. That removed replay work from Restore and made N comparisons invalid. I am changing Restore to load the nearest prior persisted checkpoint and replay only the required stored updates before starting the matrix.

Progress finding: the deterministic traces and smoke checks pass, but a run-path audit found that the runner forced middle/end checkpoints for every candidate. That removed replay work from Restore and made N comparisons invalid. I am changing Restore to load the nearest prior persisted checkpoint and replay only the required stored updates before starting the matrix.
kayg commented 2026-10-03 09:07:59 +00:00
Author
Owner
Copy link

Progress finding: the fixture's foreign edits to agent-touched items were originally placed after the undo probe, so it could not exercise the required skip/report behavior. I moved C1 overlaps into minutes 20–30 and N1 overlaps into minutes 60–70, and aligned each undo probe to the end of that ten-minute window. N1 now uses Yrs XML paragraph nodes with per-character XmlText appends in bursts; the JSON lower bound stores Note suffix patches. The small C1 and Note smoke traces pass Restore and undo checks across the Yrs stores, Loro, and JSON.

Progress finding: the fixture's foreign edits to agent-touched items were originally placed after the undo probe, so it could not exercise the required skip/report behavior. I moved C1 overlaps into minutes 20–30 and N1 overlaps into minutes 60–70, and aligned each undo probe to the end of that ten-minute window. N1 now uses Yrs XML paragraph nodes with per-character XmlText appends in bursts; the JSON lower bound stores Note suffix patches. The small C1 and Note smoke traces pass Restore and undo checks across the Yrs stores, Loro, and JSON.
kayg commented 2026-10-03 09:33:05 +00:00
Author
Owner
Copy link

Matrix audit finding: choose_y2 used canvas_bytes + note_bytes / 2, which differs from the posted average (canvas_bytes + note_bytes) / 2 and could select the wrong interval/backend. It also ranked disk first inside the 10% tie band instead of using the required CPU then RSS tie-breakers. Corrected the formula and tie handling in commit 9639b1d58; python3 -m py_compile bench/live-history/run-matrix.py and git diff --check passed. No perf runs have started.

Matrix audit finding: `choose_y2` used `canvas_bytes + note_bytes / 2`, which differs from the posted average `(canvas_bytes + note_bytes) / 2` and could select the wrong interval/backend. It also ranked disk first inside the 10% tie band instead of using the required CPU then RSS tie-breakers. Corrected the formula and tie handling in commit `9639b1d58`; `python3 -m py_compile bench/live-history/run-matrix.py` and `git diff --check` passed. No perf runs have started.
kayg commented 2026-10-03 09:50:40 +00:00
Author
Owner
Copy link

First completed sample (n=1; not an eligibility decision): C1 Canvas, Y1-N500, SQLite, repetition 1, perf VM load average 0.68 inside /root/perf.lock. Open history bytes: 189,054,976 B; after current-point fold: 17,895,424 B. Restore start/middle/end: 9.48/30.12/46.51 ms; cold open: 29.15 ms; undo: 15.66 ms, correct, with 20 later-edited elements skipped. Snapshot mean/max: 3,648.49/15,620.89 ms. Peak process RSS: 171.6 MiB; live room RSS: 136.4 MiB. One run is not enough to decide the candidate; the matrix continues.

First completed sample (n=1; not an eligibility decision): C1 Canvas, Y1-N500, SQLite, repetition 1, perf VM load average 0.68 inside `/root/perf.lock`. Open history bytes: 189,054,976 B; after current-point fold: 17,895,424 B. Restore start/middle/end: 9.48/30.12/46.51 ms; cold open: 29.15 ms; undo: 15.66 ms, correct, with 20 later-edited elements skipped. Snapshot mean/max: 3,648.49/15,620.89 ms. Peak process RSS: 171.6 MiB; live room RSS: 136.4 MiB. One run is not enough to decide the candidate; the matrix continues.
kayg commented 2026-10-03 09:52:48 +00:00
Author
Owner
Copy link

Second completed sample (n=1; provisional): C1 Canvas, Y1-N2000, segment, repetition 1, load average 2.00 inside /root/perf.lock. Open bytes: 61,255,680 B; after current-point fold: 8,888,320 B. Restore start/middle/end: 14.75/109.45/133.92 ms; cold open: 138.88 ms; undo: 18.07 ms, correct, with 20 later-edited elements skipped. Snapshot mean/max: 1,434.77/3,953.13 ms. Peak process RSS: 165.2 MiB. The one-sample restore and cold-open times exceed 50 ms. Same-run baseline: cold open 23.09 ms, peak RSS 123.9 MiB. This does not establish p95 eligibility.

Second completed sample (n=1; provisional): C1 Canvas, Y1-N2000, segment, repetition 1, load average 2.00 inside `/root/perf.lock`. Open bytes: 61,255,680 B; after current-point fold: 8,888,320 B. Restore start/middle/end: 14.75/109.45/133.92 ms; cold open: 138.88 ms; undo: 18.07 ms, correct, with 20 later-edited elements skipped. Snapshot mean/max: 1,434.77/3,953.13 ms. Peak process RSS: 165.2 MiB. The one-sample restore and cold-open times exceed 50 ms. Same-run baseline: cold open 23.09 ms, peak RSS 123.9 MiB. This does not establish p95 eligibility.
kayg commented 2026-10-03 09:55:16 +00:00
Author
Owner
Copy link

Third and fourth completed samples (each n=1; provisional): Y1-N500/segment repetition 1. C1 load 2.00: open bytes 175,648,768 B; after current-point fold 8,888,320 B; restore start/middle/end 13.20/25.94/294.73 ms; cold open 38.15 ms; undo 14.02 ms, correct, 20 skipped; snapshot mean/max 428.47/2,136.53 ms; peak RSS 146.4 MiB. N1 load 1.92: open bytes 2,146,304 B; after fold 73,728 B; restore 2.09/4.33/7.46 ms; cold open 8.17 ms; undo 3.27 ms, correct, 20 skipped; snapshot mean/max 109.60/531.44 ms; peak RSS 14.9 MiB. For C1, the single segment-backend restore-to-end result exceeds 50 ms; this remains provisional at n=1.

Third and fourth completed samples (each n=1; provisional): Y1-N500/segment repetition 1. C1 load 2.00: open bytes 175,648,768 B; after current-point fold 8,888,320 B; restore start/middle/end 13.20/25.94/294.73 ms; cold open 38.15 ms; undo 14.02 ms, correct, 20 skipped; snapshot mean/max 428.47/2,136.53 ms; peak RSS 146.4 MiB. N1 load 1.92: open bytes 2,146,304 B; after fold 73,728 B; restore 2.09/4.33/7.46 ms; cold open 8.17 ms; undo 3.27 ms, correct, 20 skipped; snapshot mean/max 109.60/531.44 ms; peak RSS 14.9 MiB. For C1, the single segment-backend restore-to-end result exceeds 50 ms; this remains provisional at n=1.
kayg commented 2026-10-03 09:58:54 +00:00
Author
Owner
Copy link

Further provisional sample (n=1): Y1-N100/SQLite. C1 load 1.99: open bytes 781,778,944 B; after current-point fold 17,899,520 B; restore start/middle/end 9.09/27.69/48.95 ms; cold open 30.63 ms; undo 16.80 ms, correct, 20 skipped; snapshot mean/max 461.91/2,173.89 ms; peak RSS 171.2 MiB. N1 load 2.00: open bytes 7,163,904 B; after fold 5,795,840 B; restore 1.51/3.24/5.83 ms; cold open 7.68 ms; undo 5.75 ms, correct, 20 skipped; snapshot mean/max 51.58/539.42 ms; peak RSS 19.2 MiB. The C1 storage total is high despite the one-sample restore staying below 50 ms; these are not p95 estimates.

Further provisional sample (n=1): Y1-N100/SQLite. C1 load 1.99: open bytes 781,778,944 B; after current-point fold 17,899,520 B; restore start/middle/end 9.09/27.69/48.95 ms; cold open 30.63 ms; undo 16.80 ms, correct, 20 skipped; snapshot mean/max 461.91/2,173.89 ms; peak RSS 171.2 MiB. N1 load 2.00: open bytes 7,163,904 B; after fold 5,795,840 B; restore 1.51/3.24/5.83 ms; cold open 7.68 ms; undo 5.75 ms, correct, 20 skipped; snapshot mean/max 51.58/539.42 ms; peak RSS 19.2 MiB. The C1 storage total is high despite the one-sample restore staying below 50 ms; these are not p95 estimates.
kayg commented 2026-10-03 10:00:21 +00:00
Author
Owner
Copy link

Matrix finding: the C1 Y2-N100/segment run (repetition 1) failed with Error: entry already exists. The trace has 11,100 updates, divisible by N=100. The loop persisted a checkpoint at sequence 11,100, then the post-loop final-checkpoint path tried to create the same immutable segment again. The matrix stopped after 15 completed runs; those raw rows remain on the perf VM. I am fixing the duplicate final write and adding a regression test before rerunning.

Matrix finding: the C1 Y2-N100/segment run (repetition 1) failed with `Error: entry already exists`. The trace has 11,100 updates, divisible by N=100. The loop persisted a checkpoint at sequence 11,100, then the post-loop final-checkpoint path tried to create the same immutable segment again. The matrix stopped after 15 completed runs; those raw rows remain on the perf VM. I am fixing the duplicate final write and adding a regression test before rerunning.
kayg commented 2026-10-03 10:15:08 +00:00
Author
Owner
Copy link

Regression result (n=1): after commit d4d592585, the full C1 Y2-N100/segment boundary probe completed; the previous duplicate snapshot-00011100.seg error did not recur. Load average was 0.28 inside /root/perf.lock. Open bytes: 117,972,992 B; after current-point fold: 1,355,776 B. Restore start/middle/end: 12.37/40.00/41.46 ms; cold open: 43.96 ms; undo: 25.47 ms, correct, 20 later-edited elements skipped. Snapshot mean/max: 702.19/5,710.99 ms. Peak RSS: 139.3 MiB. This verifies the exact boundary case; it is not a p95 result.

Regression result (n=1): after commit `d4d592585`, the full C1 Y2-N100/segment boundary probe completed; the previous duplicate `snapshot-00011100.seg` error did not recur. Load average was 0.28 inside `/root/perf.lock`. Open bytes: 117,972,992 B; after current-point fold: 1,355,776 B. Restore start/middle/end: 12.37/40.00/41.46 ms; cold open: 43.96 ms; undo: 25.47 ms, correct, 20 later-edited elements skipped. Snapshot mean/max: 702.19/5,710.99 ms. Peak RSS: 139.3 MiB. This verifies the exact boundary case; it is not a p95 result.
kayg commented 2026-10-03 10:22:04 +00:00
Author
Owner
Copy link

Y2-N500/SQLite C1 repetition 2 (load average 0.97 inside the perf lock): restore start/middle/end was 9.04/168.22/31.77 ms; cold open 30.15 ms; undo 15.48 ms and correct; peak RSS 162.7 MiB; open bytes 32,858,112 B. Repetition 1's maximum restore point was 48.33 ms. The posted rule defines p95 over five runs; because p95 of five is the maximum, this 168.22 ms sample means the configuration cannot qualify if retained in the final run set. I will finish its scheduled repetitions to report variability, but it is not an eligible winner on the current data.

Y2-N500/SQLite C1 repetition 2 (load average 0.97 inside the perf lock): restore start/middle/end was 9.04/168.22/31.77 ms; cold open 30.15 ms; undo 15.48 ms and correct; peak RSS 162.7 MiB; open bytes 32,858,112 B. Repetition 1's maximum restore point was 48.33 ms. The posted rule defines p95 over five runs; because p95 of five is the maximum, this 168.22 ms sample means the configuration cannot qualify if retained in the final run set. I will finish its scheduled repetitions to report variability, but it is not an eligible winner on the current data.
kayg commented 2026-10-03 10:40:23 +00:00
Author
Owner
Copy link

Provisional sample (n=1): C1 Y2-N100/SQLite, repetition 1, perf-lock load average 1.11. Open bytes: 123,195,392 B; after current-point fold: 119,156,736 B. Restore start/middle/end: 13.37/222.80/49.42 ms; cold open: 41.87 ms; undo: 29.45 ms, correct, 20 later-edited elements skipped. Snapshot mean/max: 282.01/3,897.23 ms. Peak process RSS: 145.8 MiB. The middle restore exceeds 50 ms; this candidate cannot meet the posted p95 rule if the sample remains in the set.

Provisional sample (n=1): C1 Y2-N100/SQLite, repetition 1, perf-lock load average 1.11. Open bytes: 123,195,392 B; after current-point fold: 119,156,736 B. Restore start/middle/end: 13.37/222.80/49.42 ms; cold open: 41.87 ms; undo: 29.45 ms, correct, 20 later-edited elements skipped. Snapshot mean/max: 282.01/3,897.23 ms. Peak process RSS: 145.8 MiB. The middle restore exceeds 50 ms; this candidate cannot meet the posted p95 rule if the sample remains in the set.
kayg commented 2026-10-03 10:50:41 +00:00
Author
Owner
Copy link

#975 Phase 1 report — no winner; stop before Phase 2

Branch: job/hist-975. Base merge from origin/dev was completed before the final gates. Head: d4d592585adbda36c05951c71badbf6469d01eb7.

Built and committed

  • Added deterministic Canvas and Note traces, a Yrs / Loro / JSON history benchmark, and a perf-VM matrix runner.
  • Fixed duplicate immutable checkpoints when the trace length lands exactly on N.
  • Files: bench/live-history/Cargo.toml, bench/live-history/Cargo.lock, bench/live-history/README.md, bench/live-history/run-matrix.py, bench/live-history/src/lib.rs, bench/live-history/src/main.rs.
  • Commits: bea46e769, 897bf005e, 9639b1d58, d4d592585.

Result

The benchmark package and deterministic C1 Canvas / N1 Note traces are built. No history design is selected. The full matrix did not finish, and every Y2 configuration with five runs on both workloads failed at least one hard limit on Canvas. Do not start Phase 2 from these partial results.

Three repeated configurations were summarized with p50 and nearest-rank p95 (p95 of five is the maximum). Disk is shown as median MiB open / after current-point fold. Snapshot is median mean / p95 maximum checkpoint time. Restore lists start / middle / end p95. Undo shows p95, correctness, and skipped later-edited items. RSS is process peak p95.

Candidate Workload (runs) Disk open / folded (MiB) Append CPU p50; snapshot mean p50 / max p95 (ms) Restore start / middle / end p95 (ms) Cold open p95 (ms) Undo p95 (ms; correct; skipped) Peak RSS p95 (MiB)
Baseline, segment C1 Canvas (5) 8.57 / 8.57 0; n/a 0 / 0 / 24.56 24.56 unsupported 123.89
Baseline, segment N1 Note (5) 0.14 / 0.14 0; n/a 0 / 0 / 5.84 5.84 unsupported 13.36
Y2-N500, SQLite C1 Canvas (5) 31.34 / 28.88 21.30; 765.53 / 8,685.66 15.63 / 168.22 / 52.71 55.24 26.29; yes; 20 163.43
Y2-N500, SQLite N1 Note (5) 0.82 / 0.20 9.38; 226.68 / 3,030.52 3.96 / 7.81 / 10.87 10.82 6.72; yes; 20 20.91
Y2-N100, segment C1 Canvas (5) 112.51 / 1.29 17.84; 264.53 / 5,710.99 22.69 / 107.08 / 46.98 53.18 28.52; yes; 20 139.29
Y2-N100, segment N1 Note (5) 0.86 / 0.02 6.23; 153.53 / 2,361.59 3.54 / 5.28 / 7.00 7.18 4.98; yes; 20 15.35
Y2-N100, SQLite C1 Canvas (5) 117.49 / 113.64 23.74; 139.20 / 3,897.23 31.78 / 222.80 / 49.42 44.96 29.85; yes; 20 145.96
Y2-N100, SQLite N1 Note (5) 1.32 / 0.28 11.11; 26.10 / 1,009.76 4.09 / 8.97 / 6.78 7.44 5.85; yes; 20 19.66

The baseline peak RSS p50 was 123.71 MiB for Canvas and 13.22 MiB for Note. Each repeated candidate stayed below 2x those baselines. The encoded live state was 8,874,769 B for Canvas and 67,493 B for Note. All six repeated candidate/workload groups had correct undo results and reported 20 later-edited overlaps.

The issue's combined disk score is (median C1 bytes + median N1 bytes) / 2. The three complete Y2 scores were: Y2-N500/SQLite 16.08 MiB; Y2-N100/segment 56.69 MiB; Y2-N100/SQLite 59.40 MiB. Y2-N500/SQLite had the smallest score in this subset, but its Canvas restore p95 was 168.22 ms and cold-open p95 was 55.24 ms. Y2-N100/segment had Canvas restore p95 107.08 ms and cold-open p95 53.18 ms. Y2-N100/SQLite had Canvas restore p95 222.80 ms. None is eligible.

One C1 Y2-N2000/segment sample had 10,342,400 B open and 126.13 ms maximum restore. It has no five-run comparison and is not a winner. Its smaller disk sample does not pass the latency rule.

Load average was recorded inside /root/perf.lock for every measured process. The 51 logged matrix/follow-up samples ranged from 0.24 to 2.23; the repaired boundary probe logged 0.28. Measurements used /mnt/hdd on root@10.69.69.63.

Matrix failure and correction

The first interleaved matrix stopped after 15 successful rows. C1 Y2-N100/segment has 11,100 updates, exactly divisible by 100. The loop wrote its final periodic checkpoint, then the final-checkpoint path attempted to create the same immutable segment and returned Error: entry already exists. Commit d4d592585 skips that redundant write and adds a boundary regression. The full C1 Y2-N100/segment probe then completed and passed restore and undo correctness.

The full 200-run matrix was not restarted. The remaining Y1/Y2/backend repetitions and the Y3, Y4, G1, Loro, and JSON performance candidates are not measured. Only the listed three Y2 candidate configurations have five runs on both workloads. The independent smoke test exercises Yrs SQLite/segment, G1, Loro, and JSON correctness, but it is not performance evidence. Restore/undo RSS values are cumulative process high-water readings, not isolated phase peaks. The after-fold number is a storage floor, not a retention policy. The Note trace uses Yrs XML paragraph/text operations shaped like y-prosemirror; it is not the app's exact editor event stream.

I stopped the full matrix before the approximately four-hour job limit. The remaining candidates and repetitions could not finish within that window. The measurements do not decide Phase 2. Raw JSONL, summary, and lock-protected load logs remain on the perf VM under /mnt/hdd/bench/hist-975/.

Decisions not specified by DESIGN §61

  • N1 is 300 stable XML paragraph nodes of about 170 bytes, with alternating six-second, 60-character typing bursts and one 20-paragraph agent rewrite. This makes foreign block edits observable while keeping the fixture deterministic.
  • JSON is a lower bound: Canvas stores full changed-element values; Note stores appended text suffixes. It was smoke-tested only.
  • SQLite uses WAL with synchronous=FULL; segment files use immutable calternal-fs root-relative writes. The fixed 64-edit flush applies to Y2 batches. No product retention rule was inferred from the current-point fold.
  • Loro is a standalone reference and was smoke-tested only; no Yjs/Loro mirror or client switch was measured.

Gates

cargo fmt --manifest-path bench/live-history/Cargo.toml --check exited 0 with no output.

cargo clippy --manifest-path bench/live-history/Cargo.toml --all-targets -- -D warnings:

    Checking calternal-live-history-bench v0.1.0 (/home/kayg/Developer/calternal-wt/hist-975/bench/live-history)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.98s

cargo test --manifest-path bench/live-history/Cargo.toml -- --test-threads=1:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.25s
     Running unittests src/lib.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-3d74a7a79e47da91)

running 2 tests
test tests::canvas_trace_has_deterministic_full_workload_shape ... ok
test tests::note_trace_has_two_authors_and_rewritten_blocks ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.38s

     Running unittests src/main.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-9158f6672573e228)

running 3 tests
test tests::candidate_parser_requires_the_gc_off_trace_for_g1 ... ok
test tests::trace_fixture_round_trips_updates_and_authors ... ok
test tests::yrs_segment_and_sqlite_restore_and_undo ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 21.99s

   Doc-tests calternal_live_history_bench

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

An initial default-parallel cargo test attempt hit a SQLx pool timeout; the isolated boundary regression and serialized full package suite passed. cargo clean --manifest-path bench/live-history/Cargo.toml completed with:

     Removed 5410 files, 2.0GiB total

Web build output: none. Worktree status is clean. No push, deployment, or Phase 2 implementation was done. The only merge was the required update from origin/dev before the final gates.

# #975 Phase 1 report — no winner; stop before Phase 2 Branch: `job/hist-975`. Base merge from `origin/dev` was completed before the final gates. Head: `d4d592585adbda36c05951c71badbf6469d01eb7`. ## Built and committed - Added deterministic Canvas and Note traces, a Yrs / Loro / JSON history benchmark, and a perf-VM matrix runner. - Fixed duplicate immutable checkpoints when the trace length lands exactly on N. - Files: `bench/live-history/Cargo.toml`, `bench/live-history/Cargo.lock`, `bench/live-history/README.md`, `bench/live-history/run-matrix.py`, `bench/live-history/src/lib.rs`, `bench/live-history/src/main.rs`. - Commits: `bea46e769`, `897bf005e`, `9639b1d58`, `d4d592585`. ## Result The benchmark package and deterministic C1 Canvas / N1 Note traces are built. No history design is selected. The full matrix did not finish, and every Y2 configuration with five runs on both workloads failed at least one hard limit on Canvas. Do not start Phase 2 from these partial results. Three repeated configurations were summarized with p50 and nearest-rank p95 (p95 of five is the maximum). Disk is shown as median MiB open / after current-point fold. Snapshot is median mean / p95 maximum checkpoint time. Restore lists start / middle / end p95. Undo shows p95, correctness, and skipped later-edited items. RSS is process peak p95. | Candidate | Workload (runs) | Disk open / folded (MiB) | Append CPU p50; snapshot mean p50 / max p95 (ms) | Restore start / middle / end p95 (ms) | Cold open p95 (ms) | Undo p95 (ms; correct; skipped) | Peak RSS p95 (MiB) | |---|---|---:|---:|---:|---:|---:|---:| | Baseline, segment | C1 Canvas (5) | 8.57 / 8.57 | 0; n/a | 0 / 0 / 24.56 | 24.56 | unsupported | 123.89 | | Baseline, segment | N1 Note (5) | 0.14 / 0.14 | 0; n/a | 0 / 0 / 5.84 | 5.84 | unsupported | 13.36 | | Y2-N500, SQLite | C1 Canvas (5) | 31.34 / 28.88 | 21.30; 765.53 / 8,685.66 | 15.63 / 168.22 / 52.71 | 55.24 | 26.29; yes; 20 | 163.43 | | Y2-N500, SQLite | N1 Note (5) | 0.82 / 0.20 | 9.38; 226.68 / 3,030.52 | 3.96 / 7.81 / 10.87 | 10.82 | 6.72; yes; 20 | 20.91 | | Y2-N100, segment | C1 Canvas (5) | 112.51 / 1.29 | 17.84; 264.53 / 5,710.99 | 22.69 / 107.08 / 46.98 | 53.18 | 28.52; yes; 20 | 139.29 | | Y2-N100, segment | N1 Note (5) | 0.86 / 0.02 | 6.23; 153.53 / 2,361.59 | 3.54 / 5.28 / 7.00 | 7.18 | 4.98; yes; 20 | 15.35 | | Y2-N100, SQLite | C1 Canvas (5) | 117.49 / 113.64 | 23.74; 139.20 / 3,897.23 | 31.78 / 222.80 / 49.42 | 44.96 | 29.85; yes; 20 | 145.96 | | Y2-N100, SQLite | N1 Note (5) | 1.32 / 0.28 | 11.11; 26.10 / 1,009.76 | 4.09 / 8.97 / 6.78 | 7.44 | 5.85; yes; 20 | 19.66 | The baseline peak RSS p50 was 123.71 MiB for Canvas and 13.22 MiB for Note. Each repeated candidate stayed below 2x those baselines. The encoded live state was 8,874,769 B for Canvas and 67,493 B for Note. All six repeated candidate/workload groups had correct undo results and reported 20 later-edited overlaps. The issue's combined disk score is `(median C1 bytes + median N1 bytes) / 2`. The three complete Y2 scores were: Y2-N500/SQLite 16.08 MiB; Y2-N100/segment 56.69 MiB; Y2-N100/SQLite 59.40 MiB. Y2-N500/SQLite had the smallest score in this subset, but its Canvas restore p95 was 168.22 ms and cold-open p95 was 55.24 ms. Y2-N100/segment had Canvas restore p95 107.08 ms and cold-open p95 53.18 ms. Y2-N100/SQLite had Canvas restore p95 222.80 ms. None is eligible. One C1 Y2-N2000/segment sample had 10,342,400 B open and 126.13 ms maximum restore. It has no five-run comparison and is not a winner. Its smaller disk sample does not pass the latency rule. Load average was recorded inside `/root/perf.lock` for every measured process. The 51 logged matrix/follow-up samples ranged from 0.24 to 2.23; the repaired boundary probe logged 0.28. Measurements used `/mnt/hdd` on `root@10.69.69.63`. ## Matrix failure and correction The first interleaved matrix stopped after 15 successful rows. C1 Y2-N100/segment has 11,100 updates, exactly divisible by 100. The loop wrote its final periodic checkpoint, then the final-checkpoint path attempted to create the same immutable segment and returned `Error: entry already exists`. Commit `d4d592585` skips that redundant write and adds a boundary regression. The full C1 Y2-N100/segment probe then completed and passed restore and undo correctness. The full 200-run matrix was not restarted. The remaining Y1/Y2/backend repetitions and the Y3, Y4, G1, Loro, and JSON performance candidates are not measured. Only the listed three Y2 candidate configurations have five runs on both workloads. The independent smoke test exercises Yrs SQLite/segment, G1, Loro, and JSON correctness, but it is not performance evidence. Restore/undo RSS values are cumulative process high-water readings, not isolated phase peaks. The after-fold number is a storage floor, not a retention policy. The Note trace uses Yrs XML paragraph/text operations shaped like y-prosemirror; it is not the app's exact editor event stream. I stopped the full matrix before the approximately four-hour job limit. The remaining candidates and repetitions could not finish within that window. The measurements do not decide Phase 2. Raw JSONL, summary, and lock-protected load logs remain on the perf VM under `/mnt/hdd/bench/hist-975/`. ## Decisions not specified by DESIGN §61 - N1 is 300 stable XML paragraph nodes of about 170 bytes, with alternating six-second, 60-character typing bursts and one 20-paragraph agent rewrite. This makes foreign block edits observable while keeping the fixture deterministic. - JSON is a lower bound: Canvas stores full changed-element values; Note stores appended text suffixes. It was smoke-tested only. - SQLite uses WAL with `synchronous=FULL`; segment files use immutable `calternal-fs` root-relative writes. The fixed 64-edit flush applies to Y2 batches. No product retention rule was inferred from the current-point fold. - Loro is a standalone reference and was smoke-tested only; no Yjs/Loro mirror or client switch was measured. ## Gates `cargo fmt --manifest-path bench/live-history/Cargo.toml --check` exited 0 with no output. `cargo clippy --manifest-path bench/live-history/Cargo.toml --all-targets -- -D warnings`: ```text Checking calternal-live-history-bench v0.1.0 (/home/kayg/Developer/calternal-wt/hist-975/bench/live-history) Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.98s ``` `cargo test --manifest-path bench/live-history/Cargo.toml -- --test-threads=1`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 0.25s Running unittests src/lib.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-3d74a7a79e47da91) running 2 tests test tests::canvas_trace_has_deterministic_full_workload_shape ... ok test tests::note_trace_has_two_authors_and_rewritten_blocks ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.38s Running unittests src/main.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-9158f6672573e228) running 3 tests test tests::candidate_parser_requires_the_gc_off_trace_for_g1 ... ok test tests::trace_fixture_round_trips_updates_and_authors ... ok test tests::yrs_segment_and_sqlite_restore_and_undo ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 21.99s Doc-tests calternal_live_history_bench running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` An initial default-parallel `cargo test` attempt hit a SQLx pool timeout; the isolated boundary regression and serialized full package suite passed. `cargo clean --manifest-path bench/live-history/Cargo.toml` completed with: ```text Removed 5410 files, 2.0GiB total ``` Web build output: none. Worktree status is clean. No push, deployment, or Phase 2 implementation was done. The only merge was the required update from `origin/dev` before the final gates.
kayg commented 2026-10-03 11:27:05 +00:00
Author
Owner
Copy link

Continuing #975 Phase 1 on branch job/hist-975. Worktree head at start: d4d592585adbda36c05951c71badbf6469d01eb7; branch base: 4a871b383864dad0d6c87bcd22b7f0d589e3506e (current origin/dev: f06679b11cde29cc0b7120fdab5f721389caf695). I will use the decision rule already posted above, retain its 50 ms and 2x limits, complete the shortlisted candidate matrix, and measure the named Y2 Canvas remedies on the locked HDD-backed perf VM. No Phase 2 implementation is in scope.

Continuing #975 Phase 1 on branch `job/hist-975`. Worktree head at start: `d4d592585adbda36c05951c71badbf6469d01eb7`; branch base: `4a871b383864dad0d6c87bcd22b7f0d589e3506e` (current `origin/dev`: `f06679b11cde29cc0b7120fdab5f721389caf695`). I will use the decision rule already posted above, retain its 50 ms and 2x limits, complete the shortlisted candidate matrix, and measure the named Y2 Canvas remedies on the locked HDD-backed perf VM. No Phase 2 implementation is in scope.
kayg commented 2026-10-03 12:00:32 +00:00
Author
Owner
Copy link

Harness finding: the earlier snapshot_ms value combined Yrs state encoding, zstd compression and durable storage write. Restore values combined checkpoint lookup/read/decode with log fetch and update replay. The old results therefore identify the failing Restore point and N, but not the expensive stage. Commit ec9ae6f7cdb73470469b3ff3c4423972a35377be adds opt-in per-stage timings, a 5-run Y2 diagnostic runner and a decoded-latest-checkpoint probe. The cache probe consumes a temporary decoded Doc and reports its RSS; it does not change the live benchmark room. The focused benchmark crate tests pass; measurements have not started with this profiler build.

Harness finding: the earlier `snapshot_ms` value combined Yrs state encoding, zstd compression and durable storage write. Restore values combined checkpoint lookup/read/decode with log fetch and update replay. The old results therefore identify the failing Restore point and N, but not the expensive stage. Commit `ec9ae6f7cdb73470469b3ff3c4423972a35377be` adds opt-in per-stage timings, a 5-run Y2 diagnostic runner and a decoded-latest-checkpoint probe. The cache probe consumes a temporary decoded Doc and reports its RSS; it does not change the live benchmark room. The focused benchmark crate tests pass; measurements have not started with this profiler build.
kayg commented 2026-10-03 12:14:06 +00:00
Author
Owner
Copy link

Raw-result audit finding: /mnt/hdd/bench/hist-975/y2-100-segment/results.jsonl contains Canvas repetitions 2, 3, 4 and 5. The repaired repetition-1 boundary probe was printed in the prior issue comment, but it was not appended to a raw JSONL file. The prior Y2-N100/segment Canvas row therefore has four auditable repetitions, not five. I will collect a fresh five-run group with stage profiles; this keeps the p95 rule based on five saved raw rows.

Raw-result audit finding: `/mnt/hdd/bench/hist-975/y2-100-segment/results.jsonl` contains Canvas repetitions 2, 3, 4 and 5. The repaired repetition-1 boundary probe was printed in the prior issue comment, but it was not appended to a raw JSONL file. The prior Y2-N100/segment Canvas row therefore has four auditable repetitions, not five. I will collect a fresh five-run group with stage profiles; this keeps the p95 rule based on five saved raw rows.
kayg commented 2026-10-03 12:19:39 +00:00
Author
Owner
Copy link

Provisional diagnostic samples (r1 only; not p95): Y2 Canvas/SQLite had maximum Restore over start/middle/end of 44.884 ms at N=25, 47.948 ms at N=500 and 138.648 ms at N=2,000. Undo was 27.430, 26.370 and 24.637 ms respectively. This points to checkpoint interval as a Restore factor, but the stage breakdown and four remaining repetitions are still running.

Provisional diagnostic samples (r1 only; not p95): Y2 Canvas/SQLite had maximum Restore over start/middle/end of 44.884 ms at N=25, 47.948 ms at N=500 and 138.648 ms at N=2,000. Undo was 27.430, 26.370 and 24.637 ms respectively. This points to checkpoint interval as a Restore factor, but the stage breakdown and four remaining repetitions are still running.
kayg commented 2026-10-03 12:25:59 +00:00
Author
Owner
Copy link

Provisional remedy result (r2): Y2-N25/segment Canvas maximum Restore was 54.887 ms, above the 50 ms limit. The N=25 SQLite Canvas r1 was 44.884 ms. A single exceedance makes the five-run nearest-rank p95 fail if retained; I will report the complete group and its replay-stage breakdown after all repetitions finish.

Provisional remedy result (r2): Y2-N25/segment Canvas maximum Restore was 54.887 ms, above the 50 ms limit. The N=25 SQLite Canvas r1 was 44.884 ms. A single exceedance makes the five-run nearest-rank p95 fail if retained; I will report the complete group and its replay-stage breakdown after all repetitions finish.
kayg commented 2026-10-03 12:34:40 +00:00
Author
Owner
Copy link

Provisional diagnostic samples (r3): maximum Restore over start/middle/end was 52.225 ms for Y2-N100/segment Canvas, 99.333 ms for Y2-N500/SQLite Canvas and 352.223 ms for Y2-N2000/SQLite Canvas. Each exceeds 50 ms. These are not final p95 values; the five-run stage profiles are still in progress.

Provisional diagnostic samples (r3): maximum Restore over start/middle/end was 52.225 ms for Y2-N100/segment Canvas, 99.333 ms for Y2-N500/SQLite Canvas and 352.223 ms for Y2-N2000/SQLite Canvas. Each exceeds 50 ms. These are not final p95 values; the five-run stage profiles are still in progress.
kayg commented 2026-10-03 12:53:28 +00:00
Author
Owner
Copy link

Y2 diagnosis and remedy measurements: 45 fresh interleaved runs, five repetitions per row, on the perf VM HDD path. Every process held /root/perf.lock; load was recorded inside the lock. The p95 of five is the maximum.

Canvas candidate Open MiB p50 Restore start / middle / end p95 (ms) Cold open p95 (ms) Undo p95 (ms) Peak RSS p95 (MiB) Result
Y2 N=100 segment 112.51 13.92 / 75.07 / 43.74 44.19 25.81 139.4 fails Restore
Y2 N=100 SQLite 117.49 14.53 / 71.81 / 45.26 47.73 33.02 146.0 fails Restore
Y2 N=500 SQLite 31.34 11.66 / 99.33 / 50.50 53.15 26.37 164.9 fails Restore and cold open
Y2 N=2,000 SQLite 11.39 29.20 / 352.22 / 142.77 138.81 26.35 165.1 fails Restore and cold open
Y2 N=25 segment 434.75 14.68 / 88.40 / 61.09 48.62 30.10 139.4 fails Restore
Y2 N=25 SQLite 436.74 15.77 / 51.46 / 39.31 45.42 38.61 146.3 fails Restore by 1.46 ms
Y2 N=500 + decoded latest checkpoint cache, SQLite 31.34 9.72 / 42.49 / 10.52 53.72 28.06 165.1 fails cold open

All measured undo p95 values are below 50 ms and peak process RSS is below 2x the baseline (Canvas baseline p50 123.7 MiB; limit 247.4 MiB). Stage timings identify checkpoint decode/apply, not undo, as the main N=100/500 Restore cost: middle-point checkpoint decode/apply p95 was 65.1 ms at N=100 SQLite and 91.6 ms at N=500 SQLite, while update replay was 4.9 and 4.4 ms. At N=2,000 both stages contribute: middle-point checkpoint decode/apply 252.8 ms and replay 86.7 ms, across 1,550 updates. At N=25 SQLite the middle point is already a checkpoint (zero replay); decode/apply p95 was 49.2 ms, just above the overall 50 ms limit once read/load overhead is included.

Checkpoint work is also expensive, but mostly in durable writes rather than state encoding. For N=500 SQLite the p95 per-run maximum state-encode time was 13.8 ms, compression 62.0 ms and durable write 2,407.9 ms. The latest-checkpoint cache lowered Restore end to 10.52 ms p95, with 10.5 ms p95 cache access and 105.5 ms p95 warm-up. It did not fix uncached cold open (53.72 ms); process peak RSS remained 165.1 MiB p95.

No measured Y2 configuration passes the posted rule. The closest are N=25 SQLite (Restore misses by 1.46 ms, but uses 436.74 MiB) and N=500 SQLite with a decoded latest-checkpoint cache (Restore and undo pass, cold open misses by 3.72 ms, with 31.34 MiB). The rule remains unchanged. I will compare these with Y1, Y3, Y4, G1, Loro and JSON before recommending any owner decision.

Y2 diagnosis and remedy measurements: 45 fresh interleaved runs, five repetitions per row, on the perf VM HDD path. Every process held `/root/perf.lock`; load was recorded inside the lock. The p95 of five is the maximum. | Canvas candidate | Open MiB p50 | Restore start / middle / end p95 (ms) | Cold open p95 (ms) | Undo p95 (ms) | Peak RSS p95 (MiB) | Result | |---|---:|---:|---:|---:|---:|---| | Y2 N=100 segment | 112.51 | 13.92 / 75.07 / 43.74 | 44.19 | 25.81 | 139.4 | fails Restore | | Y2 N=100 SQLite | 117.49 | 14.53 / 71.81 / 45.26 | 47.73 | 33.02 | 146.0 | fails Restore | | Y2 N=500 SQLite | 31.34 | 11.66 / 99.33 / 50.50 | 53.15 | 26.37 | 164.9 | fails Restore and cold open | | Y2 N=2,000 SQLite | 11.39 | 29.20 / 352.22 / 142.77 | 138.81 | 26.35 | 165.1 | fails Restore and cold open | | Y2 N=25 segment | 434.75 | 14.68 / 88.40 / 61.09 | 48.62 | 30.10 | 139.4 | fails Restore | | Y2 N=25 SQLite | 436.74 | 15.77 / 51.46 / 39.31 | 45.42 | 38.61 | 146.3 | fails Restore by 1.46 ms | | Y2 N=500 + decoded latest checkpoint cache, SQLite | 31.34 | 9.72 / 42.49 / 10.52 | 53.72 | 28.06 | 165.1 | fails cold open | All measured undo p95 values are below 50 ms and peak process RSS is below 2x the baseline (Canvas baseline p50 123.7 MiB; limit 247.4 MiB). Stage timings identify checkpoint decode/apply, not undo, as the main N=100/500 Restore cost: middle-point checkpoint decode/apply p95 was 65.1 ms at N=100 SQLite and 91.6 ms at N=500 SQLite, while update replay was 4.9 and 4.4 ms. At N=2,000 both stages contribute: middle-point checkpoint decode/apply 252.8 ms and replay 86.7 ms, across 1,550 updates. At N=25 SQLite the middle point is already a checkpoint (zero replay); decode/apply p95 was 49.2 ms, just above the overall 50 ms limit once read/load overhead is included. Checkpoint work is also expensive, but mostly in durable writes rather than state encoding. For N=500 SQLite the p95 per-run maximum state-encode time was 13.8 ms, compression 62.0 ms and durable write 2,407.9 ms. The latest-checkpoint cache lowered Restore end to 10.52 ms p95, with 10.5 ms p95 cache access and 105.5 ms p95 warm-up. It did not fix uncached cold open (53.72 ms); process peak RSS remained 165.1 MiB p95. No measured Y2 configuration passes the posted rule. The closest are N=25 SQLite (Restore misses by 1.46 ms, but uses 436.74 MiB) and N=500 SQLite with a decoded latest-checkpoint cache (Restore and undo pass, cold open misses by 3.72 ms, with 31.34 MiB). The rule remains unchanged. I will compare these with Y1, Y3, Y4, G1, Loro and JSON before recommending any owner decision.
kayg commented 2026-10-03 12:58:08 +00:00
Author
Owner
Copy link

Runner finding: resumed rows used labels such as Y1-500, but the completion specs used y1-500. The first completion attempt therefore failed to recognize saved Y1 rows. I stopped it after five new measurements, before any candidate comparison, and fixed candidate-key normalization in commit e58d5ca9addc1aa7c7382f3638f1e88f110d45d8. The deterministic helper check now covers Y1/G1 case normalization and the Loro/JSON report-label aliases. I am restarting with a clean output directory and the same saved source rows.

Runner finding: resumed rows used labels such as `Y1-500`, but the completion specs used `y1-500`. The first completion attempt therefore failed to recognize saved Y1 rows. I stopped it after five new measurements, before any candidate comparison, and fixed candidate-key normalization in commit `e58d5ca9addc1aa7c7382f3638f1e88f110d45d8`. The deterministic helper check now covers Y1/G1 case normalization and the Loro/JSON report-label aliases. I am restarting with a clean output directory and the same saved source rows.
kayg commented 2026-10-03 13:03:54 +00:00
Author
Owner
Copy link

The first completed shortlist run exposed a harness defect: Loro historical checkout is read-only by default, so Canvas selective undo failed when applying its detached branch. I enabled detached editing only after checkout, with a code comment on the isolated peer identity invariant, and added a guard that the benchmark exercises a non-empty undo. I am rerunning the focused benchmark tests before resuming the matrix.

The first completed shortlist run exposed a harness defect: Loro historical checkout is read-only by default, so Canvas selective undo failed when applying its detached branch. I enabled detached editing only after checkout, with a code comment on the isolated peer identity invariant, and added a guard that the benchmark exercises a non-empty undo. I am rerunning the focused benchmark tests before resuming the matrix.
kayg commented 2026-10-03 13:18:38 +00:00
Author
Owner
Copy link

A resumed run showed that the Note Loro trace can have an empty selective-undo action set, so the new non-empty-action assertion was too broad and stopped at repetition 2. Canvas Loro has non-empty actions and is now measured successfully. I am narrowing the invariant to Canvas only, then I will rerun the full five Note Loro repetitions with detached editing enabled for consistent measurements.

A resumed run showed that the Note Loro trace can have an empty selective-undo action set, so the new non-empty-action assertion was too broad and stopped at repetition 2. Canvas Loro has non-empty actions and is now measured successfully. I am narrowing the invariant to Canvas only, then I will rerun the full five Note Loro repetitions with detached editing enabled for consistent measurements.
kayg commented 2026-10-03 14:00:26 +00:00
Author
Owner
Copy link

Phase 1 continuation complete

Branch: job/hist-975; head: 3400b72ce8615a21913e68a046c01f075cc8b448. I merged origin/dev once before the final gates. The matrix has five fresh-process repetitions for all 13 shortlisted configurations on both C1 Canvas and N1 Note. Y2 diagnostics cover nine configuration/workload groups with five repetitions each. Each new perf-VM run held /root/perf.lock and logged load inside the lock. The 80 newly completed shortlist runs had load average 0.03–2.02 (median 1.95); reused rows retain their earlier locked-run logs.

Recommendation

Recommend Y4 at N=500 with segment storage for Phase 2 review. It is the lowest-disk eligible Yrs configuration: combined median open footprint (C1 + N1) / 2 is 16.19 MiB (C1 31.86 MiB; N1 0.52 MiB), versus 91.73 MiB for Y1-500/SQLite and 369.63 MiB for Y1-100/segment. Y4-500/segment passes every correctness, latency and memory gate. Its closest metric is Canvas Restore p95 at 48.55 ms, 1.45 ms below the limit; Canvas cold-open/undo p95 are 39.32/19.91 ms. Note Restore/cold-open/undo p95 are 6.86/7.56/3.62 ms. Peak process RSS p95 is 157.02 MiB on Canvas and 18.98 MiB on Note, below their respective 247.43/26.44 MiB limits.

Y1-100/segment and Y1-500/SQLite also pass. J (JSON element diffs) has a 13.28 MiB combined score and meets the measured gates, but remains a lower bound because it does not preserve live CRDT history. L (Loro) is 3.54× smaller than the best eligible Yrs option, but fails Canvas Restore (136.56 ms p95), Note Restore (91.14 ms), and Canvas cold-open (62.00 ms), so it does not meet the engine-switch rule. No rule change is proposed: eligible Yrs candidates exist. Phase 2 remains for owner review.

Full shortlist matrix

Rows are per candidate, backend and workload. Metric pairs are median/p95 over five runs; p95 uses nearest rank, so with five runs it is the maximum. S/M/E means Restore to start/middle/end. Disk open includes SQLite main/WAL/shared-memory files or segment metadata. Folded disk is the one-current-point storage floor, not a retention policy. Snapshot mean/max are in milliseconds. RSS is process high-water RSS; live-room RSS and encoded bytes describe the current live document.

Candidate / backend / workload Open disk MiB p50/p95 Folded disk MiB p50/p95 Append CPU µs/update p50/p95 Snapshot mean/max ms p50/p95 Restore start/middle/end ms p50/p95 Cold-open / undo ms p50/p95 Peak process RSS MiB p50/p95 Live RSS / encoded MiB p50/p95 Correctness
Y1-100 / sqlite / C1 Canvas 745.56/745.56 17.07/17.07 40.32/50.72 mean 491.31/709.48; max 3195.86/9500.61 S 6.33/10.04, M 21.38/29.12, E 27.01/53.95 cold 22.01/30.63; undo 13.51/16.80 164.17/171.16 RSS 135.64/135.66; encoded 8.46/8.46 restore/undo correct in 5/5
Y1-100 / sqlite / N1 Note 6.83/6.83 5.53/5.53 35.39/41.76 mean 52.46/93.65; max 1360.31/1460.86 S 1.72/2.22, M 3.29/3.84, E 5.12/5.91 cold 4.24/7.68; undo 3.64/5.75 19.16/19.35 RSS 15.24/15.39; encoded 0.06/0.06 restore/undo correct in 5/5
Y1-100 / segment / C1 Canvas 733.53/733.53 8.48/8.48 6.02/6.83 mean 241.75/325.18; max 2363.63/3557.00 S 9.68/17.06, M 20.68/32.45, E 28.05/33.82 cold 24.42/28.46; undo 11.76/15.86 138.75/138.79 RSS 113.44/113.50; encoded 8.46/8.46 restore/undo correct in 5/5
Y1-100 / segment / N1 Note 5.72/5.72 0.08/0.08 3.93/4.90 mean 33.67/100.98; max 311.65/427.85 S 1.87/2.47, M 3.20/5.19, E 3.44/6.31 cold 3.88/6.72; undo 2.08/3.32 15.00/15.04 RSS 11.38/11.45; encoded 0.06/0.06 restore/undo correct in 5/5
Y1-500 / sqlite / C1 Canvas 180.30/180.30 17.07/17.07 31.49/43.65 mean 462.73/3648.49; max 2150.20/15620.89 S 7.32/9.48, M 19.62/30.12, E 41.47/46.51 cold 27.60/32.94; undo 14.40/19.76 171.67/188.60 RSS 136.46/136.54; encoded 8.46/8.46 restore/undo correct in 5/5
Y1-500 / sqlite / N1 Note 3.16/3.16 1.98/1.98 28.43/31.53 mean 31.85/36.48; max 87.21/281.89 S 1.43/1.96, M 2.70/4.25, E 5.17/6.81 cold 5.27/8.21; undo 2.97/3.80 19.07/19.11 RSS 14.93/14.96; encoded 0.06/0.06 restore/undo correct in 5/5
Y1-500 / segment / C1 Canvas 167.51/167.51 8.48/8.48 4.60/6.41 mean 182.03/447.24; max 707.81/4186.94 S 9.10/13.20, M 17.20/25.94, E 23.69/294.73 cold 23.11/38.15; undo 10.51/14.02 146.64/163.43 RSS 112.70/112.77; encoded 8.46/8.46 restore/undo correct in 5/5
Y1-500 / segment / N1 Note 2.05/2.05 0.07/0.07 3.32/4.03 mean 45.95/109.60; max 166.57/615.30 S 1.64/2.16, M 2.91/4.33, E 5.12/7.46 cold 4.97/8.17; undo 2.24/3.27 14.83/14.95 RSS 11.15/11.32; encoded 0.06/0.06 restore/undo correct in 5/5
Y1-2000 / sqlite / C1 Canvas 69.77/69.77 17.07/17.07 33.93/44.53 mean 487.34/769.42; max 1026.35/3704.88 S 4.88/8.52, M 77.73/109.93, E 89.18/119.69 cold 84.75/118.20; undo 12.89/23.51 188.33/189.05 RSS 133.81/134.08; encoded 8.46/8.46 restore/undo correct in 5/5
Y1-2000 / sqlite / N1 Note 2.39/2.39 1.30/1.30 27.06/34.51 mean 22.97/29.97; max 31.37/50.77 S 1.33/1.65, M 6.57/10.60, E 9.56/15.98 cold 10.04/17.69; undo 2.25/3.48 18.47/18.50 RSS 13.95/13.97; encoded 0.06/0.06 restore/undo correct in 5/5
Y1-2000 / segment / C1 Canvas 58.42/58.42 8.48/8.48 5.59/7.36 mean 209.89/1434.77; max 252.46/3953.13 S 8.84/14.75, M 79.97/109.45, E 88.11/133.92 cold 88.67/138.88; undo 15.59/18.99 162.97/165.18 RSS 112.11/112.15; encoded 8.46/8.46 restore/undo correct in 5/5
Y1-2000 / segment / N1 Note 1.36/1.36 0.07/0.07 2.93/3.73 mean 31.51/70.06; max 43.29/156.61 S 1.42/2.33, M 3.79/7.45, E 7.93/14.66 cold 8.53/15.36; undo 1.89/3.90 14.70/14.76 RSS 11.09/11.12; encoded 0.06/0.06 restore/undo correct in 5/5
G1-500 / segment / C1 Canvas 40.77/40.77 18.19/18.19 7.08/7.52 mean 65.92/82.09; max 667.04/740.26 S 33.42/47.12, M 72.45/141.37, E 92.52/276.25 cold 25.22/31.02; undo 14.70/17.01 179.88/179.93 RSS 132.22/132.29; encoded 18.17/18.17 restore/undo correct in 5/5
G1-500 / segment / N1 Note 1.05/1.05 0.08/0.08 3.40/4.05 mean 37.31/91.06; max 200.78/1352.33 S 3.90/4.81, M 4.05/4.99, E 5.19/7.03 cold 2.32/3.33; undo 2.06/2.75 15.29/15.29 RSS 11.30/11.31; encoded 0.07/0.07 restore/undo correct in 5/5
L / native / C1 Canvas 9.10/9.10 2.89/2.89 4.35/5.49 mean 289.32/3058.36; max 289.32/3058.36 S 109.26/136.56, M 72.89/95.71, E 0.01/0.01 cold 21.60/62.00; undo 6.87/10.98 215.92/216.21 RSS 172.97/173.09; encoded 2.89/2.89 restore/undo correct in 5/5
L / native / N1 Note 0.04/0.04 0.03/0.03 4.49/5.85 mean 52.79/1261.59; max 52.79/1261.59 S 75.18/91.14, M 68.04/79.73, E 0.00/0.01 cold 0.74/0.85; undo 1.24/1.96 18.81/18.82 RSS 14.34/14.41; encoded 0.02/0.02 restore/undo correct in 5/5
J / segment / C1 Canvas 26.03/26.03 1.29/1.29 14.98/19.43 mean 95.33/109.67; max 230.16/270.08 S 7.50/8.26, M 24.05/37.44, E 25.11/31.13 cold 17.89/32.22; undo 18.21/26.64 120.26/121.44 RSS 99.58/100.52; encoded 8.49/8.49 restore/undo correct in 5/5
J / segment / N1 Note 0.52/0.52 0.01/0.01 4.43/5.16 mean 27.79/187.86; max 122.69/2579.32 S 1.29/1.62, M 1.52/1.70, E 1.69/1.81 cold 1.59/2.00; undo 3.51/4.86 10.21/11.72 RSS 9.57/9.71; encoded 0.06/0.06 restore/undo correct in 5/5
Y3-500 / sqlite / C1 Canvas 31.26/31.26 28.82/28.82 16.49/18.51 mean 165.63/294.45; max 1216.36/4264.33 S 6.80/86.71, M 27.09/36.87, E 29.36/37.06 cold 34.64/40.93; undo 18.72/23.74 162.42/164.84 RSS 119.72/119.77; encoded 8.46/8.46 restore/undo correct in 5/5
Y3-500 / sqlite / N1 Note 0.78/0.78 0.18/0.18 6.78/8.49 mean 24.59/32.82; max 55.65/128.04 S 1.69/2.76, M 3.33/5.56, E 5.43/7.17 cold 6.72/7.59; undo 3.55/4.77 18.84/18.95 RSS 13.86/13.92; encoded 0.06/0.06 restore/undo correct in 5/5
Y3-500 / segment / C1 Canvas 26.43/26.43 1.29/1.29 11.83/14.80 mean 102.93/131.70; max 480.80/758.54 S 6.88/10.63, M 29.69/43.53, E 31.13/46.98 cold 39.09/50.14; undo 18.96/22.98 154.79/155.88 RSS 113.05/113.14; encoded 8.46/8.46 restore/undo correct in 5/5
Y3-500 / segment / N1 Note 0.52/0.52 0.01/0.01 3.95/4.71 mean 21.10/94.31; max 72.46/1279.12 S 1.20/2.10, M 2.70/4.57, E 5.07/8.10 cold 5.00/7.74; undo 2.63/3.64 15.23/15.27 RSS 12.01/12.04; encoded 0.06/0.06 restore/undo correct in 5/5
Y4-500 / sqlite / C1 Canvas 37.11/37.11 34.95/34.95 17.75/23.52 mean 121.01/156.07; max 319.85/768.21 S 7.38/12.14, M 25.64/42.86, E 31.05/55.92 cold 34.82/82.17; undo 18.14/30.67 164.45/164.61 RSS 120.93/121.05; encoded 8.46/8.46 restore/undo correct in 5/5
Y4-500 / sqlite / N1 Note 0.94/0.94 0.27/0.27 7.73/8.97 mean 27.38/39.27; max 82.09/361.17 S 1.63/19.65, M 2.62/4.57, E 5.03/9.95 cold 5.27/9.70; undo 3.88/5.53 20.71/20.77 RSS 17.61/17.62; encoded 0.06/0.06 restore/undo correct in 5/5
Y4-500 / segment / C1 Canvas 31.86/31.86 1.62/1.62 13.82/17.28 mean 84.51/98.39; max 287.68/365.62 S 6.25/11.39, M 24.75/48.55, E 27.21/38.68 cold 29.73/39.32; undo 16.08/19.91 156.80/157.02 RSS 113.10/113.16; encoded 8.46/8.46 restore/undo correct in 5/5
Y4-500 / segment / N1 Note 0.52/0.52 0.02/0.02 4.42/4.96 mean 17.35/38.18; max 40.43/206.40 S 1.53/1.98, M 2.36/3.48, E 4.22/6.86 cold 4.39/7.56; undo 2.65/3.62 18.94/18.98 RSS 15.79/15.86; encoded 0.06/0.06 restore/undo correct in 5/5

All metric pairs show median/p95 over five runs. P95 uses nearest rank, so it is the maximum of five. S/M/E are Restore to start/middle/end. Snapshot mean/max reports average and maximum checkpoint times. Process RSS values are process high-water marks.

Y2 Restore profile and remedies

Times below are p95. mid/end stages use checkpoint read / decode+apply / log load / update replay, then replayed update count. Checkpoint values are maximum per-run state encode / compression / durable write.

Configuration / workload Restore S/M/E ms Mid stages ms; updates End stages ms; updates Cold / undo ms Checkpoint max encode/compress/write ms Cache warm/access/RSS Open disk MiB p50
Y2-100 / segment / C1 Canvas 13.92/75.07/43.74 1.28/65.91/3.67/4.38; 50 1.08/39.51/3.13/0.00; 0 44.19/25.81 17.83/83.21/1302.62 — 112.51
Y2-100 / sqlite / C1 Canvas 14.53/71.81/45.26 3.37/65.07/1.24/4.88; 50 2.23/42.21/0.71/0.00; 0 47.73/33.02 18.59/75.34/2618.57 — 117.49
Y2-500 / sqlite / C1 Canvas 11.66/99.33/50.50 2.35/91.57/1.38/4.43; 50 1.86/37.99/1.51/9.11; 100 53.15/26.37 13.78/62.02/2407.87 — 31.34
Y2-2000 / sqlite / C1 Canvas 29.20/352.22/142.77 1.90/252.78/12.21/86.68; 1550 2.12/41.10/8.84/91.08; 1100 138.81/26.35 28.48/57.94/1063.05 — 11.39
Y2-25 / segment / C1 Canvas 14.68/88.40/61.09 1.01/81.48/5.77/0.00; 0 1.13/54.39/5.48/0.00; 0 48.62/30.10 22.27/115.17/4837.05 — 434.75
Y2-25 / sqlite / C1 Canvas 15.77/51.46/39.31 1.90/49.20/0.94/0.00; 0 2.02/36.84/1.41/0.00; 0 45.42/38.61 43.10/193.66/4850.61 — 436.74
Y2-25 / segment / N1 Note 5.46/6.64/14.91 0.04/3.14/3.25/0.26; 10 0.07/7.45/7.05/0.29; 20 11.58/11.25 2.52/2.02/2044.05 — 2.29
Y2-25 / sqlite / N1 Note 7.12/5.34/8.07 0.70/4.00/0.70/0.21; 10 0.55/6.78/0.53/0.31; 20 8.70/11.48 5.20/0.36/1843.92 — 3.36
Y2-500 + decoded-latest cache / sqlite / C1 Canvas 9.72/42.49/10.52 2.18/35.19/1.09/4.28; 50 0.00/0.00/2.17/8.35; 100 53.72/28.06 15.30/67.86/1824.07 105.46/10.52/129.8 MiB 31.34

Y2 diagnosis and remedies

All three original Y2 Canvas configurations miss the 50 ms latency rule. At N=100 on either backend, middle Restore is dominated by checkpoint decode/apply (65.07–65.91 ms p95); replay is only 4.38–4.88 ms over 50 updates. At N=500/SQLite, middle Restore is 99.33 ms, with 91.57 ms in checkpoint decode/apply and 4.43 ms replay over 50 updates. At N=2,000/SQLite, middle Restore is 352.22 ms: 252.78 ms decode/apply plus 86.68 ms replay over 1,550 updates; end Restore is 142.77 ms, including 91.08 ms replay over 1,100 updates. Thus decode dominates at N=100/500; both decode and long-tail replay dominate at N=2,000. Undo p95 stays below 50 ms in every tested Y2 configuration (25.81–38.61 ms).

The profile separates checkpoint state encode, compression and durable write. For N=100 SQLite their max-per-run p95 is 18.59/75.34/2,618.57 ms; N=500 SQLite 13.78/62.02/2,407.87 ms; N=2,000 SQLite 28.48/57.94/1,063.05 ms. These checkpoint-write costs are substantial, but they are not the interactive Restore operation causing the 50 ms misses.

The tested remedies do not make Y2 eligible. N=25/SQLite reduces Canvas middle Restore to 51.46 ms, still 1.46 ms over the limit; N=25/segment is 88.40 ms. Their Canvas open footprints grow to 436.74/434.75 MiB. The decoded-latest-checkpoint cache at N=500 brings Canvas Restore start/middle/end to 9.72/42.49/10.52 ms, but cold-open remains 53.72 ms (3.72 ms over); cache warm-up is 105.46 ms and cache RSS is 129.8 MiB. The cache improves Restore, not a new cold open.

J is the element-level Canvas checkpoint probe: it stores a current map keyed by stable element ID plus changed-element diffs. Its Canvas Restore/cold-open/undo p95 are 37.44/32.22/26.64 ms and its combined disk score is 13.28 MiB. It demonstrates size and latency headroom, while remaining only the issue’s JSON lower bound, not a Yrs element-level history implementation.

Decisions and remaining gaps

  • Y3 and Y4 use N=500, the earlier best-measured Y2 target. I did not change the decision rule or retention rule.
  • I measured element-level Canvas checkpoints through the existing J lower-bound candidate; I did not treat that JSON format as a deployable collaboration history.
  • The Loro reference enables detached editing only after historical checkout, so selective undo can be measured on an isolated branch. Canvas must select a non-empty undo action; a Note range with no selected action is a valid no-op.
  • The after-fold metric keeps one current point only. It is a storage floor and does not define retention. No Phase 2 implementation was started.
  • No UI, route or product API changed. The full-workspace merge-round gates remain outside this job.

Final gates

cargo fmt --check — exit 0, no output.

cargo clippy --manifest-path bench/live-history/Cargo.toml --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.58s

cargo test --manifest-path bench/live-history/Cargo.toml -- --test-threads=1:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.72s
     Running unittests src/lib.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-3d74a7a79e47da91)

running 2 tests
test tests::canvas_trace_has_deterministic_full_workload_shape ... ok
test tests::note_trace_has_two_authors_and_rewritten_blocks ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.57s

     Running unittests src/main.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-9158f6672573e228)

running 3 tests
test tests::candidate_parser_requires_the_gc_off_trace_for_g1 ... ok
test tests::trace_fixture_round_trips_updates_and_authors ... ok
test tests::yrs_segment_and_sqlite_restore_and_undo ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.51s

   Doc-tests calternal_live_history_bench

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
## Phase 1 continuation complete Branch: `job/hist-975`; head: `3400b72ce8615a21913e68a046c01f075cc8b448`. I merged `origin/dev` once before the final gates. The matrix has five fresh-process repetitions for all 13 shortlisted configurations on both C1 Canvas and N1 Note. Y2 diagnostics cover nine configuration/workload groups with five repetitions each. Each new perf-VM run held `/root/perf.lock` and logged load inside the lock. The 80 newly completed shortlist runs had load average 0.03–2.02 (median 1.95); reused rows retain their earlier locked-run logs. ### Recommendation Recommend **Y4 at N=500 with segment storage** for Phase 2 review. It is the lowest-disk eligible Yrs configuration: combined median open footprint `(C1 + N1) / 2` is **16.19 MiB** (C1 31.86 MiB; N1 0.52 MiB), versus 91.73 MiB for Y1-500/SQLite and 369.63 MiB for Y1-100/segment. Y4-500/segment passes every correctness, latency and memory gate. Its closest metric is Canvas Restore p95 at 48.55 ms, 1.45 ms below the limit; Canvas cold-open/undo p95 are 39.32/19.91 ms. Note Restore/cold-open/undo p95 are 6.86/7.56/3.62 ms. Peak process RSS p95 is 157.02 MiB on Canvas and 18.98 MiB on Note, below their respective 247.43/26.44 MiB limits. Y1-100/segment and Y1-500/SQLite also pass. J (JSON element diffs) has a 13.28 MiB combined score and meets the measured gates, but remains a lower bound because it does not preserve live CRDT history. L (Loro) is 3.54× smaller than the best eligible Yrs option, but fails Canvas Restore (136.56 ms p95), Note Restore (91.14 ms), and Canvas cold-open (62.00 ms), so it does not meet the engine-switch rule. No rule change is proposed: eligible Yrs candidates exist. Phase 2 remains for owner review. ### Full shortlist matrix Rows are per candidate, backend and workload. Metric pairs are median/p95 over five runs; p95 uses nearest rank, so with five runs it is the maximum. `S/M/E` means Restore to start/middle/end. Disk open includes SQLite main/WAL/shared-memory files or segment metadata. Folded disk is the one-current-point storage floor, not a retention policy. Snapshot mean/max are in milliseconds. RSS is process high-water RSS; live-room RSS and encoded bytes describe the current live document. | Candidate / backend / workload | Open disk MiB p50/p95 | Folded disk MiB p50/p95 | Append CPU µs/update p50/p95 | Snapshot mean/max ms p50/p95 | Restore start/middle/end ms p50/p95 | Cold-open / undo ms p50/p95 | Peak process RSS MiB p50/p95 | Live RSS / encoded MiB p50/p95 | Correctness | |---|---:|---:|---:|---|---|---|---:|---|---| | Y1-100 / sqlite / C1 Canvas | 745.56/745.56 | 17.07/17.07 | 40.32/50.72 | mean 491.31/709.48; max 3195.86/9500.61 | S 6.33/10.04, M 21.38/29.12, E 27.01/53.95 | cold 22.01/30.63; undo 13.51/16.80 | 164.17/171.16 | RSS 135.64/135.66; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y1-100 / sqlite / N1 Note | 6.83/6.83 | 5.53/5.53 | 35.39/41.76 | mean 52.46/93.65; max 1360.31/1460.86 | S 1.72/2.22, M 3.29/3.84, E 5.12/5.91 | cold 4.24/7.68; undo 3.64/5.75 | 19.16/19.35 | RSS 15.24/15.39; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y1-100 / segment / C1 Canvas | 733.53/733.53 | 8.48/8.48 | 6.02/6.83 | mean 241.75/325.18; max 2363.63/3557.00 | S 9.68/17.06, M 20.68/32.45, E 28.05/33.82 | cold 24.42/28.46; undo 11.76/15.86 | 138.75/138.79 | RSS 113.44/113.50; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y1-100 / segment / N1 Note | 5.72/5.72 | 0.08/0.08 | 3.93/4.90 | mean 33.67/100.98; max 311.65/427.85 | S 1.87/2.47, M 3.20/5.19, E 3.44/6.31 | cold 3.88/6.72; undo 2.08/3.32 | 15.00/15.04 | RSS 11.38/11.45; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y1-500 / sqlite / C1 Canvas | 180.30/180.30 | 17.07/17.07 | 31.49/43.65 | mean 462.73/3648.49; max 2150.20/15620.89 | S 7.32/9.48, M 19.62/30.12, E 41.47/46.51 | cold 27.60/32.94; undo 14.40/19.76 | 171.67/188.60 | RSS 136.46/136.54; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y1-500 / sqlite / N1 Note | 3.16/3.16 | 1.98/1.98 | 28.43/31.53 | mean 31.85/36.48; max 87.21/281.89 | S 1.43/1.96, M 2.70/4.25, E 5.17/6.81 | cold 5.27/8.21; undo 2.97/3.80 | 19.07/19.11 | RSS 14.93/14.96; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y1-500 / segment / C1 Canvas | 167.51/167.51 | 8.48/8.48 | 4.60/6.41 | mean 182.03/447.24; max 707.81/4186.94 | S 9.10/13.20, M 17.20/25.94, E 23.69/294.73 | cold 23.11/38.15; undo 10.51/14.02 | 146.64/163.43 | RSS 112.70/112.77; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y1-500 / segment / N1 Note | 2.05/2.05 | 0.07/0.07 | 3.32/4.03 | mean 45.95/109.60; max 166.57/615.30 | S 1.64/2.16, M 2.91/4.33, E 5.12/7.46 | cold 4.97/8.17; undo 2.24/3.27 | 14.83/14.95 | RSS 11.15/11.32; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y1-2000 / sqlite / C1 Canvas | 69.77/69.77 | 17.07/17.07 | 33.93/44.53 | mean 487.34/769.42; max 1026.35/3704.88 | S 4.88/8.52, M 77.73/109.93, E 89.18/119.69 | cold 84.75/118.20; undo 12.89/23.51 | 188.33/189.05 | RSS 133.81/134.08; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y1-2000 / sqlite / N1 Note | 2.39/2.39 | 1.30/1.30 | 27.06/34.51 | mean 22.97/29.97; max 31.37/50.77 | S 1.33/1.65, M 6.57/10.60, E 9.56/15.98 | cold 10.04/17.69; undo 2.25/3.48 | 18.47/18.50 | RSS 13.95/13.97; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y1-2000 / segment / C1 Canvas | 58.42/58.42 | 8.48/8.48 | 5.59/7.36 | mean 209.89/1434.77; max 252.46/3953.13 | S 8.84/14.75, M 79.97/109.45, E 88.11/133.92 | cold 88.67/138.88; undo 15.59/18.99 | 162.97/165.18 | RSS 112.11/112.15; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y1-2000 / segment / N1 Note | 1.36/1.36 | 0.07/0.07 | 2.93/3.73 | mean 31.51/70.06; max 43.29/156.61 | S 1.42/2.33, M 3.79/7.45, E 7.93/14.66 | cold 8.53/15.36; undo 1.89/3.90 | 14.70/14.76 | RSS 11.09/11.12; encoded 0.06/0.06 | restore/undo correct in 5/5 | | G1-500 / segment / C1 Canvas | 40.77/40.77 | 18.19/18.19 | 7.08/7.52 | mean 65.92/82.09; max 667.04/740.26 | S 33.42/47.12, M 72.45/141.37, E 92.52/276.25 | cold 25.22/31.02; undo 14.70/17.01 | 179.88/179.93 | RSS 132.22/132.29; encoded 18.17/18.17 | restore/undo correct in 5/5 | | G1-500 / segment / N1 Note | 1.05/1.05 | 0.08/0.08 | 3.40/4.05 | mean 37.31/91.06; max 200.78/1352.33 | S 3.90/4.81, M 4.05/4.99, E 5.19/7.03 | cold 2.32/3.33; undo 2.06/2.75 | 15.29/15.29 | RSS 11.30/11.31; encoded 0.07/0.07 | restore/undo correct in 5/5 | | L / native / C1 Canvas | 9.10/9.10 | 2.89/2.89 | 4.35/5.49 | mean 289.32/3058.36; max 289.32/3058.36 | S 109.26/136.56, M 72.89/95.71, E 0.01/0.01 | cold 21.60/62.00; undo 6.87/10.98 | 215.92/216.21 | RSS 172.97/173.09; encoded 2.89/2.89 | restore/undo correct in 5/5 | | L / native / N1 Note | 0.04/0.04 | 0.03/0.03 | 4.49/5.85 | mean 52.79/1261.59; max 52.79/1261.59 | S 75.18/91.14, M 68.04/79.73, E 0.00/0.01 | cold 0.74/0.85; undo 1.24/1.96 | 18.81/18.82 | RSS 14.34/14.41; encoded 0.02/0.02 | restore/undo correct in 5/5 | | J / segment / C1 Canvas | 26.03/26.03 | 1.29/1.29 | 14.98/19.43 | mean 95.33/109.67; max 230.16/270.08 | S 7.50/8.26, M 24.05/37.44, E 25.11/31.13 | cold 17.89/32.22; undo 18.21/26.64 | 120.26/121.44 | RSS 99.58/100.52; encoded 8.49/8.49 | restore/undo correct in 5/5 | | J / segment / N1 Note | 0.52/0.52 | 0.01/0.01 | 4.43/5.16 | mean 27.79/187.86; max 122.69/2579.32 | S 1.29/1.62, M 1.52/1.70, E 1.69/1.81 | cold 1.59/2.00; undo 3.51/4.86 | 10.21/11.72 | RSS 9.57/9.71; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y3-500 / sqlite / C1 Canvas | 31.26/31.26 | 28.82/28.82 | 16.49/18.51 | mean 165.63/294.45; max 1216.36/4264.33 | S 6.80/86.71, M 27.09/36.87, E 29.36/37.06 | cold 34.64/40.93; undo 18.72/23.74 | 162.42/164.84 | RSS 119.72/119.77; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y3-500 / sqlite / N1 Note | 0.78/0.78 | 0.18/0.18 | 6.78/8.49 | mean 24.59/32.82; max 55.65/128.04 | S 1.69/2.76, M 3.33/5.56, E 5.43/7.17 | cold 6.72/7.59; undo 3.55/4.77 | 18.84/18.95 | RSS 13.86/13.92; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y3-500 / segment / C1 Canvas | 26.43/26.43 | 1.29/1.29 | 11.83/14.80 | mean 102.93/131.70; max 480.80/758.54 | S 6.88/10.63, M 29.69/43.53, E 31.13/46.98 | cold 39.09/50.14; undo 18.96/22.98 | 154.79/155.88 | RSS 113.05/113.14; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y3-500 / segment / N1 Note | 0.52/0.52 | 0.01/0.01 | 3.95/4.71 | mean 21.10/94.31; max 72.46/1279.12 | S 1.20/2.10, M 2.70/4.57, E 5.07/8.10 | cold 5.00/7.74; undo 2.63/3.64 | 15.23/15.27 | RSS 12.01/12.04; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y4-500 / sqlite / C1 Canvas | 37.11/37.11 | 34.95/34.95 | 17.75/23.52 | mean 121.01/156.07; max 319.85/768.21 | S 7.38/12.14, M 25.64/42.86, E 31.05/55.92 | cold 34.82/82.17; undo 18.14/30.67 | 164.45/164.61 | RSS 120.93/121.05; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y4-500 / sqlite / N1 Note | 0.94/0.94 | 0.27/0.27 | 7.73/8.97 | mean 27.38/39.27; max 82.09/361.17 | S 1.63/19.65, M 2.62/4.57, E 5.03/9.95 | cold 5.27/9.70; undo 3.88/5.53 | 20.71/20.77 | RSS 17.61/17.62; encoded 0.06/0.06 | restore/undo correct in 5/5 | | Y4-500 / segment / C1 Canvas | 31.86/31.86 | 1.62/1.62 | 13.82/17.28 | mean 84.51/98.39; max 287.68/365.62 | S 6.25/11.39, M 24.75/48.55, E 27.21/38.68 | cold 29.73/39.32; undo 16.08/19.91 | 156.80/157.02 | RSS 113.10/113.16; encoded 8.46/8.46 | restore/undo correct in 5/5 | | Y4-500 / segment / N1 Note | 0.52/0.52 | 0.02/0.02 | 4.42/4.96 | mean 17.35/38.18; max 40.43/206.40 | S 1.53/1.98, M 2.36/3.48, E 4.22/6.86 | cold 4.39/7.56; undo 2.65/3.62 | 18.94/18.98 | RSS 15.79/15.86; encoded 0.06/0.06 | restore/undo correct in 5/5 | All metric pairs show median/p95 over five runs. P95 uses nearest rank, so it is the maximum of five. `S/M/E` are Restore to start/middle/end. `Snapshot mean/max` reports average and maximum checkpoint times. Process RSS values are process high-water marks. ### Y2 Restore profile and remedies Times below are p95. `mid/end stages` use checkpoint read / decode+apply / log load / update replay, then replayed update count. Checkpoint values are maximum per-run state encode / compression / durable write. | Configuration / workload | Restore S/M/E ms | Mid stages ms; updates | End stages ms; updates | Cold / undo ms | Checkpoint max encode/compress/write ms | Cache warm/access/RSS | Open disk MiB p50 | |---|---|---|---|---|---|---|---:| | Y2-100 / segment / C1 Canvas | 13.92/75.07/43.74 | 1.28/65.91/3.67/4.38; 50 | 1.08/39.51/3.13/0.00; 0 | 44.19/25.81 | 17.83/83.21/1302.62 | — | 112.51 | | Y2-100 / sqlite / C1 Canvas | 14.53/71.81/45.26 | 3.37/65.07/1.24/4.88; 50 | 2.23/42.21/0.71/0.00; 0 | 47.73/33.02 | 18.59/75.34/2618.57 | — | 117.49 | | Y2-500 / sqlite / C1 Canvas | 11.66/99.33/50.50 | 2.35/91.57/1.38/4.43; 50 | 1.86/37.99/1.51/9.11; 100 | 53.15/26.37 | 13.78/62.02/2407.87 | — | 31.34 | | Y2-2000 / sqlite / C1 Canvas | 29.20/352.22/142.77 | 1.90/252.78/12.21/86.68; 1550 | 2.12/41.10/8.84/91.08; 1100 | 138.81/26.35 | 28.48/57.94/1063.05 | — | 11.39 | | Y2-25 / segment / C1 Canvas | 14.68/88.40/61.09 | 1.01/81.48/5.77/0.00; 0 | 1.13/54.39/5.48/0.00; 0 | 48.62/30.10 | 22.27/115.17/4837.05 | — | 434.75 | | Y2-25 / sqlite / C1 Canvas | 15.77/51.46/39.31 | 1.90/49.20/0.94/0.00; 0 | 2.02/36.84/1.41/0.00; 0 | 45.42/38.61 | 43.10/193.66/4850.61 | — | 436.74 | | Y2-25 / segment / N1 Note | 5.46/6.64/14.91 | 0.04/3.14/3.25/0.26; 10 | 0.07/7.45/7.05/0.29; 20 | 11.58/11.25 | 2.52/2.02/2044.05 | — | 2.29 | | Y2-25 / sqlite / N1 Note | 7.12/5.34/8.07 | 0.70/4.00/0.70/0.21; 10 | 0.55/6.78/0.53/0.31; 20 | 8.70/11.48 | 5.20/0.36/1843.92 | — | 3.36 | | Y2-500 + decoded-latest cache / sqlite / C1 Canvas | 9.72/42.49/10.52 | 2.18/35.19/1.09/4.28; 50 | 0.00/0.00/2.17/8.35; 100 | 53.72/28.06 | 15.30/67.86/1824.07 | 105.46/10.52/129.8 MiB | 31.34 | ### Y2 diagnosis and remedies All three original Y2 Canvas configurations miss the 50 ms latency rule. At N=100 on either backend, middle Restore is dominated by checkpoint decode/apply (65.07–65.91 ms p95); replay is only 4.38–4.88 ms over 50 updates. At N=500/SQLite, middle Restore is 99.33 ms, with 91.57 ms in checkpoint decode/apply and 4.43 ms replay over 50 updates. At N=2,000/SQLite, middle Restore is 352.22 ms: 252.78 ms decode/apply plus 86.68 ms replay over 1,550 updates; end Restore is 142.77 ms, including 91.08 ms replay over 1,100 updates. Thus decode dominates at N=100/500; both decode and long-tail replay dominate at N=2,000. Undo p95 stays below 50 ms in every tested Y2 configuration (25.81–38.61 ms). The profile separates checkpoint state encode, compression and durable write. For N=100 SQLite their max-per-run p95 is 18.59/75.34/2,618.57 ms; N=500 SQLite 13.78/62.02/2,407.87 ms; N=2,000 SQLite 28.48/57.94/1,063.05 ms. These checkpoint-write costs are substantial, but they are not the interactive Restore operation causing the 50 ms misses. The tested remedies do not make Y2 eligible. N=25/SQLite reduces Canvas middle Restore to 51.46 ms, still 1.46 ms over the limit; N=25/segment is 88.40 ms. Their Canvas open footprints grow to 436.74/434.75 MiB. The decoded-latest-checkpoint cache at N=500 brings Canvas Restore start/middle/end to 9.72/42.49/10.52 ms, but cold-open remains 53.72 ms (3.72 ms over); cache warm-up is 105.46 ms and cache RSS is 129.8 MiB. The cache improves Restore, not a new cold open. J is the element-level Canvas checkpoint probe: it stores a current map keyed by stable element ID plus changed-element diffs. Its Canvas Restore/cold-open/undo p95 are 37.44/32.22/26.64 ms and its combined disk score is 13.28 MiB. It demonstrates size and latency headroom, while remaining only the issue’s JSON lower bound, not a Yrs element-level history implementation. ### Decisions and remaining gaps - Y3 and Y4 use N=500, the earlier best-measured Y2 target. I did not change the decision rule or retention rule. - I measured element-level Canvas checkpoints through the existing J lower-bound candidate; I did not treat that JSON format as a deployable collaboration history. - The Loro reference enables detached editing only after historical checkout, so selective undo can be measured on an isolated branch. Canvas must select a non-empty undo action; a Note range with no selected action is a valid no-op. - The after-fold metric keeps one current point only. It is a storage floor and does not define retention. No Phase 2 implementation was started. - No UI, route or product API changed. The full-workspace merge-round gates remain outside this job. ### Final gates `cargo fmt --check` — exit 0, no output. `cargo clippy --manifest-path bench/live-history/Cargo.toml --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.58s ``` `cargo test --manifest-path bench/live-history/Cargo.toml -- --test-threads=1`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 0.72s Running unittests src/lib.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-3d74a7a79e47da91) running 2 tests test tests::canvas_trace_has_deterministic_full_workload_shape ... ok test tests::note_trace_has_two_authors_and_rewritten_blocks ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.57s Running unittests src/main.rs (/mnt/hdd/targets/jobs/hist-975/debug/deps/calternal_live_history_bench-9158f6672573e228) running 3 tests test tests::candidate_parser_requires_the_gc_off_trace_for_g1 ... ok test tests::trace_fixture_round_trips_updates_and_authors ... ok test tests::yrs_segment_and_sqlite_restore_and_undo ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.51s Doc-tests calternal_live_history_bench running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ```
kayg commented 2026-10-03 14:05:46 +00:00
Author
Owner
Copy link

Phase 2 plan (owner approved 2026-10-03): six parallel slices, one integration + review at the end

Winner from Phase 1: Y4, N=500, segment storage (Yjs v1 update rows batched per flush, zstd with a trained dictionary; v2+zstd checkpoints every 500 updates; gc on; decoded-checkpoint cache). Contract: DESIGN §61. Owner rule: cheapest footprint; review once at the end, not per slice.

Shared contract (every slice codes against this; do not change it without noting it in your report)

Module crates/calternal-collab/src/history/ (new):

pub struct DocKey { pub owner: UserId, pub item_id: ItemId }          // stable identity, never a path
pub enum Author { User(UserId), Guest(String), AgentTurn(TurnId) }    // bound to the connection, never from the update
pub struct PointId(pub u64);                                          // monotonic per DocKey
pub struct HistoryPoint { pub id: PointId, pub at_ms: i64, pub author: Author, pub bytes: u32 }

#[async_trait] pub trait HistoryStore: Send + Sync {
    async fn append(&self, doc: &DocKey, author: &Author, update_v1: &[u8], at_ms: i64) -> Result<PointId>;
    async fn points(&self, doc: &DocKey, after: Option<PointId>, limit: u16) -> Result<Vec<HistoryPoint>>;
    async fn state_at(&self, doc: &DocKey, point: PointId) -> Result<Vec<u8>>;          // v1 state update
    async fn updates_by(&self, doc: &DocKey, author: &Author, from: PointId, to: PointId) -> Result<Vec<(PointId, Vec<u8>)>>;
    async fn fold(&self, doc: &DocKey, keep_after_ms: i64) -> Result<FoldReport>;      // retention
    async fn usage(&self, owner: &UserId) -> Result<u64>;                               // bytes, counts against quota
}
pub struct MemoryHistoryStore; // in-memory reference impl + shared conformance tests (slice A owns it; others may stub until merge)

Restore and undo produce a normal Yjs update applied through the one collaboration event path (§60): no second writer, clients never reload.

Slices (each its own branch from origin/dev, own worktree, per-branch gates only)

  • A store (Sol): history/store.rs segment files via calternal-fs (append-only, torn tail dropped on open, fsync policy), zstd dictionary training + versioned dictionary IDs, checkpoints, decoded-checkpoint cache, fold, usage; MemoryHistoryStore + conformance test suite used by all impls; crash tests.
  • B write path (Sol): collab hub appends every applied update with Author from the authenticated connection; per-author batching 1–2 s; quota and per-collaborator daily write budget; Canvas and Notes rooms; refuse updates whose Yjs client ID belongs to another connection (spoofing).
  • C restore + undo (Sol): history/restore.rs: state at point → diff → one update; per-author undo with yrs UndoManager + tracked origins; Canvas element-level skip-and-report, Notes block-level; preflight report ("2 items you edited were kept"); property tests (restore(point)==state at point; undo(author) keeps others' changes).
  • D API + parity (Luna): routes: list points, preview state, restore, undo-author preflight/apply; owner-only (shares never see versions, §54); action registry entries (MCP tier: discoverable), CLI commands, OpenAPI + client; isolation matrix classification; adversarial cases.
  • E UI (Luna): Version history for Canvas (Notes behind the same component): anchored popover / phone sheet (§34), timeline grouped by author and session, preview, Restore, "Undo 's changes" with the preflight result, Copy link per point (§33: ?v=<point>); build against D's OpenAPI shapes with a typed mock until D merges; production screenshots.
  • F bench + guards (Luna): port the Phase 1 workloads into bench/ as a profile with thresholds in docs/perf/; tests/adversarial: oversized updates, history growth DoS, clock/length abuse, cross-user point access; a CI guard that history bytes per 1k edits stay under budget.

Integration + review (Sol, after all six report)

Merge A→B→C→D→E→F on one branch, replace stubs, full gates, e2e (two browsers co-editing then restoring and undoing one author), adversarial, bench vs Phase 1 numbers, independent review. One report, one owner review.

## Phase 2 plan (owner approved 2026-10-03): six parallel slices, one integration + review at the end Winner from Phase 1: **Y4, N=500, segment storage** (Yjs v1 update rows batched per flush, zstd with a trained dictionary; v2+zstd checkpoints every 500 updates; gc on; decoded-checkpoint cache). Contract: DESIGN §61. Owner rule: cheapest footprint; review once at the end, not per slice. ### Shared contract (every slice codes against this; do not change it without noting it in your report) Module `crates/calternal-collab/src/history/` (new): ```rust pub struct DocKey { pub owner: UserId, pub item_id: ItemId } // stable identity, never a path pub enum Author { User(UserId), Guest(String), AgentTurn(TurnId) } // bound to the connection, never from the update pub struct PointId(pub u64); // monotonic per DocKey pub struct HistoryPoint { pub id: PointId, pub at_ms: i64, pub author: Author, pub bytes: u32 } #[async_trait] pub trait HistoryStore: Send + Sync { async fn append(&self, doc: &DocKey, author: &Author, update_v1: &[u8], at_ms: i64) -> Result<PointId>; async fn points(&self, doc: &DocKey, after: Option<PointId>, limit: u16) -> Result<Vec<HistoryPoint>>; async fn state_at(&self, doc: &DocKey, point: PointId) -> Result<Vec<u8>>; // v1 state update async fn updates_by(&self, doc: &DocKey, author: &Author, from: PointId, to: PointId) -> Result<Vec<(PointId, Vec<u8>)>>; async fn fold(&self, doc: &DocKey, keep_after_ms: i64) -> Result<FoldReport>; // retention async fn usage(&self, owner: &UserId) -> Result<u64>; // bytes, counts against quota } pub struct MemoryHistoryStore; // in-memory reference impl + shared conformance tests (slice A owns it; others may stub until merge) ``` Restore and undo produce a **normal Yjs update applied through the one collaboration event path** (§60): no second writer, clients never reload. ### Slices (each its own branch from origin/dev, own worktree, per-branch gates only) - **A store (Sol):** `history/store.rs` segment files via `calternal-fs` (append-only, torn tail dropped on open, fsync policy), zstd dictionary training + versioned dictionary IDs, checkpoints, decoded-checkpoint cache, fold, usage; `MemoryHistoryStore` + conformance test suite used by all impls; crash tests. - **B write path (Sol):** collab hub appends every applied update with `Author` from the authenticated connection; per-author batching 1–2 s; quota and per-collaborator daily write budget; Canvas and Notes rooms; refuse updates whose Yjs client ID belongs to another connection (spoofing). - **C restore + undo (Sol):** `history/restore.rs`: state at point → diff → one update; per-author undo with yrs `UndoManager` + tracked origins; Canvas element-level skip-and-report, Notes block-level; preflight report ("2 items you edited were kept"); property tests (restore(point)==state at point; undo(author) keeps others' changes). - **D API + parity (Luna):** routes: list points, preview state, restore, undo-author preflight/apply; owner-only (shares never see versions, §54); action registry entries (MCP tier: discoverable), CLI commands, OpenAPI + client; isolation matrix classification; adversarial cases. - **E UI (Luna):** Version history for Canvas (Notes behind the same component): anchored popover / phone sheet (§34), timeline grouped by author and session, preview, Restore, "Undo <author>'s changes" with the preflight result, Copy link per point (§33: `?v=<point>`); build against D's OpenAPI shapes with a typed mock until D merges; production screenshots. - **F bench + guards (Luna):** port the Phase 1 workloads into `bench/` as a profile with thresholds in `docs/perf/`; tests/adversarial: oversized updates, history growth DoS, clock/length abuse, cross-user point access; a CI guard that history bytes per 1k edits stay under budget. ### Integration + review (Sol, after all six report) Merge A→B→C→D→E→F on one branch, replace stubs, full gates, e2e (two browsers co-editing then restoring and undoing one author), adversarial, bench vs Phase 1 numbers, independent review. One report, one owner review.
kayg commented 2026-10-03 14:07:39 +00:00
Author
Owner
Copy link

Starting Phase 2 slice A/store on job/hist2-store, base/head d4e7188810a89fb0e8e6b162279917f7e23989f9. Scope: shared history contract, segment storage through calternal-fs, crash recovery, dictionaries/checkpoints/cache, retention and usage, memory reference and conformance tests. Other Phase 2 slices remain outside this branch. I will preserve the approved trait signatures and report any additional type definitions required by the contract.

Starting Phase 2 slice A/store on `job/hist2-store`, base/head `d4e7188810a89fb0e8e6b162279917f7e23989f9`. Scope: shared history contract, segment storage through calternal-fs, crash recovery, dictionaries/checkpoints/cache, retention and usage, memory reference and conformance tests. Other Phase 2 slices remain outside this branch. I will preserve the approved trait signatures and report any additional type definitions required by the contract.
kayg commented 2026-10-03 14:07:57 +00:00
Author
Owner
Copy link

Starting Phase 2 slice B (write) on job/hist2-write, base d4e7188810. Scope: authenticated author attribution, applied-update history capture and batching, quota/write-budget checks, and connection-bound Yjs client identity. I will keep HistoryStore signatures from the approved Phase 2 plan and report any required integration hooks.

Starting Phase 2 slice B (write) on job/hist2-write, base d4e7188810a89fb0e8e6b162279917f7e23989f9. Scope: authenticated author attribution, applied-update history capture and batching, quota/write-budget checks, and connection-bound Yjs client identity. I will keep HistoryStore signatures from the approved Phase 2 plan and report any required integration hooks.
kayg commented 2026-10-03 14:08:46 +00:00
Author
Owner
Copy link

Starting Phase 2 slice API on branch job/hist2-api, based on origin/dev at d4e7188810. I am reading the approved Phase 2 contract and will keep this branch to the API-owned routes, parity surfaces, contract stubs, and focused tests.

Starting Phase 2 slice API on branch job/hist2-api, based on origin/dev at d4e7188810a89fb0e8e6b162279917f7e23989f9. I am reading the approved Phase 2 contract and will keep this branch to the API-owned routes, parity surfaces, contract stubs, and focused tests.
kayg commented 2026-10-03 14:09:12 +00:00
Author
Owner
Copy link

Starting Phase 2 slice E (UI) on job/hist2-ui, based on origin/dev at d4e7188810a89fb0e8e6b162279917f7e23989f9. Scope: Canvas Version history UI with shared Notes component, preview, Restore, author undo preflight, and ?v=<point> deep links, using the approved D API shapes and a typed mock until that slice merges. I will keep changes within UI-owned files and report any contract adjustment.

Starting Phase 2 slice E (UI) on `job/hist2-ui`, based on `origin/dev` at `d4e7188810a89fb0e8e6b162279917f7e23989f9`. Scope: Canvas Version history UI with shared Notes component, preview, Restore, author undo preflight, and `?v=<point>` deep links, using the approved D API shapes and a typed mock until that slice merges. I will keep changes within UI-owned files and report any contract adjustment.
kayg commented 2026-10-03 14:09:52 +00:00
Author
Owner
Copy link

Starting Phase 2 slice C/restore on job/hist2-restore, base d4e7188810a89fb0e8e6b162279917f7e23989f9. Scope: history/restore.rs, shared contract stub if absent, restore as one forward Yjs update, tracked-origin selective undo, whole-item skip/report, and focused seeded property tests. Persistence, event capture, routes, UI and benchmarks remain with their assigned slices. No pushes or deployment. Existing Note merge/serialization helpers will be reused. Registry verification: yrs 0.28.0 and async-trait 0.1.92.

Starting Phase 2 slice C/restore on `job/hist2-restore`, base `d4e7188810a89fb0e8e6b162279917f7e23989f9`. Scope: `history/restore.rs`, shared contract stub if absent, restore as one forward Yjs update, tracked-origin selective undo, whole-item skip/report, and focused seeded property tests. Persistence, event capture, routes, UI and benchmarks remain with their assigned slices. No pushes or deployment. Existing Note merge/serialization helpers will be reused. Registry verification: yrs 0.28.0 and async-trait 0.1.92.
kayg commented 2026-10-03 14:09:53 +00:00
Author
Owner
Copy link

Starting Phase 2 slice F (bench + guards) on branch job/hist2-bench, based on origin/dev at d4e7188810a89fb0e8e6b162279917f7e23989f9. I will port the approved Phase 1 workloads into a bench/ profile, add perf thresholds and the history-bytes CI guard, and add focused adversarial coverage. I will keep implementation within bench-owned files and add the approved shared contract signatures only if this branch does not contain them.

Starting Phase 2 slice F (bench + guards) on branch `job/hist2-bench`, based on `origin/dev` at `d4e7188810a89fb0e8e6b162279917f7e23989f9`. I will port the approved Phase 1 workloads into a `bench/` profile, add perf thresholds and the history-bytes CI guard, and add focused adversarial coverage. I will keep implementation within bench-owned files and add the approved shared contract signatures only if this branch does not contain them.
kayg commented 2026-10-03 14:16:38 +00:00
Author
Owner
Copy link

Finding: origin/dev does not contain the Phase 1 history benchmark (git ls-tree -r origin/dev bench/live-history is empty). The prior job/hist-975 result contains the deterministic C1/N1 workload generator and Y4 segment runner. I will port that measured harness into the slice and add a Y4-only profile plus a CI budget check, so the approved Phase 1 comparison remains reproducible.

Finding: `origin/dev` does not contain the Phase 1 history benchmark (`git ls-tree -r origin/dev bench/live-history` is empty). The prior `job/hist-975` result contains the deterministic C1/N1 workload generator and Y4 segment runner. I will port that measured harness into the slice and add a Y4-only profile plus a CI budget check, so the approved Phase 1 comparison remains reproducible.
kayg commented 2026-10-03 14:21:10 +00:00
Author
Owner
Copy link

Slice B finding: PluginEvent carries the Home owner and path but no author. I added a server-only notify_external_change_as hook for parity callers with a verified actor; legacy filesystem notices use the Home owner. The history writer has its own migration namespace (collab-history-write) for durable daily counters, so no other crate migration is changed. Integration must install slice A through Hub::with_history and use Home-relative segment files whose bytes are included in calternal-fs quota accounting; reservations cover queued updates until append succeeds. Canvas rooms do not exist on this base; HistoryWriter is document-neutral for their event path.

Slice B finding: PluginEvent carries the Home owner and path but no author. I added a server-only notify_external_change_as hook for parity callers with a verified actor; legacy filesystem notices use the Home owner. The history writer has its own migration namespace (collab-history-write) for durable daily counters, so no other crate migration is changed. Integration must install slice A through Hub::with_history and use Home-relative segment files whose bytes are included in calternal-fs quota accounting; reservations cover queued updates until append succeeds. Canvas rooms do not exist on this base; HistoryWriter is document-neutral for their event path.
kayg commented 2026-10-03 14:27:54 +00:00
Author
Owner
Copy link

Store progress and contract details:

  • The initial memory-store conformance test failed at its first append with Err(NotFound) against the stub. The implementation now uses the real Yjs replay engine; persistent conformance, checkpoint, dictionary, invalid-flush, clock-reversal and crash-tail tests are running.
  • All approved HistoryStore signatures are unchanged. The plan did not define UserId, ItemId, TurnId, Result or FoldReport. This branch defines the IDs as the existing server string IDs, typed errors, and a fold report with removed_points, reclaimed_bytes, and oldest_point.
  • points uses an exclusive ascending cursor, capped at 1,000. updates_by includes both range endpoints. Point zero denotes the initial empty state but is not a listed/restorable point. The first append must contain the initial full v1 state so history knows the seed.
  • Added append_batch as an inherent store API (not a trait change), so slice B can persist up to 64 attributed rows in one flush. Checkpoints are captured at exact point multiples of 500 even when a flush crosses the boundary. Each row keeps its own point and authenticated author. An idle checkpoint method and explicit idle cache evict are available to the hub.
  • The minimal calternal-fs addition is a confined history-file handle with a separate writer-lock inode, bounded range reads, durable append/truncate, and streamed atomic replacement. No existing filesystem behavior changes.
  • Retention removes only an expired contiguous prefix and preserves a full checkpoint at the first retained point. It keeps the latest point even if all rows are old, so clock reversals cannot break replay or erase current state. Dictionary-free zstd is used until real edit samples support bounded dictionary training; versioned dictionaries remain in the segment.
Store progress and contract details: - The initial memory-store conformance test failed at its first append with `Err(NotFound)` against the stub. The implementation now uses the real Yjs replay engine; persistent conformance, checkpoint, dictionary, invalid-flush, clock-reversal and crash-tail tests are running. - All approved `HistoryStore` signatures are unchanged. The plan did not define `UserId`, `ItemId`, `TurnId`, `Result` or `FoldReport`. This branch defines the IDs as the existing server string IDs, typed errors, and a fold report with `removed_points`, `reclaimed_bytes`, and `oldest_point`. - `points` uses an exclusive ascending cursor, capped at 1,000. `updates_by` includes both range endpoints. Point zero denotes the initial empty state but is not a listed/restorable point. The first append must contain the initial full v1 state so history knows the seed. - Added `append_batch` as an inherent store API (not a trait change), so slice B can persist up to 64 attributed rows in one flush. Checkpoints are captured at exact point multiples of 500 even when a flush crosses the boundary. Each row keeps its own point and authenticated author. An idle `checkpoint` method and explicit idle cache `evict` are available to the hub. - The minimal calternal-fs addition is a confined history-file handle with a separate writer-lock inode, bounded range reads, durable append/truncate, and streamed atomic replacement. No existing filesystem behavior changes. - Retention removes only an expired contiguous prefix and preserves a full checkpoint at the first retained point. It keeps the latest point even if all rows are old, so clock reversals cannot break replay or erase current state. Dictionary-free zstd is used until real edit samples support bounded dictionary training; versioned dictionaries remain in the segment.
kayg commented 2026-10-03 14:29:07 +00:00
Author
Owner
Copy link

Slice C finding and integration contract: a Yjs state vector does not change for a delete-only edit. The prepared restore/undo therefore checks the complete v1 state, plus DocKey, before bytes are exposed to the event path. Preparation captures a detached baseline before store awaits, so a concurrent edit cannot silently become part of its precondition. Focused regressions cover deletion-only staleness and a wrong-owner DocKey.

Shared contract signatures remain unchanged. This branch supplies String aliases for UserId/ItemId/TurnId, boxed store errors, and a FoldReport stub until slice A integrates. Additional required semantics: ranges are inclusive and increasing; PointId(0) is the initial empty state; a retained predecessor is required for undo. C helpers are prepare_restore and prepare_undo; PreparedChange.checked_update requires the DocKey and the live Doc under the hub room lock. Neither helper applies or appends an update. Canvas uses one atomic JSON value per stable element key, with the room map name supplied by its binding.

The first dependency build is in progress on the shared host. Restore and selective undo are being split into separate commits with focused tests.

Slice C finding and integration contract: a Yjs state vector does not change for a delete-only edit. The prepared restore/undo therefore checks the complete v1 state, plus DocKey, before bytes are exposed to the event path. Preparation captures a detached baseline before store awaits, so a concurrent edit cannot silently become part of its precondition. Focused regressions cover deletion-only staleness and a wrong-owner DocKey. Shared contract signatures remain unchanged. This branch supplies String aliases for UserId/ItemId/TurnId, boxed store errors, and a FoldReport stub until slice A integrates. Additional required semantics: ranges are inclusive and increasing; PointId(0) is the initial empty state; a retained predecessor is required for undo. C helpers are prepare_restore and prepare_undo; PreparedChange.checked_update requires the DocKey and the live Doc under the hub room lock. Neither helper applies or appends an update. Canvas uses one atomic JSON value per stable element key, with the room map name supplied by its binding. The first dependency build is in progress on the shared host. Restore and selective undo are being split into separate commits with focused tests.
kayg commented 2026-10-03 14:37:45 +00:00
Author
Owner
Copy link

Focused replay finding: using yrs 0.28 Update::merge for consecutive typing retained only the first character (expected abcd, replayed a). Its implementation merges block containers and does not merge delete sets. I replaced it with Update::merge_updates and added a separate delete-set replay regression. Initial compilation succeeded; the focused write tests caught this before commit. Client-to-author bindings now persist in the same collaboration-owned Security-state migration so unload/restart cannot reset ownership.

Focused replay finding: using yrs 0.28 Update::merge for consecutive typing retained only the first character (expected abcd, replayed a). Its implementation merges block containers and does not merge delete sets. I replaced it with Update::merge_updates and added a separate delete-set replay regression. Initial compilation succeeded; the focused write tests caught this before commit. Client-to-author bindings now persist in the same collaboration-owned Security-state migration so unload/restart cannot reset ownership.
kayg commented 2026-10-03 14:37:52 +00:00
Author
Owner
Copy link

Slice E finding: the Phase 2 plan lists the history API operations but does not define their OpenAPI response fields, and origin/dev has no history routes yet. I added a narrow typed UI adapter so the panel can use a fixture client before D merges. Its integration shape expects: list { points, before }; each point { id, at_ms, author: { kind, id, label }, bytes }; preview { kind, content }; Restore returns the new point; undo preflight returns { token, changed, kept: [{ id, label }] }; undo apply accepts the token. The HistoryStore trait and its approved signatures are unchanged. D/integration can map this adapter to the generated OpenAPI client. Timeline session groups use same-author points separated by at least 30 minutes because the approved point shape has no session ID.

Slice E finding: the Phase 2 plan lists the history API operations but does not define their OpenAPI response fields, and `origin/dev` has no history routes yet. I added a narrow typed UI adapter so the panel can use a fixture client before D merges. Its integration shape expects: list `{ points, before }`; each point `{ id, at_ms, author: { kind, id, label }, bytes }`; preview `{ kind, content }`; Restore returns the new point; undo preflight returns `{ token, changed, kept: [{ id, label }] }`; undo apply accepts the token. The `HistoryStore` trait and its approved signatures are unchanged. D/integration can map this adapter to the generated OpenAPI client. Timeline session groups use same-author points separated by at least 30 minutes because the approved point shape has no session ID.
kayg commented 2026-10-03 14:41:15 +00:00
Author
Owner
Copy link

UI review found that a point deep link can mount before ModeHeader renders the Version history action. A null PopoverSurface anchor falls back to x=0,y=0. I now anchor it to the stable header row until the action exists, so the panel remains in the expected area.

UI review found that a point deep link can mount before ModeHeader renders the Version history action. A null PopoverSurface anchor falls back to x=0,y=0. I now anchor it to the stable header row until the action exists, so the panel remains in the expected area.
kayg commented 2026-10-03 14:45:01 +00:00
Author
Owner
Copy link

Store validation update:

  • Commit 5ad7ee3d8 adds the confined history-file helper. At that step, calternal-fs clippy exited 0 and its tests passed (53 unit tests, 42 integration tests).
  • The full focused history suite passed: 10 passed; 0 failed, with --test-threads=1. It checks every point in a 1,050-edit trace after restart, exact N=500 checkpoint boundaries, a fold through a compressed batch, persisted trained dictionaries, interrupted final frames, interior corruption, invalid flushes, clock reversals, cross-User isolation, concurrent point allocation, and LRU eviction.
  • A parallel focused attempt timed out in three Db::connect calls (Sqlx(PoolTimedOut)), before store construction. The store tests now request one reader and use serialized execution. No existing assertion or expectation changed.
  • A separate decoded-checkpoint cache is now used for past checkpoint reads. Exact checkpoint reads encode the cached document directly. Replay copies its v1 state into a fresh document because Doc::clone shares mutable state; it must not mutate the cached checkpoint. The latest state has its own decoded cache.
  • The required fetch/update from origin/dev is complete: merge commit 6af52f6bc, upstream cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e. Automatic Cargo.toml merging added two blake3 declarations; I kept upstream's workspace dependency and removed the duplicate. Other upstream changes remain intact.
  • Final per-crate gates are running for calternal-fs and calternal-collab. This slice adds no route or UI. Bench/profile, real-server matrices, e2e, release builds and independent review remain for slices F/integration and the merge round under the job verification policy.
Store validation update: - Commit `5ad7ee3d8` adds the confined history-file helper. At that step, calternal-fs clippy exited 0 and its tests passed (53 unit tests, 42 integration tests). - The full focused history suite passed: `10 passed; 0 failed`, with `--test-threads=1`. It checks every point in a 1,050-edit trace after restart, exact N=500 checkpoint boundaries, a fold through a compressed batch, persisted trained dictionaries, interrupted final frames, interior corruption, invalid flushes, clock reversals, cross-User isolation, concurrent point allocation, and LRU eviction. - A parallel focused attempt timed out in three `Db::connect` calls (`Sqlx(PoolTimedOut)`), before store construction. The store tests now request one reader and use serialized execution. No existing assertion or expectation changed. - A separate decoded-checkpoint cache is now used for past checkpoint reads. Exact checkpoint reads encode the cached document directly. Replay copies its v1 state into a fresh document because `Doc::clone` shares mutable state; it must not mutate the cached checkpoint. The latest state has its own decoded cache. - The required fetch/update from `origin/dev` is complete: merge commit `6af52f6bc`, upstream `cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e`. Automatic Cargo.toml merging added two blake3 declarations; I kept upstream's workspace dependency and removed the duplicate. Other upstream changes remain intact. - Final per-crate gates are running for calternal-fs and calternal-collab. This slice adds no route or UI. Bench/profile, real-server matrices, e2e, release builds and independent review remain for slices F/integration and the merge round under the job verification policy.
kayg commented 2026-10-03 14:46:06 +00:00
Author
Owner
Copy link

Phase 2 API finding: base d4e7188 has no calternal-collab history module or HistoryStore contract; the existing Hub only exposes live Note sessions. This slice will add the approved HistoryStore signatures and a small HistoryMutations interface for Restore, undo preflight and apply. The production router will stay owner-only and return service-unavailable until the storage and mutation slices are connected during integration. I will use one stable item ID route so the API supports Canvas and Notes.

Phase 2 API finding: base d4e7188 has no calternal-collab history module or HistoryStore contract; the existing Hub only exposes live Note sessions. This slice will add the approved HistoryStore signatures and a small HistoryMutations interface for Restore, undo preflight and apply. The production router will stay owner-only and return service-unavailable until the storage and mutation slices are connected during integration. I will use one stable item ID route so the API supports Canvas and Notes.
kayg commented 2026-10-03 14:52:09 +00:00
Author
Owner
Copy link

Quota integration finding: Root::recompute_quota already walks .calternal/history, so these segment bytes are included in Home usage. Slice B must not add HistoryStore::usage to that total a second time. usage reports the history share of the same total.

The new low-level append helper currently depends on slice B's admission check. I added a focused regression for the existing Home quota invariant: appending history beyond a configured Home quota must fail before changing the segment. I will enforce the existing calternal-fs growth and free-space checks in this new helper. This does not change any existing filesystem path or move per-collaborator policy out of slice B.

Quota integration finding: `Root::recompute_quota` already walks `.calternal/history`, so these segment bytes are included in Home usage. Slice B must not add `HistoryStore::usage` to that total a second time. `usage` reports the history share of the same total. The new low-level append helper currently depends on slice B's admission check. I added a focused regression for the existing Home quota invariant: appending history beyond a configured Home quota must fail before changing the segment. I will enforce the existing calternal-fs growth and free-space checks in this new helper. This does not change any existing filesystem path or move per-collaborator policy out of slice B.
kayg commented 2026-10-03 14:53:15 +00:00
Author
Owner
Copy link

Deep-link review found that a malformed value was ignored on page load but could still be passed to the panel when opening history from the header. That showed a false 'point unavailable' message beside a valid timeline. The Note action now validates the point ID before passing it to the panel.

Deep-link review found that a malformed value was ignored on page load but could still be passed to the panel when opening history from the header. That showed a false 'point unavailable' message beside a valid timeline. The Note action now validates the point ID before passing it to the panel.
kayg commented 2026-10-03 14:53:28 +00:00
Author
Owner
Copy link

Deep-link review found that a malformed v query value was ignored on page load but could still be passed to the panel when opening history from the header. That showed a false point-unavailable message beside a valid timeline. The Note action now validates the point ID before passing it to the panel.

Deep-link review found that a malformed v query value was ignored on page load but could still be passed to the panel when opening history from the header. That showed a false point-unavailable message beside a valid timeline. The Note action now validates the point ID before passing it to the panel.
kayg commented 2026-10-03 15:01:26 +00:00
Author
Owner
Copy link

UI scope check: the merged origin/dev tree has no Canvas route or Canvas editor ( returned no files). I kept this slice to the reusable VersionHistory surface and a real Notes host, with a custom preview snippet hook for Canvas. I did not add a standalone Canvas screen without its editor/backend.

UI scope check: the merged origin/dev tree has no Canvas route or Canvas editor ( returned no files). I kept this slice to the reusable VersionHistory surface and a real Notes host, with a custom preview snippet hook for Canvas. I did not add a standalone Canvas screen without its editor/backend.
kayg commented 2026-10-03 15:01:39 +00:00
Author
Owner
Copy link

UI decisions where §61 is silent: group a same-author run into one session until there is a 30-minute idle gap (the point contract has no session ID); render Note previews as escaped plain text and let a Canvas host inject its renderer; fetch 100 points per page and cap each API JSON reply at 2 MiB so opening history stays bounded.

UI decisions where §61 is silent: group a same-author run into one session until there is a 30-minute idle gap (the point contract has no session ID); render Note previews as escaped plain text and let a Canvas host inject its renderer; fetch 100 points per page and cap each API JSON reply at 2 MiB so opening history stays bounded.
kayg commented 2026-10-03 15:01:42 +00:00
Author
Owner
Copy link

Slice B progress: core write layer committed at 2a4974013; Hub capture committed at cb79d157b. Focused tests now prove real authenticated WebSocket capture, agent-turn replay, external actor capture and refusal before agent mutation when quota is full. Additional hardening keeps presence-only claims ephemeral (no Security-state rows for Viewer cursors), verifies existing presence IDs against the Index, caches only active ownership, releases quota if admission is cancelled while waiting for the Index, and retries failed appends with one bounded background task. Pending targeted tests and the required origin/dev merge precede final gates.

Slice B progress: core write layer committed at 2a4974013; Hub capture committed at cb79d157b. Focused tests now prove real authenticated WebSocket capture, agent-turn replay, external actor capture and refusal before agent mutation when quota is full. Additional hardening keeps presence-only claims ephemeral (no Security-state rows for Viewer cursors), verifies existing presence IDs against the Index, caches only active ownership, releases quota if admission is cancelled while waiting for the Index, and retries failed appends with one bounded background task. Pending targeted tests and the required origin/dev merge precede final gates.
kayg commented 2026-10-03 15:07:19 +00:00
Author
Owner
Copy link

Slice C progress: Restore is committed in 206bbb148; selective undo in 1e08e6145. The focused suite now passes 12 tests, including 768 seeded Canvas point restores with replica convergence, 64 Canvas undo streams, 64 Note undo streams, foreign equal-value writes, root deletion/resurrection, foreign deletion, protected insertion, anchored block reorder, stale/delete-only preflight checks, concurrent store-read edits, retention gaps and author-range boundaries.

Finding: an order-only Note edit changed the document but was absent from the preflight reverted list. The guard fix now includes selected blocks whose positions changed, while excluding the foreign blocks shifted by that move. The dedicated regression passes. Preparation also retains encoded baseline bytes across awaits and drops decoded replay documents before it builds the forward edit, to reduce peak memory.

Contract additions to confirm at integration: ordered inclusive ranges, point 0 as the empty initial state, and point order matching applied event order across author batching. No approved trait signature changed. Undo requires its retained predecessor and caps one preflight at 10,000 points and 64 MiB of update bytes; states use the existing 16 MiB room-state cap. Over-limit or incomplete ranges fail without a partial apply. Exact DocKey + complete-state apply checks run under the hub lock. Canvas values are atomic JSON map entries; the binding supplies the map name. Notes use blockAnchor where present and Yjs branch identity otherwise; replay-only marker attributes never enter emitted updates.

Per-slice gates so far: cargo fmt --check, cargo clippy -p calternal-collab --all-targets -- -D warnings, focused history_restore tests and the malformed/incomplete-state unit test pass. I will fetch/merge origin/dev once, run the final per-crate gates, and post full verbatim output with the final head. No UI or route changes; the combined browser/adversarial/performance checks remain for the integration round under the verification policy.

Slice C progress: Restore is committed in 206bbb148; selective undo in 1e08e6145. The focused suite now passes 12 tests, including 768 seeded Canvas point restores with replica convergence, 64 Canvas undo streams, 64 Note undo streams, foreign equal-value writes, root deletion/resurrection, foreign deletion, protected insertion, anchored block reorder, stale/delete-only preflight checks, concurrent store-read edits, retention gaps and author-range boundaries. Finding: an order-only Note edit changed the document but was absent from the preflight reverted list. The guard fix now includes selected blocks whose positions changed, while excluding the foreign blocks shifted by that move. The dedicated regression passes. Preparation also retains encoded baseline bytes across awaits and drops decoded replay documents before it builds the forward edit, to reduce peak memory. Contract additions to confirm at integration: ordered inclusive ranges, point 0 as the empty initial state, and point order matching applied event order across author batching. No approved trait signature changed. Undo requires its retained predecessor and caps one preflight at 10,000 points and 64 MiB of update bytes; states use the existing 16 MiB room-state cap. Over-limit or incomplete ranges fail without a partial apply. Exact DocKey + complete-state apply checks run under the hub lock. Canvas values are atomic JSON map entries; the binding supplies the map name. Notes use blockAnchor where present and Yjs branch identity otherwise; replay-only marker attributes never enter emitted updates. Per-slice gates so far: cargo fmt --check, cargo clippy -p calternal-collab --all-targets -- -D warnings, focused history_restore tests and the malformed/incomplete-state unit test pass. I will fetch/merge origin/dev once, run the final per-crate gates, and post full verbatim output with the final head. No UI or route changes; the combined browser/adversarial/performance checks remain for the integration round under the verification policy.
kayg commented 2026-10-03 15:19:40 +00:00
Author
Owner
Copy link

Finding: after I moved the reusable abuse probe into collab test support to keep the benchmark's isolated lockfile small, cargo clippy -p calternal-collab --all-targets -- -D warnings reported AdversarialReport and probe as dead code. I added a test-only in-memory fixture that runs the probe, so oversized-update, growth, extreme-clock, page-length and same-item cross-User assertions execute without adding the server graph to the benchmark binary.

Finding: after I moved the reusable abuse probe into collab test support to keep the benchmark's isolated lockfile small, `cargo clippy -p calternal-collab --all-targets -- -D warnings` reported `AdversarialReport` and `probe` as dead code. I added a test-only in-memory fixture that runs the probe, so oversized-update, growth, extreme-clock, page-length and same-item cross-User assertions execute without adding the server graph to the benchmark binary.
kayg commented 2026-10-03 15:19:48 +00:00
Author
Owner
Copy link

#975 Phase 2 UI complete

Branch: job/hist2-ui
Head: 9560c48e5221226f0dce271cfb7ea253c73ea6dc

Built

  • Added a reusable live-document Version history panel. It groups points by author and session, loads previews, restores a point, preflights selective author undo, and offers Copy link for each point.
  • Added the panel to non-Daily Notes through the header action and overflow menu. A point link uses /n/<calternal-id>?v=<point>.
  • Added a typed API adapter with a 100-point page limit, a 2 MiB response bound, and input validation.

Files: apps/web/src/lib/history/{VersionHistory.svelte,api.ts,sessions.ts} and their three focused test files; apps/web/src/lib/notes/NoteView.svelte.

Contract and gaps

  • No Rust HistoryStore signature changed. The plan does not define OpenAPI response fields. This UI adapter expects list { points, before }; points { id, at_ms, author: { kind, id, label }, bytes }; preview { kind, content }; Restore returns the new point; undo preflight returns { token, changed, kept: [{ id, label }] }; undo apply accepts the token. Slice D or integration must map this adapter to the merged OpenAPI client.
  • origin/dev has no Canvas route or editor. This slice provides the shared panel and a Notes host, plus a preview snippet hook for Canvas. It does not add a Canvas screen without its editor/API.
  • The screenshots use a real Note created through the local server. Only history API replies are typed test fixtures because slice D has not merged.

UX gaps closed

  • Point deep links wait for a stable header anchor; malformed v values no longer show a false missing-point message.
  • Per-author undo names the author, shows the preflight result, and reports both undoable changes and items kept. Restore has an explicit confirmation and an Undo toast action.
  • Point rows, Copy link actions, close, Restore and Undo use keyboard-operable native controls and 44 px touch targets. Empty, loading, error and retry states are present.

UX gaps left

  • The Canvas host and live history API are not on origin/dev. Integration must mount the shared component in Canvas and replace the provisional endpoint adapter. A live two-browser restore/undo E2E remains for the merge round.

Decisions not specified by §61

  • Keep same-author points in one session when the gap is at most 30 minutes; split when it is longer. The approved point contract has no session ID.
  • Render Note preview content as escaped plain text. A Canvas host can provide its own renderer.
  • Load 100 points per page and reject JSON replies over 2 MiB to bound first paint and memory.
  • Use a 440 px desktop popover, with the shared phone sheet on narrow viewports.

Screenshots

Gates (verbatim)

bun run check

$ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
User browser caches use userStorage; only documented device/public-link exceptions remain.
Text sizes and UI shape values use shared role tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/hist2-ui/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bunx vitest run src/lib/history/api.test.ts src/lib/history/sessions.test.ts src/lib/history/VersionHistory.svelte.test.ts --maxWorkers=2

 RUN v5.0.1 /home/kayg/Developer/calternal-wt/hist2-ui/apps/web

 Test Files  3 passed (3)
      Tests  7 passed (7)
   Start at  17:17:48
   Duration  29.66s (transform 77%, tests 8%, environment 7%, import 6%, setup 2%)

  Transform  |component| transforming modules took 22.54s · 77% of tracked time, re-done on every run
             persist transforms across runs with fsModuleCache: true
             learn more at https://vitest.dev/guide/improving-performance#caching

Production build: bun run build succeeded. Screenshot server build: cargo build -p calternal-server succeeded. No Rust source changed, so Rust crate gates did not apply. cargo clean removed 4.6 GiB; web build output was deleted.

For the merge round

Run full web tests and add/run bun e2e/history-975.mjs against the merged API and Canvas. It must prove two browser contexts co-edit, restore a point, and selectively undo one author's changes with the preflight report. Slice F owns the benchmark profile.

# #975 Phase 2 UI complete Branch: `job/hist2-ui` Head: `9560c48e5221226f0dce271cfb7ea253c73ea6dc` ## Built - Added a reusable live-document Version history panel. It groups points by author and session, loads previews, restores a point, preflights selective author undo, and offers Copy link for each point. - Added the panel to non-Daily Notes through the header action and overflow menu. A point link uses `/n/<calternal-id>?v=<point>`. - Added a typed API adapter with a 100-point page limit, a 2 MiB response bound, and input validation. Files: `apps/web/src/lib/history/{VersionHistory.svelte,api.ts,sessions.ts}` and their three focused test files; `apps/web/src/lib/notes/NoteView.svelte`. ## Contract and gaps - No Rust `HistoryStore` signature changed. The plan does not define OpenAPI response fields. This UI adapter expects list `{ points, before }`; points `{ id, at_ms, author: { kind, id, label }, bytes }`; preview `{ kind, content }`; Restore returns the new point; undo preflight returns `{ token, changed, kept: [{ id, label }] }`; undo apply accepts the token. Slice D or integration must map this adapter to the merged OpenAPI client. - `origin/dev` has no Canvas route or editor. This slice provides the shared panel and a Notes host, plus a preview snippet hook for Canvas. It does not add a Canvas screen without its editor/API. - The screenshots use a real Note created through the local server. Only history API replies are typed test fixtures because slice D has not merged. ## UX gaps closed - Point deep links wait for a stable header anchor; malformed `v` values no longer show a false missing-point message. - Per-author undo names the author, shows the preflight result, and reports both undoable changes and items kept. Restore has an explicit confirmation and an Undo toast action. - Point rows, Copy link actions, close, Restore and Undo use keyboard-operable native controls and 44 px touch targets. Empty, loading, error and retry states are present. ## UX gaps left - The Canvas host and live history API are not on `origin/dev`. Integration must mount the shared component in Canvas and replace the provisional endpoint adapter. A live two-browser restore/undo E2E remains for the merge round. ## Decisions not specified by §61 - Keep same-author points in one session when the gap is at most 30 minutes; split when it is longer. The approved point contract has no session ID. - Render Note preview content as escaped plain text. A Canvas host can provide its own renderer. - Load 100 points per page and reject JSON replies over 2 MiB to bound first paint and memory. - Use a 440 px desktop popover, with the shared phone sheet on narrow viewports. ## Screenshots - 390 px: [Light](https://git.kayg.org/attachments/9dce5478-5f78-44ff-99c3-5e7f667b85e7), [Dark](https://git.kayg.org/attachments/637fc177-d867-4fe4-9a05-0ca5bd364bac) - 820 px: [Light](https://git.kayg.org/attachments/e1f7e761-fea6-4c01-b3e9-b7bf1b112abe), [Dark](https://git.kayg.org/attachments/52700e02-2712-410b-b823-0d8de5576c19) - 1440 px: [Light](https://git.kayg.org/attachments/b0e8601d-2537-4662-97f6-53d84de65312), [Dark](https://git.kayg.org/attachments/2dd4f37d-8a9b-4f6b-8a9f-c112b8e22571) ## Gates (verbatim) `bun run check` ```text $ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json User browser caches use userStorage; only documented device/public-link exceptions remain. Text sizes and UI shape values use shared role tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/hist2-ui/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bunx vitest run src/lib/history/api.test.ts src/lib/history/sessions.test.ts src/lib/history/VersionHistory.svelte.test.ts --maxWorkers=2` ```text RUN v5.0.1 /home/kayg/Developer/calternal-wt/hist2-ui/apps/web Test Files 3 passed (3) Tests 7 passed (7) Start at 17:17:48 Duration 29.66s (transform 77%, tests 8%, environment 7%, import 6%, setup 2%) Transform |component| transforming modules took 22.54s · 77% of tracked time, re-done on every run persist transforms across runs with fsModuleCache: true learn more at https://vitest.dev/guide/improving-performance#caching ``` Production build: `bun run build` succeeded. Screenshot server build: `cargo build -p calternal-server` succeeded. No Rust source changed, so Rust crate gates did not apply. `cargo clean` removed 4.6 GiB; web build output was deleted. ## For the merge round Run full web tests and add/run `bun e2e/history-975.mjs` against the merged API and Canvas. It must prove two browser contexts co-edit, restore a point, and selectively undo one author's changes with the preflight report. Slice F owns the benchmark profile.
kayg commented 2026-10-03 15:22:38 +00:00
Author
Owner
Copy link

Orchestrator visual notes on slice E (hist2-ui, 1440-light.png) for the integration round — fix there, not as a separate send-back:

  1. Copy-link buttons are bare bordered squares (stock look); use the shared icon button with warm tooltip, no box.
  2. Mixed type sizes: author time ('just now', '1 h ago') and session ranges render larger than the point rows; use the role tokens (caption for times).
  3. Session range truncates ('… at …'); show 'Today 5:07–5:11 PM · 2 changes' style compact ranges.
  4. The close × sits bottom-left; the anchored popover should anchor to the history button (top-right) it opened from, with Escape/outside-click close and no separate × (sheet on phones keeps its grabber).
  5. 'For Project plan' subtitle is redundant inside the note; drop it.
  6. Mount in Canvas once Canvas core merges (the job only had Notes).
Orchestrator visual notes on slice E (hist2-ui, 1440-light.png) for the integration round — fix there, not as a separate send-back: 1. Copy-link buttons are bare bordered squares (stock look); use the shared icon button with warm tooltip, no box. 2. Mixed type sizes: author time ('just now', '1 h ago') and session ranges render larger than the point rows; use the role tokens (caption for times). 3. Session range truncates ('… at …'); show 'Today 5:07–5:11 PM · 2 changes' style compact ranges. 4. The close × sits bottom-left; the anchored popover should anchor to the history button (top-right) it opened from, with Escape/outside-click close and no separate × (sheet on phones keeps its grabber). 5. 'For Project plan' subtitle is redundant inside the note; drop it. 6. Mount in Canvas once Canvas core merges (the job only had Notes).
kayg commented 2026-10-03 15:23:24 +00:00
Author
Owner
Copy link

Finding: replacing the broad /api/v1/collab/ authorization allowance with reviewed operation IDs exposed the existing collab_public_edit_session websocket. Its route is /api/v1/collab/public/{slug} and it uses a public Edit-link capability, so it needs the {standard, public_link, admin} policy from DESIGN §54. I added that explicit policy, kept history and signed-in sessions at {standard, admin}, and extended the owner/control request builder with Note item IDs plus point 1 and stable Undo author ranges. Evidence: python3 -m unittest test_xuser_classification passes (9 tests).

Finding: replacing the broad `/api/v1/collab/` authorization allowance with reviewed operation IDs exposed the existing `collab_public_edit_session` websocket. Its route is `/api/v1/collab/public/{slug}` and it uses a public Edit-link capability, so it needs the `{standard, public_link, admin}` policy from DESIGN §54. I added that explicit policy, kept history and signed-in sessions at `{standard, admin}`, and extended the owner/control request builder with Note item IDs plus point 1 and stable Undo author ranges. Evidence: `python3 -m unittest test_xuser_classification` passes (9 tests).
kayg commented 2026-10-03 15:37:07 +00:00
Author
Owner
Copy link

Phase 2 slice A/store complete

Branch: job/hist2-store. Head: cbb8dbeaf5c43a2cb094261716cdd05332134b03. Start/base: d4e7188810a89fb0e8e6b162279917f7e23989f9. The required origin/dev update was merged once at 6af52f6bc, using upstream cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e.

Built

  • The shared HistoryStore contract, a memory reference implementation, and a test-only conformance checker for other backends (test-hooks feature).
  • Durable attributed v1 update batches, zstd level 3, bounded per-document dictionary training with versioned IDs, v2 checkpoints at exact multiples of 500, and an initial checkpoint at point 1.
  • Decoded latest-state and checkpoint caches, with bounded LRU room admission and explicit idle eviction. Blocking filesystem and Yrs work runs outside the async workers.
  • Checksummed, length-delimited segments through calternal-fs. A torn final header/body/checksum is removed on open. Interior corruption is reported. The writer lock survives atomic segment replacement.
  • Retention that streams retained frames to an atomic replacement, preserves the anchor state, and keeps stable point IDs, author tags and timestamps. Physical history usage counts against Home quota. Appends use existing Home growth and free-space checks under the filesystem write lock.
  • A rebuildable SQLite point Index. A failed Index write does not revoke a durable append; a later mutation rebuilds the hint.
  • Ten focused history tests: shared conformance, every point in a 1,050-edit trace after restart, exact checkpoint boundaries, a fold inside a compressed batch, dictionary persistence, torn-tail recovery, interior corruption and length checks, invalid flushes, clock reversal, cross-User isolation, concurrent allocation and cache eviction. Three filesystem tests cover confinement, writer locking, replacement, symlink rejection and quota admission.

Files

  • crates/calternal-collab/src/history/mod.rs
  • crates/calternal-collab/src/history/store.rs
  • crates/calternal-collab/src/history/conformance.rs
  • crates/calternal-collab/src/lib.rs
  • crates/calternal-collab/Cargo.toml
  • crates/calternal-fs/src/history.rs
  • crates/calternal-fs/src/lib.rs
  • Cargo.lock

Commits: 5ad7ee3d8 (confined segment handles), e73dfed7e (quota and reserve checks), cbb8dbeaf5c43a2cb094261716cdd05332134b03 (history store). The merge commit only updates this job branch from origin/dev. No push or deployment was performed.

Decisions and contract additions

The approved trait signatures did not change. The plan left these details undefined:

  • UserId, ItemId and TurnId use the existing string IDs. Result uses typed errors. FoldReport has removed_points, reclaimed_bytes and oldest_point.
  • Point IDs start at 1. Zero is a pagination cursor, not a readable point. The first append must contain the initial full v1 state; an isolated delta against an unseen seed is refused.
  • points has an exclusive ascending cursor and caps a page at 1,000 rows. updates_by includes both endpoints.
  • SegmentHistoryStore::open(root, db, max_cached_documents) creates the derived Index table. No numbered migration was added or reused.
  • Inherent APIs append_batch, checkpoint and evict supplement the trait. A batch has at most 64 attributed rows; each row retains its own point. Slice B can use this for one durable flush. An idle checkpoint creates no point.
  • Dictionary ID 0 means zstd without a trained dictionary. Training uses real document edits, at most 256 KiB of samples, and an 8 KiB dictionary. Small samples can remain dictionary-free. Dictionaries remain available while retained frames refer to them.
  • Updates are capped at 16 MiB; decoded frames/checkpoints at 64 MiB. A checkpoint failure is reported in the server log after the update is committed, so a successful durable append never becomes an ambiguous retry.
  • Retention removes the expired contiguous prefix. It preserves a full state at the first retained point and always keeps the latest point. A clock reversal cannot remove a required replay row in the middle.
  • Root::recompute_quota already includes .calternal/history. Slice B must not add usage to that Home total again. It still owns collaborator budgets and admission through the collaboration event path.

Module and function comments were read again before this report. Full v1 states reuse the existing stored::encode helper. Dependencies were verified with cargo search and cargo info: zstd 0.14.0 (BSD-3-Clause), async-trait 0.1.92 (MIT or Apache-2.0).

Known gaps and integration work

  • This slice is a store library. Hub author binding, batching timers, client-ID guards, restore/undo, owner-only routes, parity, UI and benchmark guards remain in B–F and integration.
  • Slice B must record the initial seed before dependent updates, call idle checkpoints, and connect document lifecycle cleanup. The shared trait has no permanent-delete operation; integration must arrange cleanup when an item is permanently removed.
  • The new implementation was not measured on the perf VM in this slice. The numbers in the module comment are Phase 1 results, not measurements of this branch. F/integration must check disk bytes, latency and peak RSS. Cold room admission scans frame checksums to rebuild metadata; the SQLite table is a hint. Cache capacity and requested undo ranges need integration limits and the growth probes from F.
  • No UI was changed. UX gaps closed/left: not applicable to this slice.

For the merge round

  • cargo clippy --all-targets -- -D warnings and cargo test -- --test-threads=4: check the combined Rust branch once.
  • (cd apps/web && bun run check && bun run test --maxWorkers=2): check the combined web branch once.
  • (cd apps/web && bun run test:e2e && bun run test:e2e:notes), plus the history cases from E/integration: two clients keep editing through Restore and per-author undo, with no reload; point links and owner-only history work.
  • bash tests/adversarial/run.sh: run D/F history cases and the XUser, authorization and robustness matrices on the real local server.
  • Run F's history profile after it is merged, on the perf VM under flock /root/perf.lock, with load recorded inside the lock. Compare it with the Phase 1 Y4/N=500 segment result and docs/perf/baseline.json. The exact new history-profile command belongs to F and is not on this branch yet.
  • Independent review, release builds, deployed checks and macOS interop remain for integration/the merge round.

Verification notes

The final filesystem suite passed 62 unit tests and 43 integration tests. The collaboration runtime suite passed 89 tests. The final focused history run passed all 10 tests against the quota fix. Both crate clippy gates and fmt passed. The collaboration doc-test phase passed on a focused rerun.

Two test-run problems are recorded instead of omitted:

  1. An earlier parallel focused run timed out during Db::connect in cache_evicts_idle_rooms_and_reopens_durable_points, concurrent_appends_have_unique_points_and_index_rows and segment_conforms (Sqlx(PoolTimedOut)). The new tests now use one reader and serialized execution. Their assertions did not change.
  2. I overlapped a follow-up filesystem build with the final collaboration doc-test phase. All collaboration runtime tests passed, but rustdoc could not find the old filesystem rlib. This was a build-artifact race, not a failing test assertion. Remaining commands ran sequentially; cargo test -p calternal-collab --doc -- --test-threads=1 passed. The original failed gate output and its recovery are quoted below.

Gate output, verbatim

All cargo commands used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and worktree target/tmp. The preset target directory was not overridden.

cargo fmt --check: exit 0, no output.

cargo clippy -p calternal-fs --all-targets -- -D warnings:

    Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.95s

cargo test -p calternal-fs -- --test-threads=1:

   Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 12.81s
     Running unittests src/lib.rs (/home/kayg/build/targets/hist2-store/debug/deps/calternal_fs-97fdf1ec7b16d757)

running 62 tests
test history::tests::history_append_respects_home_quota ... ok
test history::tests::history_rejects_symlinked_segments_and_keeps_lock_after_fold ... ok
test history::tests::history_segments_are_confined_and_exclusively_locked ... ok
test hls::tests::failed_video_renditions_do_not_need_a_subprocess_retry ... ok
test path::tests::a_case_or_unicode_twin_is_a_conflict_in_a_large_folder ... ok
test path::tests::ascii_fast_path_matches_the_full_fold ... ok
test path::tests::relative_path_split_keeps_single_component_paths_rooted ... ok
test root::split_device_tests::split_blob_storage_rejects_a_different_device ... ok
test sidecar::tests::filesystem_pairing_ignores_a_sidecar_symlink_and_directory ... ok
test sidecar::tests::identifies_full_lightroom_and_apple_forms ... ok
test sidecar::tests::keeps_orphans_directories_and_unsupported_case_independent ... ok
test sidecar::tests::only_pairs_a_sidecar_with_one_candidate_parent ... ok
test sidecar::tests::pairs_all_standard_forms_without_treating_sidecars_as_parents ... ok
test sidecar::tests::sidecar_names_keep_the_original_standard_form ... ok
test sidecar::tests::sidecars_in_another_folder_do_not_pair ... ok
test tests::a_failure_after_the_rename_is_never_reported_as_a_lost_race ... ok
test tests::clip_model_assets_stay_private_below_the_held_root ... ok
test tests::conditional_note_replace_recovers_after_child_process_exits_mid_journal ... ok
test tests::conditional_note_replace_recovers_each_journal_step ... ok
test tests::empty_home_directory_removal_is_confined_and_refuses_contents ... ok
test tests::empty_trash_restarts_after_each_entry_without_partial_files ... ok
test tests::enospc_and_eio_leave_original_intact_at_write_upload_and_version_steps ... ok
test tests::gc_waits_for_dedup_copy_after_source_unlink ... ok
test tests::gc_waits_for_in_flight_blob_link_across_roots ... ok
test tests::hls_cache_access_is_confined_to_valid_hashes_profiles_and_outputs ... ok
test tests::photos_clip_index_directory_is_private_and_handle_relative ... ok
test tests::recovery_discards_uncommitted_intent_file ... ok
test tests::recovery_replaces_poisoned_blob_at_each_step ... ok
test tests::recovery_replays_each_move_trash_restore_and_install_step ... ok
test tests::recovery_replays_overwrites_with_versions ... ok
test tests::relative_paths_reject_ambiguous_components ... ok
test tests::search_index_directory_is_opened_below_system_directory ... ok
test tests::search_index_staging_swap_keeps_the_active_generation_intact ... ok
test tests::search_index_startup_gate_serializes_starts ... ok
test tests::search_index_startup_recovers_only_unlocked_writer_lock ... ok
test tests::semantic_model_assets_stay_private_and_use_atomic_handles ... ok
test tests::split_roots_keep_user_files_and_system_state_on_their_configured_roots ... ok
test tests::system_secret_files_are_confined_and_report_open_file_mode ... ok
test tests::system_secret_key_is_private_fixed_size_and_never_replaced ... ok
test tests::system_secret_key_rejects_wrong_lengths_and_symlinks ... ok
test tests::thinning_keeps_every_version_in_first_day ... ok
test tests::thinning_restarts_after_each_deleted_version_without_losing_kept_bytes ... ok
test tests::upload_ids_lists_staging_in_the_reserved_tree ... ok
test tests::user_search_directories_are_disjoint_and_confined ... ok
test tests::write_future_can_run_on_server_workers ... ok
test thumbnails::sealed_input_tests::decoder_snapshot_does_not_alias_later_source_changes ... ok
test thumbnails::sealed_input_tests::decoder_snapshot_has_all_kernel_seals_and_keeps_source_bytes ... ok
test thumbnails::sealed_input_tests::decoder_snapshot_refuses_input_above_the_copy_limit ... ok
test thumbnails::sealed_input_tests::large_media_snapshot_is_anonymous_read_only_and_independent ... ok
test thumbnails::sealed_input_tests::large_media_snapshot_rejects_oversize_before_copying ... ok
test thumbnails::tests::document_thumbnail_variants_keep_cache_entries_separate ... ok
test thumbnails::tests::indexed_document_kind_matches_renderer_gates ... ok
test thumbnails::tests::publishing_successful_media_clears_the_terminal_failure_marker ... ok
test thumbnails::tests::stale_unversioned_failure_marker_allows_decoder_retry ... ok
test thumbnails::tests::thumbnail_temp_is_a_precreated_regular_file_in_the_held_cache ... ok
test user_homes::tests::archive_keeps_a_fixed_expiry_and_purges_only_after_it ... ok
test user_homes::tests::each_deletion_phase_recovers_after_process_exit_without_duplication ... 
running 1 test

running 1 test

running 1 test

running 1 test
ok
test user_homes::tests::purge_is_idempotent_and_does_not_touch_another_home ... ok
test user_homes::tests::purge_removes_a_flat_home_across_bounded_batches ... ok
test user_homes::tests::transfer_keeps_the_source_tree_together_and_is_repeatable ... ok
test user_homes::tests::user_home_deletion_child_crash_hook ... ok
test write::export_tests::rooted_export_creates_a_private_new_file_and_rejects_existing_symlinks ... ok

test result: ok. 62 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 73.17s

     Running tests/storage.rs (/home/kayg/build/targets/hist2-store/debug/deps/storage-05c7d9da03af4da7)

running 43 tests
test arbitrary_paths_never_escape ... ok
test atomic_write_event_identifies_replaced_inode ... ok
test blob_scrub_cursor_checks_only_a_bounded_number_per_page ... ok
test blob_scrub_quarantines_and_repairs_shared_inode_from_private_copy ... ok
test blob_scrub_repair_skips_unaddressable_names_in_homes ... ok
test blob_scrub_resumes_quarantined_links_after_a_good_copy_arrives ... ok
test conditional_replace_serializes_competing_writers_and_versions_winner ... ok
test configured_quota_covers_copy_and_cross_home_move ... ok
test copied_file_is_read_only_with_reflink_preference ... ok
test copy_of_deduped_file_reuses_blob ... ok
test cross_home_folder_move_checks_all_file_bytes ... ok
test dedup_concurrent_writes ... ok
test directory_move_to_descendant_is_rejected_without_mutation ... ok
test directory_pages_advance_with_cursor ... ok
test excessive_delete_depth_does_not_remove_siblings ... ok
test excessive_mkdir_depth_does_not_leave_a_partial_home_tree ... ok
test existing_blob_with_wrong_bytes_is_replaced ... ok
test mirrored_versions_follow_directory_and_delete ... ok
test moving_file_carries_versions ... ok
test new_name_policy_allows_joiners_only_in_writing_sequences ... ok
test new_name_policy_normalizes_and_rejects_spoofing ... ok
test orphan_blobs_are_collected ... ok
test overwriting_move_carries_source_versions_and_preserves_target ... ok
test overwriting_with_copy_or_move_keeps_destination_version ... ok
test percent_name_survives_trash_round_trip ... ok
test public_listing_hides_reserved_names ... ok
test quota_is_shared_and_override_only_lowers ... ok
test replace_does_not_modify_another_hardlink ... ok
test reserved_internals_are_not_public_paths ... ok
test sibling_collision_uses_unicode_casefold_after_nfc ... ok
test source_mtime_is_preserved_without_changing_same_content_sibling ... ok
test symlink_parent_is_never_followed ... ok
test symlink_swap_race_never_writes_outside ... ok
test system_dedup_scrub_state_is_private_and_atomic ... ok
test system_dedup_scrub_state_rejects_oversized_checkpoints ... ok
test thinning_keeps_recent_and_bucket_representatives ... ok
test trash_collisions_get_distinct_names_and_empty ... ok
test trash_restores_versions_without_mixing_new_file ... ok
test trash_round_trip_and_quota ... ok
test traversal_is_rejected ... ok
test two_home_upload_reservations_are_isolated_and_exchange_on_install ... ok
test version_counts_against_quota ... ok
test zero_byte_file_gets_version_and_thins ... ok

test result: ok. 43 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 30.26s

   Doc-tests calternal_fs

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-collab --all-targets -- -D warnings:

    Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs)
    Checking calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-plugin)
    Checking calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-imap)
    Checking calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-location)
    Checking calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-tags)
    Checking calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/notes)
    Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/files)
    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 58.33s

cargo test -p calternal-collab -- --test-threads=1 (runtime tests pass; original doc-test artifact failure follows):

    Blocking waiting for file lock on build directory
   Compiling typenum v1.20.1
   Compiling rand_core v0.10.1
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling cmov v0.5.4
   Compiling getrandom v0.4.3
   Compiling digest v0.10.7
   Compiling generic-array v0.14.9
   Compiling hybrid-array v0.4.15
   Compiling uuid v1.26.1
   Compiling sha2 v0.10.9
   Compiling ctutils v0.4.2
   Compiling crypto-common v0.2.2
   Compiling sqlx-core v0.9.0
   Compiling inout v0.2.2
   Compiling sha1 v0.10.7
   Compiling hmac v0.12.1
   Compiling signature v2.2.0
   Compiling block-buffer v0.12.1
   Compiling sec1 v0.7.3
   Compiling cipher v0.5.2
   Compiling hkdf v0.12.4
   Compiling sqlx-sqlite v0.9.0
   Compiling tungstenite v0.29.0
   Compiling universal-hash v0.6.1
   Compiling crypto-bigint v0.5.5
   Compiling tokio-tungstenite v0.29.0
   Compiling poly1305 v0.9.1
   Compiling elliptic-curve v0.13.8
   Compiling sqlx-macros-core v0.9.0
   Compiling chacha20 v0.10.2
   Compiling aead v0.6.1
   Compiling sqlx-macros v0.9.0
   Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs)
   Compiling chacha20poly1305 v0.11.0
   Compiling axum v0.8.9
   Compiling sqlx v0.9.0
   Compiling rfc6979 v0.4.0
   Compiling primeorder v0.13.6
   Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-db)
   Compiling ecdsa v0.16.9
   Compiling curve25519-dalek v4.1.3
   Compiling headers v0.4.2
   Compiling webauthn-attestation-ca v0.5.5
   Compiling calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-api)
   Compiling ed25519 v2.2.3
   Compiling dav-server v0.11.0
   Compiling webauthn-rs-core v0.5.5
   Compiling ed25519-dalek v2.2.0
   Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-plugin)
   Compiling rsa v0.9.10
   Compiling oauth2 v5.0.0
   Compiling p256 v0.13.2
   Compiling p384 v0.13.1
   Compiling blake2 v0.10.6
   Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-imap)
   Compiling webauthn-rs v0.5.5
   Compiling argon2 v0.5.3
   Compiling openidconnect v4.0.1
   Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-tags)
   Compiling calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-dav)
   Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-location)
   Compiling calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-auth)
   Compiling tempfile v3.27.0
   Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/notes)
   Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/files)
   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 03s
     Running unittests src/lib.rs (/home/kayg/build/targets/hist2-store/debug/deps/calternal_collab-8ced68ecf00e38af)

running 40 tests
test block_merge_tests::anchored_conflict_copies_have_unique_replay_stable_ids ... ok
test block_merge_tests::changed_block_reaches_markdown_after_the_live_side_deletes_it ... ok
test block_merge_tests::conflict_keeps_both_versions_external_first ... ok
test block_merge_tests::derived_conflict_id_does_not_steal_an_existing_link ... ok
test block_merge_tests::issue_89_example ... ok
test block_merge_tests::live_only_change_is_kept ... ok
test block_merge_tests::same_change_on_both_sides_is_applied_once ... ok
test block_merge_tests::shared_change_inside_a_conflicting_chunk_is_applied_once ... ok
test history::store::tests::cache_evicts_idle_rooms_and_reopens_durable_points ... ok
test history::store::tests::checkpoints_restore_every_point_and_fold_across_a_batch ... ok
test history::store::tests::concurrent_appends_have_unique_points_and_index_rows ... ok
test history::store::tests::interior_corruption_and_oversized_headers_fail_closed ... ok
test history::store::tests::invalid_flush_does_not_advance_state_or_points ... ok
test history::store::tests::memory_conforms ... ok
test history::store::tests::retention_keeps_clock_reversals_and_the_latest_point ... ok
test history::store::tests::segment_conforms ... ok
test history::store::tests::torn_tail_recovery_keeps_prior_points_and_accepts_new_updates ... ok
test history::store::tests::trained_dictionary_survives_recovery_and_retention ... ok
test markdown::source_patch_tests::changed_line_keeps_unedited_crlf_tabs_and_trailing_spaces ... ok
test markdown::source_patch_tests::source_patch_refuses_unaligned_line_counts ... ok
test repeats::tests::adjacent_duplicates_require_identical_content_and_identity ... ok
test repeats::tests::exact_repeats_are_counted ... ok
test repeats::tests::near_repeats_and_plain_notes_are_not ... ok
test session::client_stream_limit_tests::websocket_stream_limit_uses_resolved_ip_and_retry_after ... ok
test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok
test session::conflict_shadow_tests::edit_that_reaches_server_before_delete_survives_shadow_merge ... ok
test session::conflict_shadow_tests::external_writes_then_typing_do_not_replay_the_live_keystroke ... ok
test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok
test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok
test session::conflict_shadow_tests::local_undo_that_opens_conflict_shadow_does_not_replay_its_paste ... ok
test session::conflict_shadow_tests::paste_undo_redo_from_deleting_connection_is_not_a_conflict ... ok
test session::conflict_shadow_tests::second_client_edit_to_deleted_block_survives_shadow_merge ... ok
test session::conflict_shadow_tests::stale_edit_survives_after_same_connection_deleted_another_block ... ok
test session::empty_sync_update_does_not_mark_the_room_document_changed ... ok
test session::external_persistence_race_tests::crash_before_save_keeps_epoch_and_replays_pending_edit_once ... ok
test session::external_persistence_race_tests::external_reconcile_waits_for_a_fresh_live_snapshot_to_persist ... ok
test session::public_edit_limit_tests::oversized_public_update_is_rolled_back ... ok
test session::public_edit_limit_tests::public_edit_limit_checks_the_markdown_form ... ok
test session::public_edit_limit_tests::ten_thousand_markdown_blocks_open_and_sync_within_two_seconds ... ok
test session::shared_notes_deny_without_files_share_authority ... ok

test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.38s

     Running tests/agent_turn_order.rs (/home/kayg/build/targets/hist2-store/debug/deps/agent_turn_order-27a178470289d1ea)

running 1 test
test agent_turn_with_edits_in_several_places_keeps_order ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.62s

     Running tests/block_apply_property.rs (/home/kayg/build/targets/hist2-store/debug/deps/block_apply_property-03a93cc8c6ac2651)

running 2 tests
test external_apply_yields_new_block_order_exactly ... ok
test shared_typing_beside_an_external_change_is_applied_once ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.86s

     Running tests/cross_language.rs (/home/kayg/build/targets/hist2-store/debug/deps/cross_language-dc11530f2468792d)

running 1 test
test yjs_updates_match_editor_vectors ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 96.53s

     Running tests/hostile_clients.rs (/home/kayg/build/targets/hist2-store/debug/deps/hostile_clients-8bdab6041788b3b3)

running 11 tests
test awareness_clock_at_maximum_is_refused_and_room_unloads ... ok
test continuous_typing_is_saved_within_the_maximum_wait ... ok
test cursed_bodies_open_live ... ok
test one_user_cannot_open_unbounded_sockets ... ok
test oversized_message_is_refused ... ok
test primitive_value_in_fragment_cannot_truncate_the_note ... ok
test reconnect_storm_does_not_cancel_pending_save ... ok
test trashed_note_room_unloads_instead_of_retrying_forever ... ok
test unauthenticated_websocket_routes_return_forbidden ... ok
test unrepresentable_update_is_rejected_and_room_keeps_saving ... ok
test wiki_embeds_open_live_and_save_unchanged ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 29.11s

     Running tests/journal_race.rs (/home/kayg/build/targets/hist2-store/debug/deps/journal_race-f3700b64ee4a632d)

running 1 test
test journal_log_race_with_live_hub_keeps_all_changes_and_refuses_daily_rooms ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.87s

     Running tests/restart_epoch.rs (/home/kayg/build/targets/hist2-store/debug/deps/restart_epoch-583c9b870ab44fc4)

running 5 tests
test graceful_restart_keeps_the_epoch_and_offline_edits ... ok
test out_of_band_change_starts_a_new_epoch ... ok
test stale_epoch_is_told_the_new_epoch_and_closed ... ok
test tab_open_across_a_restart_does_not_duplicate_the_note ... ok
test unload_then_reconnect_continues_the_same_room ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.67s

     Running tests/shared_notes.rs (/home/kayg/build/targets/hist2-store/debug/deps/shared_notes-da4c7d6cad8318ab)

running 1 test
test owner_editor_viewer_and_live_revoke ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.01s

     Running tests/two_clients.rs (/home/kayg/build/targets/hist2-store/debug/deps/two_clients-2742aafecff824dc)

running 2 tests
test concurrent_clients_and_external_writer_converge ... ok
test crash_before_debounce_reloads_only_complete_markdown ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.89s

     Running tests/untouched_bytes.rs (/home/kayg/build/targets/hist2-store/debug/deps/untouched_bytes-c5c2631f5fb63bb5)

running 5 tests
test clean_room_flush_keeps_noncanonical_note_bytes ... ok
test external_edit_to_an_open_room_keeps_its_bytes ... ok
test live_edit_preserves_unedited_source_lines ... ok
test random_untouched_notes_keep_every_byte ... ok
test untouched_save_keeps_every_byte ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 46.95s

     Running tests/vector_bridge.rs (/home/kayg/build/targets/hist2-store/debug/deps/vector_bridge-77ca536540770e08)

running 15 tests
test applying_the_same_external_edit_twice_changes_nothing ... ok
test concurrent_typing_in_unchanged_block_survives_external_edit ... ok
test crash_before_debounce_uses_last_complete_markdown ... ok
test editor_schema_json_survives_yrs ... ok
test external_edit_after_collaborator_insert_above_keeps_order ... ok
test external_edit_beside_concurrently_changed_block_keeps_order ... ok
test external_edit_inserts_at_start_and_end ... ok
test external_edit_keeps_independent_live_block ... ok
test external_edit_mixes_insert_delete_and_replace ... ok
test external_edit_moves_blocks ... ok
test external_edit_with_two_inserts_keeps_order ... ok
test external_insertion_keeps_existing_block_identity ... ok
test markdown_vectors_pass_through_yrs ... ok
test same_block_conflict_keeps_both_versions ... ok
test update_rebuilds_after_restart ... ok

test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s

     Running tests/wiki_embeds.rs (/home/kayg/build/targets/hist2-store/debug/deps/wiki_embeds-ed41ce2b830d274e)

running 5 tests
test cursed_bodies_open_and_keep_their_content ... ok
test embed_in_a_table_cell_stays_one_cell ... ok
test image_lines_match_the_editor_reader ... ok
test wiki_embed_stays_text_not_an_invented_node ... ok
test wiki_embeds_round_trip_byte_for_byte ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.95s

   Doc-tests calternal_collab
error: extern location for calternal_fs does not exist: /home/kayg/build/targets/hist2-store/debug/deps/libcalternal_fs-783dc72f4995ba05.rlib
  --> crates/calternal-collab/src/session.rs:46:5
   |
46 | use calternal_fs::{FileFingerprint, RelPath, Root};
   |     ^^^^^^^^^^^^

error: aborting due to 1 previous error

error: doctest failed, to rerun pass `-p calternal-collab --doc`

Caused by:
  process didn't exit successfully: `/home/kayg/.rustup/toolchains/1.98.1-x86_64-unknown-linux-gnu/bin/rustdoc --edition=2024 --crate-type lib --color auto --crate-name calternal_collab --test crates/calternal-collab/src/lib.rs --test-run-directory /home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab -L native=/home/kayg/build/targets/hist2-store/debug/build/aws-lc-sys-22e2292daf5760d8/out -L native=/home/kayg/build/targets/hist2-store/debug/build/blake3-136881d206af7920/out -L native=/home/kayg/build/targets/hist2-store/debug/build/libsqlite3-sys-122235796a52e197/out -L native=/home/kayg/build/targets/hist2-store/debug/build/libsqlite3-sys-c8822542b868e02e/out -L native=/home/kayg/build/targets/hist2-store/debug/build/ring-3111773b92f61ec3/out -L native=/home/kayg/build/targets/hist2-store/debug/build/xmp_toolkit-5fb6fc1af31f1a6d/out -L native=/home/kayg/build/targets/hist2-store/debug/build/zstd-sys-8c0dc470c8401361/out --test-args --test-threads=1 --extern async_trait=/home/kayg/build/targets/hist2-store/debug/deps/libasync_trait-588cc0803d3ae9cf.so --extern axum=/home/kayg/build/targets/hist2-store/debug/deps/libaxum-5615296a78b2ccd3.rlib --extern base64=/home/kayg/build/targets/hist2-store/debug/deps/libbase64-7e575915217cc61d.rlib --extern blake3=/home/kayg/build/targets/hist2-store/debug/deps/libblake3-b62458aedb76948b.rlib --extern calternal_auth=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_auth-1136d44da560d5d3.rlib --extern calternal_collab=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_collab-de80531be08e2161.rlib --extern calternal_db=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_db-036f5c066dbca621.rlib --extern calternal_fs=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_fs-783dc72f4995ba05.rlib --extern calternal_notes_core=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_notes_core-0d368d261f4da8bb.rlib --extern calternal_plugin=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_plugin-adacd55b0ac95122.rlib --extern calternal_plugin_files=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_plugin_files-67ba0a78b79b20bc.rlib --extern calternal_plugin_notes=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_plugin_notes-c6af46b589a821b8.rlib --extern calternal_tags=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_tags-9f76b939667e9e59.rlib --extern futures_util=/home/kayg/build/targets/hist2-store/debug/deps/libfutures_util-f3be90bd1edac511.rlib --extern ipnet=/home/kayg/build/targets/hist2-store/debug/deps/libipnet-a225065d7b729d93.rlib --extern notify=/home/kayg/build/targets/hist2-store/debug/deps/libnotify-fa8a3ade9707afd8.rlib --extern serde=/home/kayg/build/targets/hist2-store/debug/deps/libserde-45cf79a2c03a80cd.rlib --extern serde_json=/home/kayg/build/targets/hist2-store/debug/deps/libserde_json-73f174211147ba08.rlib --extern similar=/home/kayg/build/targets/hist2-store/debug/deps/libsimilar-d64eda5642c38bcc.rlib --extern sqlx=/home/kayg/build/targets/hist2-store/debug/deps/libsqlx-e6a1702116896c7a.rlib --extern tempfile=/home/kayg/build/targets/hist2-store/debug/deps/libtempfile-ae85e9649b5e3108.rlib --extern thiserror=/home/kayg/build/targets/hist2-store/debug/deps/libthiserror-7d4222164f8072f6.rlib --extern tokio=/home/kayg/build/targets/hist2-store/debug/deps/libtokio-6e8424ce77408b7d.rlib --extern tokio_tungstenite=/home/kayg/build/targets/hist2-store/debug/deps/libtokio_tungstenite-222e54b2a30de290.rlib --extern tower=/home/kayg/build/targets/hist2-store/debug/deps/libtower-efa9a82cb81f2596.rlib --extern tracing=/home/kayg/build/targets/hist2-store/debug/deps/libtracing-b41d65e1073649c0.rlib --extern utoipa=/home/kayg/build/targets/hist2-store/debug/deps/libutoipa-616642659aa5a371.rlib --extern uuid=/home/kayg/build/targets/hist2-store/debug/deps/libuuid-612694c7e931dc7a.rlib --extern yrs=/home/kayg/build/targets/hist2-store/debug/deps/libyrs-50ad96fa2c3e148c.rlib --extern zstd=/home/kayg/build/targets/hist2-store/debug/deps/libzstd-94094a1f5286493b.rlib -L dependency=/home/kayg/build/targets/hist2-store/debug/deps -C embed-bitcode=no --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("test-hooks"))' --error-format human` (exit status: 1)
note: test exited abnormally; to see the full output pass --no-capture to the harness.

cargo test -p calternal-collab --lib history:: -- --test-threads=1 (final focused regression run):

   Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs)
   Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-plugin)
   Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-location)
   Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-imap)
   Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-tags)
   Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/notes)
   Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/files)
   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 52s
     Running unittests src/lib.rs (/home/kayg/build/targets/hist2-store/debug/deps/calternal_collab-8ced68ecf00e38af)

running 10 tests
test history::store::tests::cache_evicts_idle_rooms_and_reopens_durable_points ... ok
test history::store::tests::checkpoints_restore_every_point_and_fold_across_a_batch ... ok
test history::store::tests::concurrent_appends_have_unique_points_and_index_rows ... ok
test history::store::tests::interior_corruption_and_oversized_headers_fail_closed ... ok
test history::store::tests::invalid_flush_does_not_advance_state_or_points ... ok
test history::store::tests::memory_conforms ... ok
test history::store::tests::retention_keeps_clock_reversals_and_the_latest_point ... ok
test history::store::tests::segment_conforms ... ok
test history::store::tests::torn_tail_recovery_keeps_prior_points_and_accepts_new_updates ... ok
test history::store::tests::trained_dictionary_survives_recovery_and_retention ... ok

test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 30 filtered out; finished in 33.95s

cargo test -p calternal-collab --doc -- --test-threads=1 (doc-test recovery):

    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 24.61s
   Doc-tests calternal_collab

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clean:

     Removed 9890 files, 5.6GiB total

No web build output remains. Worktree status is clean. No issue was closed.

## Phase 2 slice A/store complete Branch: `job/hist2-store`. Head: `cbb8dbeaf5c43a2cb094261716cdd05332134b03`. Start/base: `d4e7188810a89fb0e8e6b162279917f7e23989f9`. The required `origin/dev` update was merged once at `6af52f6bc`, using upstream `cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e`. ### Built - The shared `HistoryStore` contract, a memory reference implementation, and a test-only conformance checker for other backends (`test-hooks` feature). - Durable attributed v1 update batches, zstd level 3, bounded per-document dictionary training with versioned IDs, v2 checkpoints at exact multiples of 500, and an initial checkpoint at point 1. - Decoded latest-state and checkpoint caches, with bounded LRU room admission and explicit idle eviction. Blocking filesystem and Yrs work runs outside the async workers. - Checksummed, length-delimited segments through calternal-fs. A torn final header/body/checksum is removed on open. Interior corruption is reported. The writer lock survives atomic segment replacement. - Retention that streams retained frames to an atomic replacement, preserves the anchor state, and keeps stable point IDs, author tags and timestamps. Physical history usage counts against Home quota. Appends use existing Home growth and free-space checks under the filesystem write lock. - A rebuildable SQLite point Index. A failed Index write does not revoke a durable append; a later mutation rebuilds the hint. - Ten focused history tests: shared conformance, every point in a 1,050-edit trace after restart, exact checkpoint boundaries, a fold inside a compressed batch, dictionary persistence, torn-tail recovery, interior corruption and length checks, invalid flushes, clock reversal, cross-User isolation, concurrent allocation and cache eviction. Three filesystem tests cover confinement, writer locking, replacement, symlink rejection and quota admission. ### Files - `crates/calternal-collab/src/history/mod.rs` - `crates/calternal-collab/src/history/store.rs` - `crates/calternal-collab/src/history/conformance.rs` - `crates/calternal-collab/src/lib.rs` - `crates/calternal-collab/Cargo.toml` - `crates/calternal-fs/src/history.rs` - `crates/calternal-fs/src/lib.rs` - `Cargo.lock` Commits: `5ad7ee3d8` (confined segment handles), `e73dfed7e` (quota and reserve checks), `cbb8dbeaf5c43a2cb094261716cdd05332134b03` (history store). The merge commit only updates this job branch from `origin/dev`. No push or deployment was performed. ### Decisions and contract additions The approved trait signatures did not change. The plan left these details undefined: - `UserId`, `ItemId` and `TurnId` use the existing string IDs. `Result` uses typed errors. `FoldReport` has `removed_points`, `reclaimed_bytes` and `oldest_point`. - Point IDs start at 1. Zero is a pagination cursor, not a readable point. The first append must contain the initial full v1 state; an isolated delta against an unseen seed is refused. - `points` has an exclusive ascending cursor and caps a page at 1,000 rows. `updates_by` includes both endpoints. - `SegmentHistoryStore::open(root, db, max_cached_documents)` creates the derived Index table. No numbered migration was added or reused. - Inherent APIs `append_batch`, `checkpoint` and `evict` supplement the trait. A batch has at most 64 attributed rows; each row retains its own point. Slice B can use this for one durable flush. An idle checkpoint creates no point. - Dictionary ID 0 means zstd without a trained dictionary. Training uses real document edits, at most 256 KiB of samples, and an 8 KiB dictionary. Small samples can remain dictionary-free. Dictionaries remain available while retained frames refer to them. - Updates are capped at 16 MiB; decoded frames/checkpoints at 64 MiB. A checkpoint failure is reported in the server log after the update is committed, so a successful durable append never becomes an ambiguous retry. - Retention removes the expired contiguous prefix. It preserves a full state at the first retained point and always keeps the latest point. A clock reversal cannot remove a required replay row in the middle. - `Root::recompute_quota` already includes `.calternal/history`. Slice B must not add `usage` to that Home total again. It still owns collaborator budgets and admission through the collaboration event path. Module and function comments were read again before this report. Full v1 states reuse the existing `stored::encode` helper. Dependencies were verified with `cargo search` and `cargo info`: zstd 0.14.0 (BSD-3-Clause), async-trait 0.1.92 (MIT or Apache-2.0). ### Known gaps and integration work - This slice is a store library. Hub author binding, batching timers, client-ID guards, restore/undo, owner-only routes, parity, UI and benchmark guards remain in B–F and integration. - Slice B must record the initial seed before dependent updates, call idle checkpoints, and connect document lifecycle cleanup. The shared trait has no permanent-delete operation; integration must arrange cleanup when an item is permanently removed. - The new implementation was not measured on the perf VM in this slice. The numbers in the module comment are Phase 1 results, not measurements of this branch. F/integration must check disk bytes, latency and peak RSS. Cold room admission scans frame checksums to rebuild metadata; the SQLite table is a hint. Cache capacity and requested undo ranges need integration limits and the growth probes from F. - No UI was changed. UX gaps closed/left: not applicable to this slice. ### For the merge round - `cargo clippy --all-targets -- -D warnings` and `cargo test -- --test-threads=4`: check the combined Rust branch once. - `(cd apps/web && bun run check && bun run test --maxWorkers=2)`: check the combined web branch once. - `(cd apps/web && bun run test:e2e && bun run test:e2e:notes)`, plus the history cases from E/integration: two clients keep editing through Restore and per-author undo, with no reload; point links and owner-only history work. - `bash tests/adversarial/run.sh`: run D/F history cases and the XUser, authorization and robustness matrices on the real local server. - Run F's history profile after it is merged, on the perf VM under `flock /root/perf.lock`, with load recorded inside the lock. Compare it with the Phase 1 Y4/N=500 segment result and `docs/perf/baseline.json`. The exact new history-profile command belongs to F and is not on this branch yet. - Independent review, release builds, deployed checks and macOS interop remain for integration/the merge round. ### Verification notes The final filesystem suite passed 62 unit tests and 43 integration tests. The collaboration runtime suite passed 89 tests. The final focused history run passed all 10 tests against the quota fix. Both crate clippy gates and fmt passed. The collaboration doc-test phase passed on a focused rerun. Two test-run problems are recorded instead of omitted: 1. An earlier parallel focused run timed out during `Db::connect` in `cache_evicts_idle_rooms_and_reopens_durable_points`, `concurrent_appends_have_unique_points_and_index_rows` and `segment_conforms` (`Sqlx(PoolTimedOut)`). The new tests now use one reader and serialized execution. Their assertions did not change. 2. I overlapped a follow-up filesystem build with the final collaboration doc-test phase. All collaboration runtime tests passed, but rustdoc could not find the old filesystem rlib. This was a build-artifact race, not a failing test assertion. Remaining commands ran sequentially; `cargo test -p calternal-collab --doc -- --test-threads=1` passed. The original failed gate output and its recovery are quoted below. ### Gate output, verbatim All cargo commands used `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4` and worktree `target/tmp`. The preset target directory was not overridden. `cargo fmt --check`: exit 0, no output. `cargo clippy -p calternal-fs --all-targets -- -D warnings`: ```text Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs) Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.95s ``` `cargo test -p calternal-fs -- --test-threads=1`: ```text Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs) Finished `test` profile [unoptimized + debuginfo] target(s) in 12.81s Running unittests src/lib.rs (/home/kayg/build/targets/hist2-store/debug/deps/calternal_fs-97fdf1ec7b16d757) running 62 tests test history::tests::history_append_respects_home_quota ... ok test history::tests::history_rejects_symlinked_segments_and_keeps_lock_after_fold ... ok test history::tests::history_segments_are_confined_and_exclusively_locked ... ok test hls::tests::failed_video_renditions_do_not_need_a_subprocess_retry ... ok test path::tests::a_case_or_unicode_twin_is_a_conflict_in_a_large_folder ... ok test path::tests::ascii_fast_path_matches_the_full_fold ... ok test path::tests::relative_path_split_keeps_single_component_paths_rooted ... ok test root::split_device_tests::split_blob_storage_rejects_a_different_device ... ok test sidecar::tests::filesystem_pairing_ignores_a_sidecar_symlink_and_directory ... ok test sidecar::tests::identifies_full_lightroom_and_apple_forms ... ok test sidecar::tests::keeps_orphans_directories_and_unsupported_case_independent ... ok test sidecar::tests::only_pairs_a_sidecar_with_one_candidate_parent ... ok test sidecar::tests::pairs_all_standard_forms_without_treating_sidecars_as_parents ... ok test sidecar::tests::sidecar_names_keep_the_original_standard_form ... ok test sidecar::tests::sidecars_in_another_folder_do_not_pair ... ok test tests::a_failure_after_the_rename_is_never_reported_as_a_lost_race ... ok test tests::clip_model_assets_stay_private_below_the_held_root ... ok test tests::conditional_note_replace_recovers_after_child_process_exits_mid_journal ... ok test tests::conditional_note_replace_recovers_each_journal_step ... ok test tests::empty_home_directory_removal_is_confined_and_refuses_contents ... ok test tests::empty_trash_restarts_after_each_entry_without_partial_files ... ok test tests::enospc_and_eio_leave_original_intact_at_write_upload_and_version_steps ... ok test tests::gc_waits_for_dedup_copy_after_source_unlink ... ok test tests::gc_waits_for_in_flight_blob_link_across_roots ... ok test tests::hls_cache_access_is_confined_to_valid_hashes_profiles_and_outputs ... ok test tests::photos_clip_index_directory_is_private_and_handle_relative ... ok test tests::recovery_discards_uncommitted_intent_file ... ok test tests::recovery_replaces_poisoned_blob_at_each_step ... ok test tests::recovery_replays_each_move_trash_restore_and_install_step ... ok test tests::recovery_replays_overwrites_with_versions ... ok test tests::relative_paths_reject_ambiguous_components ... ok test tests::search_index_directory_is_opened_below_system_directory ... ok test tests::search_index_staging_swap_keeps_the_active_generation_intact ... ok test tests::search_index_startup_gate_serializes_starts ... ok test tests::search_index_startup_recovers_only_unlocked_writer_lock ... ok test tests::semantic_model_assets_stay_private_and_use_atomic_handles ... ok test tests::split_roots_keep_user_files_and_system_state_on_their_configured_roots ... ok test tests::system_secret_files_are_confined_and_report_open_file_mode ... ok test tests::system_secret_key_is_private_fixed_size_and_never_replaced ... ok test tests::system_secret_key_rejects_wrong_lengths_and_symlinks ... ok test tests::thinning_keeps_every_version_in_first_day ... ok test tests::thinning_restarts_after_each_deleted_version_without_losing_kept_bytes ... ok test tests::upload_ids_lists_staging_in_the_reserved_tree ... ok test tests::user_search_directories_are_disjoint_and_confined ... ok test tests::write_future_can_run_on_server_workers ... ok test thumbnails::sealed_input_tests::decoder_snapshot_does_not_alias_later_source_changes ... ok test thumbnails::sealed_input_tests::decoder_snapshot_has_all_kernel_seals_and_keeps_source_bytes ... ok test thumbnails::sealed_input_tests::decoder_snapshot_refuses_input_above_the_copy_limit ... ok test thumbnails::sealed_input_tests::large_media_snapshot_is_anonymous_read_only_and_independent ... ok test thumbnails::sealed_input_tests::large_media_snapshot_rejects_oversize_before_copying ... ok test thumbnails::tests::document_thumbnail_variants_keep_cache_entries_separate ... ok test thumbnails::tests::indexed_document_kind_matches_renderer_gates ... ok test thumbnails::tests::publishing_successful_media_clears_the_terminal_failure_marker ... ok test thumbnails::tests::stale_unversioned_failure_marker_allows_decoder_retry ... ok test thumbnails::tests::thumbnail_temp_is_a_precreated_regular_file_in_the_held_cache ... ok test user_homes::tests::archive_keeps_a_fixed_expiry_and_purges_only_after_it ... ok test user_homes::tests::each_deletion_phase_recovers_after_process_exit_without_duplication ... running 1 test running 1 test running 1 test running 1 test ok test user_homes::tests::purge_is_idempotent_and_does_not_touch_another_home ... ok test user_homes::tests::purge_removes_a_flat_home_across_bounded_batches ... ok test user_homes::tests::transfer_keeps_the_source_tree_together_and_is_repeatable ... ok test user_homes::tests::user_home_deletion_child_crash_hook ... ok test write::export_tests::rooted_export_creates_a_private_new_file_and_rejects_existing_symlinks ... ok test result: ok. 62 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 73.17s Running tests/storage.rs (/home/kayg/build/targets/hist2-store/debug/deps/storage-05c7d9da03af4da7) running 43 tests test arbitrary_paths_never_escape ... ok test atomic_write_event_identifies_replaced_inode ... ok test blob_scrub_cursor_checks_only_a_bounded_number_per_page ... ok test blob_scrub_quarantines_and_repairs_shared_inode_from_private_copy ... ok test blob_scrub_repair_skips_unaddressable_names_in_homes ... ok test blob_scrub_resumes_quarantined_links_after_a_good_copy_arrives ... ok test conditional_replace_serializes_competing_writers_and_versions_winner ... ok test configured_quota_covers_copy_and_cross_home_move ... ok test copied_file_is_read_only_with_reflink_preference ... ok test copy_of_deduped_file_reuses_blob ... ok test cross_home_folder_move_checks_all_file_bytes ... ok test dedup_concurrent_writes ... ok test directory_move_to_descendant_is_rejected_without_mutation ... ok test directory_pages_advance_with_cursor ... ok test excessive_delete_depth_does_not_remove_siblings ... ok test excessive_mkdir_depth_does_not_leave_a_partial_home_tree ... ok test existing_blob_with_wrong_bytes_is_replaced ... ok test mirrored_versions_follow_directory_and_delete ... ok test moving_file_carries_versions ... ok test new_name_policy_allows_joiners_only_in_writing_sequences ... ok test new_name_policy_normalizes_and_rejects_spoofing ... ok test orphan_blobs_are_collected ... ok test overwriting_move_carries_source_versions_and_preserves_target ... ok test overwriting_with_copy_or_move_keeps_destination_version ... ok test percent_name_survives_trash_round_trip ... ok test public_listing_hides_reserved_names ... ok test quota_is_shared_and_override_only_lowers ... ok test replace_does_not_modify_another_hardlink ... ok test reserved_internals_are_not_public_paths ... ok test sibling_collision_uses_unicode_casefold_after_nfc ... ok test source_mtime_is_preserved_without_changing_same_content_sibling ... ok test symlink_parent_is_never_followed ... ok test symlink_swap_race_never_writes_outside ... ok test system_dedup_scrub_state_is_private_and_atomic ... ok test system_dedup_scrub_state_rejects_oversized_checkpoints ... ok test thinning_keeps_recent_and_bucket_representatives ... ok test trash_collisions_get_distinct_names_and_empty ... ok test trash_restores_versions_without_mixing_new_file ... ok test trash_round_trip_and_quota ... ok test traversal_is_rejected ... ok test two_home_upload_reservations_are_isolated_and_exchange_on_install ... ok test version_counts_against_quota ... ok test zero_byte_file_gets_version_and_thins ... ok test result: ok. 43 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 30.26s Doc-tests calternal_fs running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-collab --all-targets -- -D warnings`: ```text Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs) Checking calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-plugin) Checking calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-imap) Checking calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-location) Checking calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-tags) Checking calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/notes) Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/files) Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 58.33s ``` `cargo test -p calternal-collab -- --test-threads=1 (runtime tests pass; original doc-test artifact failure follows)`: ```text Blocking waiting for file lock on build directory Compiling typenum v1.20.1 Compiling rand_core v0.10.1 Compiling crypto-common v0.1.6 Compiling block-buffer v0.10.4 Compiling cmov v0.5.4 Compiling getrandom v0.4.3 Compiling digest v0.10.7 Compiling generic-array v0.14.9 Compiling hybrid-array v0.4.15 Compiling uuid v1.26.1 Compiling sha2 v0.10.9 Compiling ctutils v0.4.2 Compiling crypto-common v0.2.2 Compiling sqlx-core v0.9.0 Compiling inout v0.2.2 Compiling sha1 v0.10.7 Compiling hmac v0.12.1 Compiling signature v2.2.0 Compiling block-buffer v0.12.1 Compiling sec1 v0.7.3 Compiling cipher v0.5.2 Compiling hkdf v0.12.4 Compiling sqlx-sqlite v0.9.0 Compiling tungstenite v0.29.0 Compiling universal-hash v0.6.1 Compiling crypto-bigint v0.5.5 Compiling tokio-tungstenite v0.29.0 Compiling poly1305 v0.9.1 Compiling elliptic-curve v0.13.8 Compiling sqlx-macros-core v0.9.0 Compiling chacha20 v0.10.2 Compiling aead v0.6.1 Compiling sqlx-macros v0.9.0 Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs) Compiling chacha20poly1305 v0.11.0 Compiling axum v0.8.9 Compiling sqlx v0.9.0 Compiling rfc6979 v0.4.0 Compiling primeorder v0.13.6 Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-db) Compiling ecdsa v0.16.9 Compiling curve25519-dalek v4.1.3 Compiling headers v0.4.2 Compiling webauthn-attestation-ca v0.5.5 Compiling calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-api) Compiling ed25519 v2.2.3 Compiling dav-server v0.11.0 Compiling webauthn-rs-core v0.5.5 Compiling ed25519-dalek v2.2.0 Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-plugin) Compiling rsa v0.9.10 Compiling oauth2 v5.0.0 Compiling p256 v0.13.2 Compiling p384 v0.13.1 Compiling blake2 v0.10.6 Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-imap) Compiling webauthn-rs v0.5.5 Compiling argon2 v0.5.3 Compiling openidconnect v4.0.1 Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-tags) Compiling calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-dav) Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-location) Compiling calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-auth) Compiling tempfile v3.27.0 Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/notes) Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/files) Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 03s Running unittests src/lib.rs (/home/kayg/build/targets/hist2-store/debug/deps/calternal_collab-8ced68ecf00e38af) running 40 tests test block_merge_tests::anchored_conflict_copies_have_unique_replay_stable_ids ... ok test block_merge_tests::changed_block_reaches_markdown_after_the_live_side_deletes_it ... ok test block_merge_tests::conflict_keeps_both_versions_external_first ... ok test block_merge_tests::derived_conflict_id_does_not_steal_an_existing_link ... ok test block_merge_tests::issue_89_example ... ok test block_merge_tests::live_only_change_is_kept ... ok test block_merge_tests::same_change_on_both_sides_is_applied_once ... ok test block_merge_tests::shared_change_inside_a_conflicting_chunk_is_applied_once ... ok test history::store::tests::cache_evicts_idle_rooms_and_reopens_durable_points ... ok test history::store::tests::checkpoints_restore_every_point_and_fold_across_a_batch ... ok test history::store::tests::concurrent_appends_have_unique_points_and_index_rows ... ok test history::store::tests::interior_corruption_and_oversized_headers_fail_closed ... ok test history::store::tests::invalid_flush_does_not_advance_state_or_points ... ok test history::store::tests::memory_conforms ... ok test history::store::tests::retention_keeps_clock_reversals_and_the_latest_point ... ok test history::store::tests::segment_conforms ... ok test history::store::tests::torn_tail_recovery_keeps_prior_points_and_accepts_new_updates ... ok test history::store::tests::trained_dictionary_survives_recovery_and_retention ... ok test markdown::source_patch_tests::changed_line_keeps_unedited_crlf_tabs_and_trailing_spaces ... ok test markdown::source_patch_tests::source_patch_refuses_unaligned_line_counts ... ok test repeats::tests::adjacent_duplicates_require_identical_content_and_identity ... ok test repeats::tests::exact_repeats_are_counted ... ok test repeats::tests::near_repeats_and_plain_notes_are_not ... ok test session::client_stream_limit_tests::websocket_stream_limit_uses_resolved_ip_and_retry_after ... ok test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok test session::conflict_shadow_tests::edit_that_reaches_server_before_delete_survives_shadow_merge ... ok test session::conflict_shadow_tests::external_writes_then_typing_do_not_replay_the_live_keystroke ... ok test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok test session::conflict_shadow_tests::local_undo_that_opens_conflict_shadow_does_not_replay_its_paste ... ok test session::conflict_shadow_tests::paste_undo_redo_from_deleting_connection_is_not_a_conflict ... ok test session::conflict_shadow_tests::second_client_edit_to_deleted_block_survives_shadow_merge ... ok test session::conflict_shadow_tests::stale_edit_survives_after_same_connection_deleted_another_block ... ok test session::empty_sync_update_does_not_mark_the_room_document_changed ... ok test session::external_persistence_race_tests::crash_before_save_keeps_epoch_and_replays_pending_edit_once ... ok test session::external_persistence_race_tests::external_reconcile_waits_for_a_fresh_live_snapshot_to_persist ... ok test session::public_edit_limit_tests::oversized_public_update_is_rolled_back ... ok test session::public_edit_limit_tests::public_edit_limit_checks_the_markdown_form ... ok test session::public_edit_limit_tests::ten_thousand_markdown_blocks_open_and_sync_within_two_seconds ... ok test session::shared_notes_deny_without_files_share_authority ... ok test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.38s Running tests/agent_turn_order.rs (/home/kayg/build/targets/hist2-store/debug/deps/agent_turn_order-27a178470289d1ea) running 1 test test agent_turn_with_edits_in_several_places_keeps_order ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.62s Running tests/block_apply_property.rs (/home/kayg/build/targets/hist2-store/debug/deps/block_apply_property-03a93cc8c6ac2651) running 2 tests test external_apply_yields_new_block_order_exactly ... ok test shared_typing_beside_an_external_change_is_applied_once ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.86s Running tests/cross_language.rs (/home/kayg/build/targets/hist2-store/debug/deps/cross_language-dc11530f2468792d) running 1 test test yjs_updates_match_editor_vectors ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 96.53s Running tests/hostile_clients.rs (/home/kayg/build/targets/hist2-store/debug/deps/hostile_clients-8bdab6041788b3b3) running 11 tests test awareness_clock_at_maximum_is_refused_and_room_unloads ... ok test continuous_typing_is_saved_within_the_maximum_wait ... ok test cursed_bodies_open_live ... ok test one_user_cannot_open_unbounded_sockets ... ok test oversized_message_is_refused ... ok test primitive_value_in_fragment_cannot_truncate_the_note ... ok test reconnect_storm_does_not_cancel_pending_save ... ok test trashed_note_room_unloads_instead_of_retrying_forever ... ok test unauthenticated_websocket_routes_return_forbidden ... ok test unrepresentable_update_is_rejected_and_room_keeps_saving ... ok test wiki_embeds_open_live_and_save_unchanged ... ok test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 29.11s Running tests/journal_race.rs (/home/kayg/build/targets/hist2-store/debug/deps/journal_race-f3700b64ee4a632d) running 1 test test journal_log_race_with_live_hub_keeps_all_changes_and_refuses_daily_rooms ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.87s Running tests/restart_epoch.rs (/home/kayg/build/targets/hist2-store/debug/deps/restart_epoch-583c9b870ab44fc4) running 5 tests test graceful_restart_keeps_the_epoch_and_offline_edits ... ok test out_of_band_change_starts_a_new_epoch ... ok test stale_epoch_is_told_the_new_epoch_and_closed ... ok test tab_open_across_a_restart_does_not_duplicate_the_note ... ok test unload_then_reconnect_continues_the_same_room ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.67s Running tests/shared_notes.rs (/home/kayg/build/targets/hist2-store/debug/deps/shared_notes-da4c7d6cad8318ab) running 1 test test owner_editor_viewer_and_live_revoke ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.01s Running tests/two_clients.rs (/home/kayg/build/targets/hist2-store/debug/deps/two_clients-2742aafecff824dc) running 2 tests test concurrent_clients_and_external_writer_converge ... ok test crash_before_debounce_reloads_only_complete_markdown ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.89s Running tests/untouched_bytes.rs (/home/kayg/build/targets/hist2-store/debug/deps/untouched_bytes-c5c2631f5fb63bb5) running 5 tests test clean_room_flush_keeps_noncanonical_note_bytes ... ok test external_edit_to_an_open_room_keeps_its_bytes ... ok test live_edit_preserves_unedited_source_lines ... ok test random_untouched_notes_keep_every_byte ... ok test untouched_save_keeps_every_byte ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 46.95s Running tests/vector_bridge.rs (/home/kayg/build/targets/hist2-store/debug/deps/vector_bridge-77ca536540770e08) running 15 tests test applying_the_same_external_edit_twice_changes_nothing ... ok test concurrent_typing_in_unchanged_block_survives_external_edit ... ok test crash_before_debounce_uses_last_complete_markdown ... ok test editor_schema_json_survives_yrs ... ok test external_edit_after_collaborator_insert_above_keeps_order ... ok test external_edit_beside_concurrently_changed_block_keeps_order ... ok test external_edit_inserts_at_start_and_end ... ok test external_edit_keeps_independent_live_block ... ok test external_edit_mixes_insert_delete_and_replace ... ok test external_edit_moves_blocks ... ok test external_edit_with_two_inserts_keeps_order ... ok test external_insertion_keeps_existing_block_identity ... ok test markdown_vectors_pass_through_yrs ... ok test same_block_conflict_keeps_both_versions ... ok test update_rebuilds_after_restart ... ok test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s Running tests/wiki_embeds.rs (/home/kayg/build/targets/hist2-store/debug/deps/wiki_embeds-ed41ce2b830d274e) running 5 tests test cursed_bodies_open_and_keep_their_content ... ok test embed_in_a_table_cell_stays_one_cell ... ok test image_lines_match_the_editor_reader ... ok test wiki_embed_stays_text_not_an_invented_node ... ok test wiki_embeds_round_trip_byte_for_byte ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.95s Doc-tests calternal_collab error: extern location for calternal_fs does not exist: /home/kayg/build/targets/hist2-store/debug/deps/libcalternal_fs-783dc72f4995ba05.rlib --> crates/calternal-collab/src/session.rs:46:5 | 46 | use calternal_fs::{FileFingerprint, RelPath, Root}; | ^^^^^^^^^^^^ error: aborting due to 1 previous error error: doctest failed, to rerun pass `-p calternal-collab --doc` Caused by: process didn't exit successfully: `/home/kayg/.rustup/toolchains/1.98.1-x86_64-unknown-linux-gnu/bin/rustdoc --edition=2024 --crate-type lib --color auto --crate-name calternal_collab --test crates/calternal-collab/src/lib.rs --test-run-directory /home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab -L native=/home/kayg/build/targets/hist2-store/debug/build/aws-lc-sys-22e2292daf5760d8/out -L native=/home/kayg/build/targets/hist2-store/debug/build/blake3-136881d206af7920/out -L native=/home/kayg/build/targets/hist2-store/debug/build/libsqlite3-sys-122235796a52e197/out -L native=/home/kayg/build/targets/hist2-store/debug/build/libsqlite3-sys-c8822542b868e02e/out -L native=/home/kayg/build/targets/hist2-store/debug/build/ring-3111773b92f61ec3/out -L native=/home/kayg/build/targets/hist2-store/debug/build/xmp_toolkit-5fb6fc1af31f1a6d/out -L native=/home/kayg/build/targets/hist2-store/debug/build/zstd-sys-8c0dc470c8401361/out --test-args --test-threads=1 --extern async_trait=/home/kayg/build/targets/hist2-store/debug/deps/libasync_trait-588cc0803d3ae9cf.so --extern axum=/home/kayg/build/targets/hist2-store/debug/deps/libaxum-5615296a78b2ccd3.rlib --extern base64=/home/kayg/build/targets/hist2-store/debug/deps/libbase64-7e575915217cc61d.rlib --extern blake3=/home/kayg/build/targets/hist2-store/debug/deps/libblake3-b62458aedb76948b.rlib --extern calternal_auth=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_auth-1136d44da560d5d3.rlib --extern calternal_collab=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_collab-de80531be08e2161.rlib --extern calternal_db=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_db-036f5c066dbca621.rlib --extern calternal_fs=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_fs-783dc72f4995ba05.rlib --extern calternal_notes_core=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_notes_core-0d368d261f4da8bb.rlib --extern calternal_plugin=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_plugin-adacd55b0ac95122.rlib --extern calternal_plugin_files=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_plugin_files-67ba0a78b79b20bc.rlib --extern calternal_plugin_notes=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_plugin_notes-c6af46b589a821b8.rlib --extern calternal_tags=/home/kayg/build/targets/hist2-store/debug/deps/libcalternal_tags-9f76b939667e9e59.rlib --extern futures_util=/home/kayg/build/targets/hist2-store/debug/deps/libfutures_util-f3be90bd1edac511.rlib --extern ipnet=/home/kayg/build/targets/hist2-store/debug/deps/libipnet-a225065d7b729d93.rlib --extern notify=/home/kayg/build/targets/hist2-store/debug/deps/libnotify-fa8a3ade9707afd8.rlib --extern serde=/home/kayg/build/targets/hist2-store/debug/deps/libserde-45cf79a2c03a80cd.rlib --extern serde_json=/home/kayg/build/targets/hist2-store/debug/deps/libserde_json-73f174211147ba08.rlib --extern similar=/home/kayg/build/targets/hist2-store/debug/deps/libsimilar-d64eda5642c38bcc.rlib --extern sqlx=/home/kayg/build/targets/hist2-store/debug/deps/libsqlx-e6a1702116896c7a.rlib --extern tempfile=/home/kayg/build/targets/hist2-store/debug/deps/libtempfile-ae85e9649b5e3108.rlib --extern thiserror=/home/kayg/build/targets/hist2-store/debug/deps/libthiserror-7d4222164f8072f6.rlib --extern tokio=/home/kayg/build/targets/hist2-store/debug/deps/libtokio-6e8424ce77408b7d.rlib --extern tokio_tungstenite=/home/kayg/build/targets/hist2-store/debug/deps/libtokio_tungstenite-222e54b2a30de290.rlib --extern tower=/home/kayg/build/targets/hist2-store/debug/deps/libtower-efa9a82cb81f2596.rlib --extern tracing=/home/kayg/build/targets/hist2-store/debug/deps/libtracing-b41d65e1073649c0.rlib --extern utoipa=/home/kayg/build/targets/hist2-store/debug/deps/libutoipa-616642659aa5a371.rlib --extern uuid=/home/kayg/build/targets/hist2-store/debug/deps/libuuid-612694c7e931dc7a.rlib --extern yrs=/home/kayg/build/targets/hist2-store/debug/deps/libyrs-50ad96fa2c3e148c.rlib --extern zstd=/home/kayg/build/targets/hist2-store/debug/deps/libzstd-94094a1f5286493b.rlib -L dependency=/home/kayg/build/targets/hist2-store/debug/deps -C embed-bitcode=no --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("test-hooks"))' --error-format human` (exit status: 1) note: test exited abnormally; to see the full output pass --no-capture to the harness. ``` `cargo test -p calternal-collab --lib history:: -- --test-threads=1 (final focused regression run)`: ```text Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-fs) Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-plugin) Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-location) Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-imap) Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-tags) Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/notes) Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/plugins/files) Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 52s Running unittests src/lib.rs (/home/kayg/build/targets/hist2-store/debug/deps/calternal_collab-8ced68ecf00e38af) running 10 tests test history::store::tests::cache_evicts_idle_rooms_and_reopens_durable_points ... ok test history::store::tests::checkpoints_restore_every_point_and_fold_across_a_batch ... ok test history::store::tests::concurrent_appends_have_unique_points_and_index_rows ... ok test history::store::tests::interior_corruption_and_oversized_headers_fail_closed ... ok test history::store::tests::invalid_flush_does_not_advance_state_or_points ... ok test history::store::tests::memory_conforms ... ok test history::store::tests::retention_keeps_clock_reversals_and_the_latest_point ... ok test history::store::tests::segment_conforms ... ok test history::store::tests::torn_tail_recovery_keeps_prior_points_and_accepts_new_updates ... ok test history::store::tests::trained_dictionary_survives_recovery_and_retention ... ok test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 30 filtered out; finished in 33.95s ``` `cargo test -p calternal-collab --doc -- --test-threads=1 (doc-test recovery)`: ```text Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-store/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 24.61s Doc-tests calternal_collab running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clean`: ```text Removed 9890 files, 5.6GiB total ``` No web build output remains. Worktree status is clean. No issue was closed.
kayg commented 2026-10-03 15:44:52 +00:00
Author
Owner
Copy link

Phase 2 slice C/restore complete on job/hist2-restore.

Head: 0886494d33454e3ad1f40132272aa6a5961eba52. Base: d4e7188810a89fb0e8e6b162279917f7e23989f9.
The required one-time merge used origin/dev at cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e (merge 9ad53bcad). The only conflict was module documentation; both descriptions were kept.

Built

  • prepare_restore: read a point and return one normal forward Yjs update. The Note path reuses the existing block bridge; Canvas changes only affected atomic JSON map entries.
  • prepare_undo: replay with a temporary GC-off document and yrs UndoManager tracked origins. Live documents keep GC on. Later foreign or out-of-range edits protect a whole Canvas element or top-level Note block, including equal-value writes, inserts and deletes.
  • PreparedChange: bind the plan to DocKey and complete live state, including deletes. checked_update must run under the hub lock immediately before the one event path applies the bytes. Preparation holds encoded baseline bytes across store awaits, so a concurrent edit invalidates the plan. No-op plans need no event.
  • Sorted preflight reverted/kept IDs. Order-only Note moves count as reverted. Temporary Note identity markers survive UndoManager resurrection and are removed before forward updates.
  • Fifteen focused tests: 768 seeded Canvas point restores, 144 Note point restores, 96 randomized undo streams against an independent model, independent-client convergence, 64 Canvas and 64 Note undo cases, root deletion/resurrection, foreign deletion/insertion, equal-value writes, stale/delete-only plans, edits during store reads, retained-point gaps, bounded ranges and author boundaries. A unit test rejects malformed or incomplete states.

Files

crates/calternal-collab/src/history/restore.rs, history/mod.rs, tests/history_restore.rs, src/lib.rs, Cargo.toml, and root Cargo.lock.
The public additions outside restore.rs are the shared contract stub, the pub mod history export, and async-trait. No other crate behavior was changed by this slice. Existing test expectations were not changed. All touched doc comments were read again before this report.

Commits

  • 206bbb148: guarded forward Restore and shared contract.
  • 1e08e6145: tracked-author undo and whole-item preservation.
  • 5c98dae39: bounded preflight, order-only reporting and race/retention regressions.
  • 9ad53bcad: required origin/dev merge.
  • 0886494d3: randomized properties and independent-client checks.

Decisions and shared contract details

No approved HistoryStore method signature changed. This branch uses String aliases for UserId/ItemId/TurnId, boxed store errors, and a FoldReport stub with kept_points/removed_points/reclaimed_bytes. Slice A must reconcile these supporting definitions and replace MemoryHistoryStore's placeholder.

Point 0 denotes the initial empty state. Ranges are inclusive and increasing. Point order must match applied event order across authors, even when physical writes are batched. Undo needs the retained predecessor of its first point and all later rows through its captured head.

One undo preflight is capped at 10,000 points and 64 MiB of update bytes. Complete states are capped at the existing 16 MiB room-state limit. Incomplete or over-limit windows fail without a partial apply. Canvas bindings supply their element map name and use atomic JSON values. Note units use blockAnchor where present, otherwise the original Yjs branch ID; replay-only identity attributes never reach clients. Head equivalence uses clocks/deletions plus semantic content, because JSON object wire order can differ after decode. Apply preconditions compare the actual room's complete encoded state.

Known gaps / integration hooks

The store is slice A's responsibility; the contract module here is a stub. Connect the preparation helpers to B/D's authenticated collaboration event path. That path must check owner access, validate the prepared bytes under its room lock, attribute the new event to the caller, append it and broadcast it. Forward bytes use a server-generated Yjs client ID; they must enter the trusted server edit path. Routes, CLI/MCP parity, preview UI, store crash/retention conformance and the full authorization matrix belong to their assigned slices and the integration round.

Nested shared Canvas values are refused; the approved atomic JSON element binding is required. Folded predecessors and windows above the limits cannot be selectively undone by this implementation. No known failing case remains within the implemented slice.

UX gaps closed

Backend cases closed: delete-only and concurrent preflight staleness; foreign equal-value writes; protected inserted/deleted blocks; accurate order-only undo reporting; normal-update convergence without changing room lineage. UI verification belongs to slice E.

UX gaps left

Owner-only API application, calm retry/error presentation, preview, Copy link and Undo controls need the combined B/D/E implementation.

For the merge round

  • cargo clippy -p calternal-server --all-targets -- -D warnings and cargo test -p calternal-server -- --test-threads=4: prove the combined route and event contracts.
  • bun --cwd apps/web run check and bun --cwd apps/web run test --maxWorkers=2: check the combined client and UI.
  • bun --cwd apps/web run test:e2e:notes and bun --cwd apps/web run test:e2e: include two-browser history Restore/undo cases; prove live convergence, room lineage, owner access and the complete history UI after D/E integrate. Use E's Canvas coverage too.
  • bash tests/adversarial/run.sh: run F's new history cases with the complete authorization, cross-User and robustness matrices.
  • On the perf VM, with F's profile integrated and the shared release build: flock /root/perf.lock bash -c 'uptime; bash bench/run.sh --measure-only --runs 5'. Prove history bytes grow with edits, compare Restore/undo/cold-open latency and peak RSS with Phase 1, and check the byte-budget guard. This slice reports functional gates; it did not measure a standalone C performance result.

Gates (verbatim)

The full per-crate suite ran once after the origin/dev merge and passed all 94 tests. The final additional test coverage then ran as the focused file (15 tests); production Rust code did not change after the full suite. This follows the per-slice verification policy. Every build used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0 and CARGO_BUILD_JOBS=4. Tests used this worktree's target/tmp; CARGO_TARGET_DIR stayed at its preset value.

cargo fmt --check: exit 0; no output.

cargo clippy -p calternal-collab --all-targets -- -D warnings:

    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.30s

cargo test -p calternal-collab -- --test-threads=4:

   Compiling typenum v1.20.1
   Compiling rand_core v0.10.1
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling cmov v0.5.4
   Compiling getrandom v0.4.3
   Compiling digest v0.10.7
   Compiling ctutils v0.4.2
   Compiling generic-array v0.14.9
   Compiling hybrid-array v0.4.15
   Compiling sha2 v0.10.9
   Compiling uuid v1.26.1
   Compiling sqlx-core v0.9.0
   Compiling crypto-common v0.2.2
   Compiling inout v0.2.2
   Compiling block-buffer v0.12.1
   Compiling sha1 v0.10.7
   Compiling hmac v0.12.1
   Compiling signature v2.2.0
   Compiling sec1 v0.7.3
   Compiling hkdf v0.12.4
   Compiling tungstenite v0.29.0
   Compiling cipher v0.5.2
   Compiling universal-hash v0.6.1
   Compiling crypto-bigint v0.5.5
   Compiling poly1305 v0.9.1
   Compiling sqlx-sqlite v0.9.0
   Compiling tokio-tungstenite v0.29.0
   Compiling chacha20 v0.10.2
   Compiling elliptic-curve v0.13.8
   Compiling aead v0.6.1
   Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-fs)
   Compiling sqlx-macros-core v0.9.0
   Compiling chacha20poly1305 v0.11.0
   Compiling axum v0.8.9
   Compiling sqlx-macros v0.9.0
   Compiling rfc6979 v0.4.0
   Compiling ecdsa v0.16.9
   Compiling primeorder v0.13.6
   Compiling sqlx v0.9.0
   Compiling curve25519-dalek v4.1.3
   Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-db)
   Compiling headers v0.4.2
   Compiling webauthn-attestation-ca v0.5.5
   Compiling calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-api)
   Compiling ed25519 v2.2.3
   Compiling dav-server v0.11.0
   Compiling webauthn-rs-core v0.5.5
   Compiling ed25519-dalek v2.2.0
   Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-plugin)
   Compiling rsa v0.9.10
   Compiling oauth2 v5.0.0
   Compiling p384 v0.13.1
   Compiling p256 v0.13.2
   Compiling blake2 v0.10.6
   Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-imap)
   Compiling argon2 v0.5.3
   Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-tags)
   Compiling openidconnect v4.0.1
   Compiling webauthn-rs v0.5.5
   Compiling calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-dav)
   Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-location)
   Compiling tempfile v3.27.0
   Compiling calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-auth)
   Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/plugins/notes)
   Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/plugins/files)
   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 48s
     Running unittests src/lib.rs (/home/kayg/build/targets/hist2-restore/debug/deps/calternal_collab-be8e57ec5480e4ec)

running 31 tests
test block_merge_tests::changed_block_reaches_markdown_after_the_live_side_deletes_it ... ok
test block_merge_tests::conflict_keeps_both_versions_external_first ... ok
test block_merge_tests::issue_89_example ... ok
test block_merge_tests::live_only_change_is_kept ... ok
test block_merge_tests::same_change_on_both_sides_is_applied_once ... ok
test block_merge_tests::derived_conflict_id_does_not_steal_an_existing_link ... ok
test block_merge_tests::anchored_conflict_copies_have_unique_replay_stable_ids ... ok
test block_merge_tests::shared_change_inside_a_conflicting_chunk_is_applied_once ... ok
test history::restore::tests::malformed_and_incomplete_states_are_refused ... ok
test markdown::source_patch_tests::changed_line_keeps_unedited_crlf_tabs_and_trailing_spaces ... ok
test markdown::source_patch_tests::source_patch_refuses_unaligned_line_counts ... ok
test repeats::tests::adjacent_duplicates_require_identical_content_and_identity ... ok
test session::client_stream_limit_tests::websocket_stream_limit_uses_resolved_ip_and_retry_after ... ok
test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok
test repeats::tests::near_repeats_and_plain_notes_are_not ... ok
test repeats::tests::exact_repeats_are_counted ... ok
test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok
test session::conflict_shadow_tests::external_writes_then_typing_do_not_replay_the_live_keystroke ... ok
test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok
test session::conflict_shadow_tests::local_undo_that_opens_conflict_shadow_does_not_replay_its_paste ... ok
test session::conflict_shadow_tests::edit_that_reaches_server_before_delete_survives_shadow_merge ... ok
test session::conflict_shadow_tests::paste_undo_redo_from_deleting_connection_is_not_a_conflict ... ok
test session::empty_sync_update_does_not_mark_the_room_document_changed ... ok
test session::conflict_shadow_tests::second_client_edit_to_deleted_block_survives_shadow_merge ... ok
test session::conflict_shadow_tests::stale_edit_survives_after_same_connection_deleted_another_block ... ok
test session::public_edit_limit_tests::oversized_public_update_is_rolled_back ... ok
test session::public_edit_limit_tests::public_edit_limit_checks_the_markdown_form ... ok
test session::shared_notes_deny_without_files_share_authority ... ok
test session::external_persistence_race_tests::crash_before_save_keeps_epoch_and_replays_pending_edit_once ... ok
test session::external_persistence_race_tests::external_reconcile_waits_for_a_fresh_live_snapshot_to_persist ... ok
test session::public_edit_limit_tests::ten_thousand_markdown_blocks_open_and_sync_within_two_seconds ... ok

test result: ok. 31 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.43s

     Running tests/agent_turn_order.rs (/home/kayg/build/targets/hist2-restore/debug/deps/agent_turn_order-671e4310f9c9624d)

running 1 test
test agent_turn_with_edits_in_several_places_keeps_order ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.54s

     Running tests/block_apply_property.rs (/home/kayg/build/targets/hist2-restore/debug/deps/block_apply_property-2b81a8f4b5fb39d6)

running 2 tests
test shared_typing_beside_an_external_change_is_applied_once ... ok
test external_apply_yields_new_block_order_exactly ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.82s

     Running tests/cross_language.rs (/home/kayg/build/targets/hist2-restore/debug/deps/cross_language-737379759b285c7c)

running 1 test
test yjs_updates_match_editor_vectors has been running for over 60 seconds
test yjs_updates_match_editor_vectors ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 95.14s

     Running tests/history_restore.rs (/home/kayg/build/targets/hist2-restore/debug/deps/history_restore-1d2e1cb611e89bd2)

running 14 tests
test an_edit_during_store_read_invalidates_restore_and_undo_plans ... ok
test author_selection_is_inclusive_and_does_not_undo_other_ranges ... ok
test note_foreign_edit_keeps_an_agent_inserted_block ... ok
test note_restore_and_undo_root_deletion_resurrect_without_markers ... ok
test note_restore_reuses_block_bridge_and_keeps_replica_lineage ... ok
test note_undo_keeps_a_foreign_deleted_block_absent ... ok
test canvas_undo_retains_foreign_edits_and_reports_equal_value_writes ... ok
test note_undo_reports_a_reordered_block_and_preserves_foreign_text ... ok
test stale_preflight_checks_delete_only_edits_and_cross_user_keys ... ok
test undo_converges_across_independent_author_clients ... ok
test undo_refuses_missing_points_and_invalid_or_oversized_ranges ... ok
test note_undo_keeps_whole_foreign_block_and_other_authors ... ok
test note_restore_every_seeded_point_is_exact ... ok
test canvas_restore_every_point_is_exact_and_converges ... ok

test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.06s

     Running tests/hostile_clients.rs (/home/kayg/build/targets/hist2-restore/debug/deps/hostile_clients-0fefa55cf60e1af2)

running 11 tests
test awareness_clock_at_maximum_is_refused_and_room_unloads ... ok
test one_user_cannot_open_unbounded_sockets ... ok
test primitive_value_in_fragment_cannot_truncate_the_note ... ok
test oversized_message_is_refused ... ok
test cursed_bodies_open_live ... ok
test trashed_note_room_unloads_instead_of_retrying_forever ... ok
test reconnect_storm_does_not_cancel_pending_save ... ok
test continuous_typing_is_saved_within_the_maximum_wait ... ok
test unauthenticated_websocket_routes_return_forbidden ... ok
test wiki_embeds_open_live_and_save_unchanged ... ok
test unrepresentable_update_is_rejected_and_room_keeps_saving ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.42s

     Running tests/journal_race.rs (/home/kayg/build/targets/hist2-restore/debug/deps/journal_race-f82017f29be604f2)

running 1 test
test journal_log_race_with_live_hub_keeps_all_changes_and_refuses_daily_rooms ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s

     Running tests/restart_epoch.rs (/home/kayg/build/targets/hist2-restore/debug/deps/restart_epoch-4fd18dc9b8605371)

running 5 tests
test stale_epoch_is_told_the_new_epoch_and_closed ... ok
test tab_open_across_a_restart_does_not_duplicate_the_note ... ok
test out_of_band_change_starts_a_new_epoch ... ok
test graceful_restart_keeps_the_epoch_and_offline_edits ... ok
test unload_then_reconnect_continues_the_same_room ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.67s

     Running tests/shared_notes.rs (/home/kayg/build/targets/hist2-restore/debug/deps/shared_notes-58186e7bf384a855)

running 1 test
test owner_editor_viewer_and_live_revoke ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.85s

     Running tests/two_clients.rs (/home/kayg/build/targets/hist2-restore/debug/deps/two_clients-d83f6d9346f8e74e)

running 2 tests
test crash_before_debounce_reloads_only_complete_markdown ... ok
test concurrent_clients_and_external_writer_converge ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.97s

     Running tests/untouched_bytes.rs (/home/kayg/build/targets/hist2-restore/debug/deps/untouched_bytes-7a7031e545613583)

running 5 tests
test clean_room_flush_keeps_noncanonical_note_bytes ... ok
test external_edit_to_an_open_room_keeps_its_bytes ... ok
test live_edit_preserves_unedited_source_lines ... ok
test untouched_save_keeps_every_byte ... ok
test random_untouched_notes_keep_every_byte ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 27.10s

     Running tests/vector_bridge.rs (/home/kayg/build/targets/hist2-restore/debug/deps/vector_bridge-051c564a2e4d1043)

running 15 tests
test crash_before_debounce_uses_last_complete_markdown ... ok
test concurrent_typing_in_unchanged_block_survives_external_edit ... ok
test applying_the_same_external_edit_twice_changes_nothing ... ok
test external_edit_after_collaborator_insert_above_keeps_order ... ok
test external_edit_keeps_independent_live_block ... ok
test external_edit_inserts_at_start_and_end ... ok
test external_edit_beside_concurrently_changed_block_keeps_order ... ok
test external_edit_mixes_insert_delete_and_replace ... ok
test external_edit_moves_blocks ... ok
test external_insertion_keeps_existing_block_identity ... ok
test external_edit_with_two_inserts_keeps_order ... ok
test update_rebuilds_after_restart ... ok
test same_block_conflict_keeps_both_versions ... ok
test editor_schema_json_survives_yrs ... ok
test markdown_vectors_pass_through_yrs ... ok

test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

     Running tests/wiki_embeds.rs (/home/kayg/build/targets/hist2-restore/debug/deps/wiki_embeds-57b661fd48bce0f9)

running 5 tests
test wiki_embed_stays_text_not_an_invented_node ... ok
test embed_in_a_table_cell_stays_one_cell ... ok
test image_lines_match_the_editor_reader ... ok
test wiki_embeds_round_trip_byte_for_byte ... ok
test cursed_bodies_open_and_keep_their_content ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.93s

   Doc-tests calternal_collab

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-collab --test history_restore -- --test-threads=4:

   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 8.39s
     Running tests/history_restore.rs (/home/kayg/build/targets/hist2-restore/debug/deps/history_restore-1d2e1cb611e89bd2)

running 15 tests
test an_edit_during_store_read_invalidates_restore_and_undo_plans ... ok
test author_selection_is_inclusive_and_does_not_undo_other_ranges ... ok
test note_foreign_edit_keeps_an_agent_inserted_block ... ok
test note_restore_and_undo_root_deletion_resurrect_without_markers ... ok
test note_restore_reuses_block_bridge_and_keeps_replica_lineage ... ok
test note_undo_keeps_a_foreign_deleted_block_absent ... ok
test canvas_undo_retains_foreign_edits_and_reports_equal_value_writes ... ok
test note_undo_reports_a_reordered_block_and_preserves_foreign_text ... ok
test note_undo_keeps_whole_foreign_block_and_other_authors ... ok
test stale_preflight_checks_delete_only_edits_and_cross_user_keys ... ok
test undo_converges_across_independent_author_clients ... ok
test undo_refuses_missing_points_and_invalid_or_oversized_ranges ... ok
test note_restore_every_seeded_point_is_exact ... ok
test seeded_undo_property_keeps_other_authors_and_reports_every_skip ... ok
test canvas_restore_every_point_is_exact_and_converges ... ok

test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.18s

Cleanup (cargo clean):

     Removed 9907 files, 5.7GiB total

Web build output and test scratch files were removed. The worktree is clean. The issue remains open for integration.

Phase 2 slice C/restore complete on `job/hist2-restore`. Head: `0886494d33454e3ad1f40132272aa6a5961eba52`. Base: `d4e7188810a89fb0e8e6b162279917f7e23989f9`. The required one-time merge used `origin/dev` at `cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e` (merge `9ad53bcad`). The only conflict was module documentation; both descriptions were kept. ## Built - `prepare_restore`: read a point and return one normal forward Yjs update. The Note path reuses the existing block bridge; Canvas changes only affected atomic JSON map entries. - `prepare_undo`: replay with a temporary GC-off document and yrs UndoManager tracked origins. Live documents keep GC on. Later foreign or out-of-range edits protect a whole Canvas element or top-level Note block, including equal-value writes, inserts and deletes. - `PreparedChange`: bind the plan to DocKey and complete live state, including deletes. `checked_update` must run under the hub lock immediately before the one event path applies the bytes. Preparation holds encoded baseline bytes across store awaits, so a concurrent edit invalidates the plan. No-op plans need no event. - Sorted preflight `reverted`/`kept` IDs. Order-only Note moves count as reverted. Temporary Note identity markers survive UndoManager resurrection and are removed before forward updates. - Fifteen focused tests: 768 seeded Canvas point restores, 144 Note point restores, 96 randomized undo streams against an independent model, independent-client convergence, 64 Canvas and 64 Note undo cases, root deletion/resurrection, foreign deletion/insertion, equal-value writes, stale/delete-only plans, edits during store reads, retained-point gaps, bounded ranges and author boundaries. A unit test rejects malformed or incomplete states. ## Files `crates/calternal-collab/src/history/restore.rs`, `history/mod.rs`, `tests/history_restore.rs`, `src/lib.rs`, `Cargo.toml`, and root `Cargo.lock`. The public additions outside restore.rs are the shared contract stub, the `pub mod history` export, and async-trait. No other crate behavior was changed by this slice. Existing test expectations were not changed. All touched doc comments were read again before this report. ## Commits - `206bbb148`: guarded forward Restore and shared contract. - `1e08e6145`: tracked-author undo and whole-item preservation. - `5c98dae39`: bounded preflight, order-only reporting and race/retention regressions. - `9ad53bcad`: required origin/dev merge. - `0886494d3`: randomized properties and independent-client checks. ## Decisions and shared contract details No approved HistoryStore method signature changed. This branch uses String aliases for UserId/ItemId/TurnId, boxed store errors, and a FoldReport stub with kept_points/removed_points/reclaimed_bytes. Slice A must reconcile these supporting definitions and replace MemoryHistoryStore's placeholder. Point 0 denotes the initial empty state. Ranges are inclusive and increasing. Point order must match applied event order across authors, even when physical writes are batched. Undo needs the retained predecessor of its first point and all later rows through its captured head. One undo preflight is capped at 10,000 points and 64 MiB of update bytes. Complete states are capped at the existing 16 MiB room-state limit. Incomplete or over-limit windows fail without a partial apply. Canvas bindings supply their element map name and use atomic JSON values. Note units use blockAnchor where present, otherwise the original Yjs branch ID; replay-only identity attributes never reach clients. Head equivalence uses clocks/deletions plus semantic content, because JSON object wire order can differ after decode. Apply preconditions compare the actual room's complete encoded state. ## Known gaps / integration hooks The store is slice A's responsibility; the contract module here is a stub. Connect the preparation helpers to B/D's authenticated collaboration event path. That path must check owner access, validate the prepared bytes under its room lock, attribute the new event to the caller, append it and broadcast it. Forward bytes use a server-generated Yjs client ID; they must enter the trusted server edit path. Routes, CLI/MCP parity, preview UI, store crash/retention conformance and the full authorization matrix belong to their assigned slices and the integration round. Nested shared Canvas values are refused; the approved atomic JSON element binding is required. Folded predecessors and windows above the limits cannot be selectively undone by this implementation. No known failing case remains within the implemented slice. ## UX gaps closed Backend cases closed: delete-only and concurrent preflight staleness; foreign equal-value writes; protected inserted/deleted blocks; accurate order-only undo reporting; normal-update convergence without changing room lineage. UI verification belongs to slice E. ## UX gaps left Owner-only API application, calm retry/error presentation, preview, Copy link and Undo controls need the combined B/D/E implementation. ## For the merge round - `cargo clippy -p calternal-server --all-targets -- -D warnings` and `cargo test -p calternal-server -- --test-threads=4`: prove the combined route and event contracts. - `bun --cwd apps/web run check` and `bun --cwd apps/web run test --maxWorkers=2`: check the combined client and UI. - `bun --cwd apps/web run test:e2e:notes` and `bun --cwd apps/web run test:e2e`: include two-browser history Restore/undo cases; prove live convergence, room lineage, owner access and the complete history UI after D/E integrate. Use E's Canvas coverage too. - `bash tests/adversarial/run.sh`: run F's new history cases with the complete authorization, cross-User and robustness matrices. - On the perf VM, with F's profile integrated and the shared release build: `flock /root/perf.lock bash -c 'uptime; bash bench/run.sh --measure-only --runs 5'`. Prove history bytes grow with edits, compare Restore/undo/cold-open latency and peak RSS with Phase 1, and check the byte-budget guard. This slice reports functional gates; it did not measure a standalone C performance result. ## Gates (verbatim) The full per-crate suite ran once after the origin/dev merge and passed all 94 tests. The final additional test coverage then ran as the focused file (15 tests); production Rust code did not change after the full suite. This follows the per-slice verification policy. Every build used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0 and CARGO_BUILD_JOBS=4. Tests used this worktree's target/tmp; CARGO_TARGET_DIR stayed at its preset value. `cargo fmt --check`: exit 0; no output. `cargo clippy -p calternal-collab --all-targets -- -D warnings`: ```text Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.30s ``` `cargo test -p calternal-collab -- --test-threads=4`: ```text Compiling typenum v1.20.1 Compiling rand_core v0.10.1 Compiling crypto-common v0.1.6 Compiling block-buffer v0.10.4 Compiling cmov v0.5.4 Compiling getrandom v0.4.3 Compiling digest v0.10.7 Compiling ctutils v0.4.2 Compiling generic-array v0.14.9 Compiling hybrid-array v0.4.15 Compiling sha2 v0.10.9 Compiling uuid v1.26.1 Compiling sqlx-core v0.9.0 Compiling crypto-common v0.2.2 Compiling inout v0.2.2 Compiling block-buffer v0.12.1 Compiling sha1 v0.10.7 Compiling hmac v0.12.1 Compiling signature v2.2.0 Compiling sec1 v0.7.3 Compiling hkdf v0.12.4 Compiling tungstenite v0.29.0 Compiling cipher v0.5.2 Compiling universal-hash v0.6.1 Compiling crypto-bigint v0.5.5 Compiling poly1305 v0.9.1 Compiling sqlx-sqlite v0.9.0 Compiling tokio-tungstenite v0.29.0 Compiling chacha20 v0.10.2 Compiling elliptic-curve v0.13.8 Compiling aead v0.6.1 Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-fs) Compiling sqlx-macros-core v0.9.0 Compiling chacha20poly1305 v0.11.0 Compiling axum v0.8.9 Compiling sqlx-macros v0.9.0 Compiling rfc6979 v0.4.0 Compiling ecdsa v0.16.9 Compiling primeorder v0.13.6 Compiling sqlx v0.9.0 Compiling curve25519-dalek v4.1.3 Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-db) Compiling headers v0.4.2 Compiling webauthn-attestation-ca v0.5.5 Compiling calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-api) Compiling ed25519 v2.2.3 Compiling dav-server v0.11.0 Compiling webauthn-rs-core v0.5.5 Compiling ed25519-dalek v2.2.0 Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-plugin) Compiling rsa v0.9.10 Compiling oauth2 v5.0.0 Compiling p384 v0.13.1 Compiling p256 v0.13.2 Compiling blake2 v0.10.6 Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-imap) Compiling argon2 v0.5.3 Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-tags) Compiling openidconnect v4.0.1 Compiling webauthn-rs v0.5.5 Compiling calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-dav) Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-location) Compiling tempfile v3.27.0 Compiling calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-auth) Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/plugins/notes) Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/plugins/files) Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 48s Running unittests src/lib.rs (/home/kayg/build/targets/hist2-restore/debug/deps/calternal_collab-be8e57ec5480e4ec) running 31 tests test block_merge_tests::changed_block_reaches_markdown_after_the_live_side_deletes_it ... ok test block_merge_tests::conflict_keeps_both_versions_external_first ... ok test block_merge_tests::issue_89_example ... ok test block_merge_tests::live_only_change_is_kept ... ok test block_merge_tests::same_change_on_both_sides_is_applied_once ... ok test block_merge_tests::derived_conflict_id_does_not_steal_an_existing_link ... ok test block_merge_tests::anchored_conflict_copies_have_unique_replay_stable_ids ... ok test block_merge_tests::shared_change_inside_a_conflicting_chunk_is_applied_once ... ok test history::restore::tests::malformed_and_incomplete_states_are_refused ... ok test markdown::source_patch_tests::changed_line_keeps_unedited_crlf_tabs_and_trailing_spaces ... ok test markdown::source_patch_tests::source_patch_refuses_unaligned_line_counts ... ok test repeats::tests::adjacent_duplicates_require_identical_content_and_identity ... ok test session::client_stream_limit_tests::websocket_stream_limit_uses_resolved_ip_and_retry_after ... ok test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok test repeats::tests::near_repeats_and_plain_notes_are_not ... ok test repeats::tests::exact_repeats_are_counted ... ok test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok test session::conflict_shadow_tests::external_writes_then_typing_do_not_replay_the_live_keystroke ... ok test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok test session::conflict_shadow_tests::local_undo_that_opens_conflict_shadow_does_not_replay_its_paste ... ok test session::conflict_shadow_tests::edit_that_reaches_server_before_delete_survives_shadow_merge ... ok test session::conflict_shadow_tests::paste_undo_redo_from_deleting_connection_is_not_a_conflict ... ok test session::empty_sync_update_does_not_mark_the_room_document_changed ... ok test session::conflict_shadow_tests::second_client_edit_to_deleted_block_survives_shadow_merge ... ok test session::conflict_shadow_tests::stale_edit_survives_after_same_connection_deleted_another_block ... ok test session::public_edit_limit_tests::oversized_public_update_is_rolled_back ... ok test session::public_edit_limit_tests::public_edit_limit_checks_the_markdown_form ... ok test session::shared_notes_deny_without_files_share_authority ... ok test session::external_persistence_race_tests::crash_before_save_keeps_epoch_and_replays_pending_edit_once ... ok test session::external_persistence_race_tests::external_reconcile_waits_for_a_fresh_live_snapshot_to_persist ... ok test session::public_edit_limit_tests::ten_thousand_markdown_blocks_open_and_sync_within_two_seconds ... ok test result: ok. 31 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.43s Running tests/agent_turn_order.rs (/home/kayg/build/targets/hist2-restore/debug/deps/agent_turn_order-671e4310f9c9624d) running 1 test test agent_turn_with_edits_in_several_places_keeps_order ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.54s Running tests/block_apply_property.rs (/home/kayg/build/targets/hist2-restore/debug/deps/block_apply_property-2b81a8f4b5fb39d6) running 2 tests test shared_typing_beside_an_external_change_is_applied_once ... ok test external_apply_yields_new_block_order_exactly ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.82s Running tests/cross_language.rs (/home/kayg/build/targets/hist2-restore/debug/deps/cross_language-737379759b285c7c) running 1 test test yjs_updates_match_editor_vectors has been running for over 60 seconds test yjs_updates_match_editor_vectors ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 95.14s Running tests/history_restore.rs (/home/kayg/build/targets/hist2-restore/debug/deps/history_restore-1d2e1cb611e89bd2) running 14 tests test an_edit_during_store_read_invalidates_restore_and_undo_plans ... ok test author_selection_is_inclusive_and_does_not_undo_other_ranges ... ok test note_foreign_edit_keeps_an_agent_inserted_block ... ok test note_restore_and_undo_root_deletion_resurrect_without_markers ... ok test note_restore_reuses_block_bridge_and_keeps_replica_lineage ... ok test note_undo_keeps_a_foreign_deleted_block_absent ... ok test canvas_undo_retains_foreign_edits_and_reports_equal_value_writes ... ok test note_undo_reports_a_reordered_block_and_preserves_foreign_text ... ok test stale_preflight_checks_delete_only_edits_and_cross_user_keys ... ok test undo_converges_across_independent_author_clients ... ok test undo_refuses_missing_points_and_invalid_or_oversized_ranges ... ok test note_undo_keeps_whole_foreign_block_and_other_authors ... ok test note_restore_every_seeded_point_is_exact ... ok test canvas_restore_every_point_is_exact_and_converges ... ok test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.06s Running tests/hostile_clients.rs (/home/kayg/build/targets/hist2-restore/debug/deps/hostile_clients-0fefa55cf60e1af2) running 11 tests test awareness_clock_at_maximum_is_refused_and_room_unloads ... ok test one_user_cannot_open_unbounded_sockets ... ok test primitive_value_in_fragment_cannot_truncate_the_note ... ok test oversized_message_is_refused ... ok test cursed_bodies_open_live ... ok test trashed_note_room_unloads_instead_of_retrying_forever ... ok test reconnect_storm_does_not_cancel_pending_save ... ok test continuous_typing_is_saved_within_the_maximum_wait ... ok test unauthenticated_websocket_routes_return_forbidden ... ok test wiki_embeds_open_live_and_save_unchanged ... ok test unrepresentable_update_is_rejected_and_room_keeps_saving ... ok test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.42s Running tests/journal_race.rs (/home/kayg/build/targets/hist2-restore/debug/deps/journal_race-f82017f29be604f2) running 1 test test journal_log_race_with_live_hub_keeps_all_changes_and_refuses_daily_rooms ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s Running tests/restart_epoch.rs (/home/kayg/build/targets/hist2-restore/debug/deps/restart_epoch-4fd18dc9b8605371) running 5 tests test stale_epoch_is_told_the_new_epoch_and_closed ... ok test tab_open_across_a_restart_does_not_duplicate_the_note ... ok test out_of_band_change_starts_a_new_epoch ... ok test graceful_restart_keeps_the_epoch_and_offline_edits ... ok test unload_then_reconnect_continues_the_same_room ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.67s Running tests/shared_notes.rs (/home/kayg/build/targets/hist2-restore/debug/deps/shared_notes-58186e7bf384a855) running 1 test test owner_editor_viewer_and_live_revoke ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.85s Running tests/two_clients.rs (/home/kayg/build/targets/hist2-restore/debug/deps/two_clients-d83f6d9346f8e74e) running 2 tests test crash_before_debounce_reloads_only_complete_markdown ... ok test concurrent_clients_and_external_writer_converge ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.97s Running tests/untouched_bytes.rs (/home/kayg/build/targets/hist2-restore/debug/deps/untouched_bytes-7a7031e545613583) running 5 tests test clean_room_flush_keeps_noncanonical_note_bytes ... ok test external_edit_to_an_open_room_keeps_its_bytes ... ok test live_edit_preserves_unedited_source_lines ... ok test untouched_save_keeps_every_byte ... ok test random_untouched_notes_keep_every_byte ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 27.10s Running tests/vector_bridge.rs (/home/kayg/build/targets/hist2-restore/debug/deps/vector_bridge-051c564a2e4d1043) running 15 tests test crash_before_debounce_uses_last_complete_markdown ... ok test concurrent_typing_in_unchanged_block_survives_external_edit ... ok test applying_the_same_external_edit_twice_changes_nothing ... ok test external_edit_after_collaborator_insert_above_keeps_order ... ok test external_edit_keeps_independent_live_block ... ok test external_edit_inserts_at_start_and_end ... ok test external_edit_beside_concurrently_changed_block_keeps_order ... ok test external_edit_mixes_insert_delete_and_replace ... ok test external_edit_moves_blocks ... ok test external_insertion_keeps_existing_block_identity ... ok test external_edit_with_two_inserts_keeps_order ... ok test update_rebuilds_after_restart ... ok test same_block_conflict_keeps_both_versions ... ok test editor_schema_json_survives_yrs ... ok test markdown_vectors_pass_through_yrs ... ok test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s Running tests/wiki_embeds.rs (/home/kayg/build/targets/hist2-restore/debug/deps/wiki_embeds-57b661fd48bce0f9) running 5 tests test wiki_embed_stays_text_not_an_invented_node ... ok test embed_in_a_table_cell_stays_one_cell ... ok test image_lines_match_the_editor_reader ... ok test wiki_embeds_round_trip_byte_for_byte ... ok test cursed_bodies_open_and_keep_their_content ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.93s Doc-tests calternal_collab running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-collab --test history_restore -- --test-threads=4`: ```text Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-restore/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 8.39s Running tests/history_restore.rs (/home/kayg/build/targets/hist2-restore/debug/deps/history_restore-1d2e1cb611e89bd2) running 15 tests test an_edit_during_store_read_invalidates_restore_and_undo_plans ... ok test author_selection_is_inclusive_and_does_not_undo_other_ranges ... ok test note_foreign_edit_keeps_an_agent_inserted_block ... ok test note_restore_and_undo_root_deletion_resurrect_without_markers ... ok test note_restore_reuses_block_bridge_and_keeps_replica_lineage ... ok test note_undo_keeps_a_foreign_deleted_block_absent ... ok test canvas_undo_retains_foreign_edits_and_reports_equal_value_writes ... ok test note_undo_reports_a_reordered_block_and_preserves_foreign_text ... ok test note_undo_keeps_whole_foreign_block_and_other_authors ... ok test stale_preflight_checks_delete_only_edits_and_cross_user_keys ... ok test undo_converges_across_independent_author_clients ... ok test undo_refuses_missing_points_and_invalid_or_oversized_ranges ... ok test note_restore_every_seeded_point_is_exact ... ok test seeded_undo_property_keeps_other_authors_and_reports_every_skip ... ok test canvas_restore_every_point_is_exact_and_converges ... ok test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.18s ``` Cleanup (`cargo clean`): ```text Removed 9907 files, 5.7GiB total ``` Web build output and test scratch files were removed. The worktree is clean. The issue remains open for integration.
kayg commented 2026-10-03 15:47:47 +00:00
Author
Owner
Copy link

Finding: the focused oversized-Undo case sent a 20,000-byte author ID through a route capped at 16 KiB. Axum 0.8.9 returned a JsonRejection with status 413, but the handler collapsed it to 400. I now preserve 413 with a generic error envelope, document 413 in both Undo OpenAPI operations, and keep malformed JSON on the 400 path. Evidence: cargo test -p calternal-server history_contract_tests -- --nocapture reports 7 passed, 0 failed.

Finding: the focused oversized-Undo case sent a 20,000-byte author ID through a route capped at 16 KiB. Axum 0.8.9 returned a `JsonRejection` with status 413, but the handler collapsed it to 400. I now preserve 413 with a generic error envelope, document 413 in both Undo OpenAPI operations, and keep malformed JSON on the 400 path. Evidence: `cargo test -p calternal-server history_contract_tests -- --nocapture` reports 7 passed, 0 failed.
kayg commented 2026-10-03 15:47:58 +00:00
Author
Owner
Copy link

Phase 2 bench finding — perf VM, 2026-10-03

The Y4-500 segment profile with decoded latest-checkpoint cache completed five locked runs per workload. The byte guard passed: Canvas was 33,411,072 B / 11,100 edits = 3,010,006 B per 1,000 edits against 3,100,000; Note was 548,864 B / 7,220 edits = 76,020 B per 1,000 edits against 80,000. Phase 1 measured 33,407,632 B and 545,260 B respectively, so the open-size deltas are +3,440 B (+0.010%) and +3,604 B (+0.661%).

C1 middle Restore p95 was 40.954 ms; undo p95 was 28.680 ms; peak RSS p95 was 156.621 MiB. N1 middle Restore p95 was 3.767 ms; undo p95 was 3.575 ms; peak RSS p95 was 18.988 MiB. All runs reported correct and supported undo.

The largest C1 trace has 5,000 starting elements, 11,100 edits, a 300-edit agent turn and 200-point freehand strokes. Its maximum checkpoint time p95 was 4,973 ms. DESIGN §61 sets no checkpoint-time threshold, and docs/perf/baseline.json had no prior history timing or RSS profile. I recorded this as the first profile for owner review; byte thresholds and full metrics are in docs/perf/live-history-budget.md and docs/perf/baseline.json. Each measured run held /root/perf.lock and recorded load while holding it.

Phase 2 bench finding — perf VM, 2026-10-03 The Y4-500 segment profile with decoded latest-checkpoint cache completed five locked runs per workload. The byte guard passed: Canvas was 33,411,072 B / 11,100 edits = 3,010,006 B per 1,000 edits against 3,100,000; Note was 548,864 B / 7,220 edits = 76,020 B per 1,000 edits against 80,000. Phase 1 measured 33,407,632 B and 545,260 B respectively, so the open-size deltas are +3,440 B (+0.010%) and +3,604 B (+0.661%). C1 middle Restore p95 was 40.954 ms; undo p95 was 28.680 ms; peak RSS p95 was 156.621 MiB. N1 middle Restore p95 was 3.767 ms; undo p95 was 3.575 ms; peak RSS p95 was 18.988 MiB. All runs reported correct and supported undo. The largest C1 trace has 5,000 starting elements, 11,100 edits, a 300-edit agent turn and 200-point freehand strokes. Its maximum checkpoint time p95 was 4,973 ms. DESIGN §61 sets no checkpoint-time threshold, and docs/perf/baseline.json had no prior history timing or RSS profile. I recorded this as the first profile for owner review; byte thresholds and full metrics are in docs/perf/live-history-budget.md and docs/perf/baseline.json. Each measured run held /root/perf.lock and recorded load while holding it.
kayg commented 2026-10-03 15:50:51 +00:00
Author
Owner
Copy link

Phase 2 slice B (write) complete

Branch: job/hist2-write.
Head: 6d327cbca21b88b7d8ebd9933b95af62157b6092.
Start/base: d4e7188810a89fb0e8e6b162279917f7e23989f9.
Required merge: origin/dev at cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e, merge commit 22dc32fbe75a689cac3994a62d3da8c1a79c6e10. The one conflict kept both the Daily note selector check (#646) and first-load serialization. No push or deployment. No merge into dev or main.

Built

  • Shared HistoryStore signatures from the approved plan. Slice A's store and MemoryHistoryStore implementation remain for integration.
  • Document-neutral HistoryWriter for Notes and future Canvas rooms. It merges only consecutive updates by the same author. Foreign updates end a batch; replay order stays correct. A one-second task drains the queue. File saves, last-client saves and shutdown also drain it. Failed appends stay queued and retry with bounded backoff.
  • Home quota reservations before mutation, cancellation-safe reservation cleanup, bounded pending bytes and durable daily collaborator counters in the Index.
  • Active connection ownership for Yjs client IDs, durable author ownership across unload/restart, server ID protection, and bounded cursor-only claims. Cursor-only claims do not create Security-state rows. Existing foreign structs can be echoed during normal sync; new clocks require ownership.
  • Hub capture before broadcast for authenticated Users, shared editors, verified guests, server-issued agent turn IDs and attributed external reconciliation. New server hook: notify_external_change_as.
  • A history seed before incremental edits; first-load serialization prevents competing seeds. History-enabled rooms reject legacy path: selectors because history requires a stable Note identity.
  • Focused regressions for author order, text/map replay, delete sets, append failures, background retry, quota, cancelled admission, durable budgets, backward clock movement, pooled agent budgets, ID ownership, presence, real WebSocket capture, agent replay and external actor capture. Unresolved Yjs dependencies roll back instead of taking effect under a later author. Existing test expectations were not changed.

Files

crates/calternal-collab/src/history/mod.rs
crates/calternal-collab/src/history/write.rs
crates/calternal-collab/src/history/write_tests.rs
crates/calternal-collab/src/history/0001_write_budget.sql
crates/calternal-collab/src/session.rs
crates/calternal-collab/src/lib.rs
crates/calternal-collab/Cargo.toml
Cargo.lock

Atomic commits

  • 2a4974013: history admission, budgets, batching and ownership boundary.
  • cb79d157b: Hub capture and real collaboration-path regressions.
  • 4067e2bb4: cancellation, retry and presence resource bounds.
  • 22dc32fbe: required origin/dev merge.
  • 6d327cbca: stable Note identity guard and its regression.

Decisions

The plan did not set numeric limits. Defaults are 64 MiB of charged input per owner/collaborator per UTC day and 64 MiB of pending quota allowance. WriteLimits can set other instance limits. All agent turns in one Home share the agent allowance, so a new turn ID cannot reset it. First client-ID writes include a 512-byte accounting allowance. Queued quota reserves twice the charged bytes plus 1 KiB for framing/expansion.

Batching has a one-second maximum healthy wait; an existing file-save flush can drain sooner. The first update supplies the batch time. All times come from the server. Failed appends retry at up to 30-second intervals with one task.

The migration uses its own collab-history-write namespace, version 1. It has no collision with Notes migration numbers; origin/dev had no history module at the required merge. IDs use the existing String representation. The shared contract left FoldReport fields and error type unspecified: this branch supplies provisional removed_points/reclaimed_bytes fields and an opaque error Result. A may reconcile those details. None of the six HistoryStore signatures changed.

I verified yrs 0.28.0 and async-trait 0.1.92 with cargo registry commands. async-trait is MIT OR Apache-2.0. No licence change.

Findings fixed

The first replay regression produced a instead of abcd: yrs::Update::merge does not preserve the complete batch and its delete set. The writer now uses Update::merge_updates; typing and deletion replay tests pass.

PluginEvent has no actor field. Server API callers can use the new verified-actor hook; legacy filesystem notices use the Home owner. No request payload supplies an Author.

Known gaps / integration requirements

  • Install A with Hub::with_history(store, limits) before rooms open. The default Hub has no history store until integration. MemoryHistoryStore is a declaration placeholder here; the test recorder is test-only. It is not shipped as a substitute store.
  • Canvas rooms do not exist on this base. Their event path must call the same HistoryWriter admission/capture functions. No Canvas protocol or second writer was added.
  • A must keep segment bytes in the Home quota accounting and enforce quota for checkpoints/dictionaries as well as update rows. The approved append signature does not carry the writer's reservation ID; conservative reservations can make a near-full Home reject earlier.
  • The integration must replace full-copy Versions for history-enabled live documents. This slice keeps the existing Notes writer and guest Version callback; it does not change another crate's file-version policy.
  • Integration must reconcile the existing history stream when a Note's cached Yrs lineage is discarded after an out-of-room file replacement. B seeds only an empty stream. The current shared contract has no explicit current-head/lineage operation; no new signature was invented here.
  • A legacy path-keyed Note needs a stable identity before history can start. This slice refuses that key and does not write a Note just because it was opened.
  • Pending batches are in memory until flush; a crash can lose that pending history window. A owns durable segment/torn-tail recovery.
  • No UI changed. UX gaps closed/left and production screenshots are not applicable to this backend slice; E owns the UI review.

Performance / for the merge round

No perf measurement was run in B. F owns the history profile and guards, and the integration job compares the complete event path with Phase 1. The Phase 1 Y4/segment reference is C1 31.86 MiB, N1 0.52 MiB; Canvas Restore/cold-open/undo p95 48.55/39.32/19.91 ms. Those are the earlier measurements, not measurements of this branch. Include admission's Index writes and configured Home quota checks in the integrated hot path; measuring append alone will omit them.

Deferred under the verification policy:

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test -- --test-threads=4 on the combined integration branch: verify all consumers and the server contract.
  • bun --cwd apps/web run check, bun --cwd apps/web run test --maxWorkers=2: verify D/E's web contracts and UI.
  • bun --cwd apps/web run test:e2e:notes: extend the runner with the integrated two-browser restore and per-author undo cases; verify convergence, kept foreign edits, and no client reload. Run E's Canvas runner too.
  • tests/adversarial/run.sh: run the combined real-server history/identity/isolation cases added by D/F.
  • F's history profile on the perf VM, under flock /root/perf.lock, with a prebuilt release binary and load recorded inside the lock: compare the complete path with Phase 1. F has not merged here, so its command cannot be named from this branch.

Gates (verbatim)

All cargo commands used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=4 and worktree target/tmp. The full crate suite ran once after the required merge: 93 tests passed. The final stable-ID guard then ran with the focused tests, per the verify-once rule. Doc comments were read again before this report.

cargo fmt --check: exit 0, no output.

cargo clippy -p calternal-collab --all-targets -- -D warnings (merged branch):

    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Checking typenum v1.20.1
   Compiling getrandom v0.4.3
    Checking rand_core v0.10.1
    Checking cmov v0.5.4
    Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-fs)
    Checking ctutils v0.4.2
    Checking calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-api)
    Checking uuid v1.26.1
    Checking generic-array v0.14.9
    Checking hybrid-array v0.4.15
    Checking webauthn-attestation-ca v0.5.5
    Checking calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-imap)
    Checking crypto-common v0.1.6
    Checking block-buffer v0.10.4
    Checking crypto-common v0.2.2
    Checking inout v0.2.2
    Checking digest v0.10.7
    Checking block-buffer v0.12.1
    Checking crypto-bigint v0.5.5
    Checking universal-hash v0.6.1
    Checking sha2 v0.10.9
    Checking sha1 v0.10.7
    Checking hmac v0.12.1
    Checking signature v2.2.0
    Checking sqlx-core v0.9.0
    Checking hkdf v0.12.4
    Checking tungstenite v0.29.0
    Checking cipher v0.5.2
    Checking sec1 v0.7.3
    Checking chacha20 v0.10.2
    Checking tokio-tungstenite v0.29.0
    Checking elliptic-curve v0.13.8
    Checking poly1305 v0.9.1
    Checking aead v0.6.1
    Checking axum v0.8.9
    Checking rfc6979 v0.4.0
    Checking chacha20poly1305 v0.11.0
    Checking ecdsa v0.16.9
    Checking primeorder v0.13.6
    Checking ed25519 v2.2.3
    Checking sqlx-sqlite v0.9.0
    Checking headers v0.4.2
    Checking curve25519-dalek v4.1.3
    Checking dav-server v0.11.0
    Checking ed25519-dalek v2.2.0
    Checking sqlx v0.9.0
    Checking p256 v0.13.2
    Checking calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-db)
    Checking p384 v0.13.1
    Checking rsa v0.9.10
    Checking oauth2 v5.0.0
    Checking blake2 v0.10.6
    Checking calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-plugin)
    Checking webauthn-rs-core v0.5.5
    Checking argon2 v0.5.3
    Checking calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-tags)
    Checking calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-dav)
    Checking openidconnect v4.0.1
    Checking webauthn-rs v0.5.5
    Checking calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-location)
    Checking tempfile v3.27.0
    Checking calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/notes)
    Checking calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-auth)
    Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/files)
    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 27s

cargo test -p calternal-collab -- --test-threads=4 (merged branch):

   Compiling typenum v1.20.1
   Compiling rand_core v0.10.1
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling digest v0.10.7
   Compiling cmov v0.5.4
   Compiling getrandom v0.4.3
   Compiling ctutils v0.4.2
   Compiling sha2 v0.10.9
   Compiling uuid v1.26.1
   Compiling generic-array v0.14.9
   Compiling hybrid-array v0.4.15
   Compiling sqlx-core v0.9.0
   Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-fs)
   Compiling crypto-common v0.2.2
   Compiling inout v0.2.2
   Compiling block-buffer v0.12.1
   Compiling universal-hash v0.6.1
   Compiling sha1 v0.10.7
   Compiling hmac v0.12.1
   Compiling sqlx-sqlite v0.9.0
   Compiling signature v2.2.0
   Compiling hkdf v0.12.4
   Compiling tungstenite v0.29.0
   Compiling cipher v0.5.2
   Compiling sec1 v0.7.3
   Compiling crypto-bigint v0.5.5
   Compiling tokio-tungstenite v0.29.0
   Compiling sqlx-macros-core v0.9.0
   Compiling chacha20 v0.10.2
   Compiling poly1305 v0.9.1
   Compiling elliptic-curve v0.13.8
   Compiling sqlx-macros v0.9.0
   Compiling aead v0.6.1
   Compiling axum v0.8.9
   Compiling rfc6979 v0.4.0
   Compiling chacha20poly1305 v0.11.0
   Compiling ecdsa v0.16.9
   Compiling primeorder v0.13.6
   Compiling curve25519-dalek v4.1.3
   Compiling headers v0.4.2
   Compiling sqlx v0.9.0
   Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-db)
   Compiling webauthn-attestation-ca v0.5.5
   Compiling calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-api)
   Compiling ed25519 v2.2.3
   Compiling dav-server v0.11.0
   Compiling ed25519-dalek v2.2.0
   Compiling rsa v0.9.10
   Compiling oauth2 v5.0.0
   Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-plugin)
   Compiling webauthn-rs-core v0.5.5
   Compiling p384 v0.13.1
   Compiling p256 v0.13.2
   Compiling blake2 v0.10.6
   Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-imap)
   Compiling openidconnect v4.0.1
   Compiling argon2 v0.5.3
   Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-tags)
   Compiling webauthn-rs v0.5.5
   Compiling calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-dav)
   Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-location)
   Compiling tempfile v3.27.0
   Compiling calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-auth)
   Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/notes)
   Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/files)
   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 42s
     Running unittests src/lib.rs (/home/kayg/build/targets/hist2-write/debug/deps/calternal_collab-be8e57ec5480e4ec)

running 44 tests
test block_merge_tests::conflict_keeps_both_versions_external_first ... ok
test block_merge_tests::issue_89_example ... ok
test block_merge_tests::derived_conflict_id_does_not_steal_an_existing_link ... ok
test block_merge_tests::live_only_change_is_kept ... ok
test block_merge_tests::same_change_on_both_sides_is_applied_once ... ok
test block_merge_tests::shared_change_inside_a_conflicting_chunk_is_applied_once ... ok
test block_merge_tests::changed_block_reaches_markdown_after_the_live_side_deletes_it ... ok
test block_merge_tests::anchored_conflict_copies_have_unique_replay_stable_ids ... ok
test history::write::tests::budget_survives_new_writer_and_backward_clock_and_groups_agent_turns ... ok
test history::write::tests::batches_keep_author_order_and_replay_notes_and_canvas ... ok
test history::write::tests::append_failure_retains_batch_and_reserved_capacity ... ok
test history::write::tests::client_ids_are_bound_to_connections_and_reconnect_author ... ok
test history::write::tests::client_author_binding_survives_room_unload_and_reopen ... ok
test history::write::tests::cancelled_index_admission_releases_home_quota_reservation ... ok
test history::write::tests::consecutive_author_batch_preserves_delete_sets ... ok
test markdown::source_patch_tests::changed_line_keeps_unedited_crlf_tabs_and_trailing_spaces ... ok
test markdown::source_patch_tests::source_patch_refuses_unaligned_line_counts ... ok
test repeats::tests::adjacent_duplicates_require_identical_content_and_identity ... ok
test repeats::tests::exact_repeats_are_counted ... ok
test repeats::tests::near_repeats_and_plain_notes_are_not ... ok
test session::client_stream_limit_tests::websocket_stream_limit_uses_resolved_ip_and_retry_after ... ok
test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok
test session::conflict_shadow_tests::edit_that_reaches_server_before_delete_survives_shadow_merge ... ok
test session::conflict_shadow_tests::external_writes_then_typing_do_not_replay_the_live_keystroke ... ok
test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok
test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok
test session::conflict_shadow_tests::local_undo_that_opens_conflict_shadow_does_not_replay_its_paste ... ok
test session::conflict_shadow_tests::paste_undo_redo_from_deleting_connection_is_not_a_conflict ... ok
test session::conflict_shadow_tests::second_client_edit_to_deleted_block_survives_shadow_merge ... ok
test session::conflict_shadow_tests::stale_edit_survives_after_same_connection_deleted_another_block ... ok
test session::empty_sync_update_does_not_mark_the_room_document_changed ... ok
test history::write::tests::presence_is_ephemeral_bounded_and_cannot_claim_another_author ... ok
test history::write::tests::quota_and_pending_limits_refuse_before_mutation ... ok
test session::external_persistence_race_tests::crash_before_save_keeps_epoch_and_replays_pending_edit_once ... ok
test session::live_history_write_tests::agent_turn_history_replays_seed_and_edit_and_quota_denies_next_turn ... ok
test session::external_persistence_race_tests::external_reconcile_waits_for_a_fresh_live_snapshot_to_persist ... ok
test session::live_history_write_tests::missing_dependencies_rollback_and_keep_server_client_identity ... ok
test session::live_history_write_tests::authenticated_socket_captures_applied_update_with_server_time ... ok
test session::public_edit_limit_tests::public_edit_limit_checks_the_markdown_form ... ok
test session::live_history_write_tests::external_reconcile_records_verified_actor ... ok
test session::shared_notes_deny_without_files_share_authority ... ok
test session::public_edit_limit_tests::oversized_public_update_is_rolled_back ... ok
test history::write::tests::background_flush_retries_without_another_edit ... ok
test session::public_edit_limit_tests::ten_thousand_markdown_blocks_open_and_sync_within_two_seconds ... ok

test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.84s

     Running tests/agent_turn_order.rs (/home/kayg/build/targets/hist2-write/debug/deps/agent_turn_order-671e4310f9c9624d)

running 1 test
test agent_turn_with_edits_in_several_places_keeps_order ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.79s

     Running tests/block_apply_property.rs (/home/kayg/build/targets/hist2-write/debug/deps/block_apply_property-2b81a8f4b5fb39d6)

running 2 tests
test shared_typing_beside_an_external_change_is_applied_once ... ok
test external_apply_yields_new_block_order_exactly ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.23s

     Running tests/cross_language.rs (/home/kayg/build/targets/hist2-write/debug/deps/cross_language-737379759b285c7c)

running 1 test
test yjs_updates_match_editor_vectors has been running for over 60 seconds
test yjs_updates_match_editor_vectors ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 90.51s

     Running tests/hostile_clients.rs (/home/kayg/build/targets/hist2-write/debug/deps/hostile_clients-0fefa55cf60e1af2)

running 11 tests
test awareness_clock_at_maximum_is_refused_and_room_unloads ... ok
test one_user_cannot_open_unbounded_sockets ... ok
test primitive_value_in_fragment_cannot_truncate_the_note ... ok
test continuous_typing_is_saved_within_the_maximum_wait ... ok
test oversized_message_is_refused ... ok
test reconnect_storm_does_not_cancel_pending_save ... ok
test trashed_note_room_unloads_instead_of_retrying_forever ... ok
test unauthenticated_websocket_routes_return_forbidden ... ok
test cursed_bodies_open_live ... ok
test wiki_embeds_open_live_and_save_unchanged ... ok
test unrepresentable_update_is_rejected_and_room_keeps_saving ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.36s

     Running tests/journal_race.rs (/home/kayg/build/targets/hist2-write/debug/deps/journal_race-f82017f29be604f2)

running 1 test
test journal_log_race_with_live_hub_keeps_all_changes_and_refuses_daily_rooms ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.42s

     Running tests/restart_epoch.rs (/home/kayg/build/targets/hist2-write/debug/deps/restart_epoch-4fd18dc9b8605371)

running 5 tests
test stale_epoch_is_told_the_new_epoch_and_closed ... ok
test tab_open_across_a_restart_does_not_duplicate_the_note ... ok
test graceful_restart_keeps_the_epoch_and_offline_edits ... ok
test out_of_band_change_starts_a_new_epoch ... ok
test unload_then_reconnect_continues_the_same_room ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.45s

     Running tests/shared_notes.rs (/home/kayg/build/targets/hist2-write/debug/deps/shared_notes-58186e7bf384a855)

running 1 test
test owner_editor_viewer_and_live_revoke ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.93s

     Running tests/two_clients.rs (/home/kayg/build/targets/hist2-write/debug/deps/two_clients-d83f6d9346f8e74e)

running 2 tests
test crash_before_debounce_reloads_only_complete_markdown ... ok
test concurrent_clients_and_external_writer_converge ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.41s

     Running tests/untouched_bytes.rs (/home/kayg/build/targets/hist2-write/debug/deps/untouched_bytes-7a7031e545613583)

running 5 tests
test clean_room_flush_keeps_noncanonical_note_bytes ... ok
test external_edit_to_an_open_room_keeps_its_bytes ... ok
test live_edit_preserves_unedited_source_lines ... ok
test untouched_save_keeps_every_byte ... ok
test random_untouched_notes_keep_every_byte ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 28.25s

     Running tests/vector_bridge.rs (/home/kayg/build/targets/hist2-write/debug/deps/vector_bridge-051c564a2e4d1043)

running 15 tests
test crash_before_debounce_uses_last_complete_markdown ... ok
test applying_the_same_external_edit_twice_changes_nothing ... ok
test concurrent_typing_in_unchanged_block_survives_external_edit ... ok
test external_edit_inserts_at_start_and_end ... ok
test external_edit_after_collaborator_insert_above_keeps_order ... ok
test external_edit_beside_concurrently_changed_block_keeps_order ... ok
test external_edit_keeps_independent_live_block ... ok
test external_edit_with_two_inserts_keeps_order ... ok
test external_edit_mixes_insert_delete_and_replace ... ok
test external_insertion_keeps_existing_block_identity ... ok
test external_edit_moves_blocks ... ok
test same_block_conflict_keeps_both_versions ... ok
test update_rebuilds_after_restart ... ok
test editor_schema_json_survives_yrs ... ok
test markdown_vectors_pass_through_yrs ... ok

test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

     Running tests/wiki_embeds.rs (/home/kayg/build/targets/hist2-write/debug/deps/wiki_embeds-57b661fd48bce0f9)

running 5 tests
test wiki_embed_stays_text_not_an_invented_node ... ok
test embed_in_a_table_cell_stays_one_cell ... ok
test image_lines_match_the_editor_reader ... ok
test wiki_embeds_round_trip_byte_for_byte ... ok
test cursed_bodies_open_and_keep_their_content ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 23.69s

   Doc-tests calternal_collab

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-collab --all-targets -- -D warnings (final guard):

    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.36s

cargo test -p calternal-collab --lib history -- --test-threads=4 (final guard):

   Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 11.25s
     Running unittests src/lib.rs (/home/kayg/build/targets/hist2-write/debug/deps/calternal_collab-be8e57ec5480e4ec)

running 17 tests
test history::write::tests::batches_keep_author_order_and_replay_notes_and_canvas ... ok
test history::write::tests::budget_survives_new_writer_and_backward_clock_and_groups_agent_turns ... ok
test history::write::tests::append_failure_retains_batch_and_reserved_capacity ... ok
test history::write::tests::client_ids_are_bound_to_connections_and_reconnect_author ... ok
test history::write::tests::consecutive_author_batch_preserves_delete_sets ... ok
test history::write::tests::cancelled_index_admission_releases_home_quota_reservation ... ok
test history::write::tests::client_author_binding_survives_room_unload_and_reopen ... ok
test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok
test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok
test history::write::tests::quota_and_pending_limits_refuse_before_mutation ... ok
test history::write::tests::presence_is_ephemeral_bounded_and_cannot_claim_another_author ... ok
test session::live_history_write_tests::authenticated_socket_captures_applied_update_with_server_time ... ok
test session::live_history_write_tests::agent_turn_history_replays_seed_and_edit_and_quota_denies_next_turn ... ok
test session::live_history_write_tests::missing_dependencies_rollback_and_keep_server_client_identity ... ok
test session::live_history_write_tests::external_reconcile_records_verified_actor ... ok
test session::live_history_write_tests::history_rooms_require_note_identity_instead_of_legacy_path_selector ... ok
test history::write::tests::background_flush_retries_without_another_edit ... ok

test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 28 filtered out; finished in 3.62s

Cleanup

cargo clean:

     Removed 9879 files, 5.4GiB total

Web build output was removed. Worktree status is clean. The issue remains open.

## Phase 2 slice B (write) complete Branch: `job/hist2-write`. Head: `6d327cbca21b88b7d8ebd9933b95af62157b6092`. Start/base: `d4e7188810a89fb0e8e6b162279917f7e23989f9`. Required merge: `origin/dev` at `cfee85c6b11537968aaa0685d1ed1c3ac68a8c3e`, merge commit `22dc32fbe75a689cac3994a62d3da8c1a79c6e10`. The one conflict kept both the Daily note selector check (#646) and first-load serialization. No push or deployment. No merge into dev or main. ### Built - Shared `HistoryStore` signatures from the approved plan. Slice A's store and `MemoryHistoryStore` implementation remain for integration. - Document-neutral `HistoryWriter` for Notes and future Canvas rooms. It merges only consecutive updates by the same author. Foreign updates end a batch; replay order stays correct. A one-second task drains the queue. File saves, last-client saves and shutdown also drain it. Failed appends stay queued and retry with bounded backoff. - Home quota reservations before mutation, cancellation-safe reservation cleanup, bounded pending bytes and durable daily collaborator counters in the Index. - Active connection ownership for Yjs client IDs, durable author ownership across unload/restart, server ID protection, and bounded cursor-only claims. Cursor-only claims do not create Security-state rows. Existing foreign structs can be echoed during normal sync; new clocks require ownership. - Hub capture before broadcast for authenticated Users, shared editors, verified guests, server-issued agent turn IDs and attributed external reconciliation. New server hook: `notify_external_change_as`. - A history seed before incremental edits; first-load serialization prevents competing seeds. History-enabled rooms reject legacy `path:` selectors because history requires a stable Note identity. - Focused regressions for author order, text/map replay, delete sets, append failures, background retry, quota, cancelled admission, durable budgets, backward clock movement, pooled agent budgets, ID ownership, presence, real WebSocket capture, agent replay and external actor capture. Unresolved Yjs dependencies roll back instead of taking effect under a later author. Existing test expectations were not changed. ### Files `crates/calternal-collab/src/history/mod.rs` `crates/calternal-collab/src/history/write.rs` `crates/calternal-collab/src/history/write_tests.rs` `crates/calternal-collab/src/history/0001_write_budget.sql` `crates/calternal-collab/src/session.rs` `crates/calternal-collab/src/lib.rs` `crates/calternal-collab/Cargo.toml` `Cargo.lock` ### Atomic commits - `2a4974013`: history admission, budgets, batching and ownership boundary. - `cb79d157b`: Hub capture and real collaboration-path regressions. - `4067e2bb4`: cancellation, retry and presence resource bounds. - `22dc32fbe`: required origin/dev merge. - `6d327cbca`: stable Note identity guard and its regression. ### Decisions The plan did not set numeric limits. Defaults are 64 MiB of charged input per owner/collaborator per UTC day and 64 MiB of pending quota allowance. `WriteLimits` can set other instance limits. All agent turns in one Home share the agent allowance, so a new turn ID cannot reset it. First client-ID writes include a 512-byte accounting allowance. Queued quota reserves twice the charged bytes plus 1 KiB for framing/expansion. Batching has a one-second maximum healthy wait; an existing file-save flush can drain sooner. The first update supplies the batch time. All times come from the server. Failed appends retry at up to 30-second intervals with one task. The migration uses its own `collab-history-write` namespace, version 1. It has no collision with Notes migration numbers; origin/dev had no history module at the required merge. IDs use the existing String representation. The shared contract left `FoldReport` fields and error type unspecified: this branch supplies provisional `removed_points`/`reclaimed_bytes` fields and an opaque error Result. A may reconcile those details. None of the six HistoryStore signatures changed. I verified `yrs` 0.28.0 and `async-trait` 0.1.92 with cargo registry commands. `async-trait` is MIT OR Apache-2.0. No licence change. ### Findings fixed The first replay regression produced `a` instead of `abcd`: `yrs::Update::merge` does not preserve the complete batch and its delete set. The writer now uses `Update::merge_updates`; typing and deletion replay tests pass. `PluginEvent` has no actor field. Server API callers can use the new verified-actor hook; legacy filesystem notices use the Home owner. No request payload supplies an Author. ### Known gaps / integration requirements - Install A with `Hub::with_history(store, limits)` before rooms open. The default Hub has no history store until integration. `MemoryHistoryStore` is a declaration placeholder here; the test recorder is test-only. It is not shipped as a substitute store. - Canvas rooms do not exist on this base. Their event path must call the same HistoryWriter admission/capture functions. No Canvas protocol or second writer was added. - A must keep segment bytes in the Home quota accounting and enforce quota for checkpoints/dictionaries as well as update rows. The approved append signature does not carry the writer's reservation ID; conservative reservations can make a near-full Home reject earlier. - The integration must replace full-copy Versions for history-enabled live documents. This slice keeps the existing Notes writer and guest Version callback; it does not change another crate's file-version policy. - Integration must reconcile the existing history stream when a Note's cached Yrs lineage is discarded after an out-of-room file replacement. B seeds only an empty stream. The current shared contract has no explicit current-head/lineage operation; no new signature was invented here. - A legacy path-keyed Note needs a stable identity before history can start. This slice refuses that key and does not write a Note just because it was opened. - Pending batches are in memory until flush; a crash can lose that pending history window. A owns durable segment/torn-tail recovery. - No UI changed. UX gaps closed/left and production screenshots are not applicable to this backend slice; E owns the UI review. ### Performance / for the merge round No perf measurement was run in B. F owns the history profile and guards, and the integration job compares the complete event path with Phase 1. The Phase 1 Y4/segment reference is C1 31.86 MiB, N1 0.52 MiB; Canvas Restore/cold-open/undo p95 48.55/39.32/19.91 ms. Those are the earlier measurements, not measurements of this branch. Include admission's Index writes and configured Home quota checks in the integrated hot path; measuring append alone will omit them. Deferred under the verification policy: - `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test -- --test-threads=4` on the combined integration branch: verify all consumers and the server contract. - `bun --cwd apps/web run check`, `bun --cwd apps/web run test --maxWorkers=2`: verify D/E's web contracts and UI. - `bun --cwd apps/web run test:e2e:notes`: extend the runner with the integrated two-browser restore and per-author undo cases; verify convergence, kept foreign edits, and no client reload. Run E's Canvas runner too. - `tests/adversarial/run.sh`: run the combined real-server history/identity/isolation cases added by D/F. - F's history profile on the perf VM, under `flock /root/perf.lock`, with a prebuilt release binary and load recorded inside the lock: compare the complete path with Phase 1. F has not merged here, so its command cannot be named from this branch. ### Gates (verbatim) All cargo commands used `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, `CARGO_INCREMENTAL=0`, `CARGO_BUILD_JOBS=4` and worktree `target/tmp`. The full crate suite ran once after the required merge: 93 tests passed. The final stable-ID guard then ran with the focused tests, per the verify-once rule. Doc comments were read again before this report. `cargo fmt --check`: exit 0, no output. `cargo clippy -p calternal-collab --all-targets -- -D warnings (merged branch)`: ```text Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Blocking waiting for file lock on package cache Checking typenum v1.20.1 Compiling getrandom v0.4.3 Checking rand_core v0.10.1 Checking cmov v0.5.4 Checking calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-fs) Checking ctutils v0.4.2 Checking calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-api) Checking uuid v1.26.1 Checking generic-array v0.14.9 Checking hybrid-array v0.4.15 Checking webauthn-attestation-ca v0.5.5 Checking calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-imap) Checking crypto-common v0.1.6 Checking block-buffer v0.10.4 Checking crypto-common v0.2.2 Checking inout v0.2.2 Checking digest v0.10.7 Checking block-buffer v0.12.1 Checking crypto-bigint v0.5.5 Checking universal-hash v0.6.1 Checking sha2 v0.10.9 Checking sha1 v0.10.7 Checking hmac v0.12.1 Checking signature v2.2.0 Checking sqlx-core v0.9.0 Checking hkdf v0.12.4 Checking tungstenite v0.29.0 Checking cipher v0.5.2 Checking sec1 v0.7.3 Checking chacha20 v0.10.2 Checking tokio-tungstenite v0.29.0 Checking elliptic-curve v0.13.8 Checking poly1305 v0.9.1 Checking aead v0.6.1 Checking axum v0.8.9 Checking rfc6979 v0.4.0 Checking chacha20poly1305 v0.11.0 Checking ecdsa v0.16.9 Checking primeorder v0.13.6 Checking ed25519 v2.2.3 Checking sqlx-sqlite v0.9.0 Checking headers v0.4.2 Checking curve25519-dalek v4.1.3 Checking dav-server v0.11.0 Checking ed25519-dalek v2.2.0 Checking sqlx v0.9.0 Checking p256 v0.13.2 Checking calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-db) Checking p384 v0.13.1 Checking rsa v0.9.10 Checking oauth2 v5.0.0 Checking blake2 v0.10.6 Checking calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-plugin) Checking webauthn-rs-core v0.5.5 Checking argon2 v0.5.3 Checking calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-tags) Checking calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-dav) Checking openidconnect v4.0.1 Checking webauthn-rs v0.5.5 Checking calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-location) Checking tempfile v3.27.0 Checking calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/notes) Checking calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-auth) Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/files) Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 27s ``` `cargo test -p calternal-collab -- --test-threads=4 (merged branch)`: ```text Compiling typenum v1.20.1 Compiling rand_core v0.10.1 Compiling crypto-common v0.1.6 Compiling block-buffer v0.10.4 Compiling digest v0.10.7 Compiling cmov v0.5.4 Compiling getrandom v0.4.3 Compiling ctutils v0.4.2 Compiling sha2 v0.10.9 Compiling uuid v1.26.1 Compiling generic-array v0.14.9 Compiling hybrid-array v0.4.15 Compiling sqlx-core v0.9.0 Compiling calternal-fs v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-fs) Compiling crypto-common v0.2.2 Compiling inout v0.2.2 Compiling block-buffer v0.12.1 Compiling universal-hash v0.6.1 Compiling sha1 v0.10.7 Compiling hmac v0.12.1 Compiling sqlx-sqlite v0.9.0 Compiling signature v2.2.0 Compiling hkdf v0.12.4 Compiling tungstenite v0.29.0 Compiling cipher v0.5.2 Compiling sec1 v0.7.3 Compiling crypto-bigint v0.5.5 Compiling tokio-tungstenite v0.29.0 Compiling sqlx-macros-core v0.9.0 Compiling chacha20 v0.10.2 Compiling poly1305 v0.9.1 Compiling elliptic-curve v0.13.8 Compiling sqlx-macros v0.9.0 Compiling aead v0.6.1 Compiling axum v0.8.9 Compiling rfc6979 v0.4.0 Compiling chacha20poly1305 v0.11.0 Compiling ecdsa v0.16.9 Compiling primeorder v0.13.6 Compiling curve25519-dalek v4.1.3 Compiling headers v0.4.2 Compiling sqlx v0.9.0 Compiling calternal-db v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-db) Compiling webauthn-attestation-ca v0.5.5 Compiling calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-api) Compiling ed25519 v2.2.3 Compiling dav-server v0.11.0 Compiling ed25519-dalek v2.2.0 Compiling rsa v0.9.10 Compiling oauth2 v5.0.0 Compiling calternal-plugin v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-plugin) Compiling webauthn-rs-core v0.5.5 Compiling p384 v0.13.1 Compiling p256 v0.13.2 Compiling blake2 v0.10.6 Compiling calternal-imap v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-imap) Compiling openidconnect v4.0.1 Compiling argon2 v0.5.3 Compiling calternal-tags v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-tags) Compiling webauthn-rs v0.5.5 Compiling calternal-dav v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-dav) Compiling calternal-location v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-location) Compiling tempfile v3.27.0 Compiling calternal-auth v0.1.0 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-auth) Compiling calternal-plugin-notes v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/notes) Compiling calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/plugins/files) Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 42s Running unittests src/lib.rs (/home/kayg/build/targets/hist2-write/debug/deps/calternal_collab-be8e57ec5480e4ec) running 44 tests test block_merge_tests::conflict_keeps_both_versions_external_first ... ok test block_merge_tests::issue_89_example ... ok test block_merge_tests::derived_conflict_id_does_not_steal_an_existing_link ... ok test block_merge_tests::live_only_change_is_kept ... ok test block_merge_tests::same_change_on_both_sides_is_applied_once ... ok test block_merge_tests::shared_change_inside_a_conflicting_chunk_is_applied_once ... ok test block_merge_tests::changed_block_reaches_markdown_after_the_live_side_deletes_it ... ok test block_merge_tests::anchored_conflict_copies_have_unique_replay_stable_ids ... ok test history::write::tests::budget_survives_new_writer_and_backward_clock_and_groups_agent_turns ... ok test history::write::tests::batches_keep_author_order_and_replay_notes_and_canvas ... ok test history::write::tests::append_failure_retains_batch_and_reserved_capacity ... ok test history::write::tests::client_ids_are_bound_to_connections_and_reconnect_author ... ok test history::write::tests::client_author_binding_survives_room_unload_and_reopen ... ok test history::write::tests::cancelled_index_admission_releases_home_quota_reservation ... ok test history::write::tests::consecutive_author_batch_preserves_delete_sets ... ok test markdown::source_patch_tests::changed_line_keeps_unedited_crlf_tabs_and_trailing_spaces ... ok test markdown::source_patch_tests::source_patch_refuses_unaligned_line_counts ... ok test repeats::tests::adjacent_duplicates_require_identical_content_and_identity ... ok test repeats::tests::exact_repeats_are_counted ... ok test repeats::tests::near_repeats_and_plain_notes_are_not ... ok test session::client_stream_limit_tests::websocket_stream_limit_uses_resolved_ip_and_retry_after ... ok test session::conflict_shadow_tests::edit_reaching_server_before_stale_peer_delete_survives ... ok test session::conflict_shadow_tests::edit_that_reaches_server_before_delete_survives_shadow_merge ... ok test session::conflict_shadow_tests::external_writes_then_typing_do_not_replay_the_live_keystroke ... ok test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok test session::conflict_shadow_tests::local_undo_that_opens_conflict_shadow_does_not_replay_its_paste ... ok test session::conflict_shadow_tests::paste_undo_redo_from_deleting_connection_is_not_a_conflict ... ok test session::conflict_shadow_tests::second_client_edit_to_deleted_block_survives_shadow_merge ... ok test session::conflict_shadow_tests::stale_edit_survives_after_same_connection_deleted_another_block ... ok test session::empty_sync_update_does_not_mark_the_room_document_changed ... ok test history::write::tests::presence_is_ephemeral_bounded_and_cannot_claim_another_author ... ok test history::write::tests::quota_and_pending_limits_refuse_before_mutation ... ok test session::external_persistence_race_tests::crash_before_save_keeps_epoch_and_replays_pending_edit_once ... ok test session::live_history_write_tests::agent_turn_history_replays_seed_and_edit_and_quota_denies_next_turn ... ok test session::external_persistence_race_tests::external_reconcile_waits_for_a_fresh_live_snapshot_to_persist ... ok test session::live_history_write_tests::missing_dependencies_rollback_and_keep_server_client_identity ... ok test session::live_history_write_tests::authenticated_socket_captures_applied_update_with_server_time ... ok test session::public_edit_limit_tests::public_edit_limit_checks_the_markdown_form ... ok test session::live_history_write_tests::external_reconcile_records_verified_actor ... ok test session::shared_notes_deny_without_files_share_authority ... ok test session::public_edit_limit_tests::oversized_public_update_is_rolled_back ... ok test history::write::tests::background_flush_retries_without_another_edit ... ok test session::public_edit_limit_tests::ten_thousand_markdown_blocks_open_and_sync_within_two_seconds ... ok test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.84s Running tests/agent_turn_order.rs (/home/kayg/build/targets/hist2-write/debug/deps/agent_turn_order-671e4310f9c9624d) running 1 test test agent_turn_with_edits_in_several_places_keeps_order ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.79s Running tests/block_apply_property.rs (/home/kayg/build/targets/hist2-write/debug/deps/block_apply_property-2b81a8f4b5fb39d6) running 2 tests test shared_typing_beside_an_external_change_is_applied_once ... ok test external_apply_yields_new_block_order_exactly ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.23s Running tests/cross_language.rs (/home/kayg/build/targets/hist2-write/debug/deps/cross_language-737379759b285c7c) running 1 test test yjs_updates_match_editor_vectors has been running for over 60 seconds test yjs_updates_match_editor_vectors ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 90.51s Running tests/hostile_clients.rs (/home/kayg/build/targets/hist2-write/debug/deps/hostile_clients-0fefa55cf60e1af2) running 11 tests test awareness_clock_at_maximum_is_refused_and_room_unloads ... ok test one_user_cannot_open_unbounded_sockets ... ok test primitive_value_in_fragment_cannot_truncate_the_note ... ok test continuous_typing_is_saved_within_the_maximum_wait ... ok test oversized_message_is_refused ... ok test reconnect_storm_does_not_cancel_pending_save ... ok test trashed_note_room_unloads_instead_of_retrying_forever ... ok test unauthenticated_websocket_routes_return_forbidden ... ok test cursed_bodies_open_live ... ok test wiki_embeds_open_live_and_save_unchanged ... ok test unrepresentable_update_is_rejected_and_room_keeps_saving ... ok test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.36s Running tests/journal_race.rs (/home/kayg/build/targets/hist2-write/debug/deps/journal_race-f82017f29be604f2) running 1 test test journal_log_race_with_live_hub_keeps_all_changes_and_refuses_daily_rooms ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.42s Running tests/restart_epoch.rs (/home/kayg/build/targets/hist2-write/debug/deps/restart_epoch-4fd18dc9b8605371) running 5 tests test stale_epoch_is_told_the_new_epoch_and_closed ... ok test tab_open_across_a_restart_does_not_duplicate_the_note ... ok test graceful_restart_keeps_the_epoch_and_offline_edits ... ok test out_of_band_change_starts_a_new_epoch ... ok test unload_then_reconnect_continues_the_same_room ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.45s Running tests/shared_notes.rs (/home/kayg/build/targets/hist2-write/debug/deps/shared_notes-58186e7bf384a855) running 1 test test owner_editor_viewer_and_live_revoke ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.93s Running tests/two_clients.rs (/home/kayg/build/targets/hist2-write/debug/deps/two_clients-d83f6d9346f8e74e) running 2 tests test crash_before_debounce_reloads_only_complete_markdown ... ok test concurrent_clients_and_external_writer_converge ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.41s Running tests/untouched_bytes.rs (/home/kayg/build/targets/hist2-write/debug/deps/untouched_bytes-7a7031e545613583) running 5 tests test clean_room_flush_keeps_noncanonical_note_bytes ... ok test external_edit_to_an_open_room_keeps_its_bytes ... ok test live_edit_preserves_unedited_source_lines ... ok test untouched_save_keeps_every_byte ... ok test random_untouched_notes_keep_every_byte ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 28.25s Running tests/vector_bridge.rs (/home/kayg/build/targets/hist2-write/debug/deps/vector_bridge-051c564a2e4d1043) running 15 tests test crash_before_debounce_uses_last_complete_markdown ... ok test applying_the_same_external_edit_twice_changes_nothing ... ok test concurrent_typing_in_unchanged_block_survives_external_edit ... ok test external_edit_inserts_at_start_and_end ... ok test external_edit_after_collaborator_insert_above_keeps_order ... ok test external_edit_beside_concurrently_changed_block_keeps_order ... ok test external_edit_keeps_independent_live_block ... ok test external_edit_with_two_inserts_keeps_order ... ok test external_edit_mixes_insert_delete_and_replace ... ok test external_insertion_keeps_existing_block_identity ... ok test external_edit_moves_blocks ... ok test same_block_conflict_keeps_both_versions ... ok test update_rebuilds_after_restart ... ok test editor_schema_json_survives_yrs ... ok test markdown_vectors_pass_through_yrs ... ok test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s Running tests/wiki_embeds.rs (/home/kayg/build/targets/hist2-write/debug/deps/wiki_embeds-57b661fd48bce0f9) running 5 tests test wiki_embed_stays_text_not_an_invented_node ... ok test embed_in_a_table_cell_stays_one_cell ... ok test image_lines_match_the_editor_reader ... ok test wiki_embeds_round_trip_byte_for_byte ... ok test cursed_bodies_open_and_keep_their_content ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 23.69s Doc-tests calternal_collab running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo clippy -p calternal-collab --all-targets -- -D warnings (final guard)`: ```text Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.36s ``` `cargo test -p calternal-collab --lib history -- --test-threads=4 (final guard)`: ```text Compiling calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-write/crates/calternal-collab) Finished `test` profile [unoptimized + debuginfo] target(s) in 11.25s Running unittests src/lib.rs (/home/kayg/build/targets/hist2-write/debug/deps/calternal_collab-be8e57ec5480e4ec) running 17 tests test history::write::tests::batches_keep_author_order_and_replay_notes_and_canvas ... ok test history::write::tests::budget_survives_new_writer_and_backward_clock_and_groups_agent_turns ... ok test history::write::tests::append_failure_retains_batch_and_reserved_capacity ... ok test history::write::tests::client_ids_are_bound_to_connections_and_reconnect_author ... ok test history::write::tests::consecutive_author_batch_preserves_delete_sets ... ok test history::write::tests::cancelled_index_admission_releases_home_quota_reservation ... ok test history::write::tests::client_author_binding_survives_room_unload_and_reopen ... ok test session::conflict_shadow_tests::history_delete_with_local_and_room_blocks_is_local ... ok test session::conflict_shadow_tests::local_history_marker_is_consumed_before_yrs_protocol_handling ... ok test history::write::tests::quota_and_pending_limits_refuse_before_mutation ... ok test history::write::tests::presence_is_ephemeral_bounded_and_cannot_claim_another_author ... ok test session::live_history_write_tests::authenticated_socket_captures_applied_update_with_server_time ... ok test session::live_history_write_tests::agent_turn_history_replays_seed_and_edit_and_quota_denies_next_turn ... ok test session::live_history_write_tests::missing_dependencies_rollback_and_keep_server_client_identity ... ok test session::live_history_write_tests::external_reconcile_records_verified_actor ... ok test session::live_history_write_tests::history_rooms_require_note_identity_instead_of_legacy_path_selector ... ok test history::write::tests::background_flush_retries_without_another_edit ... ok test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 28 filtered out; finished in 3.62s ``` ### Cleanup `cargo clean`: ```text Removed 9879 files, 5.4GiB total ``` Web build output was removed. Worktree status is clean. The issue remains open.
kayg commented 2026-10-03 16:37:57 +00:00
Author
Owner
Copy link

Phase 2 slice F complete

Branch: job/hist2-bench
Base merged: origin/dev at bd11bacb5189d39176e7cd48d5e977f1694321c1
Head: 9e3319bbf641c93145ff08ecfbbc7a8099dd9236

Built

  • Added the shared HistoryStore contract module and exposed it from calternal-collab. The contract uses stable User and item identities, monotonic PointIds, and the append/list/restore/undo/fold/quota signatures.
  • Ported the Phase 1 C1 Canvas and N1 Note Yrs traces into standalone bench/live-history. Added the owner-selected Y4-500 segment profile with decoded latest-checkpoint cache.
  • Added a strict CI byte guard and threshold docs. The CI workflow runs its one-sample local check.
  • Added a reusable test-only HistoryStore adversarial probe for valid oversized updates, bounded growth, extreme timestamps, page lengths and same-item cross-User access. Its fixture tests pass; the storage slice must call it with the concrete segment store's limits.

Files: .forgejo/workflows/ci.yml, root Cargo.lock, crates/calternal-collab/Cargo.toml, crates/calternal-collab/src/lib.rs, crates/calternal-collab/src/history/{mod.rs,adversarial.rs}, all files in bench/live-history/, docs/perf/live-history-budget.{json,md}, and docs/perf/baseline.json.

Perf profile

Used an optimized release binary built on the development host and five interleaved samples per workload on perf-test. Each sample acquired /root/perf.lock and recorded the 1-minute load under the lock (range 0.01–1.87).

Workload Open bytes / B per 1,000 edits Append CPU p50/p95 µs per edit Middle Restore p50/p95 ms Undo p50/p95 ms Peak RSS p50/p95 MiB
C1 Canvas, 11,100 edits 33,411,072 / 3,010,006 (budget 3,100,000) 14.875 / 15.502 26.569 / 40.954 17.073 / 28.680 156.219 / 156.621
N1 Note, 7,220 edits 548,864 / 76,020 (budget 80,000) 4.180 / 4.474 3.388 / 3.767 2.744 / 3.575 18.953 / 18.988

The guard passed. Against the Phase 1 open-size medians, Canvas is +3,440 B (+0.010%) and Note is +3,604 B (+0.661%). All five runs reported supported and correct undo. The largest C1 trace includes 5,000 starting elements, a 300-edit agent turn and 200-point strokes. Its maximum checkpoint-time p95 was 4,973 ms. DESIGN §61 sets no checkpoint-time threshold; this first timing/RSS profile is recorded in docs/perf/baseline.json for owner review.

Gate output (verbatim result lines)

cargo fmt --check exited 0 with no output.

$ cargo clippy -p calternal-collab --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 28s

$ cargo test -p calternal-collab -- --test-threads=1
test result: ok. 33 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.40s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.01s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.20s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 140.40s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.55s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.68s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.35s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.29s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.79s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 57.65s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.54s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

$ cargo clippy -p calternal-server --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 37.97s

$ cargo test -p calternal-server -- --test-threads=1
test result: ok. 162 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 61.52s

$ cargo clippy --manifest-path bench/live-history/Cargo.toml --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.13s

$ cargo test --manifest-path bench/live-history/Cargo.toml -- --test-threads=1
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.93s
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 26.36s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

$ python3 -B bench/live-history/test_budget.py
......
Ran 6 tests in 0.226s

OK

$ python3 -B bench/live-history/run_ci_budget.py
001 canvas-y4c-500-segment-r1 disk=33411072 restore=118.490ms undo=60.648ms rss=164.0MiB
002 note-y4c-500-segment-r1 disk=548864 restore=17.159ms undo=6.175ms rss=22.3MiB
canvas r1: 3010006 B/1000 edits (budget 3100000 B)
note r1: 76020 B/1000 edits (budget 80000 B)
live history byte budget passed

$ cargo clean
     Removed 1969 files, 1.1GiB total

The server clippy initially could not compile because apps/web/build was absent. bun install --frozen-lockfile plus bun run build supplied the real embedded frontend; the rerun passed. Build assets and node_modules were removed afterward. The build printed existing analytics-vendor use client directive warnings; no web source or lockfile changed.

Known gaps and merge round

  • The integrated branch still has no concrete history store. The reusable adversarial probe currently runs against its bounded test fixture. After Slice A adds the segment store, run cargo test -p calternal-collab with that store wired to the probe; this must prove its real size, growth, clock, paging and owner-scope rules.
  • This slice adds no API route. The full real-server adversarial round was not run under the per-branch policy. After the store and API slices integrate, run tests/adversarial/run.sh; extend it with the history endpoint cases so it proves cross-User isolation and that oversized/hostile history requests do not produce 5xx responses.
  • No UI changed, so no screenshots apply. No issues were closed.

Decisions not specified by DESIGN

  • Used uuid::Uuid for User IDs, opaque Strings for stable item and agent-turn IDs, u64 monotonic point IDs, and a boxed internal error result. The boxed error keeps storage backends out of the cross-slice contract; item identity is not a path.
  • Rounded the Phase 1 byte medians up to 3,100,000 B/1,000 Canvas edits and 80,000 B/1,000 Note edits. DESIGN §61 gives no rounding margin for the CI ceiling.
## Phase 2 slice F complete Branch: `job/hist2-bench` Base merged: `origin/dev` at `bd11bacb5189d39176e7cd48d5e977f1694321c1` Head: `9e3319bbf641c93145ff08ecfbbc7a8099dd9236` ### Built - Added the shared `HistoryStore` contract module and exposed it from `calternal-collab`. The contract uses stable User and item identities, monotonic `PointId`s, and the append/list/restore/undo/fold/quota signatures. - Ported the Phase 1 C1 Canvas and N1 Note Yrs traces into standalone `bench/live-history`. Added the owner-selected Y4-500 segment profile with decoded latest-checkpoint cache. - Added a strict CI byte guard and threshold docs. The CI workflow runs its one-sample local check. - Added a reusable test-only `HistoryStore` adversarial probe for valid oversized updates, bounded growth, extreme timestamps, page lengths and same-item cross-User access. Its fixture tests pass; the storage slice must call it with the concrete segment store's limits. Files: `.forgejo/workflows/ci.yml`, root `Cargo.lock`, `crates/calternal-collab/Cargo.toml`, `crates/calternal-collab/src/lib.rs`, `crates/calternal-collab/src/history/{mod.rs,adversarial.rs}`, all files in `bench/live-history/`, `docs/perf/live-history-budget.{json,md}`, and `docs/perf/baseline.json`. ### Perf profile Used an optimized release binary built on the development host and five interleaved samples per workload on `perf-test`. Each sample acquired `/root/perf.lock` and recorded the 1-minute load under the lock (range 0.01–1.87). | Workload | Open bytes / B per 1,000 edits | Append CPU p50/p95 µs per edit | Middle Restore p50/p95 ms | Undo p50/p95 ms | Peak RSS p50/p95 MiB | | --- | ---: | ---: | ---: | ---: | ---: | | C1 Canvas, 11,100 edits | 33,411,072 / 3,010,006 (budget 3,100,000) | 14.875 / 15.502 | 26.569 / 40.954 | 17.073 / 28.680 | 156.219 / 156.621 | | N1 Note, 7,220 edits | 548,864 / 76,020 (budget 80,000) | 4.180 / 4.474 | 3.388 / 3.767 | 2.744 / 3.575 | 18.953 / 18.988 | The guard passed. Against the Phase 1 open-size medians, Canvas is +3,440 B (+0.010%) and Note is +3,604 B (+0.661%). All five runs reported supported and correct undo. The largest C1 trace includes 5,000 starting elements, a 300-edit agent turn and 200-point strokes. Its maximum checkpoint-time p95 was 4,973 ms. DESIGN §61 sets no checkpoint-time threshold; this first timing/RSS profile is recorded in `docs/perf/baseline.json` for owner review. ### Gate output (verbatim result lines) `cargo fmt --check` exited 0 with no output. ```text $ cargo clippy -p calternal-collab --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 28s $ cargo test -p calternal-collab -- --test-threads=1 test result: ok. 33 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.40s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.01s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.20s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 140.40s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.55s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.68s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.35s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.29s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.79s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 57.65s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.54s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s $ cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 37.97s $ cargo test -p calternal-server -- --test-threads=1 test result: ok. 162 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 61.52s $ cargo clippy --manifest-path bench/live-history/Cargo.toml --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.13s $ cargo test --manifest-path bench/live-history/Cargo.toml -- --test-threads=1 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.93s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 26.36s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s $ python3 -B bench/live-history/test_budget.py ...... Ran 6 tests in 0.226s OK $ python3 -B bench/live-history/run_ci_budget.py 001 canvas-y4c-500-segment-r1 disk=33411072 restore=118.490ms undo=60.648ms rss=164.0MiB 002 note-y4c-500-segment-r1 disk=548864 restore=17.159ms undo=6.175ms rss=22.3MiB canvas r1: 3010006 B/1000 edits (budget 3100000 B) note r1: 76020 B/1000 edits (budget 80000 B) live history byte budget passed $ cargo clean Removed 1969 files, 1.1GiB total ``` The server clippy initially could not compile because `apps/web/build` was absent. `bun install --frozen-lockfile` plus `bun run build` supplied the real embedded frontend; the rerun passed. Build assets and `node_modules` were removed afterward. The build printed existing analytics-vendor `use client` directive warnings; no web source or lockfile changed. ### Known gaps and merge round - The integrated branch still has no concrete history store. The reusable adversarial probe currently runs against its bounded test fixture. After Slice A adds the segment store, run `cargo test -p calternal-collab` with that store wired to the probe; this must prove its real size, growth, clock, paging and owner-scope rules. - This slice adds no API route. The full real-server adversarial round was not run under the per-branch policy. After the store and API slices integrate, run `tests/adversarial/run.sh`; extend it with the history endpoint cases so it proves cross-User isolation and that oversized/hostile history requests do not produce 5xx responses. - No UI changed, so no screenshots apply. No issues were closed. ### Decisions not specified by DESIGN - Used `uuid::Uuid` for User IDs, opaque `String`s for stable item and agent-turn IDs, `u64` monotonic point IDs, and a boxed internal error result. The boxed error keeps storage backends out of the cross-slice contract; item identity is not a path. - Rounded the Phase 1 byte medians up to 3,100,000 B/1,000 Canvas edits and 80,000 B/1,000 Note edits. DESIGN §61 gives no rounding margin for the CI ceiling.
kayg commented 2026-10-03 16:41:40 +00:00
Author
Owner
Copy link

Finding after merge: cargo clippy -p calternal-server --all-targets -- -D warnings rejected three history route helpers because they returned Axum Response in a large Result error variant (history.rs: caller, doc_key, parse_point). I replaced the response-valued error with a small typed route error and retained the same API statuses and envelopes. I am rerunning the server gates now.

Finding after merge: `cargo clippy -p calternal-server --all-targets -- -D warnings` rejected three history route helpers because they returned Axum `Response` in a large `Result` error variant (`history.rs`: `caller`, `doc_key`, `parse_point`). I replaced the response-valued error with a small typed route error and retained the same API statuses and envelopes. I am rerunning the server gates now.
kayg commented 2026-10-03 16:54:37 +00:00
Author
Owner
Copy link

Phase 2 API slice report

Branch: job/hist2-api
Head: 1543e56074003f2fe184e9f189883763afb42952

Built

  • Added the shared live-history contracts in calternal-collab: stable DocKey/point identities, authors, store operations, errors, and the Restore/Undo mutation seam.
  • Added owner-only history list, binary Yjs v1 preview, Restore, Undo preflight, and Undo routes. The server derives owner and mutation actor from the authenticated request, validates stable IDs, and returns safe API errors.
  • Registered all five operations in OpenAPI and the shared CLI/MCP/WebMCP action registry. Generated the API client. Only these five actions receive the explicit discoverable tier; legacy actions remain without a tier.
  • Added focused route tests and explicit owner/public-link/cross-User policy coverage. Removed the broad collaboration prefix allowlist so unreviewed operations fail closed.

Files

  • crates/calternal-collab/Cargo.toml, crates/calternal-collab/src/history/mod.rs
  • crates/calternal-server/Cargo.toml, crates/calternal-server/src/history.rs, crates/calternal-server/src/history_contract_tests.rs, crates/calternal-server/src/main.rs, crates/calternal-server/src/wire.rs
  • crates/calternal-api/src/actions.rs, Cargo.lock
  • contracts/action-overrides.json, contracts/actions.json, contracts/openapi.json, packages/api-client/src/generated.ts
  • scripts/action_registry.py, scripts/test_action_registry.py, docs/parity-matrix.md
  • tests/adversarial/authz_matrix.py, tests/adversarial/test_xuser_classification.py, tests/adversarial/xuser_matrix.py

Gates

cargo fmt --check exited 0 with no output.

Commands and verbatim final success output:

cargo clippy -p calternal-collab --all-targets -- -D warnings
    Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/plugins/files)
    Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/calternal-collab)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 01s

cargo test -p calternal-collab
test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.99s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.50s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.01s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 42.56s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.72s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.94s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.87s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.50s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.92s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.92s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.93s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-api --all-targets -- -D warnings
    Checking calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/calternal-api)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.93s

cargo test -p calternal-api
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo clippy -p calternal-server --all-targets -- -D warnings
    Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/calternal-server)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.98s

cargo test -p calternal-server
test result: ok. 169 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 15.91s

cd packages/api-client && bun test
 18 pass
 0 fail
 49 expect() calls
Ran 18 tests across 1 file. [898.00ms]

Additional focused checks passed: action registry unit tests (14), XUser/admin classification tests (24), and python3 scripts/parity_matrix.py --check (345 API actions; 0 adapter gaps). The generated registry reported Action registry: 345 operations, 327 generated tools.

cd apps/web && bun run check was attempted but could not start because workspace dependencies are not installed. It exited with ERR_MODULE_NOT_FOUND: Cannot find package 'typescript' from apps/web/scripts/check-user-storage.mjs.

Known gaps and merge-round work

  • The production router still receives None for the history store and mutation service, so authorized history calls return 503 until the storage/mutation slices are wired. This is the planned integration seam; no fake history is returned.
  • The one time-boxed live adversarial round is deferred to the merge round by the verification policy. Run bash tests/adversarial/run.sh to exercise the XUser and authz matrices plus the live robustness probes against the integrated server.
  • The history benchmark profile belongs to slice F and is not part of this API slice. The UI and production screenshots belong to slice E.
  • UX gaps closed: N/A for this API-only slice. UX gaps left: UI loading/error/empty behavior is owned by slice E and needs review with the integrated services.

Decisions where the docs were silent

  • Kept the approved HistoryStore signatures and added a separate HistoryMutations interface for Restore and Undo. Undo apply revalidates the same author/range instead of trusting a stale preflight token.
  • The list response uses {points, next_after}; preview returns the Yjs v1 update as application/vnd.calternal.yjs-update-v1 bytes.
  • Kept action tier optional so this API slice does not silently reclassify existing actions; explicitly marked only its five history actions discoverable.

Cleanup: cargo clean output was Removed 16856 files, 14.0GiB total; apps/web/build was removed. No push, deploy, or issue close was performed.

## Phase 2 API slice report Branch: `job/hist2-api` Head: `1543e56074003f2fe184e9f189883763afb42952` ### Built - Added the shared live-history contracts in `calternal-collab`: stable `DocKey`/point identities, authors, store operations, errors, and the Restore/Undo mutation seam. - Added owner-only history list, binary Yjs v1 preview, Restore, Undo preflight, and Undo routes. The server derives owner and mutation actor from the authenticated request, validates stable IDs, and returns safe API errors. - Registered all five operations in OpenAPI and the shared CLI/MCP/WebMCP action registry. Generated the API client. Only these five actions receive the explicit `discoverable` tier; legacy actions remain without a tier. - Added focused route tests and explicit owner/public-link/cross-User policy coverage. Removed the broad collaboration prefix allowlist so unreviewed operations fail closed. ### Files - `crates/calternal-collab/Cargo.toml`, `crates/calternal-collab/src/history/mod.rs` - `crates/calternal-server/Cargo.toml`, `crates/calternal-server/src/history.rs`, `crates/calternal-server/src/history_contract_tests.rs`, `crates/calternal-server/src/main.rs`, `crates/calternal-server/src/wire.rs` - `crates/calternal-api/src/actions.rs`, `Cargo.lock` - `contracts/action-overrides.json`, `contracts/actions.json`, `contracts/openapi.json`, `packages/api-client/src/generated.ts` - `scripts/action_registry.py`, `scripts/test_action_registry.py`, `docs/parity-matrix.md` - `tests/adversarial/authz_matrix.py`, `tests/adversarial/test_xuser_classification.py`, `tests/adversarial/xuser_matrix.py` ### Gates `cargo fmt --check` exited 0 with no output. Commands and verbatim final success output: ```text cargo clippy -p calternal-collab --all-targets -- -D warnings Checking calternal-plugin-files v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/plugins/files) Checking calternal-collab v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/calternal-collab) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3m 01s cargo test -p calternal-collab test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.99s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.50s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.01s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 42.56s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.72s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.94s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.87s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.50s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.92s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.92s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.93s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-api --all-targets -- -D warnings Checking calternal-api v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/calternal-api) Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.93s cargo test -p calternal-api test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s cargo clippy -p calternal-server --all-targets -- -D warnings Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/hist2-api/crates/calternal-server) Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.98s cargo test -p calternal-server test result: ok. 169 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 15.91s cd packages/api-client && bun test 18 pass 0 fail 49 expect() calls Ran 18 tests across 1 file. [898.00ms] ``` Additional focused checks passed: action registry unit tests (14), XUser/admin classification tests (24), and `python3 scripts/parity_matrix.py --check` (345 API actions; 0 adapter gaps). The generated registry reported `Action registry: 345 operations, 327 generated tools`. `cd apps/web && bun run check` was attempted but could not start because workspace dependencies are not installed. It exited with `ERR_MODULE_NOT_FOUND: Cannot find package 'typescript'` from `apps/web/scripts/check-user-storage.mjs`. ### Known gaps and merge-round work - The production router still receives `None` for the history store and mutation service, so authorized history calls return 503 until the storage/mutation slices are wired. This is the planned integration seam; no fake history is returned. - The one time-boxed live adversarial round is deferred to the merge round by the verification policy. Run `bash tests/adversarial/run.sh` to exercise the XUser and authz matrices plus the live robustness probes against the integrated server. - The history benchmark profile belongs to slice F and is not part of this API slice. The UI and production screenshots belong to slice E. - UX gaps closed: N/A for this API-only slice. UX gaps left: UI loading/error/empty behavior is owned by slice E and needs review with the integrated services. ### Decisions where the docs were silent - Kept the approved `HistoryStore` signatures and added a separate `HistoryMutations` interface for Restore and Undo. Undo apply revalidates the same author/range instead of trusting a stale preflight token. - The list response uses `{points, next_after}`; preview returns the Yjs v1 update as `application/vnd.calternal.yjs-update-v1` bytes. - Kept action tier optional so this API slice does not silently reclassify existing actions; explicitly marked only its five history actions discoverable. Cleanup: `cargo clean` output was `Removed 16856 files, 14.0GiB total`; `apps/web/build` was removed. No push, deploy, or issue close was performed.
kayg commented 2026-10-03 16:58:13 +00:00
Author
Owner
Copy link

Starting Phase 2 integration on job/hist2-integrate, base bd11bacb51. I have read the approved plan, all slice reports and the six visual notes. I will merge A→B→C→D→E→F, reconcile the store and mutation seams, and report real verification evidence. Under the current verification policy, full web/e2e/adversarial suites and benchmark runs belong to the merge round; focused regressions and production screenshots run here.

Starting Phase 2 integration on job/hist2-integrate, base bd11bacb5189d39176e7cd48d5e977f1694321c1. I have read the approved plan, all slice reports and the six visual notes. I will merge A→B→C→D→E→F, reconcile the store and mutation seams, and report real verification evidence. Under the current verification policy, full web/e2e/adversarial suites and benchmark runs belong to the merge round; focused regressions and production screenshots run here.
kayg commented 2026-10-03 17:20:42 +00:00
Author
Owner
Copy link

Integration finding: the independent review found crash lineage mismatch, self-counted quota reservations, unchecked fold staging, first-session Undo at point zero, and accepted replay-only attributes. Fixes and focused regressions are being added. A further integration check found that reading missing history created directories; reads now check for existing segments before opening them. Production API wiring and the real UI adapter are implemented. Dependency compilation is still running on the shared host; logs remain active in artifacts/.

Integration finding: the independent review found crash lineage mismatch, self-counted quota reservations, unchecked fold staging, first-session Undo at point zero, and accepted replay-only attributes. Fixes and focused regressions are being added. A further integration check found that reading missing history created directories; reads now check for existing segments before opening them. Production API wiring and the real UI adapter are implemented. Dependency compilation is still running on the shared host; logs remain active in artifacts/.
kayg commented 2026-10-03 17:48:51 +00:00
Author
Owner
Copy link

Integration finding (#975): the follow-up review found that saving snapshot B while later edit C becomes durable can leave the saved hash at A after a crash. Recovery would mistake B for an external edit and reverse C. The fix writes an old:new materialization intent before the checked file replacement. Recovery accepts either known internal hash. The regression now saves an intermediate snapshot with a durable later edit and simulates loss of the process cache. The earlier focused history suite passed 36 tests and Restore suite passed 15; the new interleaving test is running. Independent review also confirmed that reservation accounting, fold space checks, first-session undo, reserved attributes, lineage recovery and idle room release are addressed.

Integration finding (#975): the follow-up review found that saving snapshot B while later edit C becomes durable can leave the saved hash at A after a crash. Recovery would mistake B for an external edit and reverse C. The fix writes an old:new materialization intent before the checked file replacement. Recovery accepts either known internal hash. The regression now saves an intermediate snapshot with a durable later edit and simulates loss of the process cache. The earlier focused history suite passed 36 tests and Restore suite passed 15; the new interleaving test is running. Independent review also confirmed that reservation accounting, fold space checks, first-session undo, reserved attributes, lineage recovery and idle room release are addressed.
kayg commented 2026-10-03 18:30:54 +00:00
Author
Owner
Copy link

#975 Phase 2 integration report

Head: c18c6ddfb5718cd236cccc9f1a8964de42daebc5. Branch: job/hist2-integrate. Base: bd11bacb5189d39176e7cd48d5e977f1694321c1.

Merged store → write → restore → API → UI → bench in the requested order. Fetched and merged origin/dev once before final gates; it was already up to date. No push, deployment, or merge into dev/main was done.

What was built

  • One typed history contract and one production SegmentHistoryStore. MemoryHistoryStore is available only for tests/test-hooks.
  • Notes live edits, agent turns, external edits, Restore and selective Undo use the shared writer. Durable history precedes checked Markdown saves. These saves omit duplicate file Versions.
  • Server bootstrap mounts the real store and mutation adapter. Reverse pagination keeps old ascending API behavior. Restore returns its actual new point so Undo can undo that operation while keeping later edits.
  • The panel reads real metadata and bounded binary Yjs previews. It uses the shared editor serializer, real author names, preflight counts, and stable point links.
  • Applied visual notes 1–5: quiet Copy link controls, caption role times, compact session ranges, the actual history-button anchor with Escape/outside close, and no redundant subtitle. Canvas core is absent from dev. The shared client and renderPreview props form its host seam; its mount and route adapter remain pending.

Independent review

The independent read-only reviewer checked author binding, quota, confined paths, crash recovery and Restore correctness. All seven reported blockers were fixed and confirmed in source:

  1. Append consumed its own Home quota reservations twice. Append now transfers only its batch's reservations under the filesystem operation lock and restores them on failure.
  2. Fold staging lacked quota/free-space checks and stale-stage cleanup. Both checks and confined cleanup now run under the item writer lock.
  3. First-session Undo requested public point zero. It now uses the internal empty v1 baseline and still verifies point one.
  4. Restart created a new Yjs lineage against old history. Rooms now resume the durable lineage.
  5. Client-supplied replay attributes could disable Restore. The bridge refuses them before broadcast.
  6. History ahead of Markdown could be reversed during recovery. Notes now publish a durable old:new file-hash intent before replacement. Recovery accepts either hash and keeps later durable edits. The regression covers file snapshot B with later durable edit C and a lost process cache.
  7. History-only requests could retain idle rooms. A request lease releases and drains them on success, error or cancellation.

Quota refusals also return a safe conflict rather than a 5xx. Module and function comments were reread and updated. Existing test expectations were preserved.

UX gaps closed

  • Signed-in User names now come from the Note's loaded identity; People lists omit self.
  • A point link now resolves the history button after header render instead of using the whole header row.
  • Binary previews replace the provisional JSON preview. Generated wire author tags and Undo ranges are mapped correctly.
  • No-op Restore produces no invented mutation point or Undo action.
  • Restore Undo uses its real forward-edit point and selective Undo.
  • Notes use the real text-edit gesture in the screenshot regression. Phone sheets use the shared sheet title.

Evidence run in this job

The real-server screenshot regression created one Note through the API, typed two edits, waited for three durable points, opened a point URL, decoded its preview, and closed the panel with Escape. It verified the signed-in author name and desktop action-button anchor. Six macOS-emulated production captures passed. Product layout uses CSS; geometry assertions exist only in the test.

The Rust tests covered authenticated socket capture, Restore broadcast, first-session Undo, persistent lineage, torn-tail recovery, quota, cross-User ownership, Share denial, idle-room release, and the concurrent save-intent crash case. The API tests confirmed that mutation actors come from authentication.

This job followed the 2026-10-02 verification policy. Full browser matrices, real SIGKILL-mid-append, full web tests, release/staging checks and a new perf run were not run here. The new script's default peer scenario currently uses two contexts of the same User; it does not prove the required two-User Collaborate flow.

Screenshots

Width Light Dark
390 px Light Dark
820 px Light Dark
1440 px Light Dark

Gates — verbatim output

All final required per-job gates passed. Full crate tests ran once; focused checks reran after the fixes above. No full workspace gate ran.

cargo fmt --check exited 0 with no output.

cargo clippy -p calternal-fs --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 12s

cargo clippy -p calternal-collab --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 29s

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 23.36s

cargo clippy -p calternal-server --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 12s

cargo clippy -p calternal-api --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 8m 51s

cargo test -p calternal-fs

test result: ok. 63 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 35.20s
test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.08s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-collab

test result: ok. 64 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 74.26s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.20s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.45s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 145.52s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.30s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.29s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.93s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.27s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.15s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.87s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 55.06s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.13s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.78s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-plugin-notes

test result: ok. 186 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 586.56s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.18s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-server

test result: ok. 170 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 56.02s

cargo test -p calternal-api

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-fs history::tests

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 59 filtered out; finished in 1.05s

cargo test -p calternal-collab durable_history_ahead

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 63 filtered out; finished in 4.13s

cargo test -p calternal-plugin-notes collab

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 181 filtered out; finished in 2.15s

bun run check

$ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
User browser caches use userStorage; only documented device/public-link exceptions remain.
Text sizes and UI shape values use shared role tokens.
UI transitions and animation options use shared motion tokens or documented exceptions.
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/hist2-integrate/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bunx vitest run src/lib/history/api.test.ts src/lib/history/VersionHistory.svelte.test.ts src/lib/history/sessions.test.ts --maxWorkers=2

 Test Files  3 passed (3)
      Tests  10 passed (10)
   Duration  31.98s (transform 78%, import 11%, tests 6%, environment 4%, setup 1%)

bun run build

✓ built in 39.42s
✓ built in 51ms
✓ built in 1m 19s
  Wrote site to "build"

cargo build -p calternal-server (OPENSSL_NO_VENDOR=1, final production assets)

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 30s

node apps/web/e2e/history-975.mjs --screenshots-only

History regression and macOS screenshots passed (390/820/1440, light/dark).

python3 -B tests/adversarial/test_xuser_classification.py

..........
----------------------------------------------------------------------
Ran 10 tests in 0.258s

OK

Bench comparison

These are the prior locked slice F measurements recorded on #975. An integrated rerun remains pending under the verification policy. They measure the F candidate harness, not the complete admission/Index/file-save event path.

Metric Phase 1 Y4/500 segment Prior slice F Delta
Combined median open footprint 16.19000 MiB 16.19336 MiB +0.00336 MiB (+0.021%)
C1 middle Restore p95 48.550 ms 40.954 ms -7.596 ms (-15.65%)
N1 middle Restore p95 6.860 ms 3.767 ms -3.093 ms (-45.09%)

F's disk guards passed: C1 3,010,006 B/1,000 edits against 3,100,000; N1 76,020 against 80,000. These are prior results, not new measurements of this head.

Decisions

  • Keep A's String identities and typed error contract. Add optional lifecycle/materialization methods with fixture defaults; the production segment store implements them.
  • Use 32 decoded history cache entries in server bootstrap and idle checkpoints/eviction. No automatic destructive retention cutoff was selected; fold remains explicit.
  • Persist old:new materialization intent in a bounded confined marker. Notes computes the exact final file hash, including frontmatter and wikilinks, before replacement.
  • Fold space admission uses a conservative old-segment plus 66 MiB staging bound. This can refuse a fold near quota even when a smaller fold might fit.
  • Add optional reverse pagination and Restore mutation metadata. Keep the existing ascending cursor and exact fixture responses.
  • Bound metadata replies at 2 MiB and binary previews at the Restore engine's 16 MiB state limit.

Known gaps / UX gaps left

  • Canvas core is not on dev. Its host, route adapter and screenshots are pending.
  • The two-User Collaborate browser flow, live update in both clients after Restore, other-author Undo with a checked preflight count, actual Copy link clipboard round trip, third-User/Share-viewer real-server isolation and SIGKILL-mid-append proof remain for the merge round. Current API and Rust coverage is described above.
  • Automatic retention scheduling and permanent-item history cleanup are not wired.
  • The integrated benchmark rerun and the owner's visual review remain pending.

For the merge round

  • cd apps/web && bun run test --maxWorkers=2: run the full web suite.
  • CALTERNAL_SERVER_BIN=<combined-build> node apps/web/e2e/history-975.mjs: run its two-context Restore/Undo smoke. Extend it to two authenticated Users through Collaborate, add the third User and Share viewer, verify both editors update without reload, assert preflight counts, and exercise Copy link through the clipboard.
  • bash tests/adversarial/run.sh: run the generated XUser/authz/robustness matrices on the combined server. Add the focused history append kill/restart phase. Verify prior points survive and only a torn tail is removed.
  • cargo test -p calternal-collab torn_tail_recovery -- --test-threads=4: retain the focused segment crash-tail proof.
  • Build the benchmark on the build host as in bench/live-history/README.md. Run python3 -B bench/live-history/run-matrix.py --phase2-y4 --binary /mnt/hdd/bench/live-history/calternal-live-history-bench --traces /mnt/hdd/bench/live-history/traces --results /mnt/hdd/bench/live-history/results.jsonl --loads /mnt/hdd/bench/live-history/load.tsv --runs /mnt/hdd/bench/live-history/runs, then python3 -B bench/live-history/check_budget.py --results /mnt/hdd/bench/live-history/results.jsonl. The runner holds /root/perf.lock and records load. Include the real admission/Index/save path in an integrated profile before claiming end-to-end numbers.
  • Run combined release, deployment and o2 checks under the merge-round policy. Review the six attached captures.

READY FOR MERGE ROUND: yes. This is readiness for the combined verification round; full Phase 2 acceptance still needs the checks listed above.

Files

Includes the six merged slices and integration fixes:

.forgejo/workflows/ci.yml
Cargo.lock
Cargo.toml
apps/web/e2e/history-975.mjs
apps/web/src/lib/history/VersionHistory.svelte
apps/web/src/lib/history/VersionHistory.svelte.test.ts
apps/web/src/lib/history/api.test.ts
apps/web/src/lib/history/api.ts
apps/web/src/lib/history/sessions.test.ts
apps/web/src/lib/history/sessions.ts
apps/web/src/lib/notes/NoteView.svelte
bench/live-history/Cargo.lock
bench/live-history/Cargo.toml
bench/live-history/README.md
bench/live-history/check_budget.py
bench/live-history/run-matrix.py
bench/live-history/run_ci_budget.py
bench/live-history/src/lib.rs
bench/live-history/src/main.rs
bench/live-history/test_budget.py
contracts/action-overrides.json
contracts/actions.json
contracts/openapi.json
crates/calternal-api/src/actions.rs
crates/calternal-collab/Cargo.toml
crates/calternal-collab/src/history/0001_write_budget.sql
crates/calternal-collab/src/history/adversarial.rs
crates/calternal-collab/src/history/conformance.rs
crates/calternal-collab/src/history/mod.rs
crates/calternal-collab/src/history/restore.rs
crates/calternal-collab/src/history/store.rs
crates/calternal-collab/src/history/write.rs
crates/calternal-collab/src/history/write_tests.rs
crates/calternal-collab/src/lib.rs
crates/calternal-collab/src/session.rs
crates/calternal-collab/tests/history_restore.rs
crates/calternal-fs/src/history.rs
crates/calternal-fs/src/lib.rs
crates/calternal-fs/src/root.rs
crates/calternal-fs/src/write.rs
crates/calternal-server/Cargo.toml
crates/calternal-server/src/history.rs
crates/calternal-server/src/history_contract_tests.rs
crates/calternal-server/src/main.rs
crates/calternal-server/src/wire.rs
crates/plugins/notes/src/lib.rs
crates/plugins/notes/src/store.rs
docs/parity-matrix.md
docs/perf/baseline.json
docs/perf/live-history-budget.json
docs/perf/live-history-budget.md
packages/api-client/src/generated.ts
scripts/action_registry.py
scripts/test_action_registry.py
tests/adversarial/authz_matrix.py
tests/adversarial/test_xuser_classification.py
tests/adversarial/xuser_matrix.py

Cleanup

Removed 18870 files, 12.3GiB total

Web build output was removed. Screenshots and gate logs remain under ignored artifacts/.

# #975 Phase 2 integration report Head: `c18c6ddfb5718cd236cccc9f1a8964de42daebc5`. Branch: `job/hist2-integrate`. Base: `bd11bacb5189d39176e7cd48d5e977f1694321c1`. Merged store → write → restore → API → UI → bench in the requested order. Fetched and merged `origin/dev` once before final gates; it was already up to date. No push, deployment, or merge into dev/main was done. ## What was built - One typed history contract and one production SegmentHistoryStore. MemoryHistoryStore is available only for tests/test-hooks. - Notes live edits, agent turns, external edits, Restore and selective Undo use the shared writer. Durable history precedes checked Markdown saves. These saves omit duplicate file Versions. - Server bootstrap mounts the real store and mutation adapter. Reverse pagination keeps old ascending API behavior. Restore returns its actual new point so Undo can undo that operation while keeping later edits. - The panel reads real metadata and bounded binary Yjs previews. It uses the shared editor serializer, real author names, preflight counts, and stable point links. - Applied visual notes 1–5: quiet Copy link controls, caption role times, compact session ranges, the actual history-button anchor with Escape/outside close, and no redundant subtitle. Canvas core is absent from dev. The shared client and renderPreview props form its host seam; its mount and route adapter remain pending. ## Independent review The independent read-only reviewer checked author binding, quota, confined paths, crash recovery and Restore correctness. All seven reported blockers were fixed and confirmed in source: 1. Append consumed its own Home quota reservations twice. Append now transfers only its batch's reservations under the filesystem operation lock and restores them on failure. 2. Fold staging lacked quota/free-space checks and stale-stage cleanup. Both checks and confined cleanup now run under the item writer lock. 3. First-session Undo requested public point zero. It now uses the internal empty v1 baseline and still verifies point one. 4. Restart created a new Yjs lineage against old history. Rooms now resume the durable lineage. 5. Client-supplied replay attributes could disable Restore. The bridge refuses them before broadcast. 6. History ahead of Markdown could be reversed during recovery. Notes now publish a durable old:new file-hash intent before replacement. Recovery accepts either hash and keeps later durable edits. The regression covers file snapshot B with later durable edit C and a lost process cache. 7. History-only requests could retain idle rooms. A request lease releases and drains them on success, error or cancellation. Quota refusals also return a safe conflict rather than a 5xx. Module and function comments were reread and updated. Existing test expectations were preserved. ## UX gaps closed - Signed-in User names now come from the Note's loaded identity; People lists omit self. - A point link now resolves the history button after header render instead of using the whole header row. - Binary previews replace the provisional JSON preview. Generated wire author tags and Undo ranges are mapped correctly. - No-op Restore produces no invented mutation point or Undo action. - Restore Undo uses its real forward-edit point and selective Undo. - Notes use the real text-edit gesture in the screenshot regression. Phone sheets use the shared sheet title. ## Evidence run in this job The real-server screenshot regression created one Note through the API, typed two edits, waited for three durable points, opened a point URL, decoded its preview, and closed the panel with Escape. It verified the signed-in author name and desktop action-button anchor. Six macOS-emulated production captures passed. Product layout uses CSS; geometry assertions exist only in the test. The Rust tests covered authenticated socket capture, Restore broadcast, first-session Undo, persistent lineage, torn-tail recovery, quota, cross-User ownership, Share denial, idle-room release, and the concurrent save-intent crash case. The API tests confirmed that mutation actors come from authentication. This job followed the 2026-10-02 verification policy. Full browser matrices, real SIGKILL-mid-append, full web tests, release/staging checks and a new perf run were not run here. The new script's default peer scenario currently uses two contexts of the same User; it does not prove the required two-User Collaborate flow. ## Screenshots | Width | Light | Dark | | --- | --- | --- | | 390 px | [Light](https://git.kayg.org/attachments/e36d95c2-6b7c-4ced-9ab8-10a89d72dc69) | [Dark](https://git.kayg.org/attachments/348ab846-5321-4d31-8ef7-ddfbe2d0ebcb) | | 820 px | [Light](https://git.kayg.org/attachments/c365e851-e59f-4c87-aaa1-31bb070d225d) | [Dark](https://git.kayg.org/attachments/a5e5fb9e-fbc1-4622-9a56-2d0692cb005f) | | 1440 px | [Light](https://git.kayg.org/attachments/eae2c15e-a5c2-4189-9edb-a214fcfd3c42) | [Dark](https://git.kayg.org/attachments/e9f12f10-170c-4458-82e5-e33f7accf40d) | ## Gates — verbatim output All final required per-job gates passed. Full crate tests ran once; focused checks reran after the fixes above. No full workspace gate ran. `cargo fmt --check` exited 0 with no output. `cargo clippy -p calternal-fs --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 12s ``` `cargo clippy -p calternal-collab --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 29s ``` `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 23.36s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 12s ``` `cargo clippy -p calternal-api --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 8m 51s ``` `cargo test -p calternal-fs` ```text test result: ok. 63 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 35.20s test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.08s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-collab` ```text test result: ok. 64 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 74.26s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.20s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.45s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 145.52s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.30s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.29s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.93s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.27s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.15s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.87s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 55.06s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.13s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.78s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-plugin-notes` ```text test result: ok. 186 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 586.56s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.18s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-server` ```text test result: ok. 170 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 56.02s ``` `cargo test -p calternal-api` ```text test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-fs history::tests` ```text test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 59 filtered out; finished in 1.05s ``` `cargo test -p calternal-collab durable_history_ahead` ```text test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 63 filtered out; finished in 4.13s ``` `cargo test -p calternal-plugin-notes collab` ```text test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 181 filtered out; finished in 2.15s ``` `bun run check` ```text $ node scripts/check-user-storage.mjs && node scripts/check-type-tokens.mjs && node scripts/check-motion-tokens.mjs && svelte-kit sync && svelte-check --tsconfig ./tsconfig.json User browser caches use userStorage; only documented device/public-link exceptions remain. Text sizes and UI shape values use shared role tokens. UI transitions and animation options use shared motion tokens or documented exceptions. Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/hist2-integrate/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bunx vitest run src/lib/history/api.test.ts src/lib/history/VersionHistory.svelte.test.ts src/lib/history/sessions.test.ts --maxWorkers=2` ```text Test Files 3 passed (3) Tests 10 passed (10) Duration 31.98s (transform 78%, import 11%, tests 6%, environment 4%, setup 1%) ``` `bun run build` ```text ✓ built in 39.42s ✓ built in 51ms ✓ built in 1m 19s Wrote site to "build" ``` `cargo build -p calternal-server` (OPENSSL_NO_VENDOR=1, final production assets) ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 30s ``` `node apps/web/e2e/history-975.mjs --screenshots-only` ```text History regression and macOS screenshots passed (390/820/1440, light/dark). ``` `python3 -B tests/adversarial/test_xuser_classification.py` ```text .......... ---------------------------------------------------------------------- Ran 10 tests in 0.258s OK ``` ## Bench comparison These are the prior locked slice F measurements recorded on #975. An integrated rerun remains pending under the verification policy. They measure the F candidate harness, not the complete admission/Index/file-save event path. | Metric | Phase 1 Y4/500 segment | Prior slice F | Delta | | --- | ---: | ---: | ---: | | Combined median open footprint | 16.19000 MiB | 16.19336 MiB | +0.00336 MiB (+0.021%) | | C1 middle Restore p95 | 48.550 ms | 40.954 ms | -7.596 ms (-15.65%) | | N1 middle Restore p95 | 6.860 ms | 3.767 ms | -3.093 ms (-45.09%) | F's disk guards passed: C1 3,010,006 B/1,000 edits against 3,100,000; N1 76,020 against 80,000. These are prior results, not new measurements of this head. ## Decisions - Keep A's String identities and typed error contract. Add optional lifecycle/materialization methods with fixture defaults; the production segment store implements them. - Use 32 decoded history cache entries in server bootstrap and idle checkpoints/eviction. No automatic destructive retention cutoff was selected; fold remains explicit. - Persist old:new materialization intent in a bounded confined marker. Notes computes the exact final file hash, including frontmatter and wikilinks, before replacement. - Fold space admission uses a conservative old-segment plus 66 MiB staging bound. This can refuse a fold near quota even when a smaller fold might fit. - Add optional reverse pagination and Restore mutation metadata. Keep the existing ascending cursor and exact fixture responses. - Bound metadata replies at 2 MiB and binary previews at the Restore engine's 16 MiB state limit. ## Known gaps / UX gaps left - Canvas core is not on dev. Its host, route adapter and screenshots are pending. - The two-User Collaborate browser flow, live update in both clients after Restore, other-author Undo with a checked preflight count, actual Copy link clipboard round trip, third-User/Share-viewer real-server isolation and SIGKILL-mid-append proof remain for the merge round. Current API and Rust coverage is described above. - Automatic retention scheduling and permanent-item history cleanup are not wired. - The integrated benchmark rerun and the owner's visual review remain pending. ## For the merge round - `cd apps/web && bun run test --maxWorkers=2`: run the full web suite. - `CALTERNAL_SERVER_BIN=<combined-build> node apps/web/e2e/history-975.mjs`: run its two-context Restore/Undo smoke. Extend it to two authenticated Users through Collaborate, add the third User and Share viewer, verify both editors update without reload, assert preflight counts, and exercise Copy link through the clipboard. - `bash tests/adversarial/run.sh`: run the generated XUser/authz/robustness matrices on the combined server. Add the focused history append kill/restart phase. Verify prior points survive and only a torn tail is removed. - `cargo test -p calternal-collab torn_tail_recovery -- --test-threads=4`: retain the focused segment crash-tail proof. - Build the benchmark on the build host as in `bench/live-history/README.md`. Run `python3 -B bench/live-history/run-matrix.py --phase2-y4 --binary /mnt/hdd/bench/live-history/calternal-live-history-bench --traces /mnt/hdd/bench/live-history/traces --results /mnt/hdd/bench/live-history/results.jsonl --loads /mnt/hdd/bench/live-history/load.tsv --runs /mnt/hdd/bench/live-history/runs`, then `python3 -B bench/live-history/check_budget.py --results /mnt/hdd/bench/live-history/results.jsonl`. The runner holds `/root/perf.lock` and records load. Include the real admission/Index/save path in an integrated profile before claiming end-to-end numbers. - Run combined release, deployment and o2 checks under the merge-round policy. Review the six attached captures. READY FOR MERGE ROUND: yes. This is readiness for the combined verification round; full Phase 2 acceptance still needs the checks listed above. ## Files Includes the six merged slices and integration fixes: ```text .forgejo/workflows/ci.yml Cargo.lock Cargo.toml apps/web/e2e/history-975.mjs apps/web/src/lib/history/VersionHistory.svelte apps/web/src/lib/history/VersionHistory.svelte.test.ts apps/web/src/lib/history/api.test.ts apps/web/src/lib/history/api.ts apps/web/src/lib/history/sessions.test.ts apps/web/src/lib/history/sessions.ts apps/web/src/lib/notes/NoteView.svelte bench/live-history/Cargo.lock bench/live-history/Cargo.toml bench/live-history/README.md bench/live-history/check_budget.py bench/live-history/run-matrix.py bench/live-history/run_ci_budget.py bench/live-history/src/lib.rs bench/live-history/src/main.rs bench/live-history/test_budget.py contracts/action-overrides.json contracts/actions.json contracts/openapi.json crates/calternal-api/src/actions.rs crates/calternal-collab/Cargo.toml crates/calternal-collab/src/history/0001_write_budget.sql crates/calternal-collab/src/history/adversarial.rs crates/calternal-collab/src/history/conformance.rs crates/calternal-collab/src/history/mod.rs crates/calternal-collab/src/history/restore.rs crates/calternal-collab/src/history/store.rs crates/calternal-collab/src/history/write.rs crates/calternal-collab/src/history/write_tests.rs crates/calternal-collab/src/lib.rs crates/calternal-collab/src/session.rs crates/calternal-collab/tests/history_restore.rs crates/calternal-fs/src/history.rs crates/calternal-fs/src/lib.rs crates/calternal-fs/src/root.rs crates/calternal-fs/src/write.rs crates/calternal-server/Cargo.toml crates/calternal-server/src/history.rs crates/calternal-server/src/history_contract_tests.rs crates/calternal-server/src/main.rs crates/calternal-server/src/wire.rs crates/plugins/notes/src/lib.rs crates/plugins/notes/src/store.rs docs/parity-matrix.md docs/perf/baseline.json docs/perf/live-history-budget.json docs/perf/live-history-budget.md packages/api-client/src/generated.ts scripts/action_registry.py scripts/test_action_registry.py tests/adversarial/authz_matrix.py tests/adversarial/test_xuser_classification.py tests/adversarial/xuser_matrix.py ``` ## Cleanup ```text Removed 18870 files, 12.3GiB total ``` Web build output was removed. Screenshots and gate logs remain under ignored artifacts/.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
wip/previewcard-1098
wip/palette2-1123
wip/palette-1093
wip/onboard2-1141
wip/onboard-1141.aborted-early
wip/onboard-1141
wip/nlpchip-1127
wip/morph-1104
wip/merge-round-7c5
wip/merge-round-7c4
wip/merge-round-7c3
wip/merge-round-7c2
wip/merge-round-7c
wip/mchrome-1084
wip/mailghost2-1094
wip/mailghost-1094
wip/kbpreview2-1118
wip/kbpreview-1118
wip/kanban-1092
wip/importhang-1121
wip/hiderev-1153
wip/hide4-1153
wip/hide3-1153
wip/hide2-1153
wip/hide-1153
wip/editreg-1132
wip/editorrail3-1113
wip/editorrail2-1113
wip/editorrail-1113
wip/e2e-b2-1071
wip/e2e-b-1071
wip/draw4-1101
wip/draw3-1101
wip/draw2-1101
wip/draw-1101
wip/directory-1199-r
wip/directory-1199
wip/delete-1119
job/merge30
wip/collabrev-1197
wip/collabloss2-1197
wip/collabloss-1197
wip/cards2-1083
wip/cards-1083
wip/canvas-visual
wip/canvasvis2-976
wip/calhdr-1112
wip/calcards-1115
wip/browserfix
wip/blocks-1125
wip/allday-1107
wip/agenda-decks
wip/agenda-1086
wip/adv7c-1105
wip/txentry-1198
wip/trayicons2-1095
wip/trayicons-1095
wip/tagperf-1186
wip/sidebar3-1094
wip/rev2-webperf
wip/rev2-money-ident
wip/restyle-mailmoney
wip/restyle-files
job/restyle-notes
job/tagperf-1186
job/adv-1202
job/notifloop-1194
job/restyle-mailmoney
job/onboard-1141
wip/restyle-notes
job/segmented-1200
job/hide-1153
wip/notifloop-1194
job/txentry-1198
job/collabloss-1197
job/perf-1124
job/perf2-1124
job/tocrail-1191
job/restyle-settings
wip/restyle-settings
wip/segmented-1200
job/restyle-files
job/tagdnd-1187
job/cards-1179
wip/cards2-1179
wip/cards-1179
wip/tocrail-1191
wip/tagdnd-1187
wip/perf-1124
wip/merge30j
job/wizchoices-1140
wip/wizchoices-1140
wip/restyle-1190
job/moneyfmt-1180
wip/moneyfmt2-1180
wip/moneyfmt-1180-r
wip/moneyfmt-1180
job/pillglass-1189
job/flags-1181
wip/flags-1181
job/restyle-1190
job/restyle-search
job/settingsreg-1195
job/wizard-1140
site/website
wip/wizardrev2-1140
wip/wizardrev-1140
wip/wizard5-1140
wip/wizard4-1140
wip/wizard3-1140
wip/wizard2-1140
wip/wizard-1140
wip/pillglass-1189
wip/settingsreg-1195
job/merge29
job/fu-1171
wip/merge29j
wip/fu-1171
job/fu-1166
job/directory-1199
job/proflog-1204
job/txresearch-1188
wip/fu-1166
job/merge28
job/search-1066
wip/search-1066
wip/merge28j
job/gateslot-1182
job/bulkimport-1157
job/mailnet-1160
wip/mailnetrev-1160
wip/mailnet-1160
wip/bulkrev-1157
wip/bulkimport-1157
job/startup-1161
wip/startup-1161
job/merge27
job/linkcards-1151
wip/linkcards3-1151
wip/linkcards2-1151
wip/linkcards-1151
job/traydate-1144
wip/traydate3-1144
wip/traydate2-1144
wip/traydate-1144
job/draw-1101
wip/merge27j
job/blockpill-1152
wip/blockpill3-1152
wip/blockpill2-1152
wip/blockpill-1152
job/minihover-1149
wip/minihover2-1149
wip/minihover-1149
job/merge25
wip/merge25-r
wip/merge25b
wip/merge25
job/inspector-1129
job/tags-1110
wip/inspector3-1129
wip/inspector2-1129
wip/inspector-1129
wip/tagsrev-1110
wip/tags2-1110
wip/tags-1110
job/dates-1148
wip/datesrev-1148
wip/dates2-1148
wip/dates-1148
job/licence-1145
wip/licence2-1145
wip/licence-1145
job/selfhost-1156
job/merge23
wip/merge23
job/tagfilter-1109
wip/tagfilter2-1109
wip/tagfilter-1109
job/kbd-1134
wip/kbd2-1134
wip/kbd-1134
job/palfoot-1137
wip/selfhost-1156
wip/palfoot2-1137
wip/palfoot-1137
job/toggle-1158
wip/toggle-1158
job/kbpreview-1118
job/docratchet-1155
job/perflint-1133
job/devtests-1159
wip/docratchet-1155
wip/devtests-1159
job/segv-1136
wip/toast-1142
wip/segv-1136
job/toast-1142
job/blockreload-1147
wip/blockreload-1147
job/font-1150
wip/font-1150
job/importui-1120
job/minimonth-1149
wip/importui-1120
wip/minimonth-1149
job/depcheck-1146
wip/perflint-1133
wip/depcheck-1146
job/calcards-1115
job/blocks-1125
job/plus-1128
job/shift-1138
wip/plus2-1128
wip/plus-1128
wip/shift-1138
job/moneyfid-1130
job/editorrail-1113
wip/moneyrev-1130
wip/moneyfid-1130
job/noext-851
wip/noext-851
wip/noext3-851
wip/noext2-851
job/week-1135
wip/week-1135
job/editreg-1132
job/smoke-1122
wip/smoke-1122
job/docs-1143
job/palette2-1123
job/calhdr-1112
job/nlpchip-1127
job/mailghost-1094
job/reconnect-1131
wip/reconnect-1131
job/trayicons-1095
job/delete-1119
job/importhang-1121
job/cards-1083
job/palette-1093
job/mchrome-1084
job/e2e-a-1071
job/canvas-visual
job/previewcard-1098
job/allday-1107
wip/e2e-a2-1071
wip/e2e-a-1071
job/e2e-b-1071
job/adv7c-1105
job/kanban-1092
job/agenda-1086
job/merge-round-7c
job/morph-1104
wip/surfaces-p2
job/merge-round-9
wip/merge-round-9
job/7cfix-small
wip/7cfix-small
job/mailui-1078
job/merge-round-8
wip/merge-round-8
wip/mailui-1078
job/mailround-1038
job/applemail-accept
wip/settitle-1068
wip/mailround2-1038
wip/mailround-1038
wip/e2e-7b
job/crash-1069
wip/crash-1069
job/searchlost-1066
wip/searchlost-1066
job/7b-reconcile
job/flake-1065
wip/flake-1065
wip/merge-round-7b7
wip/merge-round-7b6
wip/merge-round-7b5
wip/merge-round-7b4
wip/7b-reconcile
job/appupdate-1059
job/nfd-1044
wip/appupdate-1059
job/e2e-7b
job/loop-1062
wip/loop-1062
job/pdfprev-1045
job/invtoggle-1053
wip/pdfprev-1045
wip/nfd-1044
wip/invtoggle-1053
job/7bfix-e2e
job/mailstress-b
wip/7bfix-e2e
wip/mailstress-b
job/7bfix-adv
wip/7bfix-adv
job/mailstress-a
job/stack-1054
wip/stack-1054
wip/mailstress-a
job/mailstress-1038
wip/mailstress-1038
job/upload500-1051
wip/upload500-1051
job/share-1034
wip/share-1034
job/syncerr-1037
job/7bfix-photos
wip/7bfix-photos
job/paste-1036
job/setside-1039
wip/setside-1039
wip/paste-1036
job/lease-1042
wip/syncerr-1037
wip/lease-1042
job/7bfix-data
job/passkeybind-1043
wip/apprevoke-1041
job/invite-1035
wip/invite-1035
job/merge-round-7b2
wip/merge-round-7b2
job/mailproxy-486
job/apprevoke-1041
job/rebuild-1033
job/pillborder-1029
wip/pillborder-1029
wip/mailproxy-486
wip/applemail-486
job/headless-998
wip/headless-998
job/groups-1028
wip/groups-1028
job/rebuildwarn-1016
wip/rebuildwarn-1016
job/startup-1011
wip/startup-1011
job/monthpill-1009
job/bgthumb-1025
job/sharetitle-1012
wip/monthpill-1009
wip/bgthumb-1025
wip/sharetitle-1012
job/canvas-cards-977
wip/canvas-cards-977
job/canvas-pencil-978
job/canvas-sketch-990
wip/canvas-sketch-990
wip/canvas-pencil-978
job/canvas-files-989
wip/canvas-files-989
job/canvas-collab-991
wip/canvas-collab-991
job/weekscroll-1018
wip/weekscroll-1018
wip/canvas-core-976
job/canvas-core-976
job/round-drag
wip/round-drag
job/round-settings
job/browserfix
wip/oapi-974
job/oapi-974
job/hist2-integrate
job/mailhtml-726
wip/mailhtml-726
wip/hist2-integrate
job/moneyfu-984
job/drag-1015
wip/drag-1015
job/rename-1017
wip/rename-1017
job/hist2-api
wip/hist2-api
job/oneacct-1014
wip/oneacct-1014
wip/moneyfu-984
job/hist2-bench
job/hist2-restore
wip/hist2-bench
job/hist2-write
job/hotfix-724
wip/hotfix-724
wip/hist2-write
wip/hist2-restore
job/hist2-store
job/hist2-ui
wip/hist2-ui
wip/hist2-store
job/searchstarve-965
job/shutdown-963
wip/shutdown-963
wip/pubedit-981
job/pubedit-981
job/analytics-973
wip/searchstarve-965
job/authflash-850
job/weeklane-969
job/pvtitle-1004
job/hist-975
wip/authflash-850
job/voicepill-617
wip/pvtitle-1004
job/headring-1003
wip/weeklane-969
wip/voicepill-617
wip/headring-1003
wip/analytics-973
job/agentscope-980
wip/thumbsandbox-988
job/thumbsandbox-988
wip/hist-975
job/links-856
wip/links-856
job/davetag-966
wip/davetag-966
job/filesstorm-1000
job/hoverpad-725
wip/filesstorm-1000
job/ffmpegblas-993
job/merge-round-7a
wip/hoverpad-725
wip/ffmpegblas-993
job/nowdot-1002
wip/verify-7a
job/noteid-857
wip/nowdot-1002
wip/noteid-857
wip/merge-round-7a
wip/agentscope-980
job/imapedge
job/a11yfix2
wip/imapedge-941
wip/imapedge
wip/a11yfix2
job/notetask-986
job/logheading
wip/logheading-998
job/textthumb-652
job/photolive-987
wip/photolive-987
job/davactive-983
job/savefix-985
job/tabicons-607
wip/davactive-983
wip/tabicons-607
wip/notetask-986
wip/savefix-985
job/dirid-627
job/buildspeed-1007
wip/dirid-627
job/agenda-decks
job/perfguards-impl
job/undo-a11y
wip/undo-a11y
job/mailperf
job/wal-824
wip/settings-50
job/settings-50
job/notesfilter-606
wip/notesfilter-606
job/surfaces-p2
wip/wal-824
job/maillayouts
wip/mailperf
wip/maillayouts
job/taskmeta-659
job/money-ident
wip/money-ident
wip/taskmeta-659
job/errstates
wip/perfguards-impl
job/headings-881
wip/headings-881
wip/errstates
job/voice-619
job/gaps-827
job/notesperf
wip/notesperf
wip/voice-619
job/hddsql-549
job/perf-stream-668
wip/perf-stream-668
wip/deeplinks-fix
job/deeplinks-fix
job/authfix
job/docsfix-rust
wip/docsfix-rust
job/webperf
job/docsfix-web
job/datafix2
job/webdav-lock-476
job/copyfix
wip/copyfix
wip/webperf
job/focus-658
wip/protofix
job/mediafix
job/protofix
wip/mediafix
job/agentfix
job/hhmm-724
wip/agentfix
job/undo-722
job/reuse
wip/webdav-lock-476
wip/reuse
job/scopefix
job/datafix
wip/hhmm-724
wip/undo-722
job/surfaces-p1
wip/hddsql-549
job/voicememos-618
wip/datafix2
wip/surfaces-p1
job/fix-940
wip/fix-940
job/blaze-surfaces
wip/datafix
wip/blaze-surfaces
job/taskday-655
job/linknav-639
wip/linknav-639
wip/gaps-827
job/isolation-707
job/audiophotos-720
wip/audiophotos-720
job/advfind-664
wip/voicememos-618
wip/taskday-655
wip/isolation-707
wip/advfind-664
wip/scopefix
wip/focus-658
job/testgaps
wip/testgaps
job/overscroll-718
wip/authfix
job/deps
wip/overscroll-718
job/rev2-agentfix
job/rev2-money-ident
job/rev2-mailperf
wip/deps
job/hardening-728
wip/hardening-728
job/searchgen-832
wip/searchgen-832
job/photopw-849
job/mailsql-825
wip/photopw-849
job/sharefix
wip/sharefix
job/rev2-mailhtml-726
job/rev2-perfguards
job/copyval-723
job/lightglass-r2
wip/lightglass-r2
wip/docsfix-web
job/copy-audit
job/macinterop-staging-r2
job/design-sync
job/rev2-taskmeta-659
job/rev2-webperf
job/docs-audit
job/rev2-advfind-664
job/rev2-mailproxy-486
job/states-audit
job/rev2-datafix
job/design-drift
job/test-gaps
job/rev2-voicememos-618
job/rev2-mediafix
job/rev2-deps
job/rev2-datafix2
job/licence-audit
job/issue-hygiene
job/rev2-protofix
job/rev2-voice-619
job/rev2-isolation-707
job/rev2-surfaces-p1
job/deeplink-audit2
job/rev2-audiophotos-720
wip/test-gaps
job/rev2-overscroll-718
job/rev2-undo-722
wip/states-audit
job/rev2-dropmd-719
job/rev2-linknav-639
job/merge-7b-plan
wip/merge-7b-plan
job/rev2-taskday-655
wip/mailsql-825
job/rev2-webdav-lock-476
job/rev2-browserfix
wip/design-drift
job/rev2-hddsql-549
wip/deeplink-audit2
job/rev2-scopefix
job/rev2-authfix
job/rev2-hardening-728
job/rev2-wal-824
job/rev2-sharefix
job/calsidebar-638
job/chrome-audit
job/ioperf
wip/ioperf
wip/chrome-audit
wip/calsidebar-638
job/dropmd-719
wip/dropmd-719
job/ocr-build
wip/ocr-build
job/blaze-settings
wip/copyval-723
job/toastring-721
wip/toastring-721
job/deployfix-732
wip/deployfix-732
wip/blaze-settings
job/money-import-recheck
job/rev-a11y
job/perf-arch-db
job/rev-7b-data
wip/textthumb-652
wip/perf-arch-db
job/sec-protocols
job/sidehdr-660
job/rev-7b-security
job/research-surfaces
job/rev-design-gaps
job/rev-mcp-api
wip/sidehdr-660
job/perf-arch-memory
wip/sec-protocols
job/perf-arch-bundle
job/snapedge-714
wip/rev-mcp-api
job/sec-supplychain
wip/research-surfaces
job/perf-arch-sync
job/rev-consistency
job/perf-arch-server
wip/perf-arch-server
wip/perf-arch-memory
job/perf-arch-io
job/perf-arch-client
job/sec-fs
job/sec-mcp-scopes
job/sec-sharing
job/perf-guards
job/sec-browser
job/sec-admin-deploy
job/sec-auth
wip/snapedge-714
job/bgpicker-717
wip/perf-arch-bundle
wip/money-import-recheck
job/advsetup-654
wip/bgpicker-717
wip/advsetup-654
job/burst-709
job/kbdcaps-710
job/app-pw-chooser
wip/burst-709
wip/app-pw-chooser
job/imaptest-625
wip/kbdcaps-710
job/fix-499
wip/fix-499
job/perf-mut-667
job/calimg-589
job/perf-snap-666
wip/calimg-589
wip/perf-snap-666
wip/perf-mut-667
job/perf-cache-665
wip/perf-cache-665
job/voicefiles-620
wip/voicefiles-620
job/admin-burst-705
wip/admin-burst-705
job/voicememos-review
wip/voicememos-review
wip/ryw-653
job/ryw-653
job/writeonopen-661
job/instant-663
wip/writeonopen-661
job/money-import-review
wip/money-import-review
wip/importjs-610
review/integrations-407-round6
wip/integrations-review
job/dragghost-612
wip/dragghost-612
job/integrations
wip/integrations
job/decider-656
job/merge-round-6
job/perf-rerun
wip/merge-round-6
job/integrations-review-round5
job/selalign-576
wip/selalign-576
job/mcp-events-491
job/files-631
job/cal-e2e-569
wip/cal-e2e-569
job/reload-423
wip/reload-423
wip/mcp-events-491
wip/files-631
job/notesbridge-644
wip/notesbridge-644
job/editor-series
job/calcard-series
wip/calcard-series
job/mcp-events-review-491
wip/mcp-events-review
wip/editor-series
job/quirks-546
job/integrations-recheck
job/tocrail-636
wip/tocrail-636
wip/quirks-546
wip/reminders-643
job/reminders-643
wip/davscale-573
job/davscale-573
job/integrations-review
wip/ocr-eval-584
job/ocr-eval-584
job/esc-537
wip/esc-537
job/toastname-586
wip/toastname-586
job/submenu-579
wip/submenu-579
job/tasks-mode
wip/tasks-mode
job/agentdocs-630
job/dupwrite-634
wip/agentdocs-630
wip/dupwrite-634
job/lightglass-588
wip/lightglass-588
job/tabswitch-549
job/ghosttask-623
wip/ghosttask-623
job/toaststack-616
job/weekstate-609
job/mailsync-613
wip/mailsync-613
wip/weekstate-609
job/maildup-626
wip/tabswitch-549
wip/maildup-626
wip/toaststack-616
job/motion-611
wip/motion-611
job/tlstest-601
wip/tlstest-601
job/perf-495
job/floating-sheet
wip/floating-sheet
job/remdup-585
wip/remdup-585
job/fix-502
wip/fix-502
job/attachplay-622
job/perf-batch
wip/perf-batch-563
wip/perf-495
hotfix/mail-sync-diag
job/mail-m3
wip/mail-m3
job/attach-poof-603
job/calhover-608
job/editorbar-604
job/mentions-605
job/merge-round-4
job/allday-514
wip/merge-round-4
wip/allday-514
job/merge-round-4a
wip/merge-round-4a
job/sharestack-580
job/fix-501
wip/sharestack-580
wip/fix-501
job/perf-batch-563
job/apw-cache-review
wip/apw-cache-review
job/probe-520
wip/probe-520
job/mac-393
wip/mac-393
job/header-571
job/flake-513
wip/flake-513
job/docs-thumb-547
wip/header-571
job/webcal-572
wip/webcal-572
wip/shortcuts-542
job/shortcuts-542
wip/docs-thumb-547
job/caldav-stress
wip/caldav-stress
wip/sweep-478
job/apw-cache-512
wip/apw-cache-512
job/money-empty-540
wip/restart-505
wip/money-empty-540
wip/fix-510
job/restart-505
job/fix-503
job/perf-496
wip/perf-496
job/fix-498
wip/fix-498
job/info-inspector-465
wip/info-inspector-465
job/fix-510
job/fix-507
wip/fix-507
wip/fix-503
job/fix-493
job/money-kinds
wip/money-kinds
job/hygiene-548
job/merge-round-3
wip/fix-493
job/drag-snap-536
wip/merge-round-3
wip/merge-round-0930
wip/drag-snap-536
job/align-538
wip/align-538
job/bg-flash
wip/bg-flash
job/money-import
job/search-count-544
wip/search-count-544
wip/money-import
job/settings-key-541
wip/settings-key-541
job/toast-539
job/preview-421
wip/preview-421
wip/toast-539
job/tasks-500-531
job/title-plain-526
wip/title-plain-526
wip/tasks-500-531
job/notes-bridge
wip/parity-484
job/parity-484
job/files-slow
job/crash-525
wip/notes-bridge
wip/files-slow
wip/crash-525
job/kbd-motion-527
wip/bg-422
job/analytics-504
wip/analytics-504
wip/kbd-motion-527
job/upload-pill-523
wip/upload-pill-523
wip/tray-order
job/tray-order
wip/overflow-mid
wip/merge-round-2
job/perf-494
wip/perf-494
wip/mcp-fast-492
wip/motion-477
wip/asr-ab-489
wip/theme-variants-506
wip/overflow-511
wip/week-header-508
wip/attach-427
job/dav-delete-471
job/iso-435
wip/iso-435
wip/files-sel-keys
wip/dav-delete-471
job/align-253
job/siwc-490
wip/siwc-490
job/money-kinds-review
wip/align-253
wip/money-kinds-review
job/small-bugs-3
wip/overlay-title-487
wip/multiget-500
wip/hidden-420
wip/webcal-ui
wip/webcal-431
job/perf-367
job/location
wip/small-bugs-3
wip/location
wip/perf-367
wip/admin-deny-483
job/tag-unicode-473
wip/tag-unicode-473
job/blur-436
wip/photos-470
wip/blur-436
wip/small-bugs-4
wip/hunt-20260930
wip/settings-hdr-482
wip/chips-416
job/dedup-375
wip/dedup-375
job/doc-stack
wip/doc-stack
job/tokens-literals
wip/tokens-literals
job/jobs-leftovers
wip/send-fast
wip/paste-467
wip/money-numbers
job/money-plugin
wip/money-plugin
job/break-dav
wip/merge-batch
wip/crossday-469
wip/mac-verify
wip/mail-m2
wip/break-dav
wip/money-review2
job/money-md
job/modes-424
wip/money-md
wip/jobs-leftovers
job/agenda-413
wip/agenda-413
wip/modes-424
job/recog-417
wip/recog-417
wip/bounce-425
wip/ab-384-luna
job/webdav-perf
wip/webdav-perf
job/toast-ring
wip/toast-ring
job/money-review
wip/money-review
wip/micro-motion
wip/settings-card
wip/minical
job/notes-imap-428
job/least-priv
wip/ui-small-2
wip/flaky-426
wip/drag-end-418
job/jank
wip/jank
wip/least-priv
wip/docs-site
job/agenda
job/sec-batch
wip/sec-batch
wip/per-user-index
job/area-calendars
wip/area-calendars
job/parity
wip/parity
job/documents-research
wip/documents-research
job/test-infra
job/reminders-sync
wip/small-bugs-2
wip/reminders-sync
wip/gestures
job/google-oauth
wip/tags-merge
wip/tags
job/e2e-theme
wip/e2e-theme
job/icon-align
wip/test-infra
wip/select-align
wip/editor-385
job/voice
wip/webdav
job/webdav
job/app-pw-ui
job/editor-integrity
wip/editor-integrity
wip/voice
wip/quota
wip/cal-followups
wip/icon-align
job/composer-scale
wip/composer-scale
job/jobs-page
wip/jobs-page
job/hig-type
wip/hig-type
wip/app-pw-ui
job/motion-spring
job/mcp
wip/motion-spring
wip/mcp
job/small-bugs
wip/push-hosts
job/profile-sign
wip/touch-369
wip/profile-sign
job/mobile-focus
wip/mobile-focus
wip/ui-polish-354
wip/small-bugs
wip/dup-task
job/toast-polish
job/app-pw-scopes
wip/toast-polish
wip/app-pw-scopes
wip/cli-agent
wip/selection-pills
job/preview-attach
wip/preview-attach
job/dav-proppatch
wip/dav-proppatch
wip/cal-switcher
job/atomic-race
wip/atomic-race
job/photos-shared
wip/photos-shared
wip/cal-grid
wip/note-rewrite
wip/search-rebuild
job/mail-m1
job/paperless-import
wip/paperless-import
wip/mail-m1
wip/hidden-activity
wip/search-d
wip/pricing-research
wip/cursors
wip/auto-scheme
job/single-pills
wip/single-pills
wip/xuser-matrix
wip/money-format
wip/app-pw-setup
wip/purge-dos
wip/vault-health
wip/caldav-apple
wip/xuser-audit
wip/e2e-green
wip/tabbar
wip/adv-harness
wip/maple-mono
job/search-fix
wip/search-fix
wip/search-perf-c
job/adv-harness
wip/sidebar-headers
job/glass
wip/temp-index
job/polish
wip/polish
wip/file-protocols
wip/money-research
wip/glass
wip/voice-models
wip/collab-redo
job/voice-research
wip/hunt-20260928
wip/notes-actions-research
wip/search-pad
wip/search-perf
wip/search-sticky
wip/editor-undo
wip/chrome-rules
wip/motion
wip/appearance-research
wip/appearance
wip/audit-bugs
wip/cal-glass
wip/block-actions
wip/authz-order
wip/event-stripes
wip/chrome-sidebar
wip/auth-flaky
wip/robust-2
wip/gate-fix
wip/menu-blur
wip/import-calternaljs
wip/tray-fix
job/import-calternaljs
wip/index-order
wip/audit-fixes
wip/search-chevrons
research/mail
wip/phone-chrome
wip/dedup-break
wip/csp
wip/ui-audit
wip/select-toast
wip/perf
wip/flat-layout
wip/fonts
wip/event-tint
wip/sync-converge
wip/data-split
wip/glass-audit
wip/robustness
wip/sync-chaos
wip/search-thumbs
wip/fuzz
wip/menu-icons
wip/search-pill
wip/sync-changing
wip/heading-links
wip/date-formats
wip/a11y
wip/break-editor
wip/e2e-fix
wip/settings-sections
wip/sync-root-guard
wip/search-palette
wip/share-edit
job/toasts
wip/toasts
wip/cont-analytics
wip/authz-review
wip/popovers
wip/overlay-glass
wip/change-feed
wip/editor-modes
wip/composer-align
wip/cont-agenda
wip/agenda-merge
job/agent-conventions
wip/agent-conventions
wip/backend-misc
job/route-audit
wip/route-audit
wip/ui-batch
wip/heif-hardening
wip/grid-resize
wip/ask-page
wip/webmcp
job/deeplink-audit
wip/deeplinks
wip/shortcuts
wip/cont-tz-days
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#975
No description provided.