Notes storage and live rooms #24

Closed

kayg wants to merge 1 commit from issue/6-notes-storage into dev

kayg commented

2026-10-13 16:05:32 +00:00

Owner

Closes #6

Notes are plain markdown files in Notes/, kept by the new calternal-notes plugin, with every guard below backed by a test.

What's in it

calternal-fsio (new crate): the one atomic write recipe (temp, fsync, rename, fsync dir), safe renames including case-only, the exclusive data folder lock (<data>/.lock, the binary refuses to start if another server holds it), NFC names with case/normalisation-insensitive collision checks. clippy.toml bans std::fs::write, File::create, std::fs::rename everywhere else.
calternal-notes (new crate, plugin notes):
- one live room per note id (yrs, OffsetKind::Utf16). Every writer goes through it: websocket clients, API actions, outside edits.
- server-only seeding from the file, a lineage id per document (sent on join).
- per-user notes/history.db (synchronous=FULL): every change, author-tagged, written before the file. Pending-save bookkeeping so a crash between "save started" and "save done" is never mistaken for an outside edit.
- saves 750 ms after the last change by a person or agent only. Opening, joining, syncing and outside edits never write. Server-owned frontmatter keys (calternal-id, date, last edited, title ⇄ H1) are key-level edits through the room, and never mark the note dirty (no save loops).
- check-before-write: the disk is compared with the last written hash before every save; outside edits are applied on a fork of the last written base (never the live text), with a writer id derived from base state + new bytes, recorded atomically with the new base.
- idempotent API edits: a change with an idempotency key is written with a Yjs writer id derived from the key, so a retry after a crash (engine record lost) adds nothing twice.
- \n-only live text; the file's CRLF/BOM style restored on write.
- ids.jsonl id index (append-only, fsynced, torn last line tolerated, compaction), startup scan + notify watcher: outside creates, renames (by calternal-id, or by content hash for files without one), copies with a duplicate id (fresh id), deletes.
- actions: notes.list, notes.get, notes.create, notes.append, notes.patch (UTF-16 positions, base required), notes.replace (base required, three-way via fork), notes.save.
core: ActionCtx.idempotency_key, EventRecorder (journal events outside actions, implemented by the engine), rooms (RoomService, RoomSession, JoinOptions), error codes base_version_required (428) and not_saved_yet (503, retryable).
server: rooms on the existing live websocket. {"type":"join","room":"notes/<id>","tz":"+05:30"} gives joined with info (lineage, text: "md", path), then binary frames: lib0 var-string room name + one y-protocols message (same messages as y-websocket). leave, left, join_failed. Joins run off the socket loop; slow clients are dropped from the room and rejoin.
binary: data lock, notes plugin + rooms + recorder wired in, open notes saved on shutdown. CALTERNAL_NOTES_SAVE_MS (test-only knob) shortens the save delay.

Guards from #3 implemented here (each with a test)

A1 seeding only on the server · B4 no write on open · B6 echo by hash · C7 outside edit while typing merges · C8/G1 check before write · D9 delete vs typing · D10/G2 idempotent retries (engine + writer-id) · D11/G3 no blind full replace (428 + fork merge) · E13/G4 UTF-16 positions, diffs never split graphemes · E14/G5 CRLF/BOM · E15/G6 NFC + case-insensitive names, case-only renames · G20/G11 atomic write recipe + lint · G21 history before flush, pending-save settle · G24/G13 key-level frontmatter, no save loops · G26 exclusive data lock · G30 n/a here · duplicate-id copies get fresh ids · lineage id.

Deferred (other issues)

Client side of lineage + offline diff-replay, tab leader, IME, editor drift (#7).
Rename-by-title + link rewriting (#9) uses fsio::rename_safely; daily-note ids from (user, date) (#8) can already pass an explicit id to notes.create.
History compaction (rooms replay all updates on open), pruning, notes.save_failed-style UI (event notes.not_saved is emitted).
G14 version restore, G15 rollback lock handover beyond the data lock, G16 protocol handshake.
A note made via the API but never typed in is kept only in memory across its first save; after a restart before any typing it is dropped as a ghost.

Tests

property: diff (tricky Unicode: ZWJ, flags, combining, CJK, RTL) never splits a character and always reproduces the target; line ending/BOM round trip.
multi-client fuzz: 4 clients × 12 seeds, random inserts/deletes, delivery shuffled; convergence, no token twice, untouched tokens exactly once.
crash points (test-only) before and after the file write, with the runtime thrown away and a fresh server opened: no doubling, no loss.
kill -9 loop against the real binary: 30 SIGKILLs while appends stream and saves run; 458 confirmed appends each exactly once, 29 in-flight appends retried with the same key each exactly once, the file never torn.
outside edits/renames/copies/deletes with the real watcher; missed watcher event; edit while closed.
websocket: join, sync, type over a binary frame, read back via the API; 428 without base; join of a missing note fails politely.
44 notes tests + 3 binary tests; notes suite run 8× in parallel with no flakes.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Closes #6 Notes are plain markdown files in `Notes/`, kept by the new `calternal-notes` plugin, with every guard below backed by a test. ## What's in it - **`calternal-fsio`** (new crate): the one atomic write recipe (temp, fsync, rename, fsync dir), safe renames including case-only, the exclusive data folder lock (`<data>/.lock`, the binary refuses to start if another server holds it), NFC names with case/normalisation-insensitive collision checks. `clippy.toml` bans `std::fs::write`, `File::create`, `std::fs::rename` everywhere else. - **`calternal-notes`** (new crate, plugin `notes`): - one live **room** per note id (yrs, `OffsetKind::Utf16`). Every writer goes through it: websocket clients, API actions, outside edits. - **server-only seeding** from the file, a **lineage** id per document (sent on join). - per-user **`notes/history.db`** (`synchronous=FULL`): every change, author-tagged, written **before** the file. Pending-save bookkeeping so a crash between "save started" and "save done" is never mistaken for an outside edit. - saves 750 ms after the last change **by a person or agent** only. Opening, joining, syncing and outside edits never write. Server-owned frontmatter keys (`calternal-id`, `date`, `last edited`, `title` ⇄ H1) are key-level edits through the room, and never mark the note dirty (no save loops). - **check-before-write**: the disk is compared with the last written hash before every save; outside edits are applied on a fork of the **last written base** (never the live text), with a writer id derived from base state + new bytes, recorded atomically with the new base. - **idempotent API edits**: a change with an idempotency key is written with a Yjs writer id derived from the key, so a retry after a crash (engine record lost) adds nothing twice. - `\n`-only live text; the file's CRLF/BOM style restored on write. - `ids.jsonl` id index (append-only, fsynced, torn last line tolerated, compaction), startup scan + `notify` watcher: outside creates, renames (by `calternal-id`, or by content hash for files without one), copies with a duplicate id (fresh id), deletes. - actions: `notes.list`, `notes.get`, `notes.create`, `notes.append`, `notes.patch` (UTF-16 positions, base required), `notes.replace` (base required, three-way via fork), `notes.save`. - **core**: `ActionCtx.idempotency_key`, `EventRecorder` (journal events outside actions, implemented by the engine), `rooms` (`RoomService`, `RoomSession`, `JoinOptions`), error codes `base_version_required` (428) and `not_saved_yet` (503, retryable). - **server**: rooms on the existing live websocket. `{"type":"join","room":"notes/<id>","tz":"+05:30"}` gives `joined` with `info` (`lineage`, `text: "md"`, path), then **binary frames**: lib0 var-string room name + one y-protocols message (same messages as y-websocket). `leave`, `left`, `join_failed`. Joins run off the socket loop; slow clients are dropped from the room and rejoin. - **binary**: data lock, notes plugin + rooms + recorder wired in, open notes saved on shutdown. `CALTERNAL_NOTES_SAVE_MS` (test-only knob) shortens the save delay. ## Guards from #3 implemented here (each with a test) A1 seeding only on the server · B4 no write on open · B6 echo by hash · C7 outside edit while typing merges · C8/G1 check before write · D9 delete vs typing · D10/G2 idempotent retries (engine + writer-id) · D11/G3 no blind full replace (428 + fork merge) · E13/G4 UTF-16 positions, diffs never split graphemes · E14/G5 CRLF/BOM · E15/G6 NFC + case-insensitive names, case-only renames · G20/G11 atomic write recipe + lint · G21 history before flush, pending-save settle · G24/G13 key-level frontmatter, no save loops · G26 exclusive data lock · G30 n/a here · duplicate-id copies get fresh ids · lineage id. ## Deferred (other issues) - Client side of lineage + offline diff-replay, tab leader, IME, editor drift (#7). - Rename-by-title + link rewriting (#9) uses `fsio::rename_safely`; daily-note ids from (user, date) (#8) can already pass an explicit `id` to `notes.create`. - History compaction (rooms replay all updates on open), pruning, `notes.save_failed`-style UI (event `notes.not_saved` is emitted). - G14 version restore, G15 rollback lock handover beyond the data lock, G16 protocol handshake. - A note made via the API but never typed in is kept only in memory across its first save; after a restart before any typing it is dropped as a ghost. ## Tests - property: diff (tricky Unicode: ZWJ, flags, combining, CJK, RTL) never splits a character and always reproduces the target; line ending/BOM round trip. - multi-client fuzz: 4 clients × 12 seeds, random inserts/deletes, delivery shuffled; convergence, no token twice, untouched tokens exactly once. - crash points (test-only) before and after the file write, with the runtime thrown away and a fresh server opened: no doubling, no loss. - **kill -9 loop against the real binary**: 30 SIGKILLs while appends stream and saves run; 458 confirmed appends each exactly once, 29 in-flight appends retried with the same key each exactly once, the file never torn. - outside edits/renames/copies/deletes with the real watcher; missed watcher event; edit while closed. - websocket: join, sync, type over a binary frame, read back via the API; 428 without base; join of a missing note fails politely. - 44 notes tests + 3 binary tests; notes suite run 8× in parallel with no flakes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 1 commit

2026-10-13 16:05:32 +00:00

Notes storage and live rooms

CI / rust (pull_request) Failing after 54s

Details

40b77e5f9d

Notes are plain markdown files in Notes/, kept by the new notes plugin:

- One live room per note id (yrs, UTF-16 offsets). Every writer goes
  through it: websocket clients (standard y-protocols sync, multiplexed
  on the live socket as binary room frames), API actions, and edits made
  outside on disk.
- Only the server seeds a note from its file; a lineage id per document.
- Every change is recorded in a per-user history.db (synchronous=FULL,
  author-tagged) before the file is saved, 750 ms after the last change
  by a person or agent. Opening never writes.
- Saves check the disk first and merge outside edits against the last
  written base (never the live text), record a pending save before the
  atomic write, and settle half-finished saves on restart.
- Outside edits are applied on a fork of the base state with a writer id
  derived from it, recorded atomically with the new base.
- API changes carry a writer id derived from the idempotency key, so a
  retry after a crash adds nothing twice. Whole-note changes need a base
  version (428 without one).
- Line endings and BOM restored, NFC names with case-insensitive
  collision checks, key-level frontmatter edits (calternal-id, date,
  last edited, title/H1 sync), no save loops.
- ids.jsonl index, startup scan and file watcher: outside creates,
  renames (by id or content), copies (fresh id) and deletes.
- calternal-fsio: the one atomic write recipe, safe renames, the
  exclusive data folder lock; clippy bans other ways to write files.

Tests: property tests (diff, line endings), multi-client fuzz with
reordered delivery, crash points before and after the file write, and a
kill -9 loop against the real binary (30 kills, every confirmed change
exactly once, retried in-flight changes exactly once, files never torn).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-13 16:05:41 +00:00

Notes storage and live rooms #6

kayg referenced this pull request

2026-10-13 16:09:05 +00:00

Notes storage and live rooms #27

kayg commented

2026-10-13 16:09:13 +00:00

Author

Owner

Superseded by #27: the same change rebased onto dev after sign-in (#23) merged (conflicts in crates/core/src/error.rs and crates/calternal/src/main.rs resolved keeping both features). Closing this one.

kayg closed this pull request

2026-10-13 16:09:14 +00:00

kayg referenced this pull request

2026-10-13 16:14:29 +00:00

Notes storage and live rooms #28