Notes storage and live rooms #27

Closed

kayg wants to merge 0 commits from issue/6-notes-storage-2 into dev

kayg commented

2026-10-13 16:09:05 +00:00

Owner

Closes #6

Replaces #24 (rebased onto dev after sign-in #23 merged; the forge keeps stale heads on force-push).

Notes are plain markdown files in Notes/, kept by the new calternal-notes plugin, with every guard below backed by a test.

What's in it

calternal-fsio (new crate): the one atomic write recipe (temp, fsync, rename, fsync dir), safe renames including case-only, the exclusive data folder lock (<data>/.lock, the binary refuses to start if another server holds it), NFC names with case/normalisation-insensitive collision checks. clippy.toml bans std::fs::write, File::create, std::fs::rename everywhere else.
calternal-notes (new crate, plugin notes):
- one live room per note id (yrs, OffsetKind::Utf16). Every writer goes through it: websocket clients, API actions, outside edits.
- server-only seeding from the file, a lineage id per document (sent on join).
- per-user notes/history.db (synchronous=FULL): every change, author-tagged, written before the file. Pending-save bookkeeping so a crash between "save started" and "save done" is never mistaken for an outside edit.
- saves 750 ms after the last change by a person or agent only. Opening, joining, syncing and outside edits never write. Server-owned frontmatter keys (calternal-id, date, last edited, title ⇄ H1) are key-level edits through the room, and never mark the note dirty (no save loops).
- check-before-write: the disk is compared with the last written hash before every save; outside edits are applied on a fork of the last written base (never the live text), with a writer id derived from base state + new bytes, recorded atomically with the new base.
- idempotent API edits: a change with an idempotency key is written with a Yjs writer id derived from the key, so a retry after a crash (engine record lost) adds nothing twice.
- \n-only live text; the file's CRLF/BOM style restored on write.
- ids.jsonl id index (append-only, fsynced, torn last line tolerated, compaction), startup scan + notify watcher: outside creates, renames (by calternal-id, or by content hash for files without one), copies with a duplicate id (fresh id), deletes.
- actions: notes.list, notes.get, notes.create, notes.append, notes.patch (UTF-16 positions, base required), notes.replace (base required, three-way via fork), notes.save.
core: ActionCtx.idempotency_key, EventRecorder (journal events outside actions, implemented by the engine), rooms (RoomService, RoomSession, JoinOptions), error codes base_version_required (428) and not_saved_yet (503, retryable).
server: rooms on the existing live websocket. {"type":"join","room":"notes/<id>","tz":"+05:30"} gives joined with info (lineage, text: "md", path), then binary frames: lib0 var-string room name + one y-protocols message (same messages as y-websocket). leave, left, join_failed. Joins run off the socket loop; slow clients are dropped from the room and rejoin.
binary: data lock, notes plugin + rooms + recorder wired in, open notes saved on shutdown. CALTERNAL_NOTES_SAVE_MS (test-only knob) shortens the save delay.

Guards from #3 implemented here (each with a test)

A1 seeding only on the server · B4 no write on open · B6 echo by hash · C7 outside edit while typing merges · C8/G1 check before write · D9 delete vs typing · D10/G2 idempotent retries (engine + writer-id) · D11/G3 no blind full replace (428 + fork merge) · E13/G4 UTF-16 positions, diffs never split graphemes · E14/G5 CRLF/BOM · E15/G6 NFC + case-insensitive names, case-only renames · G20/G11 atomic write recipe + lint · G21 history before flush, pending-save settle · G24/G13 key-level frontmatter, no save loops · G26 exclusive data lock · G30 n/a here · duplicate-id copies get fresh ids · lineage id.

Deferred (other issues)

Client side of lineage + offline diff-replay, tab leader, IME, editor drift (#7).
Rename-by-title + link rewriting (#9) uses fsio::rename_safely; daily-note ids from (user, date) (#8) can already pass an explicit id to notes.create.
History compaction (rooms replay all updates on open), pruning, notes.save_failed-style UI (event notes.not_saved is emitted).
G14 version restore, G15 rollback lock handover beyond the data lock, G16 protocol handshake.
A note made via the API but never typed in is kept only in memory across its first save; after a restart before any typing it is dropped as a ghost.

Tests

property: diff (tricky Unicode: ZWJ, flags, combining, CJK, RTL) never splits a character and always reproduces the target; line ending/BOM round trip.
multi-client fuzz: 4 clients × 12 seeds, random inserts/deletes, delivery shuffled; convergence, no token twice, untouched tokens exactly once.
crash points (test-only) before and after the file write, with the runtime thrown away and a fresh server opened: no doubling, no loss.
kill -9 loop against the real binary: 30 SIGKILLs while appends stream and saves run; 458 confirmed appends each exactly once, 29 in-flight appends retried with the same key each exactly once, the file never torn.
outside edits/renames/copies/deletes with the real watcher; missed watcher event; edit while closed.
websocket: join, sync, type over a binary frame, read back via the API; 428 without base; join of a missing note fails politely.
44 notes tests + 3 binary tests; notes suite run 8× in parallel with no flakes.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

Closes #6 Replaces #24 (rebased onto dev after sign-in #23 merged; the forge keeps stale heads on force-push). Notes are plain markdown files in `Notes/`, kept by the new `calternal-notes` plugin, with every guard below backed by a test. ## What's in it - **`calternal-fsio`** (new crate): the one atomic write recipe (temp, fsync, rename, fsync dir), safe renames including case-only, the exclusive data folder lock (`<data>/.lock`, the binary refuses to start if another server holds it), NFC names with case/normalisation-insensitive collision checks. `clippy.toml` bans `std::fs::write`, `File::create`, `std::fs::rename` everywhere else. - **`calternal-notes`** (new crate, plugin `notes`): - one live **room** per note id (yrs, `OffsetKind::Utf16`). Every writer goes through it: websocket clients, API actions, outside edits. - **server-only seeding** from the file, a **lineage** id per document (sent on join). - per-user **`notes/history.db`** (`synchronous=FULL`): every change, author-tagged, written **before** the file. Pending-save bookkeeping so a crash between "save started" and "save done" is never mistaken for an outside edit. - saves 750 ms after the last change **by a person or agent** only. Opening, joining, syncing and outside edits never write. Server-owned frontmatter keys (`calternal-id`, `date`, `last edited`, `title` ⇄ H1) are key-level edits through the room, and never mark the note dirty (no save loops). - **check-before-write**: the disk is compared with the last written hash before every save; outside edits are applied on a fork of the **last written base** (never the live text), with a writer id derived from base state + new bytes, recorded atomically with the new base. - **idempotent API edits**: a change with an idempotency key is written with a Yjs writer id derived from the key, so a retry after a crash (engine record lost) adds nothing twice. - `\n`-only live text; the file's CRLF/BOM style restored on write. - `ids.jsonl` id index (append-only, fsynced, torn last line tolerated, compaction), startup scan + `notify` watcher: outside creates, renames (by `calternal-id`, or by content hash for files without one), copies with a duplicate id (fresh id), deletes. - actions: `notes.list`, `notes.get`, `notes.create`, `notes.append`, `notes.patch` (UTF-16 positions, base required), `notes.replace` (base required, three-way via fork), `notes.save`. - **core**: `ActionCtx.idempotency_key`, `EventRecorder` (journal events outside actions, implemented by the engine), `rooms` (`RoomService`, `RoomSession`, `JoinOptions`), error codes `base_version_required` (428) and `not_saved_yet` (503, retryable). - **server**: rooms on the existing live websocket. `{"type":"join","room":"notes/<id>","tz":"+05:30"}` gives `joined` with `info` (`lineage`, `text: "md"`, path), then **binary frames**: lib0 var-string room name + one y-protocols message (same messages as y-websocket). `leave`, `left`, `join_failed`. Joins run off the socket loop; slow clients are dropped from the room and rejoin. - **binary**: data lock, notes plugin + rooms + recorder wired in, open notes saved on shutdown. `CALTERNAL_NOTES_SAVE_MS` (test-only knob) shortens the save delay. ## Guards from #3 implemented here (each with a test) A1 seeding only on the server · B4 no write on open · B6 echo by hash · C7 outside edit while typing merges · C8/G1 check before write · D9 delete vs typing · D10/G2 idempotent retries (engine + writer-id) · D11/G3 no blind full replace (428 + fork merge) · E13/G4 UTF-16 positions, diffs never split graphemes · E14/G5 CRLF/BOM · E15/G6 NFC + case-insensitive names, case-only renames · G20/G11 atomic write recipe + lint · G21 history before flush, pending-save settle · G24/G13 key-level frontmatter, no save loops · G26 exclusive data lock · G30 n/a here · duplicate-id copies get fresh ids · lineage id. ## Deferred (other issues) - Client side of lineage + offline diff-replay, tab leader, IME, editor drift (#7). - Rename-by-title + link rewriting (#9) uses `fsio::rename_safely`; daily-note ids from (user, date) (#8) can already pass an explicit `id` to `notes.create`. - History compaction (rooms replay all updates on open), pruning, `notes.save_failed`-style UI (event `notes.not_saved` is emitted). - G14 version restore, G15 rollback lock handover beyond the data lock, G16 protocol handshake. - A note made via the API but never typed in is kept only in memory across its first save; after a restart before any typing it is dropped as a ghost. ## Tests - property: diff (tricky Unicode: ZWJ, flags, combining, CJK, RTL) never splits a character and always reproduces the target; line ending/BOM round trip. - multi-client fuzz: 4 clients × 12 seeds, random inserts/deletes, delivery shuffled; convergence, no token twice, untouched tokens exactly once. - crash points (test-only) before and after the file write, with the runtime thrown away and a fresh server opened: no doubling, no loss. - **kill -9 loop against the real binary**: 30 SIGKILLs while appends stream and saves run; 458 confirmed appends each exactly once, 29 in-flight appends retried with the same key each exactly once, the file never torn. - outside edits/renames/copies/deletes with the real watcher; missed watcher event; edit while closed. - websocket: join, sync, type over a binary frame, read back via the API; 428 without base; join of a missing note fails politely. - 44 notes tests + 3 binary tests; notes suite run 8× in parallel with no flakes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg added 1 commit

2026-10-13 16:09:06 +00:00

Notes storage and live rooms

CI / rust (pull_request) Failing after 54s

Details

b1473cf782

Notes are plain markdown files in Notes/, kept by the new notes plugin:

- One live room per note id (yrs, UTF-16 offsets). Every writer goes
  through it: websocket clients (standard y-protocols sync, multiplexed
  on the live socket as binary room frames), API actions, and edits made
  outside on disk.
- Only the server seeds a note from its file; a lineage id per document.
- Every change is recorded in a per-user history.db (synchronous=FULL,
  author-tagged) before the file is saved, 750 ms after the last change
  by a person or agent. Opening never writes.
- Saves check the disk first and merge outside edits against the last
  written base (never the live text), record a pending save before the
  atomic write, and settle half-finished saves on restart.
- Outside edits are applied on a fork of the base state with a writer id
  derived from it, recorded atomically with the new base.
- API changes carry a writer id derived from the idempotency key, so a
  retry after a crash adds nothing twice. Whole-note changes need a base
  version (428 without one).
- Line endings and BOM restored, NFC names with case-insensitive
  collision checks, key-level frontmatter edits (calternal-id, date,
  last edited, title/H1 sync), no save loops.
- ids.jsonl index, startup scan and file watcher: outside creates,
  renames (by id or content), copies (fresh id) and deletes.
- calternal-fsio: the one atomic write recipe, safe renames, the
  exclusive data folder lock; clippy bans other ways to write files.

Tests: property tests (diff, line endings), multi-client fuzz with
reordered delivery, crash points before and after the file write, and a
kill -9 loop against the real binary (30 kills, every confirmed change
exactly once, retried in-flight changes exactly once, files never torn).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yyy8fQGxPLkLa8v8NXz7T

kayg referenced this pull request

2026-10-13 16:09:13 +00:00

Notes storage and live rooms #24

kayg closed this pull request

2026-10-13 16:14:29 +00:00

kayg commented

2026-10-13 16:15:35 +00:00

Author

Owner

CI is green on db4308d.

The first two CI runs failed only in CI while everything passed locally. The likely cause is that the CI host can't start an inotify watcher (shared hosts run out of instances), so the watcher-based tests timed out. I couldn't read the job log to confirm. The second commit adds a polling fallback: without a system watcher, the notes Home compares note files' size and mtime every 5 s (tests: 100 ms) and reconciles what changed. It also moves watcher file reads off the async threads, and adds a test that forces polling mode (polling_finds_outside_changes). This matters in production too (network file systems, busy servers).

Ready for review. Not merged.

CI is green on `db4308d`. The first two CI runs failed only in CI while everything passed locally. The likely cause is that the CI host can't start an inotify watcher (shared hosts run out of instances), so the watcher-based tests timed out. I couldn't read the job log to confirm. The second commit adds a **polling fallback**: without a system watcher, the notes Home compares note files' size and mtime every 5 s (tests: 100 ms) and reconciles what changed. It also moves watcher file reads off the async threads, and adds a test that forces polling mode (`polling_finds_outside_changes`). This matters in production too (network file systems, busy servers). Ready for review. Not merged.

kayg referenced this pull request

2026-11-02 13:30:17 +00:00

AI app: chats that drive Calternal + Cals (bot assistants) — grill #839