kayg/calternal

Watch

Fork 0

Editor: make the TS and Rust Markdown converters byte-identical #87

New issue

Closed

opened 2026-09-25 12:36:46 +00:00 by kayg · 6 comments

kayg commented

2026-09-25 12:36:46 +00:00

Owner

packages/editor (owned by calternal since 2026-09-25, forked from calternal.js 118ece3d) has a TS Markdown converter; the server has a Rust one (crates/calternal-collab/src/markdown.rs). calternal.js recorded that they differ on pipe-image destinations, CRLF normalization and bare-link destination wrapping (contracts/vectors/markdown records both outputs). A mismatch means opening and saving a note can rewrite bytes the user did not touch. Make both produce identical bytes for every vector and for property-based random documents.

kayg commented

2026-09-25 12:37:51 +00:00

Author

Owner

Starting editor parity work on branch job/editor-parity from dev at 490970d66e. I am reading the shared Markdown vectors and both converter implementations before selecting canonical output.

Starting editor parity work on branch job/editor-parity from dev at 490970d66e2493a8a63307ccdf92bba680a4c48e. I am reading the shared Markdown vectors and both converter implementations before selecting canonical output.

kayg commented

2026-09-25 12:48:01 +00:00

Author

Owner

Finding: all 39 existing vectors pass through the live Rust converter and the editor/Yjs bridge, but the root fixture still carried separate expectedCoreCanonicalMarkdown bytes for pipe-image destinations, CRLF, and safe bare-link destinations. Those bytes describe a different legacy canonicalizer; crates/calternal-collab/src/markdown.rs already emits the editor's angle-wrapped destinations and LF for CRLF. I also found a remaining newline mismatch: packages/editor normalizes lone CR to LF, while the Rust reader only normalized CRLF. I am unifying the fixture on serializedMarkdown and making the Rust reader normalize lone CR too.

kayg commented

2026-09-25 13:19:20 +00:00

Author

Owner

Additional adversarial evidence: bash tests/adversarial/run.sh reported and , then exited 1 because tests/adversarial/restart.mjs reported after a graceful restart. Its baseline body contains leading and trailing blank lines. This appears to be edge-whitespace loss during room rebuild; I did not change collab session/room code, per this job's scope. I will include it as a known gap.

kayg commented

2026-09-25 13:19:48 +00:00

Author

Owner

Additional adversarial evidence: bash tests/adversarial/run.sh reported ==== FINDINGS 0 and ==== ROUND 2 FINDINGS 0, then exited 1 because tests/adversarial/restart.mjs reported restart 1 duplicate: body changed: 44 -> 42 bytes after a graceful restart. Its baseline body contains leading and trailing blank lines. This appears to be edge-whitespace loss during room rebuild; I did not change collab session/room code, per this job's scope. I will include it as a known gap.

Additional adversarial evidence: `bash tests/adversarial/run.sh` reported `==== FINDINGS 0` and `==== ROUND 2 FINDINGS 0`, then exited 1 because `tests/adversarial/restart.mjs` reported `restart 1 duplicate: body changed: 44 -> 42 bytes` after a graceful restart. Its baseline body contains leading and trailing blank lines. This appears to be edge-whitespace loss during room rebuild; I did not change collab session/room code, per this job's scope. I will include it as a known gap.

kayg commented

2026-09-25 13:55:45 +00:00

Author

Owner

Resumed the stopped job at 78f5c14 (branch job/editor-parity).

Finding: the seeded test so far only generated Markdown from a small subset. I ran the editor's own schema document generator (packages/editor/test/pmDocGen.ts, 600 documents, plain words only) through both writers: the Rust writer's bytes differed from the TypeScript writer on 530 of 600 documents. Main classes:

hard breaks: TS writes   in one-line contexts (list item line, heading, table cell) and next to blank lines; Rust wrote \n everywhere, which splits a list item or table row
mark nesting order: TS opens marks by the node's mark order and keeps runs open across neighbours; Rust used a fixed rank, so **x** became **x**
links around marks: TS **[x]()**, Rust [**x**]()
star forms: TS uses ***x*** and … where * would touch **; Rust always *
tables: TS writes no separator row for a table without a header row; Rust always added one, which turns the first row into a header on reload
list items with several blocks, code spans with backticks, and several escapes also differ
mark_rank (used by to_prosemirror) does not match the editor schema's mark order (link, bold, italic, strike, underline, code, escapedWikilink, highlight)

Plan: port the TypeScript reader and writer to Rust with the same algorithm, then extend the cross-language property test with the schema document generator and the line-mix generator.

Committed cba4125: each LF after the last content line is kept as an empty paragraph, so a Note's final newline survives a save. Leading blank lines stay dropped, because the Note body from the notes plugin starts with the separator after the frontmatter. Keeping them would show an empty paragraph above the title of every Note.

Resumed the stopped job at 78f5c14 (branch job/editor-parity). Finding: the seeded test so far only generated Markdown from a small subset. I ran the editor's own schema document generator (`packages/editor/test/pmDocGen.ts`, 600 documents, plain words only) through both writers: the Rust writer's bytes differed from the TypeScript writer on 530 of 600 documents. Main classes: - hard breaks: TS writes ` ` in one-line contexts (list item line, heading, table cell) and next to blank lines; Rust wrote `\n` everywhere, which splits a list item or table row - mark nesting order: TS opens marks by the node's mark order and keeps runs open across neighbours; Rust used a fixed rank, so `**x**` became `**x**` - links around marks: TS `**[x]()**`, Rust `[**x**]()` - star forms: TS uses `***x***` and `…` where `*` would touch `**`; Rust always `*` - tables: TS writes no separator row for a table without a header row; Rust always added one, which turns the first row into a header on reload - list items with several blocks, code spans with backticks, and several escapes also differ - `mark_rank` (used by `to_prosemirror`) does not match the editor schema's mark order (link, bold, italic, strike, underline, code, escapedWikilink, highlight) Plan: port the TypeScript reader and writer to Rust with the same algorithm, then extend the cross-language property test with the schema document generator and the line-mix generator. Committed cba4125: each LF after the last content line is kept as an empty paragraph, so a Note's final newline survives a save. Leading blank lines stay dropped, because the Note body from the notes plugin starts with the separator after the frontmatter. Keeping them would show an empty paragraph above the title of every Note.

kayg referenced this issue

2026-09-25 14:02:04 +00:00

Editor: block ids on quotes and callouts need one written form that Obsidian and both converters read #90

kayg commented

2026-09-25 15:22:57 +00:00

Author

Owner

Finished on job/editor-parity, head a01165ab58. Not pushed or merged.

What changed

crates/calternal-collab/src/markdown.rs is now a step-for-step Rust port of the editor's reader and writer (packages/editor/src/markdown.ts). Before, the Rust writer's bytes differed from the editor's on 530 of 600 schema documents. It keeps bounds for untrusted input: block depth 32, list nesting 32, an inline work budget of 128 steps per byte, a linear writer pass 3, and out-of-range attributes refused.
mark_rank now follows the editor schema's mark order. The Yrs bridge (lib.rs, read_node defaults only) now fills the codeBlock language and callout kind defaults, so the server's document equals the editor's normalized document.
The property test now has three seeded families, 5,000 cases each in CI: Markdown from the Rust generator (extended), schema documents from pmDocGen.ts, and line-mix Markdown. For each case it checks that both sides build the same document, that the document survives Yrs, that both write the same bytes, and that a second round trip writes the same bytes again. CALTERNAL_MARKDOWN_CASES sets a longer run and CALTERNAL_MARKDOWN_SEED replays a seed. A local run of 20,000 per family (60,000 cases) passed.
12 new shared vectors. Decisions are recorded in contracts/vectors/markdown/README.md and DESIGN §9.

Cases where both converters rewrote a second save (fixed on both sides): text inside a plain [[…]] span was read as marks (![[a*b*c.png]] was saved back escaped); a second pipe was rejected in a wikilink alias (![[photo.jpg|300x200|left]]); a setext heading kept block syntax at its start; an anchored paragraph kept leading whitespace after a list; the writer's  after indentation did not read back as italic; a table-cell link lost its <code> text; a 200-level list overflowed the stack.

Decisions to confirm

Final newline(s) are kept as trailing empty paragraphs. Leading blank lines are still dropped: the notes plugin's body starts with the separator line after the frontmatter, and keeping it would show an empty paragraph above every title. Follow-up for the notes plugin: move that separator into the frontmatter prefix, so the blank line survives too.
Link and image destinations stay angle-wrapped. This is the existing form, and the wiki_embeds and hostile_clients tests pin it.
Mark nesting follows the schema mark order (**y** is written **y**). The schema does not store nesting.
[[1]](url) is now read as a wikilink followed by (url), as Obsidian reads it, and as the Rust reader already did.

Known gap: text that ends with [ directly before a link whose text ends with ] is written as \[[…\]](…), which reads back as an escaped wikilink. Both sides agree, but the second save differs. The CI seed does not hit it.

Gates

cargo fmt --check: exit 0
cargo clippy --workspace --all-targets -- -D warnings: Finished \dev` profile`, exit 0
cargo test --workspace: 60 suites, 989 passed, 0 failed, exit 0
packages/editor: bun run check: COMPLETED 270 FILES 0 ERRORS 0 WARNINGS 0 FILES_WITH_PROBLEMS; bun run test: Tests 384 passed (384)
apps/web: bun run check: COMPLETED 1381 FILES 0 ERRORS 0 WARNINGS 0 FILES_WITH_PROBLEMS; bun run test: Tests 244 passed (244)
bash tests/adversarial/run.sh: ==== FINDINGS 0, ==== ROUND 2 FINDINGS 0, restart probe: 0 findings (it reported a finding before this work), exit 0

Finished on job/editor-parity, head a01165ab5898696732d1c9d1d51c597fbeadd57d. Not pushed or merged. **What changed** - `crates/calternal-collab/src/markdown.rs` is now a step-for-step Rust port of the editor's reader and writer (`packages/editor/src/markdown.ts`). Before, the Rust writer's bytes differed from the editor's on 530 of 600 schema documents. It keeps bounds for untrusted input: block depth 32, list nesting 32, an inline work budget of 128 steps per byte, a linear writer pass 3, and out-of-range attributes refused. - `mark_rank` now follows the editor schema's mark order. The Yrs bridge (`lib.rs`, `read_node` defaults only) now fills the codeBlock `language` and callout `kind` defaults, so the server's document equals the editor's normalized document. - The property test now has three seeded families, 5,000 cases each in CI: Markdown from the Rust generator (extended), schema documents from `pmDocGen.ts`, and line-mix Markdown. For each case it checks that both sides build the same document, that the document survives Yrs, that both write the same bytes, and that a second round trip writes the same bytes again. `CALTERNAL_MARKDOWN_CASES` sets a longer run and `CALTERNAL_MARKDOWN_SEED` replays a seed. A local run of 20,000 per family (60,000 cases) passed. - 12 new shared vectors. Decisions are recorded in `contracts/vectors/markdown/README.md` and DESIGN §9. **Cases where both converters rewrote a second save (fixed on both sides)**: text inside a plain `[[…]]` span was read as marks (`![[a*b*c.png]]` was saved back escaped); a second pipe was rejected in a wikilink alias (`![[photo.jpg|300x200|left]]`); a setext heading kept block syntax at its start; an anchored paragraph kept leading whitespace after a list; the writer's `` after indentation did not read back as italic; a table-cell link lost its `<code>` text; a 200-level list overflowed the stack. **Decisions to confirm** 1. Final newline(s) are kept as trailing empty paragraphs. Leading blank lines are still dropped: the notes plugin's body starts with the separator line after the frontmatter, and keeping it would show an empty paragraph above every title. Follow-up for the notes plugin: move that separator into the frontmatter prefix, so the blank line survives too. 2. Link and image destinations stay angle-wrapped. This is the existing form, and the wiki_embeds and hostile_clients tests pin it. 3. Mark nesting follows the schema mark order (`**y**` is written `**y**`). The schema does not store nesting. 4. `[[1]](url)` is now read as a wikilink followed by `(url)`, as Obsidian reads it, and as the Rust reader already did. **Known gap**: text that ends with `[` directly before a link whose text ends with `]` is written as `\[[…\]](…)`, which reads back as an escaped wikilink. Both sides agree, but the second save differs. The CI seed does not hit it. **Gates** - `cargo fmt --check`: exit 0 - `cargo clippy --workspace --all-targets -- -D warnings`: `Finished \`dev\` profile`, exit 0 - `cargo test --workspace`: 60 suites, 989 passed, 0 failed, exit 0 - packages/editor: `bun run check`: `COMPLETED 270 FILES 0 ERRORS 0 WARNINGS 0 FILES_WITH_PROBLEMS`; `bun run test`: `Tests 384 passed (384)` - apps/web: `bun run check`: `COMPLETED 1381 FILES 0 ERRORS 0 WARNINGS 0 FILES_WITH_PROBLEMS`; `bun run test`: `Tests 244 passed (244)` - `bash tests/adversarial/run.sh`: `==== FINDINGS 0`, `==== ROUND 2 FINDINGS 0`, `restart probe: 0 findings` (it reported a finding before this work), exit 0

kayg referenced this issue

2026-09-25 15:23:48 +00:00

Notes: the blank line after frontmatter is dropped on the first live save #95

kayg closed this issue

2026-09-25 15:36:13 +00:00

kayg referenced this issue from a commit

2026-09-25 15:36:13 +00:00

Merge job/editor-parity: Rust Markdown converter ported from the editor; byte-identical on 15k seeded cases (#87)