kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

docs: simpler calendar backend v2 plan #802

Open
kayg wants to merge 1 commit from docs/calendar-backend-v2-plan into dev
pull from: docs/calendar-backend-v2-plan
merge into: kayg:dev
Conversation 8 Commits 1 Files changed 1 +251
kayg commented 2026-10-31 17:50:14 +00:00
Owner
Copy link

Calendar backend steps 2–4 failed three review rounds. This plan keeps #781/#783/#785 on hold and replaces their delivery approach with a calendar-only strict writer, the existing native Rust pulldown-cmark backend and independent shared TS/Rust extraction vectors. The editor remains TypeScript.

After round-2 review, the plan chooses pulldown-cmark 0.13.4 as the sole backend parser, with option/extension mapping, offset-event extraction rules and explicit ownership of Notes link exclusions, Tasks/B3 list ancestry, Composer title handling and their test oracles. No comrak migration or production behavior change is introduced.

The isolated spike under docs/plans/0067-vectors/ and packages/editor/test/calendar-0067.spike.test.ts checks 70 authored line-classification vectors against pulldown and the actual installed editor Lezer parser. On dev-02 inside the Nix shell through ~/bin/slot: 70/70 Rust and 70/70 TS pass, zero failures or normalized classification differences. CI runs the locked standalone Rust crate and discovers the TS tests in the existing editor unit-test step. Exact commands and limits are in the spike README and review reply.

Literal-autolink/emoji node differences, full metadata/relationship rows, exact UTF-8 spans and link/Composer output compatibility remain explicit expanded-corpus gates. The spike establishes line-eligibility feasibility only. The plan retains the existing-property compatibility matrix, durable last-good/unresolved recovery, bounded workers, 100k-note scale gates and risk-first delivery slices. Promotion remains default-off.

No full CI, production implementation, native sync, recovery or performance pass is claimed. No merge or fourth patch round on the held implementation stack.

Calendar backend steps 2–4 failed three review rounds. This plan keeps #781/#783/#785 on hold and replaces their delivery approach with a calendar-only strict writer, the existing native Rust pulldown-cmark backend and independent shared TS/Rust extraction vectors. The editor remains TypeScript. After round-2 review, the plan chooses pulldown-cmark 0.13.4 as the sole backend parser, with option/extension mapping, offset-event extraction rules and explicit ownership of Notes link exclusions, Tasks/B3 list ancestry, Composer title handling and their test oracles. No comrak migration or production behavior change is introduced. The isolated spike under `docs/plans/0067-vectors/` and `packages/editor/test/calendar-0067.spike.test.ts` checks 70 authored line-classification vectors against pulldown and the actual installed editor Lezer parser. On dev-02 inside the Nix shell through `~/bin/slot`: **70/70 Rust and 70/70 TS pass, zero failures or normalized classification differences**. CI runs the locked standalone Rust crate and discovers the TS tests in the existing editor unit-test step. Exact commands and limits are in the spike README and review reply. Literal-autolink/emoji node differences, full metadata/relationship rows, exact UTF-8 spans and link/Composer output compatibility remain explicit expanded-corpus gates. The spike establishes line-eligibility feasibility only. The plan retains the existing-property compatibility matrix, durable last-good/unresolved recovery, bounded workers, 100k-note scale gates and risk-first delivery slices. Promotion remains default-off. No full CI, production implementation, native sync, recovery or performance pass is claimed. No merge or fourth patch round on the held implementation stack.
0067-calendar-backend-v2.md
50 KiB
docs: plan a smaller calendar backend v2
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
292e725fa1
kayg commented 2026-10-31 17:56:14 +00:00
Author
Owner
Copy link

Plan review

verdict: rework. reviewed plan head 292e725fa101fac68321b7d4a53917fb30ba13f6, all three independent review rounds and fix reports on #781/#783/#785, ADRs 0067/0068/0056, and feedback-wasm-boundaries.md. this is a plan review, with no implementation, build, benchmark or passing-gate claim.

the writer and delivery boundaries are materially simpler. rejecting hashes and unsupported syntax before planning removes the recurring YAML comment-detection problem instead of adding another quote heuristic. original-offset exclusions shared by events, tasks, relationships and identity address the recurring code/comment visibility problem. the one-pass parent walk addresses the nested rescan bug. default-off bulk replay, preserved receipts and deferred promotion are the right containment. those are good contracts, not proof of an implementation yet.

blocking

  1. the whole-frontmatter gate changes existing safe-edit behavior, despite claiming to preserve settled decisions (lines 44–85; delivery slice 2). ADR 0068's Odd cases explicitly says “unrelated safe edits remain possible.” a valid repeating task with a nonempty map-based log would become entirely property-read-only, including unrelated priority/title changes. so would a note with hidden: ["^id"], table-width arrays, an unrelated comment, or a quoted link fragment. the plan acknowledges log is read-only but understates that every property action on that note is refused. this is broader than refusing an unsafe target and reaches existing Tasks, not just unfinished Calendar actions. keep the narrow writer initially scoped to the new supported calendar operations and preserve proven existing unrelated edits, or explicitly reconcile this behavior with the normative contract before calling it an unchanged decision. do not restore a permissive scanner fallback. add an action/fixture matrix covering ordinary notes, tracked/repeating tasks, hidden blocks and display properties, with both target and unrelated-field edits. blanket refusal is safe for bytes, but is a product regression, not merely an implementation detail.

  2. the parser choice has not earned the claim of a simpler backend (lines 106–138). a bounded embedded QuickJS bridge is technically plausible, and correctly avoids per-note processes and client-submitted trees. however, sharing Lezer eliminates implementation drift only for a pinned parser/configuration/span ABI; it adds runtime ownership, JS bundling, native runtime packaging, string/result marshalling, interruption and worker recovery. the plan makes it the chosen dependency before comparing the existing Rust option. a feasibility spike alone cannot establish that this is the simplest solution. revise the parser decision as recommended below, or require a side-by-side bridge/native comparison before choosing and before switching existing Tasks.

  3. “suppress exports” on parser failure needs a durable state contract (lines 132–138; failure table). retain bindings alone does not say what happens to old task/event/relationship rows, cached DAV objects, sync tokens or pending work when a new revision cannot be parsed. removing a resource from a projection can itself become a remote deletion. specify a persisted successful/unresolved result with source revision and parser version; failed extraction must not commit an authoritative empty replacement or emit deletion/outcome/identity work. retain the last accepted published generation without publishing a changed interpretation; cold failures with no prior generation stay explicitly unresolved. gate management of stale targets. test timeout/OOM after a previously successful revision, restart, deletion/rebuild of the derived index, parser upgrade, then successful retry. state where accepted publication/binding state survives an index rebuild. apply the policy to Tasks as well as events. the failure table's “no false tombstones” intent is correct but not yet an implementable recovery contract.

parser recommendation

option concrete tradeoff recommendation
embedded Lezer/QuickJS one JS extraction implementation; still copies/decodes source, builds a UTF offset map, walks/serializes spans and validates returned data. startup, JS heap, host allocations and worker queues all cost. pin and ship the bundle/runtime for Linux x86_64/aarch64 and macOS; no Node service is needed at runtime. exceptions, interrupts and OOM need worker reset; native process failure is not contained by disabling filesystem/network. viable only with a demonstrated end-to-end advantage over the native option; not the default simplification.
Rust CommonMark parser, preferably the already-used pulldown-cmark native byte offsets, no JS deployment or runtime/string bridge. comrak is another candidate if the required container/source-position API warrants its extra structure. neither is automatically identical to Lezer, and source-range fidelity must be checked. choose this, with one explicit extraction contract implemented against Rust parser events and the editor's Lezer tree, and shared independent expected vectors run against both.
narrow backend line grammar alone cheap for positively recognized lines, but cannot know a bare 10:00 is inside a multiline backtick span, lazy list continuation or inline comment without surrounding Markdown structure. refusing all uncertain context either recreates a scanner or drops valid bare/nested/quoted-checkbox cases. use it only inside parser-approved spans, or as a deliberately smaller temporary read-only slice. not a substitute for structural exclusions.

for the Rust choice, specify exactly the same block/inline eligibility, code/HTML/comment exclusions, direct-child relationship rules, paragraph segmentation, metadata rules and original source spans as the TS editor. share independent expected extraction spans/rows, not entire implementation-specific syntax trees. include all nine review probes and adversarial container combinations. gate both implementations and parser upgrades on the corpus; unresolved parity cases refuse dependent management rather than guess. one immutable Rust parsed view supplies all backend consumers, with no masked reparse or separate child scanner. two implementations with executable parity is compatible with kayg's explicit boundary decision.

this proposal does not introduce client WASM, so it does not literally violate the no-Rust-to-WASM rule. equally, that rule does not justify embedding JS in Rust to force a single implementation: it explicitly names shared goldens as the drift control. per-keystroke work stays TS; any shared editor span walk must preserve incremental/bounded editor behavior rather than rebuilding a full-note view on each keystroke.

should change

  1. make the write grammar executable on both input and output. its lexical restrictions are strict enough to exclude the demonstrated comment-loss class. require exact output validation too: creating absent frontmatter or writing a new string containing a newline, quote or hash must not emit unsafe YAML or immediately strand the note outside the subset. spell out final-newline/empty-envelope handling, mapping/list indentation, key typing and integer range/overflow; JSON syntax alone does not settle YAML decoder differences. verify independent YAML 1.2 typed meaning. define overlapping batch targets, ancestor removal and conflicting changes as deterministic refusal or a single specified result. count changes/output bytes in the complexity bound. predeclare allowed changed spans in fixtures, including ancestor lines, independently of production span selection. supplement the already-good byte oracle with punctuation/escape/duplicate-key mutations and an accepted-edit semantic model that does not call production helpers. preserve refusal as exact whole-file bytes with no claims/destination/history side effects; test all property entrypoints.

  2. turn the scale section into falsifiable acceptance criteria. the sizes and doubling/nesting tests are a substantial improvement. keep O(N+B+L) as an extraction target, not a proof about Lezer, and account separately for database inserts/indexes, bridge copies/result size, offset maps and queueing. define worker count, bounded queue/backpressure, total RSS/native stack limits and restart behavior; 64 MiB of JS heap does not bound Rust buffers or process RSS. run parsing outside long-held database/write locks and test that a timeout storm leaves other notes/users responsive. specify absolute startup/cold-index time and the baseline distribution, bytes and nesting of the 100k-note Home. at 0.2 ms extra per note, 100k notes already represent 20 seconds of CPU before other indexing work; at 50 ms each, a pathological corpus consumes 5,000 seconds on one worker. the 50 ms ceiling is containment only, as the plan says. include full source-in/result-out bridge cost, cold/warm runs, concurrent foreground edits, normal-fixture extraction success counts and unresolved counts. rejecting most fixtures must never count as passing the latency/cold-index gates. measure Tasks' total describe regression separately from the extra-event budget.

  3. slice by risk before broad migration. the containment PR first is right. move the parser/contract comparison ahead of the Tasks-wide switch; separate read-only events from changing existing task/link semantics, each with independently expected rows and transaction-before-journal evidence. slice 7 combines permanent claim inventory, allocation, concurrency, crash recovery and undo: split durable claim inventory/readiness from enabling identity. list read-only rebuild/upgrade/failure gates per slice and later feature gates separately. retain the real process-kill/new-key tests, native sync gates and default-off promotion. no fourth patch round on the held stack.

nits and settled decisions

  • clarify that [] is the explicit empty-container exception, and that event keys use lowercase Crockford, matching 0056.
  • keep “editor parser” precise: the source does install markdownLanguage; its current blockAt helper is not itself the proposed pure shared extraction walk.
  • frontmatter authority, 16-character IDs/legacy reads, invisible markers, server allocation authority, permanent claims, UUID-bound links and deferred promotion otherwise respect kayg's decisions. refusing ambiguous outside copies is the safe interim state; it is not delivery of 0056's eventual proven-copy rekey behavior.

no code pushed or changed. recommendation: retain the narrow writer/containment direction, reconcile the unrelated-edit regression, choose the native Rust parser plus shared TS/Rust vectors, and make unresolved extraction recovery explicit before implementation.

# Plan review **verdict: rework.** reviewed plan head `292e725fa101fac68321b7d4a53917fb30ba13f6`, all three independent review rounds and fix reports on #781/#783/#785, ADRs 0067/0068/0056, and `feedback-wasm-boundaries.md`. this is a plan review, with no implementation, build, benchmark or passing-gate claim. the writer and delivery boundaries are materially simpler. rejecting hashes and unsupported syntax before planning removes the recurring YAML comment-detection problem instead of adding another quote heuristic. original-offset exclusions shared by events, tasks, relationships and identity address the recurring code/comment visibility problem. the one-pass parent walk addresses the nested rescan bug. default-off bulk replay, preserved receipts and deferred promotion are the right containment. those are good contracts, not proof of an implementation yet. ## blocking 1. **the whole-frontmatter gate changes existing safe-edit behavior, despite claiming to preserve settled decisions** (lines 44–85; delivery slice 2). ADR 0068's Odd cases explicitly says “unrelated safe edits remain possible.” a valid repeating task with a nonempty map-based `log` would become entirely property-read-only, including unrelated priority/title changes. so would a note with `hidden: ["^id"]`, table-width arrays, an unrelated comment, or a quoted link fragment. the plan acknowledges `log` is read-only but understates that *every* property action on that note is refused. this is broader than refusing an unsafe target and reaches existing Tasks, not just unfinished Calendar actions. keep the narrow writer initially scoped to the new supported calendar operations and preserve proven existing unrelated edits, or explicitly reconcile this behavior with the normative contract before calling it an unchanged decision. do not restore a permissive scanner fallback. add an action/fixture matrix covering ordinary notes, tracked/repeating tasks, hidden blocks and display properties, with both target and unrelated-field edits. blanket refusal is safe for bytes, but is a product regression, not merely an implementation detail. 2. **the parser choice has not earned the claim of a simpler backend** (lines 106–138). a bounded embedded QuickJS bridge is technically plausible, and correctly avoids per-note processes and client-submitted trees. however, sharing Lezer eliminates implementation drift only for a pinned parser/configuration/span ABI; it adds runtime ownership, JS bundling, native runtime packaging, string/result marshalling, interruption and worker recovery. the plan makes it the chosen dependency before comparing the existing Rust option. a feasibility spike alone cannot establish that this is the simplest solution. revise the parser decision as recommended below, or require a side-by-side bridge/native comparison before choosing and before switching existing Tasks. 3. **“suppress exports” on parser failure needs a durable state contract** (lines 132–138; failure table). retain bindings alone does not say what happens to old task/event/relationship rows, cached DAV objects, sync tokens or pending work when a new revision cannot be parsed. removing a resource from a projection can itself become a remote deletion. specify a persisted successful/unresolved result with source revision and parser version; failed extraction must not commit an authoritative empty replacement or emit deletion/outcome/identity work. retain the last accepted published generation without publishing a changed interpretation; cold failures with no prior generation stay explicitly unresolved. gate management of stale targets. test timeout/OOM after a previously successful revision, restart, deletion/rebuild of the derived index, parser upgrade, then successful retry. state where accepted publication/binding state survives an index rebuild. apply the policy to Tasks as well as events. the failure table's “no false tombstones” intent is correct but not yet an implementable recovery contract. ## parser recommendation | option | concrete tradeoff | recommendation | | --- | --- | --- | | embedded Lezer/QuickJS | one JS extraction implementation; still copies/decodes source, builds a UTF offset map, walks/serializes spans and validates returned data. startup, JS heap, host allocations and worker queues all cost. pin and ship the bundle/runtime for Linux x86_64/aarch64 and macOS; no Node service is needed at runtime. exceptions, interrupts and OOM need worker reset; native process failure is not contained by disabling filesystem/network. | viable only with a demonstrated end-to-end advantage over the native option; not the default simplification. | | Rust CommonMark parser, preferably the already-used pulldown-cmark | native byte offsets, no JS deployment or runtime/string bridge. comrak is another candidate if the required container/source-position API warrants its extra structure. neither is automatically identical to Lezer, and source-range fidelity must be checked. | **choose this**, with one explicit extraction contract implemented against Rust parser events and the editor's Lezer tree, and shared independent expected vectors run against both. | | narrow backend line grammar alone | cheap for positively recognized lines, but cannot know a bare `10:00` is inside a multiline backtick span, lazy list continuation or inline comment without surrounding Markdown structure. refusing all uncertain context either recreates a scanner or drops valid bare/nested/quoted-checkbox cases. | use it only *inside parser-approved spans*, or as a deliberately smaller temporary read-only slice. not a substitute for structural exclusions. | for the Rust choice, specify exactly the same block/inline eligibility, code/HTML/comment exclusions, direct-child relationship rules, paragraph segmentation, metadata rules and original source spans as the TS editor. share independent expected extraction spans/rows, not entire implementation-specific syntax trees. include all nine review probes and adversarial container combinations. gate both implementations and parser upgrades on the corpus; unresolved parity cases refuse dependent management rather than guess. one immutable Rust parsed view supplies all backend consumers, with no masked reparse or separate child scanner. two implementations with executable parity is compatible with kayg's explicit boundary decision. this proposal does **not** introduce client WASM, so it does not literally violate the no-Rust-to-WASM rule. equally, that rule does not justify embedding JS in Rust to force a single implementation: it explicitly names shared goldens as the drift control. per-keystroke work stays TS; any shared editor span walk must preserve incremental/bounded editor behavior rather than rebuilding a full-note view on each keystroke. ## should change 4. **make the write grammar executable on both input and output.** its lexical restrictions are strict enough to exclude the demonstrated comment-loss class. require exact output validation too: creating absent frontmatter or writing a new string containing a newline, quote or hash must not emit unsafe YAML or immediately strand the note outside the subset. spell out final-newline/empty-envelope handling, mapping/list indentation, key typing and integer range/overflow; JSON syntax alone does not settle YAML decoder differences. verify independent YAML 1.2 typed meaning. define overlapping batch targets, ancestor removal and conflicting changes as deterministic refusal or a single specified result. count changes/output bytes in the complexity bound. predeclare allowed changed spans in fixtures, including ancestor lines, independently of production span selection. supplement the already-good byte oracle with punctuation/escape/duplicate-key mutations and an accepted-edit semantic model that does not call production helpers. preserve refusal as exact whole-file bytes with no claims/destination/history side effects; test all property entrypoints. 5. **turn the scale section into falsifiable acceptance criteria.** the sizes and doubling/nesting tests are a substantial improvement. keep `O(N+B+L)` as an extraction target, not a proof about Lezer, and account separately for database inserts/indexes, bridge copies/result size, offset maps and queueing. define worker count, bounded queue/backpressure, total RSS/native stack limits and restart behavior; 64 MiB of JS heap does not bound Rust buffers or process RSS. run parsing outside long-held database/write locks and test that a timeout storm leaves other notes/users responsive. specify absolute startup/cold-index time and the baseline distribution, bytes and nesting of the 100k-note Home. at 0.2 ms extra per note, 100k notes already represent 20 seconds of CPU before other indexing work; at 50 ms each, a pathological corpus consumes 5,000 seconds on one worker. the 50 ms ceiling is containment only, as the plan says. include full source-in/result-out bridge cost, cold/warm runs, concurrent foreground edits, normal-fixture extraction success counts and unresolved counts. rejecting most fixtures must never count as passing the latency/cold-index gates. measure Tasks' total describe regression separately from the extra-event budget. 6. **slice by risk before broad migration.** the containment PR first is right. move the parser/contract comparison ahead of the Tasks-wide switch; separate read-only events from changing existing task/link semantics, each with independently expected rows and transaction-before-journal evidence. slice 7 combines permanent claim inventory, allocation, concurrency, crash recovery and undo: split durable claim inventory/readiness from enabling identity. list read-only rebuild/upgrade/failure gates per slice and later feature gates separately. retain the real process-kill/new-key tests, native sync gates and default-off promotion. no fourth patch round on the held stack. ## nits and settled decisions - clarify that `[]` is the explicit empty-container exception, and that event keys use lowercase Crockford, matching 0056. - keep “editor parser” precise: the source does install `markdownLanguage`; its current `blockAt` helper is not itself the proposed pure shared extraction walk. - frontmatter authority, 16-character IDs/legacy reads, invisible markers, server allocation authority, permanent claims, UUID-bound links and deferred promotion otherwise respect kayg's decisions. refusing ambiguous outside copies is the safe interim state; it is not delivery of 0056's eventual proven-copy rekey behavior. no code pushed or changed. recommendation: retain the narrow writer/containment direction, reconcile the unrelated-edit regression, choose the native Rust parser plus shared TS/Rust vectors, and make unresolved extraction recovery explicit before implementation.
kayg commented 2026-10-31 18:03:42 +00:00
Author
Owner
Copy link

reworked the plan in 45c211f5 on the same branch. docs only; the held implementation stack stays held.

review finding change in the plan
blocker 1: whole-frontmatter gate regresses unrelated edits limited the new API to calendar operations, with no fallback and no replacement of generic Props/capture. inventoried every production props writer caller, merge path and adjacent pin/stamp/editor text route at 133cf2c2. added target/unrelated-field fixtures for map-list log, repeating tasks, hidden/table-width arrays, comments and link fragments. existing non-calendar actions gain no new refusal; calendar-only limits are explicitly user-visible.
blocker 2: Lezer/QuickJS adds backend complexity removed the bridge. audited locked lang-markdown 6.5.2 / Lezer 1.8.0: GFM tables/tasks/strikethrough/autolinks plus subscript/superscript/emoji. compared current pulldown-cmark 0.13.6 and comrak 0.59.0 against that inventory, cited upstream docs and checked registry versions. chose pinned native comrak for its complete extension options; source-range and behavioral parity remain mandatory, not assumed. editor stays TS with incremental tree work, no WASM.
parser extraction recommendation specified containers, paragraph segmentation, direct-child relationships, code/HTML/comment exclusions, metadata and original spans. one immutable Rust view, no masked reparse or child scanner. shared independent expected JSON vectors run in required CI against BOTH production editor Lezer and Rust comrak adapters; upgrades must pass both. listed all nine concrete review probes and positive/adversarial cases.
blocker 3: parser failure lacks durable publication state added a backed-up private accepted/unresolved ledger with source revision, parser/config/contract version, rows/snapshots, published generation, UUID bindings, wire bytes/ETags/tokens and pending bases. retain last-good Tasks/events/relationships and DAV state; no empty replacement, deletion, identity or outcome work. cold failures stay unresolved; stale management gates. specified crash replay, restart/index hydration, shadow upgrades, quiet indicator and retry policy.
error, timeout, huge-note and invalid-UTF-8 recovery explicit individual failure rows/tests, no source repair or lossy writeback. quarantine invalid-byte revisions from flush/stamping until explicit correction. each consumer must pass good → failure → restart → derived-index deletion/rebuild → failed upgrade → successful retry, plus cold failures and ledger/index/journal crash boundaries.
should-change 4: executable input/output grammar exact envelope/final-newline behavior, indentation, key typing, i64 bounds, escaped output validation and independent YAML 1.2 meaning; deterministic overlapping/ancestor/batch refusal. independent predeclared byte spans and accepted-edit model; refusal has zero claims/destination/history effects. complexity includes change sorting and output bytes.
should-change 5: falsifiable scale criteria corrected byte-sized 100k-note distribution (925,696,000 bytes), depths and expected row/success counts; cold index ≤120s and ≤10% growth, startup bounds, separate Tasks/event budgets, per-note reparse cost and 20s/5,000s CPU examples. bounded two-process native pool, queue/backpressure/fairness, OS RSS/stack/deadline containment, IPC costs, timeout storms, foreground responsiveness and no-lock parsing. rejecting normal fixtures cannot pass. these are proposed gates, not measured results.
should-change 6: risk-first slices containment → parser/contract parity → recovery ledger → read-only events → separate Tasks and relationship migrations. separate calendar writer, claim inventory/readiness, UI hiding and identity enablement; promotion/settlement later. per-slice read-only rebuild/upgrade/failure/publication-order gates stay separate from mutation/native-sync gates.
nits / settled decisions [] is explicitly the empty-container exception; event keys use lowercase Crockford. distinguished installed editor language from blockAt. frontmatter authority, server IDs, private permanent claims, hidden markers, UUID links and default-off promotion stay; ambiguous-copy refusal is explicitly not eventual 0056 rekey delivery.

validation: git diff --cached --check passed before commit; all relative document links exist; fixture-size arithmetic checked; only docs/plans/0067-calendar-backend-v2.md changed. no build, test, native sync or performance run, consistent with this docs-only task and the control-host rule. parser parity, process-pool cost/platform limits and recovery-schema implementation remain delivery gates. no merge or fourth implementation patch round.

reworked the plan in `45c211f5` on the same branch. docs only; the held implementation stack stays held. | review finding | change in the plan | | --- | --- | | blocker 1: whole-frontmatter gate regresses unrelated edits | limited the new API to calendar operations, with no fallback and no replacement of generic Props/capture. inventoried every production props writer caller, merge path and adjacent pin/stamp/editor text route at `133cf2c2`. added target/unrelated-field fixtures for map-list log, repeating tasks, hidden/table-width arrays, comments and link fragments. existing non-calendar actions gain no new refusal; calendar-only limits are explicitly user-visible. | | blocker 2: Lezer/QuickJS adds backend complexity | removed the bridge. audited locked lang-markdown 6.5.2 / Lezer 1.8.0: GFM tables/tasks/strikethrough/autolinks plus subscript/superscript/emoji. compared current pulldown-cmark 0.13.6 and comrak 0.59.0 against that inventory, cited upstream docs and checked registry versions. chose pinned native comrak for its complete extension options; source-range and behavioral parity remain mandatory, not assumed. editor stays TS with incremental tree work, no WASM. | | parser extraction recommendation | specified containers, paragraph segmentation, direct-child relationships, code/HTML/comment exclusions, metadata and original spans. one immutable Rust view, no masked reparse or child scanner. shared independent expected JSON vectors run in required CI against BOTH production editor Lezer and Rust comrak adapters; upgrades must pass both. listed all nine concrete review probes and positive/adversarial cases. | | blocker 3: parser failure lacks durable publication state | added a backed-up private accepted/unresolved ledger with source revision, parser/config/contract version, rows/snapshots, published generation, UUID bindings, wire bytes/ETags/tokens and pending bases. retain last-good Tasks/events/relationships and DAV state; no empty replacement, deletion, identity or outcome work. cold failures stay unresolved; stale management gates. specified crash replay, restart/index hydration, shadow upgrades, quiet indicator and retry policy. | | error, timeout, huge-note and invalid-UTF-8 recovery | explicit individual failure rows/tests, no source repair or lossy writeback. quarantine invalid-byte revisions from flush/stamping until explicit correction. each consumer must pass good → failure → restart → derived-index deletion/rebuild → failed upgrade → successful retry, plus cold failures and ledger/index/journal crash boundaries. | | should-change 4: executable input/output grammar | exact envelope/final-newline behavior, indentation, key typing, i64 bounds, escaped output validation and independent YAML 1.2 meaning; deterministic overlapping/ancestor/batch refusal. independent predeclared byte spans and accepted-edit model; refusal has zero claims/destination/history effects. complexity includes change sorting and output bytes. | | should-change 5: falsifiable scale criteria | corrected byte-sized 100k-note distribution (925,696,000 bytes), depths and expected row/success counts; cold index ≤120s and ≤10% growth, startup bounds, separate Tasks/event budgets, per-note reparse cost and 20s/5,000s CPU examples. bounded two-process native pool, queue/backpressure/fairness, OS RSS/stack/deadline containment, IPC costs, timeout storms, foreground responsiveness and no-lock parsing. rejecting normal fixtures cannot pass. these are proposed gates, not measured results. | | should-change 6: risk-first slices | containment → parser/contract parity → recovery ledger → read-only events → separate Tasks and relationship migrations. separate calendar writer, claim inventory/readiness, UI hiding and identity enablement; promotion/settlement later. per-slice read-only rebuild/upgrade/failure/publication-order gates stay separate from mutation/native-sync gates. | | nits / settled decisions | `[]` is explicitly the empty-container exception; event keys use lowercase Crockford. distinguished installed editor language from `blockAt`. frontmatter authority, server IDs, private permanent claims, hidden markers, UUID links and default-off promotion stay; ambiguous-copy refusal is explicitly not eventual 0056 rekey delivery. | validation: `git diff --cached --check` passed before commit; all relative document links exist; fixture-size arithmetic checked; only `docs/plans/0067-calendar-backend-v2.md` changed. no build, test, native sync or performance run, consistent with this docs-only task and the control-host rule. parser parity, process-pool cost/platform limits and recovery-schema implementation remain delivery gates. no merge or fourth implementation patch round.
kayg commented 2026-10-31 18:08:42 +00:00
Author
Owner
Copy link

Plan review, round 2

verdict: rework. reviewed 45c211f51113c7698e318e96203b15b9232a70c6, the first review/revision comments, current property/parser routes at the plan base, the held containment boundary, and the published Lezer 1.8.0/lang-markdown 6.5.2 and comrak 0.59.0 sources. docs review only; no implementation tests or performance gates run, no code pushed.

blocking

  1. commit to one backend Markdown parser, including existing consumers (docs/plans/0067-calendar-backend-v2.md:180–185, 471–486). the explicit permission for existing pulldown to remain “until their separate migration” leaves two backend rule sets with no complete removal gate. both crates/notes/Cargo.toml:17 and crates/tasks/Cargo.toml:12 depend on pulldown-cmark 0.13.4. production uses are:

    • notes/src/links.rs:66: fenced/indented code exclusion ranges for note links;
    • tasks/src/found.rs:257: real list-item ancestry for checkbox extraction, reused by Tasks B3;
    • tasks/src/create.rs:38: composer title link-to-label conversion.

    the last route is not covered by “Tasks then relationships” migration as written; parser-based test oracles also need updating (notes/src/links.rs, tasks/src/found/tests.rs). either choose pulldown if the shared editor/backend extraction vectors establish the required parity, or justify comrak and explicitly migrate every pulldown use, remove both dependencies, and name the delivery gate that enforces this. a temporary isolated comparison adapter is reasonable; shipping new production consumers on comrak while leaving the old ones indefinitely on pulldown is not. retain one pinned backend configuration and one shared vector suite with the editor, extended for link exclusion and composer title behavior. do not infer that missing rendered emoji support alone makes pulldown unsuitable for a source-preserving extraction contract.

first-round blockers

  • unrelated property edits: resolved at plan level (lines 40–109). the separate calendar API, no rerouting of generic Props/capture/merge/pin/stamps/display edits, and target/unrelated-field matrix address the regression. the inventory matches the inspected routes. map-list logs, hidden/table-width arrays, comments and fragment backlinks explicitly retain existing safe edits. capturing independently checked base results before routing changes makes this falsifiable. this is a preservation contract, not proof that every legacy edit is safe; the plan correctly says so.
  • unjustified JS bridge: resolved. no runtime/bundle/UTF marshalling bridge remains. native parsing plus independently expected TS/Rust vectors is a defensible direction. parser consolidation above remains blocking.
  • parser-failure recovery: resolved at plan level (lines 262–322). the backed-up ledger outside the derived index specifies revision/config identity, accepted rows and wire bases, cold unresolved state, stale mutation admission, crash replay and index hydration. failure cannot publish an empty replacement or tombstones. the per-consumer good → failure → restart → index loss → failed upgrade → successful retry sequence covers the previously missing contract. schema/crash correctness remains an implementation gate.

should change

  1. make the concrete extension differences explicit in the corpus, rather than equating option names. the editor really installs markdownLanguage, whose published configuration includes GFM, subscript, superscript and emoji. comrak has the broad capabilities, including CommonMark multiline code spans and HTML comments, but exact extraction parity is still unproven. for example, Lezer's Emoji parser accepts any [a-zA-Z_0-9]+ name, including unknown names; comrak shortcodes use emojis::get_by_shortcode. preserving literal source can normalize this harmlessly, but the plan should say how and test it. include unknown/uppercase emoji, sub/superscript whitespace/escapes and link nesting, and autolink label boundaries where metadata must be excluded. none requires identical ASTs.

    the existing vector design is strong: both production adapters independently compare spans/slices, rows, parents and unresolved reasons to the same authored expectations, then compare normalized results; upgrades run both required jobs. keep its positive controls and all nine probes. explicitly instantiate matched/unmatched and multi-backtick spans crossing physical lines in paragraphs/list/quote containers, plus inline/block and multiline HTML comments with visible adjacent prose. these checks catch drift in exclusions and offsets rather than merely confirming matching parser names. the stop-before-migration gate is essential; passing tests are not yet claimed.

  2. spell out existing checkbox compatibility before the Tasks switch. tasks/src/found.rs:60–63, 255–276, 325–352 intentionally recognizes doing/cancelled/later and unknown single-character boxes, including empty boxes at end of line. Lezer's TaskMarker recognizes only space/x/X followed by space/tab; comrak with relaxed matching off also does not classify all current boxes as task items. extract the app's supported box grammar from parser-approved list/paragraph source spans in both adapters, not only TaskMarker/TaskItem nodes. add positive expected rows for [ ], [x], [X], [/], [-], [>], [?] and end-of-line boxes. reconcile “unknown custom boxes refuse classification” (line 217) with today's lenient task reader explicitly, so the property compatibility fix does not conceal a later task-reading regression.

slice 1 and nits

  • slice 1 is small and safe as specified: bulk execution deferral plus production replay tests, preserving plans, partial owners, receipts and restart/undo behavior. the held implementation already has the single bulk::drive boundary suitable for a narrow port. require ordinary note identity/non-calendar bulk jobs to remain executable, and deferred calendar replay/undo/direct calls to perform zero additional writes. no parser, writer, ledger, allocation or UI enablement belongs in this slice.
  • nits: none worth holding this plan for. the scale limits and later risk separation are materially clearer; they remain proposed gates, not measured evidence.

rework the parser ownership/migration decision before implementation. the separate calendar writer and durable recovery direction can stay.

# Plan review, round 2 **verdict: rework.** reviewed `45c211f51113c7698e318e96203b15b9232a70c6`, the first review/revision comments, current property/parser routes at the plan base, the held containment boundary, and the published Lezer 1.8.0/lang-markdown 6.5.2 and comrak 0.59.0 sources. docs review only; no implementation tests or performance gates run, no code pushed. ## blocking 1. **commit to one backend Markdown parser, including existing consumers** (`docs/plans/0067-calendar-backend-v2.md:180–185, 471–486`). the explicit permission for existing pulldown to remain “until their separate migration” leaves two backend rule sets with no complete removal gate. both `crates/notes/Cargo.toml:17` and `crates/tasks/Cargo.toml:12` depend on `pulldown-cmark` 0.13.4. production uses are: - `notes/src/links.rs:66`: fenced/indented code exclusion ranges for note links; - `tasks/src/found.rs:257`: real list-item ancestry for checkbox extraction, reused by Tasks B3; - `tasks/src/create.rs:38`: composer title link-to-label conversion. the last route is not covered by “Tasks then relationships” migration as written; parser-based test oracles also need updating (`notes/src/links.rs`, `tasks/src/found/tests.rs`). **either choose pulldown if the shared editor/backend extraction vectors establish the required parity, or justify comrak and explicitly migrate every pulldown use, remove both dependencies, and name the delivery gate that enforces this.** a temporary isolated comparison adapter is reasonable; shipping new production consumers on comrak while leaving the old ones indefinitely on pulldown is not. retain one pinned backend configuration and one shared vector suite with the editor, extended for link exclusion and composer title behavior. do not infer that missing rendered emoji support alone makes pulldown unsuitable for a source-preserving extraction contract. ## first-round blockers - **unrelated property edits: resolved at plan level** (lines 40–109). the separate calendar API, no rerouting of generic Props/capture/merge/pin/stamps/display edits, and target/unrelated-field matrix address the regression. the inventory matches the inspected routes. map-list logs, hidden/table-width arrays, comments and fragment backlinks explicitly retain existing safe edits. capturing independently checked base results before routing changes makes this falsifiable. this is a preservation contract, not proof that every legacy edit is safe; the plan correctly says so. - **unjustified JS bridge: resolved.** no runtime/bundle/UTF marshalling bridge remains. native parsing plus independently expected TS/Rust vectors is a defensible direction. parser consolidation above remains blocking. - **parser-failure recovery: resolved at plan level** (lines 262–322). the backed-up ledger outside the derived index specifies revision/config identity, accepted rows and wire bases, cold unresolved state, stale mutation admission, crash replay and index hydration. failure cannot publish an empty replacement or tombstones. the per-consumer good → failure → restart → index loss → failed upgrade → successful retry sequence covers the previously missing contract. schema/crash correctness remains an implementation gate. ## should change 2. **make the concrete extension differences explicit in the corpus, rather than equating option names.** the editor really installs `markdownLanguage`, whose published configuration includes GFM, subscript, superscript and emoji. comrak has the broad capabilities, including CommonMark multiline code spans and HTML comments, but exact extraction parity is still unproven. for example, Lezer's Emoji parser accepts any `[a-zA-Z_0-9]+` name, including unknown names; comrak shortcodes use `emojis::get_by_shortcode`. preserving literal source can normalize this harmlessly, but the plan should say how and test it. include unknown/uppercase emoji, sub/superscript whitespace/escapes and link nesting, and autolink label boundaries where metadata must be excluded. none requires identical ASTs. the existing vector design is strong: both **production** adapters independently compare spans/slices, rows, parents and unresolved reasons to the same authored expectations, then compare normalized results; upgrades run both required jobs. keep its positive controls and all nine probes. explicitly instantiate matched/unmatched and multi-backtick spans crossing physical lines in paragraphs/list/quote containers, plus inline/block and multiline HTML comments with visible adjacent prose. these checks catch drift in exclusions and offsets rather than merely confirming matching parser names. the stop-before-migration gate is essential; passing tests are not yet claimed. 3. **spell out existing checkbox compatibility before the Tasks switch.** `tasks/src/found.rs:60–63, 255–276, 325–352` intentionally recognizes doing/cancelled/later and unknown single-character boxes, including empty boxes at end of line. Lezer's TaskMarker recognizes only space/x/X followed by space/tab; comrak with relaxed matching off also does not classify all current boxes as task items. extract the app's supported box grammar from parser-approved list/paragraph source spans in both adapters, not only TaskMarker/TaskItem nodes. add positive expected rows for `[ ]`, `[x]`, `[X]`, `[/]`, `[-]`, `[>]`, `[?]` and end-of-line boxes. reconcile “unknown custom boxes refuse classification” (line 217) with today's lenient task reader explicitly, so the property compatibility fix does not conceal a later task-reading regression. ## slice 1 and nits - **slice 1 is small and safe as specified:** bulk execution deferral plus production replay tests, preserving plans, partial owners, receipts and restart/undo behavior. the held implementation already has the single `bulk::drive` boundary suitable for a narrow port. require ordinary note identity/non-calendar bulk jobs to remain executable, and deferred calendar replay/undo/direct calls to perform zero additional writes. no parser, writer, ledger, allocation or UI enablement belongs in this slice. - **nits:** none worth holding this plan for. the scale limits and later risk separation are materially clearer; they remain proposed gates, not measured evidence. rework the parser ownership/migration decision before implementation. the separate calendar writer and durable recovery direction can stay.
kayg commented 2026-10-31 18:18:06 +00:00
Author
Owner
Copy link

updated in c19747b2be60cdcd6a24c929127cdf181526c3c3.

round-2 parser decision: retain pulldown-cmark 0.13.4, already used by the backend. no comrak dependency or parser migration, and no change to Tasks B3 (#798) or Composer behavior in this spike.

updated the plan's option mapping and offset-event extraction contract. the single-parser ownership inventory explicitly includes Notes link-code exclusions, Tasks list ancestry/B3, Composer link-to-label conversion and both existing parser-based test oracles. the later shared-view/configuration change must preserve their independently expected outputs.

dev-02 spike: Rust 70/70 and editor Lezer 70/70 vectors pass; 0 failures and 0 normalized line-classification differences. both independently compare to the same authored JSON expectations, using the existing pulldown version and the actual installed markdownLanguage.parser (lang-markdown 6.5.2 / Lezer 1.8.0). no expected output was generated from either parser. includes positive and negative custom/EOL/Unicode task boxes, matched/unmatched multiline single/multi-backticks in paragraph/list/quote containers, fenced/indented code, inline/block/multiline HTML comments with adjacent visible prose, nested lists/quotes, frontmatter boundaries, CRLF, non-BMP Unicode and extension prose. the editor does not install Obsidian %% comments, so those bytes intentionally leave enclosed candidates visible.

reproducible commands, run inside the Nix shell on calternal-dev-02-std, with the slot wrapper:

~/bin/slot cargo test -j4 --locked --manifest-path docs/plans/0067-vectors/rust/Cargo.toml -- --nocapture
(cd packages/editor && ~/bin/slot bunx vitest run test/calendar-0067.spike.test.ts)
(cd packages/editor && ~/bin/slot bun run check)

editor typechecking (tsc --noEmit) also passed; the final locked Rust run had no warnings. staged diff whitespace and relative plan links passed.

spike files: docs/plans/0067-vectors/ and packages/editor/test/calendar-0067.spike.test.ts. the Rust CI job now runs the isolated locked crate; existing editor unit-test discovery runs the TS adapter. both use the same checked-in JSON. no production code or production dependency changes.

remaining differences are explicit: pulldown has no literal-autolink/emoji extension, and sub/superscript delimiter details may differ. source-preserving emoji/title spelling requires no renderer parity. before metadata migration, the plan requires a narrow pinned Lezer literal-autolink boundary rule and expected URL/label spans. this spike proves line eligibility only: exact UTF-8 spans, complete clock/metadata rows, relationships, link exclusions/Composer output compatibility, recovery, worker limits and scale still need the expanded production-adapter corpus and delivery gates. no full extraction, full CI, recovery or performance pass is claimed. no tested mismatch requires switching to comrak.

updated in `c19747b2be60cdcd6a24c929127cdf181526c3c3`. round-2 parser decision: retain **pulldown-cmark 0.13.4**, already used by the backend. no comrak dependency or parser migration, and no change to Tasks B3 (#798) or Composer behavior in this spike. updated the plan's option mapping and offset-event extraction contract. the single-parser ownership inventory explicitly includes Notes link-code exclusions, Tasks list ancestry/B3, Composer link-to-label conversion and both existing parser-based test oracles. the later shared-view/configuration change must preserve their independently expected outputs. **dev-02 spike: Rust 70/70 and editor Lezer 70/70 vectors pass; 0 failures and 0 normalized line-classification differences.** both independently compare to the same authored JSON expectations, using the existing pulldown version and the actual installed `markdownLanguage.parser` (lang-markdown 6.5.2 / Lezer 1.8.0). no expected output was generated from either parser. includes positive and negative custom/EOL/Unicode task boxes, matched/unmatched multiline single/multi-backticks in paragraph/list/quote containers, fenced/indented code, inline/block/multiline HTML comments with adjacent visible prose, nested lists/quotes, frontmatter boundaries, CRLF, non-BMP Unicode and extension prose. the editor does not install Obsidian `%%` comments, so those bytes intentionally leave enclosed candidates visible. reproducible commands, run inside the Nix shell on `calternal-dev-02-std`, with the slot wrapper: ```sh ~/bin/slot cargo test -j4 --locked --manifest-path docs/plans/0067-vectors/rust/Cargo.toml -- --nocapture (cd packages/editor && ~/bin/slot bunx vitest run test/calendar-0067.spike.test.ts) (cd packages/editor && ~/bin/slot bun run check) ``` editor typechecking (`tsc --noEmit`) also passed; the final locked Rust run had no warnings. staged diff whitespace and relative plan links passed. spike files: `docs/plans/0067-vectors/` and `packages/editor/test/calendar-0067.spike.test.ts`. the Rust CI job now runs the isolated locked crate; existing editor unit-test discovery runs the TS adapter. both use the same checked-in JSON. no production code or production dependency changes. remaining differences are explicit: pulldown has no literal-autolink/emoji extension, and sub/superscript delimiter details may differ. source-preserving emoji/title spelling requires no renderer parity. before metadata migration, the plan requires a narrow pinned Lezer literal-autolink boundary rule and expected URL/label spans. this spike proves line eligibility only: exact UTF-8 spans, complete clock/metadata rows, relationships, link exclusions/Composer output compatibility, recovery, worker limits and scale still need the expanded production-adapter corpus and delivery gates. no full extraction, full CI, recovery or performance pass is claimed. no tested mismatch requires switching to comrak.
kayg commented 2026-10-31 19:45:14 +00:00
Author
Owner
Copy link

updated in 6a4289fa: calendar writes now explicitly depend on the single NoteWriteExecutor defined by #808 (calternal-notes, crates/notes/src/write_executor/). both domains share exact-splice plans/inverses, verification, WAL, guarded apply, recovery, undo, Home coordination and ancestry deferral.

executor lands once, before either plan enables writes. read-only/parser/pure-writer prerequisites can proceed; Tasks ports retain B1 → B2 → B3 ordering. calendar identity/promotion/settlement cannot use ordinary Flush. added overlap and cross-domain replay/undo gates. ADR 0066 product choices remain pending kayg, not approved here.

validation: git diff --check passed; docs only, no builds or runtime tests. no merge.

updated in `6a4289fa`: calendar writes now explicitly depend on the single `NoteWriteExecutor` defined by #808 (`calternal-notes`, `crates/notes/src/write_executor/`). both domains share exact-splice plans/inverses, verification, WAL, guarded apply, recovery, undo, Home coordination and ancestry deferral. executor lands once, before either plan enables writes. read-only/parser/pure-writer prerequisites can proceed; Tasks ports retain B1 → B2 → B3 ordering. calendar identity/promotion/settlement cannot use ordinary Flush. added overlap and cross-domain replay/undo gates. ADR 0066 product choices remain pending kayg, not approved here. validation: `git diff --check` passed; docs only, no builds or runtime tests. no merge.
kayg commented 2026-11-02 12:42:33 +00:00
Author
Owner
Copy link

ADR 0003 evidence migration: docs/plans/0067-calendar-backend-v2.md

Source commit: 75a1577f5b98909836146e3f3309a23932a6aca2; snapshot from dev 55f35fcd. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable.

Original file (51,379 bytes): 0067-calendar-backend-v2.md

ADR 0003 evidence migration: `docs/plans/0067-calendar-backend-v2.md` Source commit: [75a1577f5b98909836146e3f3309a23932a6aca2](https://git.kayg.org/kayg/calternal/commit/75a1577f5b98909836146e3f3309a23932a6aca2); snapshot from dev `55f35fcd`. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable. Original file (51,379 bytes): [0067-calendar-backend-v2.md](https://git.kayg.org/attachments/efeb5c10-eb89-4e52-8194-1a4ba6af6882)
kayg commented 2026-11-02 12:44:42 +00:00
Author
Owner
Copy link

ADR 0003 evidence migration: docs/plans/0067-vectors/README.md

Source commit: fec18a59ded981f79c9c8841585bb9b67e6c0145; snapshot from dev 55f35fcd. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable.

# 0067 parser parity spike

Research only, 2026-10-31. No production parser, Tasks B3 or Composer behavior is
changed. This is the line-eligibility feasibility gate for retaining the existing
pulldown-cmark backend, not a completed calendar extraction implementation.

## Result

On dev-02 (`calternal-dev-02-std`, aarch64), inside the Nix development shell and
through `~/bin/slot`: **70/70 Rust vectors and 70/70 TS vectors pass**, zero failures
and zero normalized line-classification differences. Both adapters independently
compare to the same authored expectations, so equality to those expectations
also establishes equality between adapters for this output. No expected results
are produced by either parser. The installed editor parser is used directly,
not a hand-configured approximation: lang-markdown 6.5.2 / Lezer Markdown 1.8.0.
Rust is pinned to the existing backend's pulldown-cmark 0.13.4.
The final locked Rust rerun had no warnings; editor `bun run check`
(`tsc --noEmit`) also passed on dev-02.

## Format and scope

`lines.json` has a contract version and `vectors: [{id, source, expected}]`.
`source` is exact Markdown, including CRLF and Unicode. `expected` has one entry
per physical line (`source.split('\n')`), including a final empty line:

| Entry | Meaning |
| --- | --- |
| `.` | No task/event candidate or exclusion on this line |
| `t` | App checkbox at a parser-approved list-item opening |
| `e` | Clock candidate in parser-approved prose |
| `te` | Both a checkbox and a clock candidate |
| `x` | Line intersects inline/block code or HTML/comment; withhold dependent extraction |
| `f` | Frontmatter envelope; no body interpretation |

Exclusion takes precedence over candidates, including visible adjacent prose.
This is a proposed extraction rule, not a change to existing task reading.
Task boxes include the existing non-GFM marks and EOL boxes, rather than relying
on Lezer TaskMarker or pulldown TaskListMarker alone. The clock recognizer is
intentionally just `H:MM`/`HH:MM` plus whitespace/EOL; full calendar clock syntax
is already specified separately by `contracts/log-entries.json` and 0068.

The 70 vectors cover checked/unchecked/custom/Unicode/EOL boxes, invalid boxes,
ordered/bullet/nested lists, lazy continuation, bare/list clocks, single/double
backticks matched and unmatched across physical lines (paragraph/list/quote/task),
fenced/indented/tab code and unclosed fences, block/inline/multiline/unclosed HTML
comments with adjacent visible prose, HTML blocks/tags, `%%` prose, quoted/callout
checkboxes, nested quotes, frontmatter boundaries/unclosed envelopes, CRLF,
Unicode/non-BMP text, tables/headings, emoji names, sub/superscript, escapes and
link/autolink prose. The editor has no `%%` comment extension; both deliberately
keep enclosed candidates visible.

The envelope rule is separate from Markdown: a byte-zero `---` line (LF/CRLF)
through its matching `---` line is excluded, including the empty envelope.
An unclosed opener withholds the rest of the source. This is the planned shared
extraction boundary, not a claim that today's editor frontmatter-hiding helper
already matches it. Neither adapter calls that UI helper. Unsupported/BOM YAML
and typed frontmatter validation remain full-contract gates.

Rust consumes offset events with TABLES/TASKLISTS/STRIKETHROUGH/SUBSCRIPT/SUPERSCRIPT.
TS walks the installed editor tree. They each derive list starts, prose coverage
and exclusions independently. Tight-list text, whose Rust events need not include
Paragraph, is eligible. Quoted prose clocks remain unsupported; quoted boxes are
positive controls. Regexes recognize only the app's box and small clock grammars
inside parser-approved context; they do not identify Markdown containers or code.

## Run and CI

From the repository root in a VM Nix shell:

```sh
~/bin/slot cargo test -j4 --locked --manifest-path docs/plans/0067-vectors/rust/Cargo.toml -- --nocapture
(cd packages/editor && ~/bin/slot bunx vitest run test/calendar-0067.spike.test.ts)
```

The isolated Rust crate has its own lockfile and no production workspace member
or dependency changes. The Rust CI job explicitly runs it with `--locked`.
The TS file is discovered by the existing editor unit-test step in the web job.
Either job failing blocks parity; both depend on the same checked-in JSON.

## Remaining differences and delivery gates

- No pulldown literal-autolink or emoji extension. Literal titles retain exact
  source, including unknown/uppercase emoji. These AST differences do not change
  the measured line classifications. Before metadata migration, port the pinned
  Lezer literal-autolink lexical boundaries as a narrow source-span exclusion
  rule within eligible prose; compare URL/link label spans on punctuation,
  parentheses, escapes, www/email/mailto/xmpp and adjacent metadata vectors.
  Angle autolinks and explicit links already have native pulldown ranges.
- Sub/superscript options have potentially different delimiter rules. The spike
  proves eligibility for the sampled prose, not identical inline nodes or full
  metadata recognition across formatting.
- This spike compares physical lines, not exact UTF-8 ranges, link/Composer
  outputs, direct-child relationship edges, metadata rows, failure recovery or
  all date/time grammar. The full suite must add independently authored spans,
  source slices, rows, parent edges and unresolved reasons, then run the two
  production adapters on the same files. Include existing Notes link-exclusion
  and Composer title outputs and the parser-based test oracles before routing
  legacy consumers through shared options. No parser or configuration migration
  is hidden inside this spike.
- Adapters use intentionally simple full-document walks and line/range scans;
  this is no complexity, scale, worker-containment or incremental-editor proof.
  Full production gates in plan 0067 still apply. Passing by refusing supported
  normal fixtures is forbidden.

No tested material mismatch requires comrak. A later full-contract mismatch
that cannot be resolved by an option or a narrow documented extraction rule
stops the affected migration and reopens the parser decision.
ADR 0003 evidence migration: `docs/plans/0067-vectors/README.md` Source commit: [fec18a59ded981f79c9c8841585bb9b67e6c0145](https://git.kayg.org/kayg/calternal/commit/fec18a59ded981f79c9c8841585bb9b67e6c0145); snapshot from dev `55f35fcd`. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable. ````text # 0067 parser parity spike Research only, 2026-10-31. No production parser, Tasks B3 or Composer behavior is changed. This is the line-eligibility feasibility gate for retaining the existing pulldown-cmark backend, not a completed calendar extraction implementation. ## Result On dev-02 (`calternal-dev-02-std`, aarch64), inside the Nix development shell and through `~/bin/slot`: **70/70 Rust vectors and 70/70 TS vectors pass**, zero failures and zero normalized line-classification differences. Both adapters independently compare to the same authored expectations, so equality to those expectations also establishes equality between adapters for this output. No expected results are produced by either parser. The installed editor parser is used directly, not a hand-configured approximation: lang-markdown 6.5.2 / Lezer Markdown 1.8.0. Rust is pinned to the existing backend's pulldown-cmark 0.13.4. The final locked Rust rerun had no warnings; editor `bun run check` (`tsc --noEmit`) also passed on dev-02. ## Format and scope `lines.json` has a contract version and `vectors: [{id, source, expected}]`. `source` is exact Markdown, including CRLF and Unicode. `expected` has one entry per physical line (`source.split('\n')`), including a final empty line: | Entry | Meaning | | --- | --- | | `.` | No task/event candidate or exclusion on this line | | `t` | App checkbox at a parser-approved list-item opening | | `e` | Clock candidate in parser-approved prose | | `te` | Both a checkbox and a clock candidate | | `x` | Line intersects inline/block code or HTML/comment; withhold dependent extraction | | `f` | Frontmatter envelope; no body interpretation | Exclusion takes precedence over candidates, including visible adjacent prose. This is a proposed extraction rule, not a change to existing task reading. Task boxes include the existing non-GFM marks and EOL boxes, rather than relying on Lezer TaskMarker or pulldown TaskListMarker alone. The clock recognizer is intentionally just `H:MM`/`HH:MM` plus whitespace/EOL; full calendar clock syntax is already specified separately by `contracts/log-entries.json` and 0068. The 70 vectors cover checked/unchecked/custom/Unicode/EOL boxes, invalid boxes, ordered/bullet/nested lists, lazy continuation, bare/list clocks, single/double backticks matched and unmatched across physical lines (paragraph/list/quote/task), fenced/indented/tab code and unclosed fences, block/inline/multiline/unclosed HTML comments with adjacent visible prose, HTML blocks/tags, `%%` prose, quoted/callout checkboxes, nested quotes, frontmatter boundaries/unclosed envelopes, CRLF, Unicode/non-BMP text, tables/headings, emoji names, sub/superscript, escapes and link/autolink prose. The editor has no `%%` comment extension; both deliberately keep enclosed candidates visible. The envelope rule is separate from Markdown: a byte-zero `---` line (LF/CRLF) through its matching `---` line is excluded, including the empty envelope. An unclosed opener withholds the rest of the source. This is the planned shared extraction boundary, not a claim that today's editor frontmatter-hiding helper already matches it. Neither adapter calls that UI helper. Unsupported/BOM YAML and typed frontmatter validation remain full-contract gates. Rust consumes offset events with TABLES/TASKLISTS/STRIKETHROUGH/SUBSCRIPT/SUPERSCRIPT. TS walks the installed editor tree. They each derive list starts, prose coverage and exclusions independently. Tight-list text, whose Rust events need not include Paragraph, is eligible. Quoted prose clocks remain unsupported; quoted boxes are positive controls. Regexes recognize only the app's box and small clock grammars inside parser-approved context; they do not identify Markdown containers or code. ## Run and CI From the repository root in a VM Nix shell: ```sh ~/bin/slot cargo test -j4 --locked --manifest-path docs/plans/0067-vectors/rust/Cargo.toml -- --nocapture (cd packages/editor && ~/bin/slot bunx vitest run test/calendar-0067.spike.test.ts) ``` The isolated Rust crate has its own lockfile and no production workspace member or dependency changes. The Rust CI job explicitly runs it with `--locked`. The TS file is discovered by the existing editor unit-test step in the web job. Either job failing blocks parity; both depend on the same checked-in JSON. ## Remaining differences and delivery gates - No pulldown literal-autolink or emoji extension. Literal titles retain exact source, including unknown/uppercase emoji. These AST differences do not change the measured line classifications. Before metadata migration, port the pinned Lezer literal-autolink lexical boundaries as a narrow source-span exclusion rule within eligible prose; compare URL/link label spans on punctuation, parentheses, escapes, www/email/mailto/xmpp and adjacent metadata vectors. Angle autolinks and explicit links already have native pulldown ranges. - Sub/superscript options have potentially different delimiter rules. The spike proves eligibility for the sampled prose, not identical inline nodes or full metadata recognition across formatting. - This spike compares physical lines, not exact UTF-8 ranges, link/Composer outputs, direct-child relationship edges, metadata rows, failure recovery or all date/time grammar. The full suite must add independently authored spans, source slices, rows, parent edges and unresolved reasons, then run the two production adapters on the same files. Include existing Notes link-exclusion and Composer title outputs and the parser-based test oracles before routing legacy consumers through shared options. No parser or configuration migration is hidden inside this spike. - Adapters use intentionally simple full-document walks and line/range scans; this is no complexity, scale, worker-containment or incremental-editor proof. Full production gates in plan 0067 still apply. Passing by refusing supported normal fixtures is forbidden. No tested material mismatch requires comrak. A later full-contract mismatch that cannot be resolved by an option or a narrow documented extraction rule stops the affected migration and reopens the parser decision. ````
kayg commented 2026-11-07 19:33:42 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+green, left open for review.

earlier plan commits have semantic equivalents or newer text in dev; rebased the final missing plan amendment.

head: f59b367a23bfea29e60addd895b4908265272a44. rebased onto dev cd7f84432febfc63b788fbb9b9f9950452982d74 and pushed with the original exact force-with-lease.

validation on the Mac VM, through Nix and the shared Mac lock/slot: root bun install --frozen-lockfile; in apps/web, bun run check, bun run lint, bun run build, bun run test; PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. exact commands, logs, and dev bundle control.

original-head missing-commit/patch inventory, file pr-802.md.

housekeeping #1009: rebased+green, left open for review. earlier plan commits have semantic equivalents or newer text in dev; rebased the final missing plan amendment. head: `f59b367a23bfea29e60addd895b4908265272a44`. rebased onto dev `cd7f84432febfc63b788fbb9b9f9950452982d74` and pushed with the original exact force-with-lease. validation on the Mac VM, through Nix and the shared Mac lock/slot: root `bun install --frozen-lockfile`; in `apps/web`, `bun run check`, `bun run lint`, `bun run build`, `bun run test`; `PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate`. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. [exact commands, logs, and dev bundle control](https://git.kayg.org/attachments/64244177-fd81-4992-860e-4d5d22ca01cd). [original-head missing-commit/patch inventory](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5), file `pr-802.md`.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
This pull request has changes conflicting with the target branch.
  • docs/plans/0067-calendar-backend-v2.md
  • docs/plans/0067-vectors/README.md
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin docs/calendar-backend-v2-plan:docs/calendar-backend-v2-plan
git switch docs/calendar-backend-v2-plan

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff docs/calendar-backend-v2-plan
git switch docs/calendar-backend-v2-plan
git rebase dev
git switch dev
git merge --ff-only docs/calendar-backend-v2-plan
git switch docs/calendar-backend-v2-plan
git rebase dev
git switch dev
git merge --no-ff docs/calendar-backend-v2-plan
git switch dev
git merge --squash docs/calendar-backend-v2-plan
git switch dev
git merge --ff-only docs/calendar-backend-v2-plan
git switch dev
git merge docs/calendar-backend-v2-plan
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!802
No description provided.