kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

feat(notes): shared note-write executor, default-off slice 0 #809

Closed
kayg wants to merge 0 commits from feat/note-write-executor into dev
pull from: feat/note-write-executor
merge into: kayg:dev
Conversation 5 Commits - Files changed -
kayg commented 2026-10-31 19:50:33 +00:00
Owner
Copy link

Round 1 update: pushed f30b760afdac1de32973a7c74f3bf78c472c099c. P1/P2 addressed; fmt and clippy passed; executor 28/28; full notes 471 passed, 0 failed, 6 existing ignored. Independent re-review pending. Never merged.

Round 1 implementation plan: retain full-image journal hashes and verify every participant against them; require directional placement evidence before accepting an already-applied goal; persist outside edits as Conflicted, retaining the plan and displaced files with no undo; derive note and record staging names from the journal identity and clean them on recovery and terminal paths. No production callsites change.

Costs remain O(J + K × (N + W)) recovery/placement, with O(J + N + W) memory. Each image remains bounded by the existing admission caps; staging bytes are included in journal usage. Creation evidence additionally checks the retained/public inode identity because a staged post-image alone does not prove creation.

failure effect detection recovery test
outside exact post before placement no commit or undo hash plus missing directional evidence durable conflict; preserve files and journal process kill after PREPARE, initial execution, rollback, create/delete
outside unknown bytes no commit or rollback of that note neither recorded full-image hash durable conflict; preserve evidence placed-note edit/delete restart regressions
interrupted staging or journal write abandoned intermediate journal-derived staging identity collect intermediate, retain displaced files real process kills during partial write, full write, fsync, publication and placement; recover twice
changed committed post before undo refuse whole inverse committed post-image hashes preserve original existing strict undo regression

Gates: VM-only Nix fmt check, locked clippy with -D warnings, locked executor tests and full notes tests, all heavy commands through ~/bin/slot. Never merge.

Shared note-write executor, slice 0

Plan: add the isolated calternal-notes::write_executor byte transaction boundary,
then exercise independent byte properties, real process crashes, recovery, strict
own-author undo and compaction. References: #808 at 5be50361,
#802 at 6a4289fa.
No production caller, Home room, task, calendar, legacy replay or publication route
is switched. There is no runtime enable switch. This is the byte protocol slice,
not approval to port actions or a claim that the complete executor-first delivery
(room isolation, semantic validation, private-state/outbox adapters) is complete.
Those participants must refuse here instead of pretending they were committed.

Costs: pure sorted-splice verification/application is O(N + W + S). Admission
and recovery load the retained journal, O(J). Placement deliberately rechecks all
participants before and after each file: worst-case O(K × (N + W) + J), plus K
file/directory syncs; memory O(N + W + J), with JSON/image copies counted. This
conservative implementation does not claim the plan's linear placement target or
any latency pass. No Home walk. Image/splice payloads are bounded before PREPARE:
1,024 participant/dependency paths, 64 MiB pre/post/dependency images, 4,096 total
forward + inverse splices and 4 MiB encoded metadata. Journal accounting includes
encoded records, abandoned private record temporaries and retained placement
files. Admission reserves space under a 1 GiB cap and refuses rather than deleting
pinned evidence. Pending/conflicted plans are always roots. Caller-released terminal
history payloads compact atomically; operation digests, author, Home, consumed undo
ancestry and displaced inodes remain. Compaction also removes unreferenced private
record temporaries. Displaced inode collection needs a later reviewed retention
protocol; it is deliberately unavailable, so those bytes can cause backpressure.

failure effect detection recovery test
stale input/dependency or invalid splice refusal, no mutation full bytes plus every span and inverse explicit new plan properties and outside-edit cases
crash writing journal no placement without durable PREPARE atomic checksummed record old record or complete new record process exit at journal boundaries
crash between files pending, no committed receipt all pre/post images recorded direction only kill each placement, restart twice
late outside edit conflict, evidence retained displaced bytes and final rechecks preserve journal and outside inode placement race and pending replay
I/O failure pending, never success I/O result retry or durable rollback injected phase failures
crash after COMMIT committed action durable terminal record return receipt, never replay old files edit after commit and restart
changed state before undo whole inverse refused recorded post-image and author/ancestry retain original changed later note and duplicate keys
corrupt/unknown journal recovery withheld schema, digest, state validation explicit recovery, never guess corruption and unknown version
crash during compaction evidence/receipts survive atomic record replacement reopen checkpoint compact/reopen/retry and consumed undo

Conservative policy points implement the recommendations, not answers on kayg's
behalf: Q1 exact_undo; Q2 recorded_inverse_only; Q3 success_after_commit;
Q4 conflict_without_replan; Q5 refuse_unsupported_participants; Q6
refuse_oversized_action. Explicit adaptive Trash Restore is a later planner.

Placement feasibility is gated separately on each supported platform. This slice
uses exclusive create and retained atomic exchange on Linux/Apple; unsupported
platforms refuse. No route is enabled by passing Linux tests. Partial raw files
can be observed by outside editors. The private receipt boundary is not yet an
integrated Calternal generation/publication barrier. Uncooperative namespace
mutation and in-place writers prevent claiming portable filesystem CAS.

Validation on dev-01 through Nix and ~/bin/slot:

  • cargo clippy -j4 -p calternal-notes --all-targets -- -D warnings: passed.
  • cargo test -j4 -p calternal-notes: 467 passed, 0 failed, 6 existing ignored;
    doc tests passed. Includes 24 executor tests, independent generated byte models,
    real subprocess kills at partial journal write/file fsync/rename/directory sync,
    each note placement, COMMIT, inverse, rollback and compaction boundaries, plus
    simulated ENOSPC errors at protocol boundaries. Real power-loss/filesystem fault
    injection is not claimed. Initial stub runs demonstrated expected test failures.
  • cargo fmt --all -- --check: passed on the VM.
  • 500-note and 300-note/1,200-task-line byte admission fixtures pass. These are
    size/protocol fixtures, not task planner/relationship/semantic compatibility proof.
  • Callsite search: only this module and its tests reference NoteWriteExecutor.
    No Home/service/room/task/calendar/legacy replay caller changed. Production
    writes remain on their existing paths; this slice freezes no existing workflow.

The six ignored tests are repository timing/manual stress gates. No Apple gate,
shared semantic writer/parser integration, ranks/claims/bindings/outbox or room
publication proof is claimed. All corresponding plan requirements refuse; later
ports remain blocked on those integrations and platform/performance gates.

Final reproducible VM commands (Nix, dev-01):

cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 -p calternal-notes --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib
~/bin/slot cargo test -j4 -p calternal-notes

Full notes gate log: ~/note-executor-gates4.log; final locked-module gate log: ~/note-executor-final.log (passed: fmt check, locked clippy, 24/24 executor tests). Never merged; no action port or production enablement.

Pre-review head: 64c3983f9b3bac857c083ecd638df3b100ca9b7d. Full notes: 467 passed, 0 failed, 6 existing ignored. Final locked executor run: 24 passed, 0 failed. fmt check and clippy -D warnings passed. Local tree clean, pushed head matches origin. Independent PR review is still pending; no merge.

Round 1 update: pushed `f30b760afdac1de32973a7c74f3bf78c472c099c`. P1/P2 addressed; fmt and clippy passed; executor **28/28**; full notes **471 passed, 0 failed, 6 existing ignored**. Independent re-review pending. Never merged. Round 1 implementation plan: retain full-image journal hashes and verify every participant against them; require directional placement evidence before accepting an already-applied goal; persist outside edits as `Conflicted`, retaining the plan and displaced files with no undo; derive note and record staging names from the journal identity and clean them on recovery and terminal paths. No production callsites change. Costs remain O(J + K × (N + W)) recovery/placement, with O(J + N + W) memory. Each image remains bounded by the existing admission caps; staging bytes are included in journal usage. Creation evidence additionally checks the retained/public inode identity because a staged post-image alone does not prove creation. | failure | effect | detection | recovery | test | | --- | --- | --- | --- | --- | | outside exact post before placement | no commit or undo | hash plus missing directional evidence | durable conflict; preserve files and journal | process kill after PREPARE, initial execution, rollback, create/delete | | outside unknown bytes | no commit or rollback of that note | neither recorded full-image hash | durable conflict; preserve evidence | placed-note edit/delete restart regressions | | interrupted staging or journal write | abandoned intermediate | journal-derived staging identity | collect intermediate, retain displaced files | real process kills during partial write, full write, fsync, publication and placement; recover twice | | changed committed post before undo | refuse whole inverse | committed post-image hashes | preserve original | existing strict undo regression | Gates: VM-only Nix fmt check, locked clippy with `-D warnings`, locked executor tests and full notes tests, all heavy commands through `~/bin/slot`. Never merge. # Shared note-write executor, slice 0 Plan: add the isolated `calternal-notes::write_executor` byte transaction boundary, then exercise independent byte properties, real process crashes, recovery, strict own-author undo and compaction. References: [#808](https://git.kayg.org/kayg/calternal/pulls/808) at 5be50361, [#802](https://git.kayg.org/kayg/calternal/pulls/802) at 6a4289fa. No production caller, Home room, task, calendar, legacy replay or publication route is switched. There is no runtime enable switch. This is the byte protocol slice, not approval to port actions or a claim that the complete executor-first delivery (room isolation, semantic validation, private-state/outbox adapters) is complete. Those participants must refuse here instead of pretending they were committed. Costs: pure sorted-splice verification/application is O(N + W + S). Admission and recovery load the retained journal, O(J). Placement deliberately rechecks all participants before and after each file: worst-case O(K × (N + W) + J), plus K file/directory syncs; memory O(N + W + J), with JSON/image copies counted. This conservative implementation does not claim the plan's linear placement target or any latency pass. No Home walk. Image/splice payloads are bounded before PREPARE: 1,024 participant/dependency paths, 64 MiB pre/post/dependency images, 4,096 total forward + inverse splices and 4 MiB encoded metadata. Journal accounting includes encoded records, abandoned private record temporaries and retained placement files. Admission reserves space under a 1 GiB cap and refuses rather than deleting pinned evidence. Pending/conflicted plans are always roots. Caller-released terminal history payloads compact atomically; operation digests, author, Home, consumed undo ancestry and displaced inodes remain. Compaction also removes unreferenced private record temporaries. Displaced inode collection needs a later reviewed retention protocol; it is deliberately unavailable, so those bytes can cause backpressure. | failure | effect | detection | recovery | test | | --- | --- | --- | --- | --- | | stale input/dependency or invalid splice | refusal, no mutation | full bytes plus every span and inverse | explicit new plan | properties and outside-edit cases | | crash writing journal | no placement without durable PREPARE | atomic checksummed record | old record or complete new record | process exit at journal boundaries | | crash between files | pending, no committed receipt | all pre/post images | recorded direction only | kill each placement, restart twice | | late outside edit | conflict, evidence retained | displaced bytes and final rechecks | preserve journal and outside inode | placement race and pending replay | | I/O failure | pending, never success | I/O result | retry or durable rollback | injected phase failures | | crash after COMMIT | committed action | durable terminal record | return receipt, never replay old files | edit after commit and restart | | changed state before undo | whole inverse refused | recorded post-image and author/ancestry | retain original | changed later note and duplicate keys | | corrupt/unknown journal | recovery withheld | schema, digest, state validation | explicit recovery, never guess | corruption and unknown version | | crash during compaction | evidence/receipts survive | atomic record replacement | reopen checkpoint | compact/reopen/retry and consumed undo | Conservative policy points implement the recommendations, not answers on kayg's behalf: Q1 `exact_undo`; Q2 `recorded_inverse_only`; Q3 `success_after_commit`; Q4 `conflict_without_replan`; Q5 `refuse_unsupported_participants`; Q6 `refuse_oversized_action`. Explicit adaptive Trash Restore is a later planner. Placement feasibility is gated separately on each supported platform. This slice uses exclusive create and retained atomic exchange on Linux/Apple; unsupported platforms refuse. No route is enabled by passing Linux tests. Partial raw files can be observed by outside editors. The private receipt boundary is not yet an integrated Calternal generation/publication barrier. Uncooperative namespace mutation and in-place writers prevent claiming portable filesystem CAS. Validation on dev-01 through Nix and `~/bin/slot`: - `cargo clippy -j4 -p calternal-notes --all-targets -- -D warnings`: passed. - `cargo test -j4 -p calternal-notes`: 467 passed, 0 failed, 6 existing ignored; doc tests passed. Includes 24 executor tests, independent generated byte models, real subprocess kills at partial journal write/file fsync/rename/directory sync, each note placement, COMMIT, inverse, rollback and compaction boundaries, plus simulated ENOSPC errors at protocol boundaries. Real power-loss/filesystem fault injection is not claimed. Initial stub runs demonstrated expected test failures. - `cargo fmt --all -- --check`: passed on the VM. - 500-note and 300-note/1,200-task-line byte admission fixtures pass. These are size/protocol fixtures, not task planner/relationship/semantic compatibility proof. - Callsite search: only this module and its tests reference `NoteWriteExecutor`. No Home/service/room/task/calendar/legacy replay caller changed. Production writes remain on their existing paths; this slice freezes no existing workflow. The six ignored tests are repository timing/manual stress gates. No Apple gate, shared semantic writer/parser integration, ranks/claims/bindings/outbox or room publication proof is claimed. All corresponding plan requirements refuse; later ports remain blocked on those integrations and platform/performance gates. Final reproducible VM commands (Nix, dev-01): ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 -p calternal-notes --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib ~/bin/slot cargo test -j4 -p calternal-notes ``` Full notes gate log: `~/note-executor-gates4.log`; final locked-module gate log: `~/note-executor-final.log` (passed: fmt check, locked clippy, 24/24 executor tests). Never merged; no action port or production enablement. Pre-review head: `64c3983f9b3bac857c083ecd638df3b100ca9b7d`. Full notes: **467 passed, 0 failed, 6 existing ignored**. Final locked executor run: **24 passed, 0 failed**. fmt check and clippy `-D warnings` passed. Local tree clean, pushed head matches origin. Independent PR review is still pending; no merge.
docs: scope shared note-write executor byte protocol slice
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
0d951932b1
kayg changed title from WIP: shared note-write executor, default-off slice 0 to feat(notes): shared note-write executor, default-off slice 0 2026-10-31 20:11:08 +00:00
kayg commented 2026-10-31 20:18:54 +00:00
Author
Owner
Copy link

Independent review, round 1

Reviewed head 64c3983f9b3bac857c083ecd638df3b100ca9b7d, including admission, placement, forward recovery, rollback, committed-only undo, journal loading/compaction and the generated/crash tests. Scope is the disconnected byte executor; later room/publication/private-state integration and Apple/performance gates remain deferred as described in the PR.

Verdict: fix first. P0: none. Findings:

P1: a matching post-image is treated as proof of an applied change

Location: crates/notes/src/write_executor/recovery.rs:111 (also the permissive pre/post check at lines 66–77).

Scenario: Kill the existing crash-test child at JournalDurable, before staging or placing either note. An outside editor changes only a.md from its pre-image to the exact planned post-image. On restart, check accepts that image even though both recorded placement-evidence paths for this note are absent; finish skips the note, applies b.md, and records COMMIT. Undo now changes a.md back to the pre-image, reversing an outside change the executor never applied. The same goal-only shortcut is used by initial execution and rollback. Exact bytes establish content equality, but do not establish placement provenance.

Evidence: Reproduced on dev-01 against this head's compiled library and existing crash-test child, using a two-note old -> NEW plan. Output:

recovery: RecoveryReport { committed: ["change"], aborted: [], pending: [] }
a.md committed without either placement evidence file
undo replaced the outside NEW edit with old

The reproduction asserted both evidence paths were absent before undo.

Fix: Classify each participant using its directional placement evidence as well as its public bytes. An unplaced participant must still match the directional source; accepting an already placed goal must require evidence proving that placement, including the create/delete cases. Retain ambiguous states as conflicts, and ensure rollback only reverses proven applied placements. Add regressions for an outside exact-post edit after PREPARE/before placement, after restart, and before rollback; prove such edits cannot become undoable executor work.

P2: interrupted staging leaves bytes outside journal accounting and collection

Location: crates/notes/src/write_executor/placement.rs:85; accounting at journal.rs:192, collection at journal.rs:222.

Scenario: Prepared::new creates a separate, uniquely named fsio temporary beside the deterministic evidence path, then writes and fsyncs it before place. A process exit during that write, or after its fsync but before placement, leaves that temporary behind because Drop does not run. The WAL records only the deterministic evidence paths. usage counts those paths and private-journal files; compact only collects private record temporaries. Neither visits the abandoned staging temporary in the notes directory. Repeated interrupted recovery attempts can therefore accumulate unaccounted note images despite the stated 1 GiB budget.

Fix: Give intermediate staging files a journal-owned lifecycle: record/derive and account for their identities, and safely collect abandoned intermediates without collecting displaced outside inodes or unrelated fsio files. Cover interruption inside staging write, file fsync and placement, then repeated recovery and compaction. The current Staged(i) hook runs after the entire Prepared::new(...).place(...) call, so it does not exercise these windows.

Validation: on dev-01, through Nix and the slot wrapper, ~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib passed 24/24. The focused P1 reproduction above also completed with its assertions passing; it demonstrates a missing regression despite the green suite. No build/test ran on the control host and no repository files were changed.

The properties are genuinely generated and do execute two-note do/undo plans, although both participants receive the same generated replacement and the operation topology is fixed. The six conservative policies are named separately; no answer to the pending policy questions is inferred here. Journal version/checksum checks and terminal-only payload compaction are present. These observations do not resolve the placement-provenance and staging-lifecycle findings above.

## Independent review, round 1 Reviewed head `64c3983f9b3bac857c083ecd638df3b100ca9b7d`, including admission, placement, forward recovery, rollback, committed-only undo, journal loading/compaction and the generated/crash tests. Scope is the disconnected byte executor; later room/publication/private-state integration and Apple/performance gates remain deferred as described in the PR. **Verdict: fix first.** P0: none. Findings: ### P1: a matching post-image is treated as proof of an applied change **Location:** `crates/notes/src/write_executor/recovery.rs:111` (also the permissive pre/post check at lines 66–77). **Scenario:** Kill the existing crash-test child at `JournalDurable`, before staging or placing either note. An outside editor changes only `a.md` from its pre-image to the exact planned post-image. On restart, `check` accepts that image even though both recorded placement-evidence paths for this note are absent; `finish` skips the note, applies `b.md`, and records COMMIT. Undo now changes `a.md` back to the pre-image, reversing an outside change the executor never applied. The same goal-only shortcut is used by initial execution and rollback. Exact bytes establish content equality, but do not establish placement provenance. **Evidence:** Reproduced on dev-01 against this head's compiled library and existing crash-test child, using a two-note `old -> NEW` plan. Output: ```text recovery: RecoveryReport { committed: ["change"], aborted: [], pending: [] } a.md committed without either placement evidence file undo replaced the outside NEW edit with old ``` The reproduction asserted both evidence paths were absent before undo. **Fix:** Classify each participant using its directional placement evidence as well as its public bytes. An unplaced participant must still match the directional source; accepting an already placed goal must require evidence proving that placement, including the create/delete cases. Retain ambiguous states as conflicts, and ensure rollback only reverses proven applied placements. Add regressions for an outside exact-post edit after PREPARE/before placement, after restart, and before rollback; prove such edits cannot become undoable executor work. ### P2: interrupted staging leaves bytes outside journal accounting and collection **Location:** `crates/notes/src/write_executor/placement.rs:85`; accounting at `journal.rs:192`, collection at `journal.rs:222`. **Scenario:** `Prepared::new` creates a separate, uniquely named fsio temporary beside the deterministic evidence path, then writes and fsyncs it before `place`. A process exit during that write, or after its fsync but before placement, leaves that temporary behind because Drop does not run. The WAL records only the deterministic evidence paths. `usage` counts those paths and private-journal files; `compact` only collects private record temporaries. Neither visits the abandoned staging temporary in the notes directory. Repeated interrupted recovery attempts can therefore accumulate unaccounted note images despite the stated 1 GiB budget. **Fix:** Give intermediate staging files a journal-owned lifecycle: record/derive and account for their identities, and safely collect abandoned intermediates without collecting displaced outside inodes or unrelated fsio files. Cover interruption inside staging write, file fsync and placement, then repeated recovery and compaction. The current `Staged(i)` hook runs after the entire `Prepared::new(...).place(...)` call, so it does not exercise these windows. Validation: on dev-01, through Nix and the slot wrapper, `~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib` passed **24/24**. The focused P1 reproduction above also completed with its assertions passing; it demonstrates a missing regression despite the green suite. No build/test ran on the control host and no repository files were changed. The properties are genuinely generated and do execute two-note do/undo plans, although both participants receive the same generated replacement and the operation topology is fixed. The six conservative policies are named separately; no answer to the pending policy questions is inferred here. Journal version/checksum checks and terminal-only payload compaction are present. These observations do not resolve the placement-provenance and staging-lifecycle findings above.
kayg commented 2026-10-31 20:34:09 +00:00
Author
Owner
Copy link

Implementation response, round 1 of 3

Head: f30b760afdac1de32973a7c74f3bf78c472c099c (pushed). Never merged.

item status evidence
P1: recovery adopts an outside exact-post edit addressed real process exit at JournalDurable, then outside exact-post write; regression failed on the reviewed implementation and now passes
recovery evidence rule addressed every participant must match its journaled full-content pre/post hash; journal loading verifies forward splices reconstruct the recorded post-image; accepting an applied goal also requires directional placement evidence
outside edits and undo addressed durable Conflicted, no replay/rollback/undo; plan and displaced evidence retained through repeated recovery/compaction; new undo checks every committed post-image hash
P2: interrupted staging addressed note intermediates derive from journal evidence id, record intermediates from operation-key hash; usage includes note intermediates; recovery collects partial files and terminal paths clean intermediates while retaining displaced files
crash coverage passed partial/full staging write, file fsync, staging publication, public placement, forward/inverse/rollback, repeated recovery; no stray note or record staging after recovery
fmt passed final pushed source, VM-only Nix
clippy passed locked, all notes targets, -D warnings
executor tests passed 28 passed, 0 failed
full notes tests passed 471 passed, 0 failed, 6 existing ignored; doc tests passed

Exact commands on dev-01, in the existing worktree through nix develop -c bash -c:

cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 -p calternal-notes --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib
~/bin/slot cargo test --locked -j4 -p calternal-notes

Logs: ~/note-executor-r1-red.log (expected pre-fix regression failure), ~/note-executor-r1-gates.log (executor and full notes gates), ~/note-executor-r1-final.log (final fmt/clippy). All builds/tests ran on the VM; none on the control host.

Decisions: exact post bytes alone do not establish placement provenance. Replacement/deletion require retained source evidence; creation requires the retained/public inode identity as well as exact bytes. Conflicted actions remain in the existing pending report list and are also reported separately in conflicted. Disposable staging is collected; retained displaced evidence is not.

Open limits: no production caller enabled; existing Apple, room/publication, namespace/in-place writer and power-loss limitations remain. The six ignored tests remain existing manual/timing gates. Independent re-review is pending; this comment does not claim approval.

## Implementation response, round 1 of 3 Head: `f30b760afdac1de32973a7c74f3bf78c472c099c` (pushed). Never merged. | item | status | evidence | | --- | --- | --- | | P1: recovery adopts an outside exact-post edit | addressed | real process exit at `JournalDurable`, then outside exact-post write; regression failed on the reviewed implementation and now passes | | recovery evidence rule | addressed | every participant must match its journaled full-content pre/post hash; journal loading verifies forward splices reconstruct the recorded post-image; accepting an applied goal also requires directional placement evidence | | outside edits and undo | addressed | durable `Conflicted`, no replay/rollback/undo; plan and displaced evidence retained through repeated recovery/compaction; new undo checks every committed post-image hash | | P2: interrupted staging | addressed | note intermediates derive from journal evidence id, record intermediates from operation-key hash; usage includes note intermediates; recovery collects partial files and terminal paths clean intermediates while retaining displaced files | | crash coverage | passed | partial/full staging write, file fsync, staging publication, public placement, forward/inverse/rollback, repeated recovery; no stray note or record staging after recovery | | fmt | passed | final pushed source, VM-only Nix | | clippy | passed | locked, all notes targets, `-D warnings` | | executor tests | passed | 28 passed, 0 failed | | full notes tests | passed | 471 passed, 0 failed, 6 existing ignored; doc tests passed | Exact commands on dev-01, in the existing worktree through `nix develop -c bash -c`: ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 -p calternal-notes --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib ~/bin/slot cargo test --locked -j4 -p calternal-notes ``` Logs: `~/note-executor-r1-red.log` (expected pre-fix regression failure), `~/note-executor-r1-gates.log` (executor and full notes gates), `~/note-executor-r1-final.log` (final fmt/clippy). All builds/tests ran on the VM; none on the control host. Decisions: exact post bytes alone do not establish placement provenance. Replacement/deletion require retained source evidence; creation requires the retained/public inode identity as well as exact bytes. Conflicted actions remain in the existing `pending` report list and are also reported separately in `conflicted`. Disposable staging is collected; retained displaced evidence is not. Open limits: no production caller enabled; existing Apple, room/publication, namespace/in-place writer and power-loss limitations remain. The six ignored tests remain existing manual/timing gates. Independent re-review is pending; this comment does not claim approval.
kayg commented 2026-10-31 20:37:25 +00:00
Author
Owner
Copy link

Independent review, round 2

Reviewed f30b760afdac1de32973a7c74f3bf78c472c099c, focusing on the round 1 fixes, directional recovery, conflict retention, journal accounting and strict undo. Scope remains the disconnected byte executor; this does not approve production enablement or the deferred platform/publication integrations.

Verdict: ready. No new P0, P1 or P2 findings. One nonblocking P3 documentation finding below.

round 1 finding status does its test demonstrate the fix?
P1: recovery could adopt an outside exact-post edit and later undo it resolved; no regression found yes. outside_post_image_without_placement_is_never_undoable kills after durable PREPARE, asserts all evidence paths are absent, writes the exact outside post-image, and verifies no commit/undo, unchanged second note, and durable conflict through repeated recovery and compaction. It covers forward recovery and rollback. Separate initial-execution and create/delete regressions cover those paths, including staged creation without public inode identity.
P2: interrupted staging left unaccounted files resolved; no regression found yes. interrupted_staging_is_collected_on_recovery kills during partial/full staging writes, file sync, staging publication and public placement; repeated recovery checks note/private staging cleanup. The forward/inverse/rollback crash suites also check cleanup. Accounting now includes deterministic note intermediates; retained displaced evidence remains separate.

Additional checks:

  • Unplaced pre-images and multiple notes: recovery accepts a pre-image only as an unplaced forward participant, then finishes the recorded direction. Already placed post-images require directional evidence. The two-note forward crash suite covers PREPARE, staging and each placement; after recovery and idempotent retry both notes are at the post-image. The rollback suite begins after the first forward placement and proves both notes return to pre-images across rollback crashes. No new crash-only half-application path found. Outside edits deliberately stop as conflicts, preserving evidence rather than claiming atomic completion.
  • Conflict visibility and retries: RecoveryReport.conflicted explicitly surfaces conflicts, also retaining them in pending. Recovery skips durable conflicts; execute with the same key returns pending without replay; rollback and undo reject them. records() exposes the durable state. New keys require a freshly verified plan.
  • Many conflicts / journal growth: conflicted plans and evidence remain roots and cannot be compacted. Their records, evidence and staging contribute to admission accounting, which refuses at the 1 GiB budget rather than discarding evidence. Recovery/admission still load O(J) retained data. Eventual backpressure and the lack of a conflict-resolution/retention protocol are documented limits. There is no dedicated many-conflict budget regression; the current tests establish individual conflict retention, not that aggregate workload.
  • Undo at invocation: a new undo checks every current full-content post-image hash, then exact-byte inverse admission verifies the complete snapshot before PREPARE. strict_undo_refuses_entire_inverse_after_later_edit changes the second note and proves the first note remains unchanged with no inverse journal record. Same-key retries of an already committed undo return its receipt without writing again.

P3: document the hash equality assumption

Location: crates/notes/src/write_executor/journal.rs:46, recovery.rs:82–111, and docs/implementation/note-write-executor.md:76.

Recovery uses full-length BLAKE3 digest equality to classify public content, but the protocol documentation does not state the collision assumption. Document that distinct contents are assumed not to share a 256-bit BLAKE3 digest, that these checks are integrity checks rather than journal authentication, and that content equality still requires the separate placement evidence before accepting an applied goal. This is a documentation gap, not a demonstrated collision or a blocker. No collision test is requested.

Validation: independently reran on dev-01 at the reviewed SHA, via Nix and the slot wrapper:

netbird ssh claude@calternal-dev-01-compute02.vpn.kayg.org 'export PATH=/nix/var/nix/profiles/default/bin:$PATH; cd ~/Developer/Worktrees/note-write-executor && nix develop -c bash -c "~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib"'

Result: 28 passed, 0 failed, 449 filtered out. Also inspected the existing full-notes gate log (471 passed, 0 failed, 6 ignored) and final clippy log; those were not independently rerun in this round. No builds/tests ran on the control host. No code changed or pushed.

## Independent review, round 2 Reviewed `f30b760afdac1de32973a7c74f3bf78c472c099c`, focusing on the round 1 fixes, directional recovery, conflict retention, journal accounting and strict undo. Scope remains the disconnected byte executor; this does not approve production enablement or the deferred platform/publication integrations. **Verdict: ready.** No new P0, P1 or P2 findings. One nonblocking P3 documentation finding below. | round 1 finding | status | does its test demonstrate the fix? | | --- | --- | --- | | P1: recovery could adopt an outside exact-post edit and later undo it | resolved; no regression found | yes. `outside_post_image_without_placement_is_never_undoable` kills after durable PREPARE, asserts all evidence paths are absent, writes the exact outside post-image, and verifies no commit/undo, unchanged second note, and durable conflict through repeated recovery and compaction. It covers forward recovery and rollback. Separate initial-execution and create/delete regressions cover those paths, including staged creation without public inode identity. | | P2: interrupted staging left unaccounted files | resolved; no regression found | yes. `interrupted_staging_is_collected_on_recovery` kills during partial/full staging writes, file sync, staging publication and public placement; repeated recovery checks note/private staging cleanup. The forward/inverse/rollback crash suites also check cleanup. Accounting now includes deterministic note intermediates; retained displaced evidence remains separate. | Additional checks: - **Unplaced pre-images and multiple notes:** recovery accepts a pre-image only as an unplaced forward participant, then finishes the recorded direction. Already placed post-images require directional evidence. The two-note forward crash suite covers PREPARE, staging and each placement; after recovery and idempotent retry both notes are at the post-image. The rollback suite begins after the first forward placement and proves both notes return to pre-images across rollback crashes. No new crash-only half-application path found. Outside edits deliberately stop as conflicts, preserving evidence rather than claiming atomic completion. - **Conflict visibility and retries:** `RecoveryReport.conflicted` explicitly surfaces conflicts, also retaining them in `pending`. Recovery skips durable conflicts; execute with the same key returns pending without replay; rollback and undo reject them. `records()` exposes the durable state. New keys require a freshly verified plan. - **Many conflicts / journal growth:** conflicted plans and evidence remain roots and cannot be compacted. Their records, evidence and staging contribute to admission accounting, which refuses at the 1 GiB budget rather than discarding evidence. Recovery/admission still load O(J) retained data. Eventual backpressure and the lack of a conflict-resolution/retention protocol are documented limits. There is no dedicated many-conflict budget regression; the current tests establish individual conflict retention, not that aggregate workload. - **Undo at invocation:** a new undo checks every current full-content post-image hash, then exact-byte inverse admission verifies the complete snapshot before PREPARE. `strict_undo_refuses_entire_inverse_after_later_edit` changes the second note and proves the first note remains unchanged with no inverse journal record. Same-key retries of an already committed undo return its receipt without writing again. ### P3: document the hash equality assumption **Location:** `crates/notes/src/write_executor/journal.rs:46`, `recovery.rs:82–111`, and `docs/implementation/note-write-executor.md:76`. Recovery uses full-length BLAKE3 digest equality to classify public content, but the protocol documentation does not state the collision assumption. Document that distinct contents are assumed not to share a 256-bit BLAKE3 digest, that these checks are integrity checks rather than journal authentication, and that content equality still requires the separate placement evidence before accepting an applied goal. This is a documentation gap, not a demonstrated collision or a blocker. No collision test is requested. Validation: independently reran on dev-01 at the reviewed SHA, via Nix and the slot wrapper: ```sh netbird ssh claude@calternal-dev-01-compute02.vpn.kayg.org 'export PATH=/nix/var/nix/profiles/default/bin:$PATH; cd ~/Developer/Worktrees/note-write-executor && nix develop -c bash -c "~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor --lib"' ``` Result: **28 passed, 0 failed**, 449 filtered out. Also inspected the existing full-notes gate log (471 passed, 0 failed, 6 ignored) and final clippy log; those were not independently rerun in this round. No builds/tests ran on the control host. No code changed or pushed.
kayg commented 2026-11-02 12:42:34 +00:00
Author
Owner
Copy link

ADR 0003 evidence migration: docs/implementation/note-write-executor.md

Source commit: 98eff7e4bd71c576e8cd3580bfa890d37f3a828f; snapshot from dev 55f35fcd. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable.

# Shared note-write executor, slice 0

Plan: add the isolated `calternal-notes::write_executor` byte transaction boundary,
then exercise independent byte properties, real process crashes, recovery, strict
own-author undo and compaction. References: [#808](https://git.kayg.org/kayg/calternal/pulls/808) at 5be50361,
[#802](https://git.kayg.org/kayg/calternal/pulls/802) at 6a4289fa.
No production caller, Home room, task, calendar, legacy replay or publication route
is switched. There is no runtime enable switch. This is the byte protocol slice,
not approval to port actions or a claim that the complete executor-first delivery
(room isolation, semantic validation, private-state/outbox adapters) is complete.
Those participants must refuse here instead of pretending they were committed.

Costs: pure sorted-splice verification/application is O(N + W + S). Admission
and recovery load the retained journal, O(J). Placement deliberately rechecks all
participants before and after each file: worst-case O(K × (N + W) + J), plus K
file/directory syncs; memory O(N + W + J), with JSON/image copies counted. This
conservative implementation does not claim the plan's linear placement target or
any latency pass. No Home walk. Image/splice payloads are bounded before PREPARE:
1,024 participant/dependency paths, 64 MiB pre/post/dependency images, 4,096 total
forward + inverse splices and 4 MiB encoded metadata. Journal accounting includes
encoded records, abandoned private record temporaries and retained placement
files. Admission reserves space under a 1 GiB cap and refuses rather than deleting
pinned evidence. Pending/conflicted plans are always roots. Caller-released terminal
history payloads compact atomically; operation digests, author, Home, consumed undo
ancestry and displaced inodes remain. Compaction also removes unreferenced private
record temporaries. Displaced inode collection needs a later reviewed retention
protocol; it is deliberately unavailable, so those bytes can cause backpressure.

| failure | effect | detection | recovery | test |
| --- | --- | --- | --- | --- |
| stale input/dependency or invalid splice | refusal, no mutation | full bytes plus every span and inverse | explicit new plan | properties and outside-edit cases |
| crash writing journal | no placement without durable PREPARE | atomic checksummed record | old record or complete new record | process exit at journal boundaries |
| crash between files | pending, no committed receipt | all pre/post images | recorded direction only | kill each placement, restart twice |
| late outside edit | conflict, evidence retained | displaced bytes and final rechecks | preserve journal and outside inode | placement race and pending replay |
| I/O failure | pending, never success | I/O result | retry or durable rollback | injected phase failures |
| crash after COMMIT | committed action | durable terminal record | return receipt, never replay old files | edit after commit and restart |
| changed state before undo | whole inverse refused | recorded post-image and author/ancestry | retain original | changed later note and duplicate keys |
| corrupt/unknown journal | recovery withheld | schema, digest, state validation | explicit recovery, never guess | corruption and unknown version |
| crash during compaction | evidence/receipts survive | atomic record replacement | reopen checkpoint | compact/reopen/retry and consumed undo |

Conservative policy points implement the recommendations, not answers on kayg's
behalf: Q1 `exact_undo`; Q2 `recorded_inverse_only`; Q3 `success_after_commit`;
Q4 `conflict_without_replan`; Q5 `refuse_unsupported_participants`; Q6
`refuse_oversized_action`. Explicit adaptive Trash Restore is a later planner.

Placement feasibility is gated separately on each supported platform. This slice
uses exclusive create and retained atomic exchange on Linux/Apple; unsupported
platforms refuse. No route is enabled by passing Linux tests. Partial raw files
can be observed by outside editors. The private receipt boundary is not yet an
integrated Calternal generation/publication barrier. Uncooperative namespace
mutation and in-place writers prevent claiming portable filesystem CAS.


Validation on dev-01 through Nix and `~/bin/slot`:

- `cargo clippy -j4 -p calternal-notes --all-targets -- -D warnings`: passed.
- `cargo test -j4 -p calternal-notes`: 467 passed, 0 failed, 6 existing ignored;
  doc tests passed. Includes 24 executor tests, independent generated byte models,
  real subprocess kills at partial journal write/file fsync/rename/directory sync,
  each note placement, COMMIT, inverse, rollback and compaction boundaries, plus
  simulated ENOSPC errors at protocol boundaries. Real power-loss/filesystem fault
  injection is not claimed. Initial stub runs demonstrated expected test failures.
- `cargo fmt --all -- --check`: passed on the VM.
- 500-note and 300-note/1,200-task-line byte admission fixtures pass. These are
  size/protocol fixtures, not task planner/relationship/semantic compatibility proof.
- Callsite search: only this module and its tests reference `NoteWriteExecutor`.
  No Home/service/room/task/calendar/legacy replay caller changed. Production
  writes remain on their existing paths; this slice freezes no existing workflow.

The six ignored tests are repository timing/manual stress gates. No Apple gate,
shared semantic writer/parser integration, ranks/claims/bindings/outbox or room
publication proof is claimed. All corresponding plan requirements refuse; later
ports remain blocked on those integrations and platform/performance gates.

## Review round 1: recovery evidence and staging

Recovery now compares every public participant with the full-content pre/post
hashes retained in the journal. Loading the journal validates those hashes and
reconstructs the post-image from the forward splices over the recorded pre-image.
An already-applied goal also requires directional placement evidence. Replacements
and removals retain the source bytes; creations retain the same inode as the public
file. A staged image alone cannot establish placement.

An outside image, including an exact post-image without placement evidence, makes
the action durably `Conflicted`. Recovery never replays that action, rollback is
unavailable, undo is unavailable, and compaction keeps the plan and displaced
files. Recovery reports conflicts separately and also includes them in `pending`
for callers that already treat all unresolved actions as pending. A new undo must
match every committed post-image hash before admitting any inverse.

Note intermediates use the journal-derived evidence name plus `.staging`; private
record intermediates use the operation-key hash plus `.staging`. Recovery removes
abandoned intermediates after validating the journal, including for terminal and
conflicted records. Placement cleans intermediates on return. The retained evidence
files are distinct from disposable staging and are never collected by this path.
Journal usage counts both deterministic evidence and intermediate note files.

The regression reproduces a real process exit after durable PREPARE followed by
an outside exact-post write, and verifies no commit or undo, no second-note change,
and a retained conflict across repeated recovery and compaction. Further cases
cover initial execution, rollback, outside create/delete, and process exits during
partial/full staging writes, file sync, staging publication and public placement.
The forward, inverse and rollback crash suites recover twice and check that neither
note nor private-journal staging files remain.
ADR 0003 evidence migration: `docs/implementation/note-write-executor.md` Source commit: [98eff7e4bd71c576e8cd3580bfa890d37f3a828f](https://git.kayg.org/kayg/calternal/commit/98eff7e4bd71c576e8cd3580bfa890d37f3a828f); snapshot from dev `55f35fcd`. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable. ```text # Shared note-write executor, slice 0 Plan: add the isolated `calternal-notes::write_executor` byte transaction boundary, then exercise independent byte properties, real process crashes, recovery, strict own-author undo and compaction. References: [#808](https://git.kayg.org/kayg/calternal/pulls/808) at 5be50361, [#802](https://git.kayg.org/kayg/calternal/pulls/802) at 6a4289fa. No production caller, Home room, task, calendar, legacy replay or publication route is switched. There is no runtime enable switch. This is the byte protocol slice, not approval to port actions or a claim that the complete executor-first delivery (room isolation, semantic validation, private-state/outbox adapters) is complete. Those participants must refuse here instead of pretending they were committed. Costs: pure sorted-splice verification/application is O(N + W + S). Admission and recovery load the retained journal, O(J). Placement deliberately rechecks all participants before and after each file: worst-case O(K × (N + W) + J), plus K file/directory syncs; memory O(N + W + J), with JSON/image copies counted. This conservative implementation does not claim the plan's linear placement target or any latency pass. No Home walk. Image/splice payloads are bounded before PREPARE: 1,024 participant/dependency paths, 64 MiB pre/post/dependency images, 4,096 total forward + inverse splices and 4 MiB encoded metadata. Journal accounting includes encoded records, abandoned private record temporaries and retained placement files. Admission reserves space under a 1 GiB cap and refuses rather than deleting pinned evidence. Pending/conflicted plans are always roots. Caller-released terminal history payloads compact atomically; operation digests, author, Home, consumed undo ancestry and displaced inodes remain. Compaction also removes unreferenced private record temporaries. Displaced inode collection needs a later reviewed retention protocol; it is deliberately unavailable, so those bytes can cause backpressure. | failure | effect | detection | recovery | test | | --- | --- | --- | --- | --- | | stale input/dependency or invalid splice | refusal, no mutation | full bytes plus every span and inverse | explicit new plan | properties and outside-edit cases | | crash writing journal | no placement without durable PREPARE | atomic checksummed record | old record or complete new record | process exit at journal boundaries | | crash between files | pending, no committed receipt | all pre/post images | recorded direction only | kill each placement, restart twice | | late outside edit | conflict, evidence retained | displaced bytes and final rechecks | preserve journal and outside inode | placement race and pending replay | | I/O failure | pending, never success | I/O result | retry or durable rollback | injected phase failures | | crash after COMMIT | committed action | durable terminal record | return receipt, never replay old files | edit after commit and restart | | changed state before undo | whole inverse refused | recorded post-image and author/ancestry | retain original | changed later note and duplicate keys | | corrupt/unknown journal | recovery withheld | schema, digest, state validation | explicit recovery, never guess | corruption and unknown version | | crash during compaction | evidence/receipts survive | atomic record replacement | reopen checkpoint | compact/reopen/retry and consumed undo | Conservative policy points implement the recommendations, not answers on kayg's behalf: Q1 `exact_undo`; Q2 `recorded_inverse_only`; Q3 `success_after_commit`; Q4 `conflict_without_replan`; Q5 `refuse_unsupported_participants`; Q6 `refuse_oversized_action`. Explicit adaptive Trash Restore is a later planner. Placement feasibility is gated separately on each supported platform. This slice uses exclusive create and retained atomic exchange on Linux/Apple; unsupported platforms refuse. No route is enabled by passing Linux tests. Partial raw files can be observed by outside editors. The private receipt boundary is not yet an integrated Calternal generation/publication barrier. Uncooperative namespace mutation and in-place writers prevent claiming portable filesystem CAS. Validation on dev-01 through Nix and `~/bin/slot`: - `cargo clippy -j4 -p calternal-notes --all-targets -- -D warnings`: passed. - `cargo test -j4 -p calternal-notes`: 467 passed, 0 failed, 6 existing ignored; doc tests passed. Includes 24 executor tests, independent generated byte models, real subprocess kills at partial journal write/file fsync/rename/directory sync, each note placement, COMMIT, inverse, rollback and compaction boundaries, plus simulated ENOSPC errors at protocol boundaries. Real power-loss/filesystem fault injection is not claimed. Initial stub runs demonstrated expected test failures. - `cargo fmt --all -- --check`: passed on the VM. - 500-note and 300-note/1,200-task-line byte admission fixtures pass. These are size/protocol fixtures, not task planner/relationship/semantic compatibility proof. - Callsite search: only this module and its tests reference `NoteWriteExecutor`. No Home/service/room/task/calendar/legacy replay caller changed. Production writes remain on their existing paths; this slice freezes no existing workflow. The six ignored tests are repository timing/manual stress gates. No Apple gate, shared semantic writer/parser integration, ranks/claims/bindings/outbox or room publication proof is claimed. All corresponding plan requirements refuse; later ports remain blocked on those integrations and platform/performance gates. ## Review round 1: recovery evidence and staging Recovery now compares every public participant with the full-content pre/post hashes retained in the journal. Loading the journal validates those hashes and reconstructs the post-image from the forward splices over the recorded pre-image. An already-applied goal also requires directional placement evidence. Replacements and removals retain the source bytes; creations retain the same inode as the public file. A staged image alone cannot establish placement. An outside image, including an exact post-image without placement evidence, makes the action durably `Conflicted`. Recovery never replays that action, rollback is unavailable, undo is unavailable, and compaction keeps the plan and displaced files. Recovery reports conflicts separately and also includes them in `pending` for callers that already treat all unresolved actions as pending. A new undo must match every committed post-image hash before admitting any inverse. Note intermediates use the journal-derived evidence name plus `.staging`; private record intermediates use the operation-key hash plus `.staging`. Recovery removes abandoned intermediates after validating the journal, including for terminal and conflicted records. Placement cleans intermediates on return. The retained evidence files are distinct from disposable staging and are never collected by this path. Journal usage counts both deterministic evidence and intermediate note files. The regression reproduces a real process exit after durable PREPARE followed by an outside exact-post write, and verifies no commit or undo, no second-note change, and a retained conflict across repeated recovery and compaction. Further cases cover initial execution, rollback, outside create/delete, and process exits during partial/full staging writes, file sync, staging publication and public placement. The forward, inverse and rollback crash suites recover twice and check that neither note nor private-journal staging files remain. ```
kayg commented 2026-11-04 13:04:44 +00:00
Author
Owner
Copy link

Closing: head f30b760a is already in dev (reached via 43a7f740); nothing left to merge. Branch kept.

Closing: head f30b760a is already in dev (reached via 43a7f740); nothing left to merge. Branch kept.
kayg closed this pull request 2026-11-04 13:04:44 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!809
No description provided.