kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

notes: integrate shared executor with Home save publication #814

Closed
kayg wants to merge 0 commits from feat/note-write-executor-home into dev
pull from: feat/note-write-executor-home
merge into: kayg:dev
Conversation 10 Commits - Files changed -
kayg commented 2026-10-31 20:46:22 +00:00
Owner
Copy link

round 2 design, implemented

readers pin immutable accepted source versions with an Arc token; writers publish a new SQLite generation without waiting for readers. retain at most four pinned old generations plus current, reclaim unpinned intermediate versions, and revoke the oldest reader before reclamation when the cap is exceeded. stale search batches discard staging and retry. placement releases the Home metadata lock; participant edits/Flush refuse promptly, and executor metadata waits use 25 ms checks.

N source bytes, K changed notes, R <= 4 old generations: O(1) snapshot pins, O((R+1)N) derived source disk, O(K R log N) normal version work; pin-set changes can examine O(R N) retired rows.

failure effect detection recovery gate
reader never returns bounded retained rows; writer and other readers continue Arc pin cap oldest pin gets stale error stuck snapshot and stalled cross-crate rebuild
old pin cap exceeded stale reader cannot publish mixed/removed rows validation before/after reads and before publication discard staging, open fresh snapshot cap, reclamation and retry tests
crash with readers process-local pins disappear; accepted SQLite commit remains atomic executor WAL and source generation replay accepted publication, remove obsolete versions real process kills with pinned snapshots
executor publication contends with typing bounded retryable admission response 25 ms metadata wait checks; leased actor refusal retry without placing partial ordinary bytes Edit/Flush response and latency harness

final implementation, all gates and measured latency: round 2 status. head 99cfe9416becedf2831b9354314419a8eff8062a. final independent review pending; never merge.

The byte executor had no safe admission or publication path for a live Home. This change adds Home coordination for exact replacements of existing notes: actor leases, authored private history, atomic derived rows and a permanent journal publication receipt. No task/calendar action is enabled.

Stacked on #809 at f30b760a; prerequisite audit #812. Target dev. Never merge.

Shared note-write executor, slice 0b

Stack: #809 at f30b760a. Prerequisite audit: #812. Target: dev. No action is enabled. Never merge.

Plan:

  1. Add failing Home integration and publication replay tests.
  2. Add room leases, durable Home payloads and recovery before Home startup. Use the existing jobs/lifecycle/room/meta_update lock order. Refuse unsupported lifecycle, metadata, rank, claim and occurrence participants.
  3. Install authored CRDT history and bases after COMMIT, publish one journal batch with a permanent deduplication key, then commit one accepted metadata batch. Release room leases only after the complete generation is ready. A pending action keeps affected rooms unavailable.
  4. Exercise crash/failure boundaries, ordinary-save concurrency, recovery and existing notes behavior. Run VM gates and record results.

Scope: existing-note replacements with stable ids and exact room byte round trips. Create/delete/rename and domain-specific private stores remain refused. The shared strict metadata writer remains a separate prerequisite for Tasks B1.

Costs: K participants, N retained image/state bytes, J retained WAL bytes. Room acquisition O(K log K), payload/lease matching O(K²), history/index installation O(K + N), existing conservative placement O(KN + J). Memory O(N + J). Startup additionally replays H retained history bytes for the distinct identities named by the WAL, O(H + J + N), in bounded identity chunks before exposing the Home. Existing 1,024 path / 64 MiB image / 1 GiB journal caps remain; private CRDT payloads must be bounded too. No Home walk during admission.

failure effect detection recovery test
stale room/file/binding refuse before PREPARE exact bytes and stable id/path new plan concurrent save and stale input
interrupted PREPARE no file placement checksummed WAL drop staging or resume complete record existing process crash tests
interrupted placement no accepted partial generation WAL + placement evidence recorded direction every placement boundary
COMMIT before history committed, not published Home payload in WAL idempotent history install restart after COMMIT
history before index affected rooms unavailable transaction receipt install complete index batch injected history boundary
journal before index accepted rows remain before-state outbox cursor install whole accepted batch injected publication boundary
journal before cursor no completion reply permanent journal dedup key retry without duplicate events publication boundary
caller cancellation coordinator continues Home-owned background task finish or retain pending dropped caller
unsupported participant no writes admission retain existing action path refusal coverage

No power-loss, Apple or performance pass is claimed without those gates. BLAKE3 equality assumes distinct content does not share a 256-bit digest; integrity checks do not authenticate the WAL. Placement evidence is required separately.

Implemented boundaries:

  • Home owns each active executor under its jobs and lifecycle locks. UUID-ordered actor leases drain ordinary saves, pin revisions and prevent timers, websocket edits, outside ingestion or Flush from interleaving. Peek holds a generation read lock, including its disk fallback.
  • PREPARE includes bounded, checksummed exact CRDT pre/post states and the authored update. File placement and COMMIT keep #809's evidence checks. One conditional history transaction installs every authored update/base and its permanent receipt after COMMIT.
  • One SQLite transaction installs all note/plugin rows; one journal transaction publishes the batch and a permanent deduplication receipt. Publication transactions use FULL sync; ordinary journal writes retain their prior sync policy. The WAL publication cursor follows that receipt. Same-key retries repair live room release even if cursor installation already succeeded.
  • Recovery runs before scan, watcher, bulk replay and readers. Published files are never replayed. Derived rows are rebuilt from current history for WAL participants, preserving later ordinary saves. Conflicted identities retain accepted history/index state and refuse source fallback; unrelated notes remain available.

Delivery limits: no task/calendar action or enable switch; stable existing-note replacements only. Namespace creation/removal/rename, metadata validation, ranks, claims, occurrences and Home undo remain refused pending their adapters. Home WAL payload compaction is withheld, so retained records can cause bounded backpressure. No Apple, power-loss or performance pass is claimed. Uncooperative outside tools can observe partial raw filesystem placement, as in #809.

Validation on dev-01 in ~/Developer/Worktrees/note-write-executor-home, through
nix develop -c bash -c, implementation at a479a812:

cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal
~/bin/slot cargo test --locked -j4 -p calternal-store

All passed. Notes: 480 passed, 0 failed, 6 existing ignored; doc tests passed.
All calternal unit/integration suites passed with their existing ignored tests.
Store: 9 passed, 0 failed; doc tests passed. Focused executor run: 37 passed,
including nine Home tests and 24 real process-kill Home phases with repeated
recovery, live publication retry, cursor I/O errors, caller cancellation,
concurrent ordinary save, unsupported admission and cache rebuild after a later
ordinary save. Existing notes tests were not edited.

Final gate log: ~/note-home-final.log. Expected failing runs are retained in
~/note-home-red.log, ~/note-home-live-red.log, ~/note-home-cursor-red.log
and ~/note-home-index-red.log. No build or test ran on the control host.
The final documentation commit changes comments and this report only.
Independent review is pending. Never merged.

Initial delivery head: 0f38d06f441c5f0c02752dce8d71fffe57ab5a12. Superseded by the round 1 implementation and gate results below.

round 1 implementation status

head: 966a97bb14d1ebee3e4c40342399477c1c2508d0. both P1 findings addressed; ready for independent re-review. never merged; repository settings unchanged.

item status evidence
list/plugin rows before journal receipt fixed journal receipt now precedes one accepted list/plugin/link commit; page counts and rows share a read transaction
startup/pending publication visibility fixed retained executor records gate startup lists through Home recovery; pre-receipt failure retains accepted rows; crash tests query the startup path
search/backlink reads of working files fixed shared generation leases cover file reads and complete builds/snippet responses; pending/conflicted actions refuse source reads, including after restart
search publication split across fetch batches fixed private file-backed staging; one final SQLite transaction; related action participants expand together even with one journal event and batch size one; queries pin one read snapshot
concurrent reader regressions passed readers at 24 two-note action boundaries; pinned before-state search build; conflict/restart and recorder failure; concurrent search result polling through atomic refresh, removals, failure and cancellation
fmt passed cargo fmt --all -- --check
workspace clippy passed ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
notes passed 484 passed, 0 failed, 6 existing ignored
calternal passed all unit/integration suites passed, existing ignored tests retained
store passed 9 passed, 0 failed
search passed 16 unit + 12 integration passed, 0 failed, 1 existing ignored

all gates ran on dev-01 in ~/Developer/Worktrees/note-write-executor-home, using the existing Nix shell and VM slots:

cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search

final log: ~/note-home-r1-final2.log, GATES_PASSED, service result success. expected failing regressions are in ~/note-home-r1-red.log; focused Home coverage passed in ~/note-home-r1-focused2.log. no builds/tests ran on the control host. commits were pushed after each step; the control worktree is clean.

decisions and remaining limits: search remains an asynchronous journal follower, but each refresh exposes a complete accepted generation. source reads wait for a generation lease or refuse pending recovery; no working-file fallback follows a transaction refusal. the retained participant graph is rebuilt from the WAL and expands overlapping actions. search staging uses O(D) temporary disk with bounded fetch memory, and group tracking uses O(G) retained memberships. a full search reconciliation can delay a note action while holding its shared lease; startup lists with retained executor records can wait for recovery. existing executor scope, WAL limits and outside-tool filesystem visibility limits remain. no performance, Apple or power-loss pass is claimed. implementation details and failure/cost notes are in docs/implementation/note-write-executor-home.md.

round 1 of max 3; independent re-review pending. never merge.

## round 2 design, implemented readers pin immutable accepted source versions with an Arc token; writers publish a new SQLite generation without waiting for readers. retain at most four pinned old generations plus current, reclaim unpinned intermediate versions, and revoke the oldest reader before reclamation when the cap is exceeded. stale search batches discard staging and retry. placement releases the Home metadata lock; participant edits/Flush refuse promptly, and executor metadata waits use 25 ms checks. N source bytes, K changed notes, R <= 4 old generations: O(1) snapshot pins, O((R+1)N) derived source disk, O(K R log N) normal version work; pin-set changes can examine O(R N) retired rows. | failure | effect | detection | recovery | gate | | --- | --- | --- | --- | --- | | reader never returns | bounded retained rows; writer and other readers continue | Arc pin cap | oldest pin gets stale error | stuck snapshot and stalled cross-crate rebuild | | old pin cap exceeded | stale reader cannot publish mixed/removed rows | validation before/after reads and before publication | discard staging, open fresh snapshot | cap, reclamation and retry tests | | crash with readers | process-local pins disappear; accepted SQLite commit remains atomic | executor WAL and source generation | replay accepted publication, remove obsolete versions | real process kills with pinned snapshots | | executor publication contends with typing | bounded retryable admission response | 25 ms metadata wait checks; leased actor refusal | retry without placing partial ordinary bytes | Edit/Flush response and latency harness | final implementation, all gates and measured latency: [round 2 status](https://git.kayg.org/kayg/calternal/pulls/814#issuecomment-71925). head `99cfe9416becedf2831b9354314419a8eff8062a`. final independent review pending; never merge. --- The byte executor had no safe admission or publication path for a live Home. This change adds Home coordination for exact replacements of existing notes: actor leases, authored private history, atomic derived rows and a permanent journal publication receipt. No task/calendar action is enabled. Stacked on #809 at f30b760a; prerequisite audit #812. Target dev. Never merge. # Shared note-write executor, slice 0b Stack: #809 at f30b760a. Prerequisite audit: #812. Target: dev. No action is enabled. Never merge. Plan: 1. Add failing Home integration and publication replay tests. 2. Add room leases, durable Home payloads and recovery before Home startup. Use the existing jobs/lifecycle/room/meta_update lock order. Refuse unsupported lifecycle, metadata, rank, claim and occurrence participants. 3. Install authored CRDT history and bases after COMMIT, publish one journal batch with a permanent deduplication key, then commit one accepted metadata batch. Release room leases only after the complete generation is ready. A pending action keeps affected rooms unavailable. 4. Exercise crash/failure boundaries, ordinary-save concurrency, recovery and existing notes behavior. Run VM gates and record results. Scope: existing-note replacements with stable ids and exact room byte round trips. Create/delete/rename and domain-specific private stores remain refused. The shared strict metadata writer remains a separate prerequisite for Tasks B1. Costs: K participants, N retained image/state bytes, J retained WAL bytes. Room acquisition O(K log K), payload/lease matching O(K²), history/index installation O(K + N), existing conservative placement O(KN + J). Memory O(N + J). Startup additionally replays H retained history bytes for the distinct identities named by the WAL, O(H + J + N), in bounded identity chunks before exposing the Home. Existing 1,024 path / 64 MiB image / 1 GiB journal caps remain; private CRDT payloads must be bounded too. No Home walk during admission. | failure | effect | detection | recovery | test | | --- | --- | --- | --- | --- | | stale room/file/binding | refuse before PREPARE | exact bytes and stable id/path | new plan | concurrent save and stale input | | interrupted PREPARE | no file placement | checksummed WAL | drop staging or resume complete record | existing process crash tests | | interrupted placement | no accepted partial generation | WAL + placement evidence | recorded direction | every placement boundary | | COMMIT before history | committed, not published | Home payload in WAL | idempotent history install | restart after COMMIT | | history before index | affected rooms unavailable | transaction receipt | install complete index batch | injected history boundary | | journal before index | accepted rows remain before-state | outbox cursor | install whole accepted batch | injected publication boundary | | journal before cursor | no completion reply | permanent journal dedup key | retry without duplicate events | publication boundary | | caller cancellation | coordinator continues | Home-owned background task | finish or retain pending | dropped caller | | unsupported participant | no writes | admission | retain existing action path | refusal coverage | No power-loss, Apple or performance pass is claimed without those gates. BLAKE3 equality assumes distinct content does not share a 256-bit digest; integrity checks do not authenticate the WAL. Placement evidence is required separately. Implemented boundaries: - Home owns each active executor under its jobs and lifecycle locks. UUID-ordered actor leases drain ordinary saves, pin revisions and prevent timers, websocket edits, outside ingestion or Flush from interleaving. Peek holds a generation read lock, including its disk fallback. - PREPARE includes bounded, checksummed exact CRDT pre/post states and the authored update. File placement and COMMIT keep #809's evidence checks. One conditional history transaction installs every authored update/base and its permanent receipt after COMMIT. - One SQLite transaction installs all note/plugin rows; one journal transaction publishes the batch and a permanent deduplication receipt. Publication transactions use FULL sync; ordinary journal writes retain their prior sync policy. The WAL publication cursor follows that receipt. Same-key retries repair live room release even if cursor installation already succeeded. - Recovery runs before scan, watcher, bulk replay and readers. Published files are never replayed. Derived rows are rebuilt from current history for WAL participants, preserving later ordinary saves. Conflicted identities retain accepted history/index state and refuse source fallback; unrelated notes remain available. Delivery limits: no task/calendar action or enable switch; stable existing-note replacements only. Namespace creation/removal/rename, metadata validation, ranks, claims, occurrences and Home undo remain refused pending their adapters. Home WAL payload compaction is withheld, so retained records can cause bounded backpressure. No Apple, power-loss or performance pass is claimed. Uncooperative outside tools can observe partial raw filesystem placement, as in #809. Validation on dev-01 in `~/Developer/Worktrees/note-write-executor-home`, through `nix develop -c bash -c`, implementation at `a479a812`: ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal ~/bin/slot cargo test --locked -j4 -p calternal-store ``` All passed. Notes: 480 passed, 0 failed, 6 existing ignored; doc tests passed. All calternal unit/integration suites passed with their existing ignored tests. Store: 9 passed, 0 failed; doc tests passed. Focused executor run: 37 passed, including nine Home tests and 24 real process-kill Home phases with repeated recovery, live publication retry, cursor I/O errors, caller cancellation, concurrent ordinary save, unsupported admission and cache rebuild after a later ordinary save. Existing notes tests were not edited. Final gate log: `~/note-home-final.log`. Expected failing runs are retained in `~/note-home-red.log`, `~/note-home-live-red.log`, `~/note-home-cursor-red.log` and `~/note-home-index-red.log`. No build or test ran on the control host. The final documentation commit changes comments and this report only. Independent review is pending. Never merged. Initial delivery head: `0f38d06f441c5f0c02752dce8d71fffe57ab5a12`. Superseded by the round 1 implementation and gate results below. ## round 1 implementation status head: `966a97bb14d1ebee3e4c40342399477c1c2508d0`. both P1 findings addressed; ready for independent re-review. never merged; repository settings unchanged. | item | status | evidence | | --- | --- | --- | | list/plugin rows before journal receipt | fixed | journal receipt now precedes one accepted list/plugin/link commit; page counts and rows share a read transaction | | startup/pending publication visibility | fixed | retained executor records gate startup lists through Home recovery; pre-receipt failure retains accepted rows; crash tests query the startup path | | search/backlink reads of working files | fixed | shared generation leases cover file reads and complete builds/snippet responses; pending/conflicted actions refuse source reads, including after restart | | search publication split across fetch batches | fixed | private file-backed staging; one final SQLite transaction; related action participants expand together even with one journal event and batch size one; queries pin one read snapshot | | concurrent reader regressions | passed | readers at 24 two-note action boundaries; pinned before-state search build; conflict/restart and recorder failure; concurrent search result polling through atomic refresh, removals, failure and cancellation | | fmt | passed | `cargo fmt --all -- --check` | | workspace clippy | passed | `~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings` | | notes | passed | 484 passed, 0 failed, 6 existing ignored | | calternal | passed | all unit/integration suites passed, existing ignored tests retained | | store | passed | 9 passed, 0 failed | | search | passed | 16 unit + 12 integration passed, 0 failed, 1 existing ignored | all gates ran on dev-01 in `~/Developer/Worktrees/note-write-executor-home`, using the existing Nix shell and VM slots: ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search ``` final log: `~/note-home-r1-final2.log`, `GATES_PASSED`, service result success. expected failing regressions are in `~/note-home-r1-red.log`; focused Home coverage passed in `~/note-home-r1-focused2.log`. no builds/tests ran on the control host. commits were pushed after each step; the control worktree is clean. decisions and remaining limits: search remains an asynchronous journal follower, but each refresh exposes a complete accepted generation. source reads wait for a generation lease or refuse pending recovery; no working-file fallback follows a transaction refusal. the retained participant graph is rebuilt from the WAL and expands overlapping actions. search staging uses O(D) temporary disk with bounded fetch memory, and group tracking uses O(G) retained memberships. a full search reconciliation can delay a note action while holding its shared lease; startup lists with retained executor records can wait for recovery. existing executor scope, WAL limits and outside-tool filesystem visibility limits remain. no performance, Apple or power-loss pass is claimed. implementation details and failure/cost notes are in `docs/implementation/note-write-executor-home.md`. round 1 of max 3; independent re-review pending. never merge.
kayg added 18 commits 2026-10-31 20:46:23 +00:00
docs: scope shared note-write executor byte protocol slice
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
0d951932b1
docs(notes): plan Home executor integration
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
b93aaa844c
kayg changed title from WIP: notes executor Home integration, slice 0b to notes: integrate shared executor with Home save publication 2026-10-31 21:20:12 +00:00
kayg commented 2026-10-31 21:24:15 +00:00
Author
Owner
Copy link

Independent review, round 1

Head reviewed: 0f38d06f441c5f0c02752dce8d71fffe57ab5a12, delta from #809 at f30b760a. Read both plans’ shared-executor sections and the existing room/save/publication paths.

Verdict: fix first.

P1: list/plugin rows become readable before journal publication

crates/notes/src/write_executor/home.rs:367–390; reader: crates/notes/src/meta.rs:1387–1406.

executor_meta commits the new rows before record_once runs. Notes::page reads those rows without the generation lock or pending-publication checks, including its list-only startup path. Pause at Phase::IndexInstalled, or let the recorder fail before committing: the list returns both NEW rows while the journal still has no transaction events and room reads refuse. After a crash at this boundary, the persisted list can also be served before Home recovery completes. Atomic installation of all index rows prevents a split within that database, but does not provide the requested consistent index/journal publication boundary.

Fix: keep a separately accepted generation, or gate every affected derived reader through publication/recovery, including list-only startup reads. Preserve the old accepted rows or refuse while publication is pending; a live generation lock alone does not cover a returned publication error or unopened Home. Add tests that query list/plugin rows at IndexInstalled, with a recorder failure before commit, and immediately after restart before Home recovery.

P1: internal file readers bypass executor isolation

crates/notes/src/search.rs:58–72; crates/notes/src/backlinks.rs:177–187; integration boundary: crates/notes/src/write_executor/home.rs:242.

The new guards cover rooms and peek, but search’s provider still reads the mutable note files directly. Pause a two-note action at Placed(0): an initial/reconciliation search build can read the first post-image and second pre-image, then publish that mixed state to its own index without an executor journal event. This is a cooperating internal reader, distinct from the documented limitation for outside tools. The backlink fallback has the same issue after a conflicted action: live_text turns the room’s refusal into None, then text_of reads the partially placed file anyway.

Fix: route internal text consumers through an accepted-generation reader. Pin reads against placement and check persistent pending/conflict state; never treat a transaction refusal as permission to fall back to disk. Search batch publication must also avoid combining documents from opposite sides of a generation switch. Add a search-build test paused after the first placement and a backlink-snippet test after a multi-note conflict.

No P0 findings. No additional P2/P3 findings.

Validation: on dev-01, in ~/Developer/Worktrees/note-write-executor-home, through nix develop -c bash -c:

~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor::tests::home:: -- --test-threads=1

9 passed, 0 failed. These existing tests cover save serialization, crash recovery, idempotent publication retry and live room updates, but do not exercise the reader scenarios above. Findings are from code-path inspection; no new regression tests were added. Existing notes tests are unchanged. No code pushed and no control-host builds/tests run.

## Independent review, round 1 Head reviewed: `0f38d06f441c5f0c02752dce8d71fffe57ab5a12`, delta from #809 at `f30b760a`. Read both plans’ shared-executor sections and the existing room/save/publication paths. **Verdict: fix first.** ### P1: list/plugin rows become readable before journal publication `crates/notes/src/write_executor/home.rs:367–390`; reader: `crates/notes/src/meta.rs:1387–1406`. `executor_meta` commits the new rows before `record_once` runs. `Notes::page` reads those rows without the generation lock or pending-publication checks, including its list-only startup path. Pause at `Phase::IndexInstalled`, or let the recorder fail before committing: the list returns both NEW rows while the journal still has no transaction events and room reads refuse. After a crash at this boundary, the persisted list can also be served before Home recovery completes. Atomic installation of all index rows prevents a split within that database, but does not provide the requested consistent index/journal publication boundary. Fix: keep a separately accepted generation, or gate every affected derived reader through publication/recovery, including list-only startup reads. Preserve the old accepted rows or refuse while publication is pending; a live generation lock alone does not cover a returned publication error or unopened Home. Add tests that query list/plugin rows at IndexInstalled, with a recorder failure before commit, and immediately after restart before Home recovery. ### P1: internal file readers bypass executor isolation `crates/notes/src/search.rs:58–72`; `crates/notes/src/backlinks.rs:177–187`; integration boundary: `crates/notes/src/write_executor/home.rs:242`. The new guards cover rooms and peek, but search’s provider still reads the mutable note files directly. Pause a two-note action at `Placed(0)`: an initial/reconciliation search build can read the first post-image and second pre-image, then publish that mixed state to its own index without an executor journal event. This is a cooperating internal reader, distinct from the documented limitation for outside tools. The backlink fallback has the same issue after a conflicted action: `live_text` turns the room’s refusal into `None`, then `text_of` reads the partially placed file anyway. Fix: route internal text consumers through an accepted-generation reader. Pin reads against placement and check persistent pending/conflict state; never treat a transaction refusal as permission to fall back to disk. Search batch publication must also avoid combining documents from opposite sides of a generation switch. Add a search-build test paused after the first placement and a backlink-snippet test after a multi-note conflict. No P0 findings. No additional P2/P3 findings. Validation: on dev-01, in `~/Developer/Worktrees/note-write-executor-home`, through `nix develop -c bash -c`: ```sh ~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor::tests::home:: -- --test-threads=1 ``` 9 passed, 0 failed. These existing tests cover save serialization, crash recovery, idempotent publication retry and live room updates, but do not exercise the reader scenarios above. Findings are from code-path inspection; no new regression tests were added. Existing notes tests are unchanged. No code pushed and no control-host builds/tests run.
kayg commented 2026-10-31 21:28:36 +00:00
Author
Owner
Copy link

round 1 implementation plan: keep list/plugin rows at the accepted generation until the journal receipt commits, then install the whole derived batch atomically. shared generation leases cover search fetch-through-publication and backlink rows-through-snippets; pending/conflicted actions refuse file-derived reads after a returned error. list-only startup keeps the last accepted rows. recovery repeats publication idempotently.

cost: K action participants; unchanged executor caps. search stages documents in bounded batches on its writer connection and publishes a complete refresh in one SQLite transaction, avoiding a full-Home document buffer.

failure effect detection recovery regression
placement in progress file-derived readers wait shared generation lease coordinator finishes concurrent readers at every step
recorder fails before receipt list stays before-state; file readers refuse pending action same-key retry/restart recorder failure + unopened list
receipt before derived commit list stays before-state WAL publication cursor replay whole batch crash boundaries
conflict no working-file fallback pending participant check explicit recovery backlink/search refusal
search refresh interrupted staged rows remain invisible private staging batch discard/reconcile atomic refresh regression

VM gates: fmt; workspace clippy with -D warnings; notes, calternal, store and search tests. never merge. no repository settings changes.

round 1 implementation plan: keep list/plugin rows at the accepted generation until the journal receipt commits, then install the whole derived batch atomically. shared generation leases cover search fetch-through-publication and backlink rows-through-snippets; pending/conflicted actions refuse file-derived reads after a returned error. list-only startup keeps the last accepted rows. recovery repeats publication idempotently. cost: K action participants; unchanged executor caps. search stages documents in bounded batches on its writer connection and publishes a complete refresh in one SQLite transaction, avoiding a full-Home document buffer. | failure | effect | detection | recovery | regression | | --- | --- | --- | --- | --- | | placement in progress | file-derived readers wait | shared generation lease | coordinator finishes | concurrent readers at every step | | recorder fails before receipt | list stays before-state; file readers refuse | pending action | same-key retry/restart | recorder failure + unopened list | | receipt before derived commit | list stays before-state | WAL publication cursor | replay whole batch | crash boundaries | | conflict | no working-file fallback | pending participant check | explicit recovery | backlink/search refusal | | search refresh interrupted | staged rows remain invisible | private staging batch | discard/reconcile | atomic refresh regression | VM gates: fmt; workspace clippy with -D warnings; notes, calternal, store and search tests. never merge. no repository settings changes.
kayg commented 2026-10-31 21:43:27 +00:00
Author
Owner
Copy link

round 1 implementation status

head: 966a97bb14d1ebee3e4c40342399477c1c2508d0. both P1 findings addressed; ready for independent re-review. never merged; repository settings unchanged.

item status evidence
list/plugin rows before journal receipt fixed journal receipt now precedes one accepted list/plugin/link commit; page counts and rows share a read transaction
startup/pending publication visibility fixed retained executor records gate startup lists through Home recovery; pre-receipt failure retains accepted rows; crash tests query the startup path
search/backlink reads of working files fixed shared generation leases cover file reads and complete builds/snippet responses; pending/conflicted actions refuse source reads, including after restart
search publication split across fetch batches fixed private file-backed staging; one final SQLite transaction; related action participants expand together even with one journal event and batch size one; queries pin one read snapshot
concurrent reader regressions passed readers at 24 two-note action boundaries; pinned before-state search build; conflict/restart and recorder failure; concurrent search result polling through atomic refresh, removals, failure and cancellation
fmt passed cargo fmt --all -- --check
workspace clippy passed ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
notes passed 484 passed, 0 failed, 6 existing ignored
calternal passed all unit/integration suites passed, existing ignored tests retained
store passed 9 passed, 0 failed
search passed 16 unit + 12 integration passed, 0 failed, 1 existing ignored

all gates ran on dev-01 in ~/Developer/Worktrees/note-write-executor-home, using the existing Nix shell and VM slots:

cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search

final log: ~/note-home-r1-final2.log, GATES_PASSED, service result success. expected failing regressions are in ~/note-home-r1-red.log; focused Home coverage passed in ~/note-home-r1-focused2.log. no builds/tests ran on the control host. commits were pushed after each step; the control worktree is clean.

decisions and remaining limits: search remains an asynchronous journal follower, but each refresh exposes a complete accepted generation. source reads wait for a generation lease or refuse pending recovery; no working-file fallback follows a transaction refusal. the retained participant graph is rebuilt from the WAL and expands overlapping actions. search staging uses O(D) temporary disk with bounded fetch memory, and group tracking uses O(G) retained memberships. a full search reconciliation can delay a note action while holding its shared lease; startup lists with retained executor records can wait for recovery. existing executor scope, WAL limits and outside-tool filesystem visibility limits remain. no performance, Apple or power-loss pass is claimed. implementation details and failure/cost notes are in docs/implementation/note-write-executor-home.md.

round 1 of max 3; independent re-review pending. never merge.

## round 1 implementation status head: `966a97bb14d1ebee3e4c40342399477c1c2508d0`. both P1 findings addressed; ready for independent re-review. never merged; repository settings unchanged. | item | status | evidence | | --- | --- | --- | | list/plugin rows before journal receipt | fixed | journal receipt now precedes one accepted list/plugin/link commit; page counts and rows share a read transaction | | startup/pending publication visibility | fixed | retained executor records gate startup lists through Home recovery; pre-receipt failure retains accepted rows; crash tests query the startup path | | search/backlink reads of working files | fixed | shared generation leases cover file reads and complete builds/snippet responses; pending/conflicted actions refuse source reads, including after restart | | search publication split across fetch batches | fixed | private file-backed staging; one final SQLite transaction; related action participants expand together even with one journal event and batch size one; queries pin one read snapshot | | concurrent reader regressions | passed | readers at 24 two-note action boundaries; pinned before-state search build; conflict/restart and recorder failure; concurrent search result polling through atomic refresh, removals, failure and cancellation | | fmt | passed | `cargo fmt --all -- --check` | | workspace clippy | passed | `~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings` | | notes | passed | 484 passed, 0 failed, 6 existing ignored | | calternal | passed | all unit/integration suites passed, existing ignored tests retained | | store | passed | 9 passed, 0 failed | | search | passed | 16 unit + 12 integration passed, 0 failed, 1 existing ignored | all gates ran on dev-01 in `~/Developer/Worktrees/note-write-executor-home`, using the existing Nix shell and VM slots: ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search ``` final log: `~/note-home-r1-final2.log`, `GATES_PASSED`, service result success. expected failing regressions are in `~/note-home-r1-red.log`; focused Home coverage passed in `~/note-home-r1-focused2.log`. no builds/tests ran on the control host. commits were pushed after each step; the control worktree is clean. decisions and remaining limits: search remains an asynchronous journal follower, but each refresh exposes a complete accepted generation. source reads wait for a generation lease or refuse pending recovery; no working-file fallback follows a transaction refusal. the retained participant graph is rebuilt from the WAL and expands overlapping actions. search staging uses O(D) temporary disk with bounded fetch memory, and group tracking uses O(G) retained memberships. a full search reconciliation can delay a note action while holding its shared lease; startup lists with retained executor records can wait for recovery. existing executor scope, WAL limits and outside-tool filesystem visibility limits remain. no performance, Apple or power-loss pass is claimed. implementation details and failure/cost notes are in `docs/implementation/note-write-executor-home.md`. round 1 of max 3; independent re-review pending. never merge.
kayg commented 2026-10-31 21:48:12 +00:00
Author
Owner
Copy link

Independent review, round 2

head reviewed: 966a97bb14d1ebee3e4c40342399477c1c2508d0, stacked on #809 (f30b760a); round 1 delta from 0f38d06f. verdict: fix first.

round 1 item status does the concurrent-reader test demonstrate the fix?
P1: list/plugin rows visible before journal publication, including startup resolved yes for the reported failure: derived_readers_wait_through_every_action_step checks complete old/new list rows and requires both transaction events before NEW rows, at 24 boundaries including IndexInstalled. failed_receipt_preserves_list_before_and_after_restart checks pre-receipt refusal and persisted old rows. The process-crash test now calls public page before explicitly opening Home and checks publication. Plugin rows share the same accepted SQLite commit, although the new test does not separately assert a plugin-specific filter.
P1: search/backlink readers see partially applied changes resolved yes, with complementary tests: the 24-boundary test shows direct search and backlink readers wait and return both NEW documents/snippets; derived_readers_refuse_conflicted_working_files checks refusal live and after restart. pinned_search_generation_serializes_an_entire_build_and_expands_actions checks a retained before-state reader and action-group expansion. The three search indexer tests exercise batch-size-one publication, polling readers, removals, failure and cancellation. The actual Notes snapshot and search publication are tested separately rather than in one end-to-end case.

new P1: an unbounded search lease can indefinitely stall executor admission and subsequent Home readers

Locations: crates/notes/src/search.rs:74–85, 108–123, 136–143; crates/search/src/indexer.rs:211–248, 264–271; crates/notes/src/write_executor/home.rs:154–157.

A reconciliation pins the Home-wide generation read guard before listing and retains it through every fetch, staging operation and final index publication. Batch yields do not release it. Neither the lease nor the executor's write().await has a time bound or shutdown-select. The source already describes full builds at 100,000 notes as taking minutes; this review did not measure that duration. Actual writer delay is the remaining entire rebuild/publication time, with no enforced upper bound. A retained snapshot or stuck file read can keep it waiting indefinitely.

Tokio's write-preferring RwLock also queues subsequent generation readers behind that waiting writer. Thus one stalled rebuild plus one queued action can stall peek, backlinks and new provider snapshots for unrelated notes in that Home. Aborting the action caller does not remove the waiter: the coordinator is Home-owned. A pending coordinator is also joined by Home shutdown without a bound.

The new pinned-generation test demonstrates the dependency: the action remains pending for 40 ms and completes only after drop(before). It does not demonstrate bounded writer progress. This conflicts with ADR 0014's short-lock/background-work requirement. Please provide bounded pre-PREPARE admission/refusal and a generation-reading design that preserves complete generations without holding a Home-wide exclusion lease throughout an arbitrarily long rebuild. Merely timing out an async wrapper around spawn_blocking does not reclaim a guard held by the still-running closure. Add a deliberately stalled-reader test that verifies bounded action response, subsequent reader progress, and shutdown behavior.

save latency, crash lifetime and lock ordering

  • Ordinary typing saves/Flush do not acquire executor_generation; the coordinator waits for the generation lock before jobs, lifecycle, actor leases and meta_update. A rebuild or an action queued behind it therefore adds no direct lock wait to ordinary saves. No performance budget pass is established by these functional tests.
  • Once an action obtains actor leases, edits/Flush to its participants wait for the action, and other rooms can wait on its Home-wide meta_update during publication. This behavior predates the round 1 fixes. ordinary_save_serializes_behind_action_and_caller_cancellation explicitly verifies waiting, not a fast-save upper bound. The fixes do not add a rebuild wait while those actor leases are held.
  • No durable generation-lease leak on process crash: guards are in-memory RAII objects and Home recreates the lock. Normal drops/errors release them. Cancellation of a running blocking read deliberately retains its owned guard until the closure returns; a stuck closure is the live-process stall above.
  • No new lock-order cycle found in the inspected production paths: generation precedes jobs/lifecycle/room requests/meta, actors do not acquire generation, backlink snippet helpers reuse the existing guard, and search writer jobs do not call Home. The availability issue above does not require a deadlock cycle.

No new P0 or separate P2/P3 findings.

Validation independently run on dev-01 at the reviewed SHA, in ~/Developer/Worktrees/note-write-executor-home, through nix develop -c bash -c:

~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor::tests::home:: -- --test-threads=1
~/bin/slot cargo test --locked -j4 -p calternal-search indexer::tests:: -- --test-threads=1

13 Home tests and 3 search indexer tests passed, 0 failed. No new test/code edits, no code pushed, no control-host builds/tests, no merge. The new P1 is supported by the lock/lifetime paths and the existing pinned-reader test; no stalled-filesystem experiment or latency measurement was performed.

## Independent review, round 2 head reviewed: `966a97bb14d1ebee3e4c40342399477c1c2508d0`, stacked on #809 (`f30b760a`); round 1 delta from `0f38d06f`. **verdict: fix first.** | round 1 item | status | does the concurrent-reader test demonstrate the fix? | | --- | --- | --- | | P1: list/plugin rows visible before journal publication, including startup | resolved | yes for the reported failure: `derived_readers_wait_through_every_action_step` checks complete old/new list rows and requires both transaction events before NEW rows, at 24 boundaries including `IndexInstalled`. `failed_receipt_preserves_list_before_and_after_restart` checks pre-receipt refusal and persisted old rows. The process-crash test now calls public `page` before explicitly opening Home and checks publication. Plugin rows share the same accepted SQLite commit, although the new test does not separately assert a plugin-specific filter. | | P1: search/backlink readers see partially applied changes | resolved | yes, with complementary tests: the 24-boundary test shows direct search and backlink readers wait and return both NEW documents/snippets; `derived_readers_refuse_conflicted_working_files` checks refusal live and after restart. `pinned_search_generation_serializes_an_entire_build_and_expands_actions` checks a retained before-state reader and action-group expansion. The three search indexer tests exercise batch-size-one publication, polling readers, removals, failure and cancellation. The actual Notes snapshot and search publication are tested separately rather than in one end-to-end case. | ### new P1: an unbounded search lease can indefinitely stall executor admission and subsequent Home readers Locations: `crates/notes/src/search.rs:74–85, 108–123, 136–143`; `crates/search/src/indexer.rs:211–248, 264–271`; `crates/notes/src/write_executor/home.rs:154–157`. A reconciliation pins the Home-wide generation read guard before listing and retains it through every fetch, staging operation and final index publication. Batch yields do not release it. Neither the lease nor the executor's `write().await` has a time bound or shutdown-select. The source already describes full builds at 100,000 notes as taking minutes; this review did not measure that duration. Actual writer delay is the remaining entire rebuild/publication time, with no enforced upper bound. A retained snapshot or stuck file read can keep it waiting indefinitely. Tokio's write-preferring RwLock also queues subsequent generation readers behind that waiting writer. Thus one stalled rebuild plus one queued action can stall peek, backlinks and new provider snapshots for unrelated notes in that Home. Aborting the action caller does not remove the waiter: the coordinator is Home-owned. A pending coordinator is also joined by Home shutdown without a bound. The new pinned-generation test demonstrates the dependency: the action remains pending for 40 ms and completes only after `drop(before)`. It does not demonstrate bounded writer progress. This conflicts with ADR 0014's short-lock/background-work requirement. Please provide bounded pre-PREPARE admission/refusal and a generation-reading design that preserves complete generations without holding a Home-wide exclusion lease throughout an arbitrarily long rebuild. Merely timing out an async wrapper around `spawn_blocking` does not reclaim a guard held by the still-running closure. Add a deliberately stalled-reader test that verifies bounded action response, subsequent reader progress, and shutdown behavior. ### save latency, crash lifetime and lock ordering - Ordinary typing saves/Flush do **not** acquire `executor_generation`; the coordinator waits for the generation lock **before** jobs, lifecycle, actor leases and `meta_update`. A rebuild or an action queued behind it therefore adds no direct lock wait to ordinary saves. No performance budget pass is established by these functional tests. - Once an action obtains actor leases, edits/Flush to its participants wait for the action, and other rooms can wait on its Home-wide `meta_update` during publication. This behavior predates the round 1 fixes. `ordinary_save_serializes_behind_action_and_caller_cancellation` explicitly verifies waiting, not a fast-save upper bound. The fixes do not add a rebuild wait while those actor leases are held. - No durable generation-lease leak on process crash: guards are in-memory RAII objects and Home recreates the lock. Normal drops/errors release them. Cancellation of a running blocking read deliberately retains its owned guard until the closure returns; a stuck closure is the live-process stall above. - No new lock-order cycle found in the inspected production paths: generation precedes jobs/lifecycle/room requests/meta, actors do not acquire generation, backlink snippet helpers reuse the existing guard, and search writer jobs do not call Home. The availability issue above does not require a deadlock cycle. No new P0 or separate P2/P3 findings. Validation independently run on dev-01 at the reviewed SHA, in `~/Developer/Worktrees/note-write-executor-home`, through `nix develop -c bash -c`: ```sh ~/bin/slot cargo test --locked -j4 -p calternal-notes write_executor::tests::home:: -- --test-threads=1 ~/bin/slot cargo test --locked -j4 -p calternal-search indexer::tests:: -- --test-threads=1 ``` 13 Home tests and 3 search indexer tests passed, 0 failed. No new test/code edits, no code pushed, no control-host builds/tests, no merge. The new P1 is supported by the lock/lifetime paths and the existing pinned-reader test; no stalled-filesystem experiment or latency measurement was performed.
kayg commented 2026-10-31 21:52:11 +00:00
Author
Owner
Copy link

round 2 plan: remove Home reader exclusion. commit accepted note text, path and a content stamp alongside the list/plugin/link rows. readers pin a cheap immutable generation token and query versioned accepted rows, never working files. each writer creates a new generation without waiting for readers. keep at most four old generations; expired snapshots return a retryable stale-snapshot error. unpinned versions are reclaimed on the next write. no long SQLite read transaction or reader-owned lock enters admission.

sizes/cost: N accepted source bytes on derived disk, K changed notes, R <= 4 retained old generations. O(K log N) version updates plus indexed reclamation; source bodies are read per document, not copied per snapshot. snapshots carry O(1) token state, with bounded query batches. existing executor image/WAL caps remain.

failure effect detection recovery test
idle/stuck snapshot retains bounded old rows only generation age revoke oldest, retry retained reader + writer/readers/shutdown
retention cap exceeded old reader gets stale snapshot token validation before/after query discard staged search batch, retry repeated publication with retained tokens
interrupted publication accepted rows stay entirely old/new SQLite transaction + executor cursor existing WAL recovery process kill with pinned snapshots
concurrent ordinary save bounded wait/refusal timed admission retry without placing partial bytes save timing during executor + rebuild

validation: failing stuck-reader regression first; then fmt, workspace clippy -D warnings and notes/calternal/store/search tests on dev-01. measure ordinary save p95 on a perf VM against current dev using the same fixture. never merge; no repository settings changes.

round 2 plan: remove Home reader exclusion. commit accepted note text, path and a content stamp alongside the list/plugin/link rows. readers pin a cheap immutable generation token and query versioned accepted rows, never working files. each writer creates a new generation without waiting for readers. keep at most four old generations; expired snapshots return a retryable stale-snapshot error. unpinned versions are reclaimed on the next write. no long SQLite read transaction or reader-owned lock enters admission. sizes/cost: N accepted source bytes on derived disk, K changed notes, R <= 4 retained old generations. O(K log N) version updates plus indexed reclamation; source bodies are read per document, not copied per snapshot. snapshots carry O(1) token state, with bounded query batches. existing executor image/WAL caps remain. | failure | effect | detection | recovery | test | | --- | --- | --- | --- | --- | | idle/stuck snapshot | retains bounded old rows only | generation age | revoke oldest, retry | retained reader + writer/readers/shutdown | | retention cap exceeded | old reader gets stale snapshot | token validation before/after query | discard staged search batch, retry | repeated publication with retained tokens | | interrupted publication | accepted rows stay entirely old/new | SQLite transaction + executor cursor | existing WAL recovery | process kill with pinned snapshots | | concurrent ordinary save | bounded wait/refusal | timed admission | retry without placing partial bytes | save timing during executor + rebuild | validation: failing stuck-reader regression first; then fmt, workspace clippy -D warnings and notes/calternal/store/search tests on dev-01. measure ordinary save p95 on a perf VM against current dev using the same fixture. never merge; no repository settings changes.
kayg commented 2026-10-31 22:33:25 +00:00
Author
Owner
Copy link

round 2 implementation status

head: 99cfe9416becedf2831b9354314419a8eff8062a. ready for final independent review. never merged; repository settings unchanged.

item status evidence
unbounded reader exclusion lease fixed Home generation RwLock removed. readers pin an Arc token for immutable accepted SQLite source rows; admission never waits for reader completion or drop
writer and later reader progress passed deliberately retained snapshot plus stalled real Notes-to-search rebuild; action and fresh readers complete while the original reader remains stalled; shutdown completes with a retained snapshot
retained generations bounded four pinned old generations plus current. a lone reader survives arbitrary writes; intermediate versions are reclaimed. a fifth old pin revokes the oldest with a retryable stale-snapshot error
last-reader drop and retry passed 20 writes preserve exactly one pinned old source version; dropped pins permit reclamation on the next write; expired snapshots fail before/after reads and before search publication; search follower retries without advancing its cursor
accepted source consistency passed search, backlink snippets and peek fallback use accepted rows; journal receipt precedes the atomic source/list/plugin/link commit; readers at 24 action boundaries see complete old/new state; conflicts retain accepted text
pinned-reader crash recovery passed existing real process-kill Home tests now pin a snapshot before every action; repeated recovery preserves authored history, accepted rows and one journal batch
ordinary save admission bounded file placement no longer holds meta_update; leased actors promptly refuse Edit/Flush for retry, including caller cancellation. ordinary metadata waits check executor publication every 25 ms and return NotSavedYet; regression requires participant edit and Flush responses within 100 ms
fmt passed cargo fmt --all -- --check
workspace clippy passed ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
notes passed 485 passed, 0 failed, 6 existing ignored; doc tests passed
calternal passed 61 passed across 11 unit/integration suites, 0 failed, 4 existing ignored
store passed 9 passed, 0 failed; doc tests passed
search passed 29 passed across unit and integration suites, 0 failed, 2 ignored (existing perf gate and the explicitly run new latency harness); doc tests passed
concurrent ordinary save latency measured median of three run p95s: head 142.943 ms versus dev 132.197 ms, +10.746 ms / +8.13%; full results below

all gates ran at this head on dev-01, in ~/Developer/Worktrees/note-write-executor-home, through the Nix shell and VM slots:

cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search

final log: ~/note-home-r2-verified2.log, GATES_PASSED. the original regression failed with “writer waited for an idle reader” in ~/note-home-r2-red2.log before the fix. no build or test ran on the control host. commits were pushed after each step; the control worktree is clean.

latency comparison

three paired runs on calternal-perf-02-backup01, under the existing ~/bin/perf-lock shared lock. binaries were built on dev-01 using the default optimized release profile and identical Nix tools, relayed through the control host, with SHA-256 checked at every destination. data lived on disk at /home/claude/note-home-r2-perf-tmp.

same fixture: 1,000 notes of about 4 KiB, four runtime workers, 200 one-character appends with an explicit save each, automatic flush/watch/rename disabled. a new empty search index is fully rebuilt repeatedly throughout each run. head concurrently repeats 16-note executor actions. current dev baseline is a55521f94ef7337f0793037ae2cb2c27266d5720; it has no executor API, so the baseline fixture replaces only the executor helper with a no-op. it still runs the same save and full-rebuild workload.

paired run dev save p95 ms head save p95 ms dev edit + save p95 ms head edit + save p95 ms
1 132.197 113.452 152.038 156.094
2 174.485 256.825 211.238 297.260
3 117.388 142.943 137.126 190.369
median run p95 132.197 142.943 152.038 190.369

head completed 22 executor actions and 120 full rebuilds across 600 successful saves, with zero save retries; baseline completed 131 full rebuilds. edit + save median run p95 increased by 38.331 ms / 25.21%. every appended character was verified exactly once. variability is visible in the per-run table: these measurements do not establish an absolute storage completion deadline. the 25 ms mechanism bounds waiting for executor metadata publication, not disk I/O.

reproduction:

# build on dev-01 through nix develop -c bash -c
~/bin/slot cargo test --locked --release -j4 -p calternal-search --test note_generation --no-run
# run the copied binary on the perf VM under ~/bin/perf-lock
<binary> --ignored --exact ordinary_save_p95_during_executor_and_rebuild --nocapture --test-threads=1

perf log: ~/note-home-r2-perf3.log on perf-02, PERF_FINISHED. binary SHA-256:

  • head: a3774e9412f1e44b7203a8d4d64a18fa78799bb668f4da86aa805d6bfb3ee484
  • dev: 24cc8199bc182c97c52ee5ae96651fd9acf88c9655c0e0e847117cd6429014b0

costs and limits: snapshots are O(1) pins with per-operation SQLite connections. retained source disk is O((R+1)N), R <= 4. normal updates/reclamation are O(K R log N); a changed retained-pin set may scan O(R N) retired rows. Arc drops permit lazy SQL reclamation on the next write; restart removes all obsolete versions. no reader-owned transaction spans a rebuild. oldest-reader revocation and concurrent backlink generation changes require retry. existing executor scope/WAL/image limits remain. implementation design is documented in docs/implementation/note-write-executor-home.md. no Apple or power-loss pass is claimed.

## round 2 implementation status head: `99cfe9416becedf2831b9354314419a8eff8062a`. ready for final independent review. never merged; repository settings unchanged. | item | status | evidence | | --- | --- | --- | | unbounded reader exclusion lease | fixed | Home generation RwLock removed. readers pin an Arc token for immutable accepted SQLite source rows; admission never waits for reader completion or drop | | writer and later reader progress | passed | deliberately retained snapshot plus stalled real Notes-to-search rebuild; action and fresh readers complete while the original reader remains stalled; shutdown completes with a retained snapshot | | retained generations | bounded | four pinned old generations plus current. a lone reader survives arbitrary writes; intermediate versions are reclaimed. a fifth old pin revokes the oldest with a retryable stale-snapshot error | | last-reader drop and retry | passed | 20 writes preserve exactly one pinned old source version; dropped pins permit reclamation on the next write; expired snapshots fail before/after reads and before search publication; search follower retries without advancing its cursor | | accepted source consistency | passed | search, backlink snippets and peek fallback use accepted rows; journal receipt precedes the atomic source/list/plugin/link commit; readers at 24 action boundaries see complete old/new state; conflicts retain accepted text | | pinned-reader crash recovery | passed | existing real process-kill Home tests now pin a snapshot before every action; repeated recovery preserves authored history, accepted rows and one journal batch | | ordinary save admission | bounded | file placement no longer holds meta_update; leased actors promptly refuse Edit/Flush for retry, including caller cancellation. ordinary metadata waits check executor publication every 25 ms and return NotSavedYet; regression requires participant edit and Flush responses within 100 ms | | fmt | passed | `cargo fmt --all -- --check` | | workspace clippy | passed | `~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings` | | notes | passed | 485 passed, 0 failed, 6 existing ignored; doc tests passed | | calternal | passed | 61 passed across 11 unit/integration suites, 0 failed, 4 existing ignored | | store | passed | 9 passed, 0 failed; doc tests passed | | search | passed | 29 passed across unit and integration suites, 0 failed, 2 ignored (existing perf gate and the explicitly run new latency harness); doc tests passed | | concurrent ordinary save latency | measured | median of three run p95s: head 142.943 ms versus dev 132.197 ms, +10.746 ms / +8.13%; full results below | all gates ran at this head on dev-01, in `~/Developer/Worktrees/note-write-executor-home`, through the Nix shell and VM slots: ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search ``` final log: `~/note-home-r2-verified2.log`, `GATES_PASSED`. the original regression failed with “writer waited for an idle reader” in `~/note-home-r2-red2.log` before the fix. no build or test ran on the control host. commits were pushed after each step; the control worktree is clean. ### latency comparison three paired runs on `calternal-perf-02-backup01`, under the existing `~/bin/perf-lock` shared lock. binaries were built on dev-01 using the default optimized release profile and identical Nix tools, relayed through the control host, with SHA-256 checked at every destination. data lived on disk at `/home/claude/note-home-r2-perf-tmp`. same fixture: 1,000 notes of about 4 KiB, four runtime workers, 200 one-character appends with an explicit save each, automatic flush/watch/rename disabled. a new empty search index is fully rebuilt repeatedly throughout each run. head concurrently repeats 16-note executor actions. current dev baseline is `a55521f94ef7337f0793037ae2cb2c27266d5720`; it has no executor API, so the baseline fixture replaces only the executor helper with a no-op. it still runs the same save and full-rebuild workload. | paired run | dev save p95 ms | head save p95 ms | dev edit + save p95 ms | head edit + save p95 ms | | --- | ---: | ---: | ---: | ---: | | 1 | 132.197 | 113.452 | 152.038 | 156.094 | | 2 | 174.485 | 256.825 | 211.238 | 297.260 | | 3 | 117.388 | 142.943 | 137.126 | 190.369 | | median run p95 | 132.197 | 142.943 | 152.038 | 190.369 | head completed 22 executor actions and 120 full rebuilds across 600 successful saves, with zero save retries; baseline completed 131 full rebuilds. edit + save median run p95 increased by 38.331 ms / 25.21%. every appended character was verified exactly once. variability is visible in the per-run table: these measurements do not establish an absolute storage completion deadline. the 25 ms mechanism bounds waiting for executor metadata publication, not disk I/O. reproduction: ```sh # build on dev-01 through nix develop -c bash -c ~/bin/slot cargo test --locked --release -j4 -p calternal-search --test note_generation --no-run # run the copied binary on the perf VM under ~/bin/perf-lock <binary> --ignored --exact ordinary_save_p95_during_executor_and_rebuild --nocapture --test-threads=1 ``` perf log: `~/note-home-r2-perf3.log` on perf-02, `PERF_FINISHED`. binary SHA-256: - head: `a3774e9412f1e44b7203a8d4d64a18fa78799bb668f4da86aa805d6bfb3ee484` - dev: `24cc8199bc182c97c52ee5ae96651fd9acf88c9655c0e0e847117cd6429014b0` costs and limits: snapshots are O(1) pins with per-operation SQLite connections. retained source disk is O((R+1)N), R <= 4. normal updates/reclamation are O(K R log N); a changed retained-pin set may scan O(R N) retired rows. Arc drops permit lazy SQL reclamation on the next write; restart removes all obsolete versions. no reader-owned transaction spans a rebuild. oldest-reader revocation and concurrent backlink generation changes require retry. existing executor scope/WAL/image limits remain. implementation design is documented in `docs/implementation/note-write-executor-home.md`. no Apple or power-loss pass is claimed.
kayg commented 2026-10-31 22:47:18 +00:00
Author
Owner
Copy link

Independent review, round 3 (final)

reviewed head 99cfe9416becedf2831b9354314419a8eff8062a into dev, stacked on #809. no code pushed.

prior findings

finding status evidence
round 1 p1: list/plugin rows visible before journal receipt, including startup fixed the journal receipt precedes the atomic accepted source/list/plugin/link transaction. the process-kill test checks the startup page path and observes either 0 or 2 updated notes, never 1.
round 1 p1: search/backlink readers could mix text from file placement fixed search, peek and backlink snippets read accepted source rows. search validates before staged publication; conflicted or pending rows do not fall back to working files. the publication-boundary test checks old-or-new results.
round 2 p1: a Home generation read lease could hold action admission, later readers and shutdown fixed snapshots pin an Arc token; each query uses its own short SQLite connection and no reader lock or transaction spans awaits. the stalled-rebuild probe passes while the writer and fresh readers finish.
round 2 retention/recovery gates passed at most four distinct old epochs are retained; expired snapshots return retryable stale errors and startup deletes ended rows. the process-kill test covers 24 publication and recovery phases.
round 2 save p95: +10.746 ms (+8.13%), edit+save +38.331 ms (+25.21%) measurement confirmed, attribution open three paired runs vary substantially. the benchmark also runs executor publications that use a durable FULL-sync journal receipt, so it does not isolate source snapshot cost.

new findings

priority finding evidence and recommendation
p1 avoidable full-body copy and duplicate source serialization on saves crates/notes/src/meta.rs:946 clones SearchDoc, including its cleaned body; Source::new immediately clears that body (crates/notes/src/generation.rs:20-22). it then copies borrowed text and serializes the full source into a temporary buffer for BLAKE3 (:24-32); generation::store serializes the same text again (:135-137). have source construction consume the document and owned text, hash canonical fields without a full JSON buffer, and persist one source encoding. this is avoidable hot-path work and is the p1 basis for the verdict. the exact share of the measured 10.7 ms needs an A/B after the fix.
p2 retained bytes have no absolute ceiling the count is bounded to four old generations plus current, so reader duration/retries do not grow row count. bytes are only bounded relative to the total source data in those generations (about 5x one full Home image); Home-wide source bytes have no hard cap, and deleting rows does not shrink SQLite's high-water file allocation. if an absolute byte ceiling is required, add a Home-wide source budget and define admission behavior when it is reached.
p2 retry fairness is not guaranteed under sustained newer-pin churn rapid writes alone do not stale a lone reader: its generation survives arbitrary commits. with more than four distinct pinned old generations, the oldest token is revoked; if four newer long-lived pins repeatedly arrive before each search retry finishes, that reader can keep retrying. the indexer backs off by its quiet interval, but has no fairness policy. this does not block writers.
p0 none
p3 none

focused checks on dev-01

all passed in the Nix shell via ~/bin/slot:

  • cargo test --locked -j4 -p calternal-notes retained_generations_are_capped_and_dropped_pins_are_reclaimed
  • cargo test --locked -j4 -p calternal-notes home_process_crashes_recover_history_index_and_one_journal_batch (covers 24 process-kill phases)
  • cargo test --locked -j4 -p calternal-notes stuck_search_snapshot_never_blocks_writer_readers_or_shutdown
  • cargo test --locked -j4 -p calternal-search --test note_generation stalled_rebuild_uses_old_sources_while_writer_and_new_readers_finish

verdict: fix first.

## Independent review, round 3 (final) reviewed head `99cfe9416becedf2831b9354314419a8eff8062a` into `dev`, stacked on #809. no code pushed. ### prior findings | finding | status | evidence | | --- | --- | --- | | round 1 p1: list/plugin rows visible before journal receipt, including startup | fixed | the journal receipt precedes the atomic accepted source/list/plugin/link transaction. the process-kill test checks the startup page path and observes either 0 or 2 updated notes, never 1. | | round 1 p1: search/backlink readers could mix text from file placement | fixed | search, peek and backlink snippets read accepted source rows. search validates before staged publication; conflicted or pending rows do not fall back to working files. the publication-boundary test checks old-or-new results. | | round 2 p1: a Home generation read lease could hold action admission, later readers and shutdown | fixed | snapshots pin an Arc token; each query uses its own short SQLite connection and no reader lock or transaction spans awaits. the stalled-rebuild probe passes while the writer and fresh readers finish. | | round 2 retention/recovery gates | passed | at most four distinct old epochs are retained; expired snapshots return retryable stale errors and startup deletes ended rows. the process-kill test covers 24 publication and recovery phases. | | round 2 save p95: +10.746 ms (+8.13%), edit+save +38.331 ms (+25.21%) | measurement confirmed, attribution open | three paired runs vary substantially. the benchmark also runs executor publications that use a durable FULL-sync journal receipt, so it does not isolate source snapshot cost. | ### new findings | priority | finding | evidence and recommendation | | --- | --- | --- | | p1 | avoidable full-body copy and duplicate source serialization on saves | `crates/notes/src/meta.rs:946` clones `SearchDoc`, including its cleaned body; `Source::new` immediately clears that body (`crates/notes/src/generation.rs:20-22`). it then copies borrowed text and serializes the full source into a temporary buffer for BLAKE3 (`:24-32`); `generation::store` serializes the same text again (`:135-137`). have source construction consume the document and owned text, hash canonical fields without a full JSON buffer, and persist one source encoding. this is avoidable hot-path work and is the p1 basis for the verdict. the exact share of the measured 10.7 ms needs an A/B after the fix. | | p2 | retained bytes have no absolute ceiling | the count is bounded to four old generations plus current, so reader duration/retries do not grow row count. bytes are only bounded relative to the total source data in those generations (about 5x one full Home image); Home-wide source bytes have no hard cap, and deleting rows does not shrink SQLite's high-water file allocation. if an absolute byte ceiling is required, add a Home-wide source budget and define admission behavior when it is reached. | | p2 | retry fairness is not guaranteed under sustained newer-pin churn | rapid writes alone do not stale a lone reader: its generation survives arbitrary commits. with more than four distinct pinned old generations, the oldest token is revoked; if four newer long-lived pins repeatedly arrive before each search retry finishes, that reader can keep retrying. the indexer backs off by its quiet interval, but has no fairness policy. this does not block writers. | | p0 | none | | | p3 | none | | ### focused checks on dev-01 all passed in the Nix shell via `~/bin/slot`: - `cargo test --locked -j4 -p calternal-notes retained_generations_are_capped_and_dropped_pins_are_reclaimed` - `cargo test --locked -j4 -p calternal-notes home_process_crashes_recover_history_index_and_one_journal_batch` (covers 24 process-kill phases) - `cargo test --locked -j4 -p calternal-notes stuck_search_snapshot_never_blocks_writer_readers_or_shutdown` - `cargo test --locked -j4 -p calternal-search --test note_generation stalled_rebuild_uses_old_sources_while_writer_and_new_readers_finish` **verdict: fix first.**
kayg commented 2026-10-31 22:53:40 +00:00
Author
Owner
Copy link

Round-3 P1 fixed (orchestrator) @ 2e08de74

finding status
p1: full-body clone + temporary full-source buffer per changed note fixed: described moves the SearchDoc into Source (only title, kind, tags, stamp cloned for the list entry); the stamp hash streams serde_json::to_writer into blake3::Hasher, with no buffer. Stamps are byte-for-byte unchanged (same serializer output), so no stored rows change. One more encoding stays in generation::store; it can't be dropped without moving the stamp out of the stored JSON, which is a format change that isn't worth it here.
p2: no absolute byte ceiling accepted for now: 4 old generations is about 5× one Home image. Revisit with tiering quotas.
p2: retry fairness under newer-pin churn accepted: writers are never blocked; this only affects a background reader under unusual churn.

Gates on dev-01 (Nix shell, ~/bin/slot): cargo fmt --all -- --check, cargo clippy --locked -j4 -p calternal-notes -p calternal-search --all-targets -- -D warnings, cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search → 584 passed, 0 failed, 12 ignored.

Save p95 attribution (journal FULL sync vs. snapshot cost) will be measured in the BM5 train perf run. Ready for BM5 (stack #809 → #814 → #813 → #812).

## Round-3 P1 fixed (orchestrator) @ `2e08de74` | finding | status | | --- | --- | | p1: full-body clone + temporary full-source buffer per changed note | fixed: `described` moves the `SearchDoc` into `Source` (only title, kind, tags, stamp cloned for the list entry); the stamp hash streams `serde_json::to_writer` into `blake3::Hasher`, with no buffer. Stamps are byte-for-byte unchanged (same serializer output), so no stored rows change. One more encoding stays in `generation::store`; it can't be dropped without moving the stamp out of the stored JSON, which is a format change that isn't worth it here. | | p2: no absolute byte ceiling | accepted for now: 4 old generations is about 5× one Home image. Revisit with tiering quotas. | | p2: retry fairness under newer-pin churn | accepted: writers are never blocked; this only affects a background reader under unusual churn. | Gates on dev-01 (Nix shell, `~/bin/slot`): `cargo fmt --all -- --check`, `cargo clippy --locked -j4 -p calternal-notes -p calternal-search --all-targets -- -D warnings`, `cargo test --locked -j4 -p calternal-notes -p calternal -p calternal-store -p calternal-search` → 584 passed, 0 failed, 12 ignored. Save p95 attribution (journal FULL sync vs. snapshot cost) will be measured in the BM5 train perf run. Ready for BM5 (stack #809 → #814 → #813 → #812).
kayg commented 2026-11-02 12:42:34 +00:00
Author
Owner
Copy link

ADR 0003 evidence migration: docs/implementation/note-write-executor-home.md

Source commit: 7e2032a0b88abc1cd4c377c4feb88f9f9582a2e9; snapshot from dev 55f35fcd. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable.

# Shared note-write executor, slice 0b

Stack: #809 at f30b760a. Prerequisite audit: #812. Target: dev. No action is enabled. Never merge.

Plan:
1. Add failing Home integration and publication replay tests.
2. Add room leases, durable Home payloads and recovery before Home startup. Use the existing jobs/lifecycle/room/meta_update lock order. Refuse unsupported lifecycle, metadata, rank, claim and occurrence participants.
3. Install authored CRDT history and bases after COMMIT, publish one journal batch with a permanent deduplication key, then commit one accepted metadata batch. Release room leases only after the complete generation is ready. A pending action keeps affected rooms unavailable.
4. Exercise crash/failure boundaries, ordinary-save concurrency, recovery and existing notes behavior. Run VM gates and record results.

Scope: existing-note replacements with stable ids and exact room byte round trips. Create/delete/rename and domain-specific private stores remain refused. The shared strict metadata writer remains a separate prerequisite for Tasks B1.

Costs: K participants, N retained image/state bytes, J retained WAL bytes. Room acquisition O(K log K), payload/lease matching O(K²), history/index installation O(K + N), existing conservative placement O(KN + J). Memory O(N + J). Startup additionally replays H retained history bytes for the distinct identities named by the WAL, O(H + J + N), in bounded identity chunks before exposing the Home. Existing 1,024 path / 64 MiB image / 1 GiB journal caps remain; private CRDT payloads must be bounded too. No Home walk during admission.

| failure | effect | detection | recovery | test |
| --- | --- | --- | --- | --- |
| stale room/file/binding | refuse before PREPARE | exact bytes and stable id/path | new plan | concurrent save and stale input |
| interrupted PREPARE | no file placement | checksummed WAL | drop staging or resume complete record | existing process crash tests |
| interrupted placement | no accepted partial generation | WAL + placement evidence | recorded direction | every placement boundary |
| COMMIT before history | committed, not published | Home payload in WAL | idempotent history install | restart after COMMIT |
| history before index | affected rooms unavailable | transaction receipt | install complete index batch | injected history boundary |
| journal before index | accepted rows remain before-state | outbox cursor | install whole accepted batch | injected publication boundary |
| journal before cursor | no completion reply | permanent journal dedup key | retry without duplicate events | publication boundary |
| caller cancellation | coordinator continues | Home-owned background task | finish or retain pending | dropped caller |
| unsupported participant | no writes | admission | retain existing action path | refusal coverage |

No power-loss, Apple or performance pass is claimed without those gates. BLAKE3 equality assumes distinct content does not share a 256-bit digest; integrity checks do not authenticate the WAL. Placement evidence is required separately.


Implemented boundaries:
- Home owns each active executor under its jobs and lifecycle locks. UUID-ordered actor leases drain ordinary saves, pin revisions and prevent timers, websocket edits, outside ingestion or Flush from interleaving. Peek reads accepted source versions without a working-file fallback.
- PREPARE includes bounded, checksummed exact CRDT pre/post states and the authored update. File placement and COMMIT keep #809's evidence checks. One conditional history transaction installs every authored update/base and its permanent receipt after COMMIT.
- One journal transaction publishes the batch and a permanent deduplication receipt, then one SQLite transaction installs all accepted note/plugin rows. Publication transactions use FULL sync; ordinary journal writes retain their prior sync policy. The WAL publication cursor follows both commits. Same-key retries repair live room release even if cursor installation already succeeded.
- Recovery runs before scan, watcher, bulk replay and readers. Homes with retained executor records also wait for recovery on the startup list path. Published files are never replayed. Derived rows are rebuilt from current history for WAL participants, preserving later ordinary saves. Conflicted identities retain accepted history/index state and refuse source fallback; unrelated notes remain available.

Delivery limits: no task/calendar action or enable switch; stable existing-note replacements only. Namespace creation/removal/rename, metadata validation, ranks, claims, occurrences and Home undo remain refused pending their adapters. Home WAL payload compaction is withheld, so retained records can cause bounded backpressure. No Apple, power-loss or performance pass is claimed. Uncooperative outside tools can observe partial raw filesystem placement, as in #809.


Validation on dev-01 in `~/Developer/Worktrees/note-write-executor-home`, through
`nix develop -c bash -c`, implementation at `a479a812`:

```sh
cargo fmt --all -- --check
~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings
~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal
~/bin/slot cargo test --locked -j4 -p calternal-store
```

All passed. Notes: 480 passed, 0 failed, 6 existing ignored; doc tests passed.
All calternal unit/integration suites passed with their existing ignored tests.
Store: 9 passed, 0 failed; doc tests passed. Focused executor run: 37 passed,
including nine Home tests and 24 real process-kill Home phases with repeated
recovery, live publication retry, cursor I/O errors, caller cancellation,
concurrent ordinary save, unsupported admission and cache rebuild after a later
ordinary save. Existing notes tests were not edited.

Final gate log: `~/note-home-final.log`. Expected failing runs are retained in
`~/note-home-red.log`, `~/note-home-live-red.log`, `~/note-home-cursor-red.log`
and `~/note-home-index-red.log`. No build or test ran on the control host.
The final documentation commit changes comments and this report only.
Independent review is pending. Never merged.

## B1 stacked integration

The B1 port in #812 adds shared strict input/output metadata validation,
expected room versions, a conservative incoming-inventory revision check and
request fingerprints with immutable recorded-plan lookup. Unchanged participants
are read dependencies, so they receive no authored history update. The Tasks
adapter calls this coordinator; it never calls the offline writer or implements
placement/recovery/undo. `notes.undo` routes executor records to the existing
inverse boundary, which still refuses Home payloads. Namespace, rank, claim,
occurrence and source-authority adapters remain unavailable. The validation above
is the original #814 evidence; current B1 gates are recorded on #812. The
recorded-plan probe reads atomically replaced journal records without the
exclusive writer lock; recovery and admission still go through the Home writer.

Round 1 reader publication changes (the reader lease below is replaced by round 2):
- The journal receipt precedes the single accepted list/plugin/link commit.
  List counts, filters and rows use one read transaction. The startup list
  waits for Home recovery when executor records exist, including after an
  index rebuild; intermediate recovery chunks cannot escape through that path.
- Search uses a shared Home generation lease from listing through index
  publication. Direct provider reads use the same boundary. Backlink rows and
  snippets share a lease. Blocking file reads retain owned leases after caller
  cancellation. Pending or conflicted actions refuse file-derived batches;
  unrelated ordinary saves remain available.
- Search stages serialized documents and removals in the writer connection's
  private file-backed TEMP database, with bounded fetch batches. One final
  SQLite transaction publishes the entire refresh. Failure or cancellation
  discards staging; search queries pin one read transaction. Failed refreshes
  retry without advancing the caught-up cursor.
- Retained action participants form an adjacency graph, rebuilt from the WAL
  before Home opens. A journal refresh expands through overlapping actions,
  so even one received event and batch size one publish all related documents.
  Recovery repeats publication with the existing durable receipt.

Additional costs: G retained participant memberships, D bytes in a search
refresh. Group memory and rebuild work O(G); expansion O(G log G) with ordered
sets. Fetch memory stays bounded by the configured batch; private temporary
storage is O(D), with one final O(D) publication transaction. A full search
reconciliation holds its read lease until publication and can delay a shared
note action. Startup list reads for Homes with executor records can wait for
recovery. Search retains its existing asynchronous journal-following behavior,
with complete old or complete new refreshes visible. No performance budget is
claimed by these functional gates.

Regression coverage includes concurrent readers at 24 two-note action steps,
recorder refusal before its receipt, list reads after process crashes before
Home recovery, retained conflicts after restart, a pinned pre-action search
build, batch-size-one search refreshes, concurrent search result polling,
removals, provider failures and cancellation. Existing crash recovery and
ordinary-save coverage remains in place.


Round 2 accepted source generations:
- `meta.db` stores canonical note text, path, search fields and a content stamp
  in the same transaction as accepted note/plugin/link rows. A schema rebuild
  fills these derived rows from files and retained authored history. Search,
  peek and backlink snippets read these rows, including after a conflict.
- A snapshot clones an Arc generation token. Each query has its own short
  SQLite connection; no transaction, file read or reader-owned exclusion lock
  spans the snapshot lifetime. Admission never waits for a reader to drop.
- Publication retains at most four distinct pinned old generations plus the
  current generation. A lone pinned rebuild survives arbitrary writes. SQL
  keeps only source versions visible to retained tokens; intermediate versions
  are removed. A fifth old generation revokes the oldest token before SQL
  reclamation, returning retryable `NotSavedYet` with a stale-snapshot detail.
  Dropping the last Arc permits reclamation on the next write; startup removes
  obsolete versions because process-local pins cannot survive a crash.
- Source queries validate before and after reading. Search validates again
  before publishing its staged batch; a stale read discards staging and the
  journal follower retries. Backlink rows/snippets validate against the current
  generation so separate metadata queries cannot return a mixed response.
- File placement holds affected actor leases and lifecycle ownership, but no
  Home-wide metadata lock. Leased actors promptly refuse edits/Flush with a
  retryable error; websocket clients are asked to rejoin and resend. Ordinary
  room metadata waits for executor publication in 25 ms checks and refuses
  rather than waiting for the action. Other metadata contention retains its
  normal queue position. Shutdown attempts a checkpoint with zero busy timeout, without waiting
  for an outstanding reader statement.

Costs: N accepted source bytes, K changed notes, R <= 4 pinned old generations.
Snapshot creation is O(1) memory/work and performs no I/O. Derived source disk
is O((R + 1)N); unchanged rows are shared between generations. Normal version
updates/reclamation are O(K R log N). When the retained pin set changes,
reclamation may examine O(R N) old rows; reader operations never enter its
critical section. A full list is O(number of source versions) plus sorting;
source bodies are fetched per document, in bounded search batches. The old
executor image, participant and journal caps remain. A slow SQLite/disk write
can still cost I/O time; 25 ms bounds the metadata lock wait caused by executor
publication, not an operating-system I/O completion deadline.

Regression coverage: a retained reader that never cooperates with admission;
a stalled real Notes-to-search rebuild while an action and new readers finish;
successive writes retaining only the pinned source version; four-old-generation
cap and stale errors; reclamation after last-reader drop; accepted readers at
24 publication boundaries; process crashes with snapshots pinned; prompt
participant edit/Flush refusal followed by an idempotent retry. The ignored
cross-crate latency harness measures save and edit-plus-save p95 during repeated
16-note actions and search reconciliation on a 1,000-note fixture. Performance
evidence and exact gate results belong in PR #814.
ADR 0003 evidence migration: `docs/implementation/note-write-executor-home.md` Source commit: [7e2032a0b88abc1cd4c377c4feb88f9f9582a2e9](https://git.kayg.org/kayg/calternal/commit/7e2032a0b88abc1cd4c377c4feb88f9f9582a2e9); snapshot from dev `55f35fcd`. Full original source preserved below before repository removal/stripping. Design, code and test inputs are retained where applicable. ````text # Shared note-write executor, slice 0b Stack: #809 at f30b760a. Prerequisite audit: #812. Target: dev. No action is enabled. Never merge. Plan: 1. Add failing Home integration and publication replay tests. 2. Add room leases, durable Home payloads and recovery before Home startup. Use the existing jobs/lifecycle/room/meta_update lock order. Refuse unsupported lifecycle, metadata, rank, claim and occurrence participants. 3. Install authored CRDT history and bases after COMMIT, publish one journal batch with a permanent deduplication key, then commit one accepted metadata batch. Release room leases only after the complete generation is ready. A pending action keeps affected rooms unavailable. 4. Exercise crash/failure boundaries, ordinary-save concurrency, recovery and existing notes behavior. Run VM gates and record results. Scope: existing-note replacements with stable ids and exact room byte round trips. Create/delete/rename and domain-specific private stores remain refused. The shared strict metadata writer remains a separate prerequisite for Tasks B1. Costs: K participants, N retained image/state bytes, J retained WAL bytes. Room acquisition O(K log K), payload/lease matching O(K²), history/index installation O(K + N), existing conservative placement O(KN + J). Memory O(N + J). Startup additionally replays H retained history bytes for the distinct identities named by the WAL, O(H + J + N), in bounded identity chunks before exposing the Home. Existing 1,024 path / 64 MiB image / 1 GiB journal caps remain; private CRDT payloads must be bounded too. No Home walk during admission. | failure | effect | detection | recovery | test | | --- | --- | --- | --- | --- | | stale room/file/binding | refuse before PREPARE | exact bytes and stable id/path | new plan | concurrent save and stale input | | interrupted PREPARE | no file placement | checksummed WAL | drop staging or resume complete record | existing process crash tests | | interrupted placement | no accepted partial generation | WAL + placement evidence | recorded direction | every placement boundary | | COMMIT before history | committed, not published | Home payload in WAL | idempotent history install | restart after COMMIT | | history before index | affected rooms unavailable | transaction receipt | install complete index batch | injected history boundary | | journal before index | accepted rows remain before-state | outbox cursor | install whole accepted batch | injected publication boundary | | journal before cursor | no completion reply | permanent journal dedup key | retry without duplicate events | publication boundary | | caller cancellation | coordinator continues | Home-owned background task | finish or retain pending | dropped caller | | unsupported participant | no writes | admission | retain existing action path | refusal coverage | No power-loss, Apple or performance pass is claimed without those gates. BLAKE3 equality assumes distinct content does not share a 256-bit digest; integrity checks do not authenticate the WAL. Placement evidence is required separately. Implemented boundaries: - Home owns each active executor under its jobs and lifecycle locks. UUID-ordered actor leases drain ordinary saves, pin revisions and prevent timers, websocket edits, outside ingestion or Flush from interleaving. Peek reads accepted source versions without a working-file fallback. - PREPARE includes bounded, checksummed exact CRDT pre/post states and the authored update. File placement and COMMIT keep #809's evidence checks. One conditional history transaction installs every authored update/base and its permanent receipt after COMMIT. - One journal transaction publishes the batch and a permanent deduplication receipt, then one SQLite transaction installs all accepted note/plugin rows. Publication transactions use FULL sync; ordinary journal writes retain their prior sync policy. The WAL publication cursor follows both commits. Same-key retries repair live room release even if cursor installation already succeeded. - Recovery runs before scan, watcher, bulk replay and readers. Homes with retained executor records also wait for recovery on the startup list path. Published files are never replayed. Derived rows are rebuilt from current history for WAL participants, preserving later ordinary saves. Conflicted identities retain accepted history/index state and refuse source fallback; unrelated notes remain available. Delivery limits: no task/calendar action or enable switch; stable existing-note replacements only. Namespace creation/removal/rename, metadata validation, ranks, claims, occurrences and Home undo remain refused pending their adapters. Home WAL payload compaction is withheld, so retained records can cause bounded backpressure. No Apple, power-loss or performance pass is claimed. Uncooperative outside tools can observe partial raw filesystem placement, as in #809. Validation on dev-01 in `~/Developer/Worktrees/note-write-executor-home`, through `nix develop -c bash -c`, implementation at `a479a812`: ```sh cargo fmt --all -- --check ~/bin/slot cargo clippy --locked -j4 --workspace --all-targets -- -D warnings ~/bin/slot cargo test --locked -j4 -p calternal-notes -p calternal ~/bin/slot cargo test --locked -j4 -p calternal-store ``` All passed. Notes: 480 passed, 0 failed, 6 existing ignored; doc tests passed. All calternal unit/integration suites passed with their existing ignored tests. Store: 9 passed, 0 failed; doc tests passed. Focused executor run: 37 passed, including nine Home tests and 24 real process-kill Home phases with repeated recovery, live publication retry, cursor I/O errors, caller cancellation, concurrent ordinary save, unsupported admission and cache rebuild after a later ordinary save. Existing notes tests were not edited. Final gate log: `~/note-home-final.log`. Expected failing runs are retained in `~/note-home-red.log`, `~/note-home-live-red.log`, `~/note-home-cursor-red.log` and `~/note-home-index-red.log`. No build or test ran on the control host. The final documentation commit changes comments and this report only. Independent review is pending. Never merged. ## B1 stacked integration The B1 port in #812 adds shared strict input/output metadata validation, expected room versions, a conservative incoming-inventory revision check and request fingerprints with immutable recorded-plan lookup. Unchanged participants are read dependencies, so they receive no authored history update. The Tasks adapter calls this coordinator; it never calls the offline writer or implements placement/recovery/undo. `notes.undo` routes executor records to the existing inverse boundary, which still refuses Home payloads. Namespace, rank, claim, occurrence and source-authority adapters remain unavailable. The validation above is the original #814 evidence; current B1 gates are recorded on #812. The recorded-plan probe reads atomically replaced journal records without the exclusive writer lock; recovery and admission still go through the Home writer. Round 1 reader publication changes (the reader lease below is replaced by round 2): - The journal receipt precedes the single accepted list/plugin/link commit. List counts, filters and rows use one read transaction. The startup list waits for Home recovery when executor records exist, including after an index rebuild; intermediate recovery chunks cannot escape through that path. - Search uses a shared Home generation lease from listing through index publication. Direct provider reads use the same boundary. Backlink rows and snippets share a lease. Blocking file reads retain owned leases after caller cancellation. Pending or conflicted actions refuse file-derived batches; unrelated ordinary saves remain available. - Search stages serialized documents and removals in the writer connection's private file-backed TEMP database, with bounded fetch batches. One final SQLite transaction publishes the entire refresh. Failure or cancellation discards staging; search queries pin one read transaction. Failed refreshes retry without advancing the caught-up cursor. - Retained action participants form an adjacency graph, rebuilt from the WAL before Home opens. A journal refresh expands through overlapping actions, so even one received event and batch size one publish all related documents. Recovery repeats publication with the existing durable receipt. Additional costs: G retained participant memberships, D bytes in a search refresh. Group memory and rebuild work O(G); expansion O(G log G) with ordered sets. Fetch memory stays bounded by the configured batch; private temporary storage is O(D), with one final O(D) publication transaction. A full search reconciliation holds its read lease until publication and can delay a shared note action. Startup list reads for Homes with executor records can wait for recovery. Search retains its existing asynchronous journal-following behavior, with complete old or complete new refreshes visible. No performance budget is claimed by these functional gates. Regression coverage includes concurrent readers at 24 two-note action steps, recorder refusal before its receipt, list reads after process crashes before Home recovery, retained conflicts after restart, a pinned pre-action search build, batch-size-one search refreshes, concurrent search result polling, removals, provider failures and cancellation. Existing crash recovery and ordinary-save coverage remains in place. Round 2 accepted source generations: - `meta.db` stores canonical note text, path, search fields and a content stamp in the same transaction as accepted note/plugin/link rows. A schema rebuild fills these derived rows from files and retained authored history. Search, peek and backlink snippets read these rows, including after a conflict. - A snapshot clones an Arc generation token. Each query has its own short SQLite connection; no transaction, file read or reader-owned exclusion lock spans the snapshot lifetime. Admission never waits for a reader to drop. - Publication retains at most four distinct pinned old generations plus the current generation. A lone pinned rebuild survives arbitrary writes. SQL keeps only source versions visible to retained tokens; intermediate versions are removed. A fifth old generation revokes the oldest token before SQL reclamation, returning retryable `NotSavedYet` with a stale-snapshot detail. Dropping the last Arc permits reclamation on the next write; startup removes obsolete versions because process-local pins cannot survive a crash. - Source queries validate before and after reading. Search validates again before publishing its staged batch; a stale read discards staging and the journal follower retries. Backlink rows/snippets validate against the current generation so separate metadata queries cannot return a mixed response. - File placement holds affected actor leases and lifecycle ownership, but no Home-wide metadata lock. Leased actors promptly refuse edits/Flush with a retryable error; websocket clients are asked to rejoin and resend. Ordinary room metadata waits for executor publication in 25 ms checks and refuses rather than waiting for the action. Other metadata contention retains its normal queue position. Shutdown attempts a checkpoint with zero busy timeout, without waiting for an outstanding reader statement. Costs: N accepted source bytes, K changed notes, R <= 4 pinned old generations. Snapshot creation is O(1) memory/work and performs no I/O. Derived source disk is O((R + 1)N); unchanged rows are shared between generations. Normal version updates/reclamation are O(K R log N). When the retained pin set changes, reclamation may examine O(R N) old rows; reader operations never enter its critical section. A full list is O(number of source versions) plus sorting; source bodies are fetched per document, in bounded search batches. The old executor image, participant and journal caps remain. A slow SQLite/disk write can still cost I/O time; 25 ms bounds the metadata lock wait caused by executor publication, not an operating-system I/O completion deadline. Regression coverage: a retained reader that never cooperates with admission; a stalled real Notes-to-search rebuild while an action and new readers finish; successive writes retaining only the pinned source version; four-old-generation cap and stale errors; reclamation after last-reader drop; accepted readers at 24 publication boundaries; process crashes with snapshots pinned; prompt participant edit/Flush refusal followed by an idempotent retry. The ignored cross-crate latency harness measures save and edit-plus-save p95 during repeated 16-note actions and search reconciliation on a 1,000-note fixture. Performance evidence and exact gate results belong in PR #814. ````
kayg commented 2026-11-04 13:04:46 +00:00
Author
Owner
Copy link

Closing: head 2e08de74 is already in dev (reached via 7e2032a0); nothing left to merge. Branch kept.

Closing: head 2e08de74 is already in dev (reached via 7e2032a0); nothing left to merge. Branch kept.
kayg closed this pull request 2026-11-04 13:04:49 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!814
No description provided.