kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

WIP: perf: executor writes cost what they touch (5-20x faster; 40 ms gate not yet met) #898

Draft
kayg wants to merge 3 commits from perf/executor-hot-path into dev
pull from: perf/executor-hot-path
merge into: kayg:dev
Conversation 5 Commits 3 Files changed 16 +944 -180
kayg commented 2026-11-03 14:05:59 +00:00
Owner
Copy link

The exact-byte note-write executor (#809, #814) was far over the 40 ms write budget on perf-01 with the 10k fixture (#894 comment 76971). This PR makes the executor's CPU and I/O per write scale with the notes it touches, not with every write it has ever made. It also fixes the two spurious block_changed refusals.

Status: a 5–20× faster write path, but the 40 ms and move-many 200 ms gates are NOT met. What's left is about 9 durable flushes per write; getting under them needs a journal format change, proposed as the next PR in the results comment. Some other-gate medians read +6–9% under heavy VM contention and need a quiet rerun before merge.

  • Plan: first comment. Results and before/after: results comment.
  • Cause: every write read, parsed and re-validated every retained journal record (with Yjs replays) about 6 times, and ran the full recovery sweep. The journal grows by one record per write.
  • Changes, one commit each:
    1. a cached journal view: stat every record, parse only changed ones;
    2. skip the sweep when nothing is in flight;
    3. ExpectedOwners: a digest of the write's own owner rows instead of the global index epoch;
    4. file tokens instead of version: 0 for unopened notes;
    5. batched staging and placement syncs (no k² rereads);
    6. drop folder syncs that only repeat this process's own flushes.

Coordinates with #894: rebase guide in #894 comment 77087; a worked rebase is on scratch/executor-on-894.

🤖 Generated with Claude Code

Merge order: #905 (tolerant journal reader) must be in prod BEFORE this PR. Rolling back past this PR is only safe to a build that has #905.

The exact-byte note-write executor (#809, #814) was far over the 40 ms write budget on perf-01 with the 10k fixture (#894 comment 76971). This PR makes the executor's CPU and I/O per write scale with the notes it touches, not with every write it has ever made. It also fixes the two spurious `block_changed` refusals. **Status: a 5–20× faster write path, but the 40 ms and move-many 200 ms gates are NOT met.** What's left is about 9 durable flushes per write; getting under them needs a journal format change, proposed as the next PR in the results comment. Some other-gate medians read +6–9% under heavy VM contention and need a quiet rerun before merge. - Plan: first comment. Results and before/after: results comment. - Cause: every write read, parsed and re-validated every retained journal record (with Yjs replays) about 6 times, and ran the full recovery sweep. The journal grows by one record per write. - Changes, one commit each: 1. a cached journal view: stat every record, parse only changed ones; 2. skip the sweep when nothing is in flight; 3. `ExpectedOwners`: a digest of the write's own owner rows instead of the global index epoch; 4. file tokens instead of `version: 0` for unopened notes; 5. batched staging and placement syncs (no k² rereads); 6. drop folder syncs that only repeat this process's own flushes. Coordinates with #894: rebase guide in #894 comment 77087; a worked rebase is on `scratch/executor-on-894`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) **Merge order:** #905 (tolerant journal reader) must be in prod BEFORE this PR. Rolling back past this PR is only safe to a build that has #905.
Each write used to read, parse and re-check every retained journal record
(including forking and replaying their Yjs images) several times. The view
stats every record file, so any changed or damaged record still refuses the
write, but parses only new or changed files. The full recovery sweep runs
only when something is in flight or the last sweep left work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The B1 adapter guarded its owner query with the index connection's global
change count, so any index write anywhere (another note saved, a large
move-many settling) refused with 'changed note inventory'. Plans now carry
ExpectedOwners, a digest of the exact owner rows they were built from; the
Home re-reads those rows under the index write lock at admission. The old
ExpectedInventory variant stays readable for retained records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
notes: list rows can carry a file token for notes nobody opened
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
8a266bc2ed 
A note with no history had no version to list, so rows said 0 and every
first write to it was refused. Home::row_versions gives the newest change,
or a negative token from the file's index stamp. Home::version_matches
accepts a token only while the history is just the first seed and the file
on disk still has that stamp, so any change since listing still refuses.
set-status checks versions through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kayg commented 2026-11-03 14:06:39 +00:00
Author
Owner
Copy link

Plan: executor hot path

Where the time goes (from reading the code, spans to confirm)

Profiling binaries with timing spans (scratch/executor-spans = #894 head cac156dd plus CALTERNAL_EXEC_SPANS=1 spans, never merged) are queued on dev-01 behind train BM14c. The span table on perf-01 will follow in the next comment. What the code already shows, per Home write:

step cost now why
journal::load / records() / read_records O(R) full parses, ~7× per write every call reads every retained record file, re-checks checksum and digest, and runs Payload::validate (3 Yjs forks + an update apply per image). Home records are never compacted, so R grows by one per write (the bench: ~1000 records when set-status ran)
recover() on every write O(R·k) syscalls runs cleanup_staging for every retained record, plus a journal dir scan and fsync, even when nothing is in flight
journal::usage O(R·k) stats stats every evidence file of every record
finish_inner check() per note O(k²) small reads rereads all k participants before each placement (move-many 50 ⇒ ~10k reads)
fsyncs ~4 per note + 3 journal stage file + dir, placement dir, final file + dir, Prepared/Committed/published records

R = retained journal records, k = notes the write touches, N = notes in the Home. Nothing above is O(N); the dominant term is O(R) with a large constant (Yjs work), which is why set-status after a few hundred writes takes 1.7 s.

Big-O per write

  • now: O(R · (record bytes + Yjs validate)) × ~7, plus O(R·k) syscalls, plus O(k²) reads
  • target: O(k) reads/fsyncs + O(k) Yjs work for the touched images + one stat per record file (≈1 µs each; the one O(R) term left, and it keeps the "damaged record refuses every write" guarantee)

The fix

  1. Journal view (write_executor/index.rs): a per-journal cache of small summaries (key, state, author, undo_of, published, image ids, evidence names), never images. Each write lists the journal and stats every record file. A file is read again only if its inode/size/mtime/ctime changed, or if it was written within the last 2 s (a same-tick edit could keep the stamp equal). Even then it is fully parsed and checked only if its BLAKE3 differs; the check is a pure function of the bytes. Disk use = Σ record files + other journal files + evidence (re-counted while a record is in flight or recent). The hot path reads full records by key only: replay, commit, publish, undo.
  2. Recovery sweep only when needed: recover() runs when any record is non-terminal, a Home record is unpublished or needs release, or the previous write failed (executor_swept is cleared on any error). Startup always sweeps.
  3. Scoped inventory guard: a new Requirement::ExpectedOwners { digest } holds a digest of the exact task_lines and links owner rows the plan was built from. Home re-reads those rows with an adapter probe under meta_update, on the index connection, at the same point the old epoch was checked. The old ExpectedInventory { epoch } stays readable for retained records.
  4. Real row versions: Home::row_versions(ids) gives each note's newest change. A note nobody has opened gets a negative file token from its index stamp instead. Home::version_matches accepts a token only if the history is just the first seed and the file on disk still has that stamp. A token never equals a real version (those are ≥ 0).
  5. If the spans show it after 1–4: batch the per-note fsyncs by directory, and cheapen the O(k²) recheck. The final complete-byte checks before COMMIT stay as they are.

Failure table

failure effect detection recovery test
a record file is damaged or rewritten in place the write must refuse stamp differs, so the file is re-read, its hash differs and the full parse fails; not cached, so every write keeps refusing operator repair; startup sweep journal_corruption_gates_every_write_and_recovery (unchanged)
same-tick in-place edit keeps the stamp equal stale summary files under 2 s old are always re-read and hashed as above new: rewrite a fresh record same size, same mtime
silent media corruption of an old, untouched record not seen by an unrelated write (before: seen) that record's next full read (replay, undo, commit, recovery) checks its checksum; startup sweep as before covered by startup recovery tests
outside writer grows an old record's evidence file budget undercounts until restart re-counted at startup and while in flight 1 GB budget has headroom (reservation = 2×record + 64 MB) — (documented trade-off)
write errors after PREPARE staging may remain executor_swept cleared next write runs the full sweep crash tests (tests/crash.rs, Tasks B1 crash child)
process crash at any phase as today startup recover_note_writes sweeps fully unchanged all NOTE_EXECUTOR_STOP / NOTE_HOME_STOP crash tests
a task's own owners change between plan and admission stale plan digest mismatch under meta_update block_changed, client re-plans new owner_digest_ignores_unrelated_index_writes_and_refuses_new_owners
an unrelated note changes the index used to refuse spuriously — write goes through same test (legacy epoch still refuses)
plan with ExpectedOwners but no probe cannot check refuses "owner probe missing" — no-probe variant
token sent, file changed outside stale view stamp now ≠ token block_changed new file_tokens_write_and_go_stale_on_any_change
token sent, note changed in the app stale view history no longer seed-only block_changed same test
token sent, note only opened same note seed-only + same stamp write proceeds same test
new binary journals ExpectedOwners, then an older binary is deployed back the older binary fails to read that record and the whole Home fails to open (Home records are never compacted, so draining is impossible) journal read error at Home open #905 (fix/journal-tolerant-reader) ships in a train BEFORE this PR; roll back past this PR only to a build that contains it (also in docs/deploy.md on #905); otherwise roll forward #905 write_executor::tests::tolerant: a record written by this PR's head is read, recovered and compacted

Gates (perf-01 under perf-lock, 10k tasks-bench fixture, tasks-write-bench --n 200)

  • p95 ≤ 40 ms: tasks.set-status, tasks.move (both kinds), tasks.undo, tasks.delete; move-many 50 ≤ 200 ms
  • set-status: dev (41f2efa3) vs this PR. move/undo/delete/move-many: #894 (cac156dd) vs #894 + this PR
  • no regression: storage-bench appends, open.note, list, search, cold index (tasks-bench), each before/after on the same VM
  • refusal counts in the bench: 0 version refetches, 0 inventory refusals
  • tests: cargo test -p calternal-notes write_executor, -p calternal-tasks, -p calternal-engine, the crash children, clippy -D warnings

Update 11-03 (review 77166 fixes): P2-1 above; P3-1 creations synced before the first removal; P3-2 sweep flag fail-safe; P3-3 journal cache dropped on a failed folder sync; P3-4/P3-5 rows added to docs/implementation/note-write-executor.md.

## Plan: executor hot path ### Where the time goes (from reading the code, spans to confirm) Profiling binaries with timing spans (`scratch/executor-spans` = #894 head cac156dd plus `CALTERNAL_EXEC_SPANS=1` spans, never merged) are queued on dev-01 behind train BM14c. The span table on perf-01 will follow in the next comment. What the code already shows, per Home write: | step | cost now | why | |---|---|---| | `journal::load` / `records()` / `read_records` | **O(R) full parses, ~7× per write** | every call reads every retained record file, re-checks checksum and digest, and runs `Payload::validate` (3 Yjs forks + an update apply per image). Home records are never compacted, so R grows by one per write (the bench: ~1000 records when set-status ran) | | `recover()` on every write | O(R·k) syscalls | runs `cleanup_staging` for every retained record, plus a journal dir scan and fsync, even when nothing is in flight | | `journal::usage` | O(R·k) stats | stats every evidence file of every record | | `finish_inner` `check()` per note | O(k²) small reads | rereads all k participants before each placement (move-many 50 ⇒ ~10k reads) | | fsyncs | ~4 per note + 3 journal | stage file + dir, placement dir, final file + dir, Prepared/Committed/published records | R = retained journal records, k = notes the write touches, N = notes in the Home. Nothing above is O(N); the dominant term is O(R) with a large constant (Yjs work), which is why set-status after a few hundred writes takes 1.7 s. ### Big-O per write - now: O(R · (record bytes + Yjs validate)) × ~7, plus O(R·k) syscalls, plus O(k²) reads - target: O(k) reads/fsyncs + O(k) Yjs work for the touched images + **one `stat` per record file** (≈1 µs each; the one O(R) term left, and it keeps the "damaged record refuses every write" guarantee) ### The fix 1. **Journal view** (`write_executor/index.rs`): a per-journal cache of small summaries (key, state, author, undo_of, published, image ids, evidence names), never images. Each write lists the journal and stats every record file. A file is read again only if its inode/size/mtime/ctime changed, or if it was written within the last 2 s (a same-tick edit could keep the stamp equal). Even then it is fully parsed and checked only if its BLAKE3 differs; the check is a pure function of the bytes. Disk use = Σ record files + other journal files + evidence (re-counted while a record is in flight or recent). The hot path reads full records by key only: replay, commit, publish, undo. 2. **Recovery sweep only when needed**: `recover()` runs when any record is non-terminal, a Home record is unpublished or needs release, or the previous write failed (`executor_swept` is cleared on any error). Startup always sweeps. 3. **Scoped inventory guard**: a new `Requirement::ExpectedOwners { digest }` holds a digest of the exact `task_lines` and `links` owner rows the plan was built from. Home re-reads those rows with an adapter probe under `meta_update`, on the index connection, at the same point the old epoch was checked. The old `ExpectedInventory { epoch }` stays readable for retained records. 4. **Real row versions**: `Home::row_versions(ids)` gives each note's newest change. A note nobody has opened gets a negative **file token** from its index stamp instead. `Home::version_matches` accepts a token only if the history is just the first `seed` and the file on disk still has that stamp. A token never equals a real version (those are ≥ 0). 5. If the spans show it after 1–4: batch the per-note fsyncs by directory, and cheapen the O(k²) recheck. The final complete-byte checks before COMMIT stay as they are. ### Failure table | failure | effect | detection | recovery | test | |---|---|---|---|---| | a record file is damaged or rewritten in place | the write must refuse | stamp differs, so the file is re-read, its hash differs and the full parse fails; not cached, so every write keeps refusing | operator repair; startup sweep | `journal_corruption_gates_every_write_and_recovery` (unchanged) | | same-tick in-place edit keeps the stamp equal | stale summary | files under 2 s old are always re-read and hashed | as above | new: rewrite a fresh record same size, same mtime | | silent media corruption of an old, untouched record | not seen by an unrelated write (before: seen) | that record's next full read (replay, undo, commit, recovery) checks its checksum; startup sweep | as before | covered by startup recovery tests | | outside writer grows an old record's evidence file | budget undercounts until restart | re-counted at startup and while in flight | 1 GB budget has headroom (reservation = 2×record + 64 MB) | — (documented trade-off) | | write errors after PREPARE | staging may remain | `executor_swept` cleared | next write runs the full sweep | crash tests (`tests/crash.rs`, Tasks B1 crash child) | | process crash at any phase | as today | startup `recover_note_writes` sweeps fully | unchanged | all `NOTE_EXECUTOR_STOP` / `NOTE_HOME_STOP` crash tests | | a task's own owners change between plan and admission | stale plan | digest mismatch under `meta_update` | `block_changed`, client re-plans | new `owner_digest_ignores_unrelated_index_writes_and_refuses_new_owners` | | an unrelated note changes the index | used to refuse spuriously | — | write goes through | same test (legacy epoch still refuses) | | plan with `ExpectedOwners` but no probe | cannot check | refuses "owner probe missing" | — | `no-probe` variant | | token sent, file changed outside | stale view | stamp now ≠ token | `block_changed` | new `file_tokens_write_and_go_stale_on_any_change` | | token sent, note changed in the app | stale view | history no longer seed-only | `block_changed` | same test | | token sent, note only opened | same note | seed-only + same stamp | write proceeds | same test | | new binary journals `ExpectedOwners`, then an older binary is deployed back | the older binary fails to read that record and the whole Home fails to open (Home records are never compacted, so draining is impossible) | journal read error at Home open | **#905 (`fix/journal-tolerant-reader`) ships in a train BEFORE this PR**; roll back past this PR only to a build that contains it (also in `docs/deploy.md` on #905); otherwise roll forward | #905 `write_executor::tests::tolerant`: a record written by this PR's head is read, recovered and compacted | ### Gates (perf-01 under `perf-lock`, 10k `tasks-bench` fixture, `tasks-write-bench --n 200`) - p95 ≤ 40 ms: `tasks.set-status`, `tasks.move` (both kinds), `tasks.undo`, `tasks.delete`; `move-many 50` ≤ 200 ms - set-status: dev (41f2efa3) vs this PR. move/undo/delete/move-many: #894 (cac156dd) vs #894 + this PR - no regression: `storage-bench` appends, `open.note`, list, search, cold index (`tasks-bench`), each before/after on the same VM - refusal counts in the bench: 0 version refetches, 0 inventory refusals - tests: `cargo test -p calternal-notes write_executor`, `-p calternal-tasks`, `-p calternal-engine`, the crash children, clippy `-D warnings` **Update 11-03 (review 77166 fixes):** P2-1 above; P3-1 creations synced before the first removal; P3-2 sweep flag fail-safe; P3-3 journal cache dropped on a failed folder sync; P3-4/P3-5 rows added to `docs/implementation/note-write-executor.md`.
kayg commented 2026-11-03 18:03:47 +00:00
Author
Owner
Copy link

Results: perf-01 under perf-lock, 10k tasks-bench fixture

Builds (release, built on dev-01, relayed through the control host, sha256 checked on both ends):

build what
before scratch/executor-spans = #894 head cac156dd plus no-op spans (env unset) · tasks-write-bench 11928a10…
after scratch/executor-on-894 = cac156dd + this PR (head 1b81f852), rebased the way the #894 comment 77087 describes · tasks-write-bench bc6df09a…
gates calternal-perf at dev 41f2efa3 (5185e941…, server 6bace468…) vs this PR 1b81f852 (57fb1d2f…, server d0591511…)

Command: tasks-write-bench --base data/base-10000 --state <tmp> --n 200. Before and after ran interleaved (B A B A B A). The VM was heavily loaded the whole evening: other agents' browser and perf runs on perf-01 and perf-02, which share the backup01 disk. An fsync probe (file + folder) measured p50 63 ms, p95 338 ms at 18:28; the earlier quiet window measured ~6.5 ms. Absolute numbers are noisy; the before/after ratio is consistent.

Write actions (ms; three runs each, p50 / p95)

measure before p50 before p95 after p50 after p95 budget
tasks.set-status 1718 / 1175 / 1101 2052 / 1373 / 1380 101 / 103 / 126 131 / 137 / 165 p95 ≤ 40 ✗
tasks.move When 308 / 266 / 259 508 / 362 / 351 88 / 93 / 109 125 / 121 / 171 p95 ≤ 40 ✗
tasks.move Someday → day 816 / 524 / 531 1072 / 683 / 669 98 / 104 / 135 133 / 149 / 215 p95 ≤ 40 ✗
tasks.undo 587 / 412 / 388 1159 / 718 / 728 85 / 90 / 113 114 / 133 / 175 p95 ≤ 40 ✗
tasks.delete 2100 / – / – 2187 / – / – 140 / 183 / 171 201 / 234 / 258 p95 ≤ 40 ✗
move-many 50 2226 / 2829 / 2787 – 383 / 384 / 453 – ≤ 200 ✗
move-many 50 undo 2338 / 2997 / 2353 – 264 / 271 / 317 – —
refetched versions 400 / 200 / 200 0 / 0 / 0 0 ✓
inventory refusals retried 100 / 100 / 100 0 / 0 / 0 0 ✓

The before runs 2–3 used --n 100 (shorter journal, so faster than run 1: the O(R) cost); delete has no rows at n=100. In a quiet window (after1, before the batching commit) the after numbers were: set-status p95 130, move p95 93, undo p95 87.

Verdict: 5–20× faster, and the cost no longer grows with the journal. Both spurious refusals are gone. The 40 ms and 200 ms gates are not met.

Where the remaining time goes (spans, scratch/executor-spans2, mean ms per write)

step ms what
3 record persists (Prepared, Commit, published) 13.5 each sync_all of the record file + fsync of the journal folder
stage files 14 sync the new images, link, sync the folder
placement folder sync 6.4 —
final barrier + checks 6.5 removed for notes this call placed (commit 1b81f852)
record_once + history install 13 the change journal and history SQLite commits
open / view / recover folder syncs ~20 cut to once per process / only when needed (1b81f852)
journal view (300 stats, recent records hashed) <1 was 38 ms per load × 6

So a one-note write is now about 9 durable flushes. At ~6.5 ms per flush on a quiet perf-01 that is ~60 ms, above budget before any other work. Under tonight's contention it's worse. Getting to 40 ms needs fewer flushes, not less CPU. That means changing the journal's on-disk format. Per the foundational-work rule I have not done it here; I propose it as the next PR, plan first:

  • one preallocated, append-only journal segment per Home: each state change is one framed append + fdatasync, with no per-record file or folder sync (−4 flushes);
  • the published mark rides on the next frame or a lazy marker. Recovery already republishes idempotently from receipts, so losing it costs one republish (−1 flush);
  • stage image and Prepared frame flushed together (−1).

That brings it to ~4 flushes per write, or ~26 ms at 6.5 ms per flush. move-many scales the same way.

Other gates: dev vs this PR, interleaved 3 × 3 (calternal-perf --sizes 10k --runs 3 --cold-runs 2 --idle-secs 0)

metric dev p50 (3 runs) this PR p50 (3 runs) dev p95 med PR p95 med Δ p50 median
open.note (ms) 0.687 / 0.993 / 0.908 0.912 / 1.01 / 0.912 1.64 1.78 +0.4%
open.api_get (ms) 4.11 / 4.13 / 4.47 4.41 / 4.6 / 4.28 13 8.13 +6.6%
open.long_note (ms) 16.2 / 19.6 / 57.5 18.3 / 18.5 / 18.1 65.7 65.4 −6.9%
home.first_page (ms) 2.68 / 3.22 / 3.15 2.35 / 2.94 / 2.99 4.66 3.12 −6.5%
home.page (ms) 1.04 / 0.973 / 0.99 1.08 / 1.15 / 0.949 2.48 2.78 +9.2%
home.page_tag (ms) 5.75 / 6.24 / 6.36 6.79 / 6.71 / 6.29 8.3 9.55 +7.5%
home.page_big (ms) 18.2 / 19.7 / 19.9 19.5 / 19.6 / 20.8 22.6 22.7 −0.8%
search.query (ms) 1.78 / 2.11 / 1.86 1.9 / 2 / 1.98 4.49 4.78 +6.6%
search.first_query (ms) 5.27 / 5.93 / 10.4 6.32 / 5.53 / 8.47 32.8 31.6 +6.6%
cold.index (s) 1.63 / 1.78 / 1.77 1.95 / 1.79 / 1.77 1.79 1.82 +0.9%
cold.first_page (ms) 305 / 343 / 324 411 / 352 / 324 349 352 +8.6%
cold.mem_peak (MB) 143 / 142 / 143 143 / 143 / 142 143 143 −0.1%
sync.save_ack (ms) 3.8 / 4.16 / 4.13 4.01 / 4.08 / 3.98 7.01 8.98 −2.9%
sync.durable_write (ms) 26.3 / 28 / 26.6 25.9 / 26.7 / 25.6 45.3 31.4 −3.0%
storm.save_ack (ms) 7.5 / 10.3 / 7.8 7.72 / 7.71 / 7.87 18.6 16.8 −1.1%
uploads.save_ack (ms) 5.08 / 5.65 / 5.61 5.38 / 5.46 / 5.84 12.4 14 −2.7%
journal.page (ms) 2.78 / 3.32 / 3.26 3.09 / 3.38 / 3.46 3.84 4.14 +3.6%
trash.delete (ms) 79.9 / 76.8 / 85 89.2 / 86.4 / 82.6 120 136 +8.2%
trash.restore (ms) 76.4 / 75.5 / 73.5 77.5 / 78.9 / 72.8 129 135 +2.7%
mem.idle (MB) 18.8 / 19.3 / 19 19.3 / 18.4 / 18.6 19.5 19.5 −2.1%
mem.peak (MB) 48.9 / 47.6 / 46.9 47.8 / 47.9 / 48.5 74.7 74.5 +0.7%

Every PR range overlaps dev's own run-to-run range. None of these paths calls the changed code: the executor runs only when write-executor/ exists, which it doesn't in this suite, and row_versions/stamps_of have no caller on dev. But under this contention I cannot show them flat. home.page, home.page_tag, search.query and trash.delete read +6–9% at the median. They need a rerun in a quiet window before this merges.

Tests (dev-01, Nix shell, ~/bin/slot, at 1b81f852)

cargo clippy --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-engine --all-targets -- -D warnings ✓ · cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-engine ✓: notes 549 (includes write_executor/tests/crash.rs and the io_failure_at_every_durability_boundary sweep), tasks 69 (includes the B1 Home crash children), engine 20. New tests: same_length_record_edit_with_restored_mtime_still_gates_writes, owner_digest_ignores_unrelated_index_writes_and_refuses_new_owners, the owners / no-probe refusal variants, file_tokens_write_and_go_stale_on_any_change, tokens_are_negative_stable_and_json_exact.

Not run: the full workspace and the Playwright suite (that's for trains).

## Results: perf-01 under `perf-lock`, 10k `tasks-bench` fixture **Builds** (release, built on dev-01, relayed through the control host, sha256 checked on both ends): | build | what | |---|---| | before | `scratch/executor-spans` = #894 head cac156dd plus no-op spans (env unset) · `tasks-write-bench` `11928a10…` | | after | `scratch/executor-on-894` = cac156dd + this PR (head 1b81f852), rebased the way the #894 comment 77087 describes · `tasks-write-bench` `bc6df09a…` | | gates | `calternal-perf` at dev 41f2efa3 (`5185e941…`, server `6bace468…`) vs this PR 1b81f852 (`57fb1d2f…`, server `d0591511…`) | Command: `tasks-write-bench --base data/base-10000 --state <tmp> --n 200`. Before and after ran interleaved (B A B A B A). **The VM was heavily loaded the whole evening**: other agents' browser and perf runs on perf-01 and perf-02, which share the backup01 disk. An fsync probe (file + folder) measured **p50 63 ms, p95 338 ms** at 18:28; the earlier quiet window measured ~6.5 ms. Absolute numbers are noisy; the before/after ratio is consistent. ### Write actions (ms; three runs each, p50 / p95) | measure | before p50 | before p95 | after p50 | after p95 | budget | |---|---|---|---|---|---| | `tasks.set-status` | 1718 / 1175 / 1101 | 2052 / 1373 / 1380 | 101 / 103 / 126 | 131 / 137 / 165 | p95 ≤ 40 ✗ | | `tasks.move` When | 308 / 266 / 259 | 508 / 362 / 351 | 88 / 93 / 109 | 125 / 121 / 171 | p95 ≤ 40 ✗ | | `tasks.move` Someday → day | 816 / 524 / 531 | 1072 / 683 / 669 | 98 / 104 / 135 | 133 / 149 / 215 | p95 ≤ 40 ✗ | | `tasks.undo` | 587 / 412 / 388 | 1159 / 718 / 728 | 85 / 90 / 113 | 114 / 133 / 175 | p95 ≤ 40 ✗ | | `tasks.delete` | 2100 / – / – | 2187 / – / – | 140 / 183 / 171 | 201 / 234 / 258 | p95 ≤ 40 ✗ | | `move-many` 50 | 2226 / 2829 / 2787 | – | 383 / 384 / 453 | – | ≤ 200 ✗ | | `move-many` 50 undo | 2338 / 2997 / 2353 | – | 264 / 271 / 317 | – | — | | refetched versions | 400 / 200 / 200 | | **0 / 0 / 0** | | 0 ✓ | | inventory refusals retried | 100 / 100 / 100 | | **0 / 0 / 0** | | 0 ✓ | The before runs 2–3 used `--n 100` (shorter journal, so faster than run 1: the O(R) cost); `delete` has no rows at n=100. In a quiet window (after1, before the batching commit) the after numbers were: set-status p95 130, move p95 93, undo p95 87. **Verdict:** 5–20× faster, and the cost no longer grows with the journal. Both spurious refusals are gone. **The 40 ms and 200 ms gates are not met.** ### Where the remaining time goes (spans, `scratch/executor-spans2`, mean ms per write) | step | ms | what | |---|---|---| | 3 record persists (Prepared, Commit, published) | 13.5 each | `sync_all` of the record file + `fsync` of the journal folder | | stage files | 14 | sync the new images, link, sync the folder | | placement folder sync | 6.4 | — | | final barrier + checks | 6.5 | removed for notes this call placed (commit 1b81f852) | | `record_once` + history install | 13 | the change journal and history SQLite commits | | open / view / recover folder syncs | ~20 | cut to once per process / only when needed (1b81f852) | | journal view (300 stats, recent records hashed) | <1 | was 38 ms per load × 6 | So a one-note write is now about 9 durable flushes. At ~6.5 ms per flush on a quiet perf-01 that is ~60 ms, above budget before any other work. Under tonight's contention it's worse. **Getting to 40 ms needs fewer flushes, not less CPU.** That means changing the journal's on-disk format. Per the foundational-work rule I have not done it here; I propose it as the next PR, plan first: - one preallocated, append-only journal segment per Home: each state change is one framed append + `fdatasync`, with no per-record file or folder sync (−4 flushes); - the `published` mark rides on the next frame or a lazy marker. Recovery already republishes idempotently from receipts, so losing it costs one republish (−1 flush); - stage image and Prepared frame flushed together (−1). That brings it to ~4 flushes per write, or ~26 ms at 6.5 ms per flush. move-many scales the same way. ### Other gates: dev vs this PR, interleaved 3 × 3 (`calternal-perf --sizes 10k --runs 3 --cold-runs 2 --idle-secs 0`) | metric | dev p50 (3 runs) | this PR p50 (3 runs) | dev p95 med | PR p95 med | Δ p50 median | |---|---|---|---|---|---| | open.note (ms) | 0.687 / 0.993 / 0.908 | 0.912 / 1.01 / 0.912 | 1.64 | 1.78 | +0.4% | | open.api_get (ms) | 4.11 / 4.13 / 4.47 | 4.41 / 4.6 / 4.28 | 13 | 8.13 | +6.6% | | open.long_note (ms) | 16.2 / 19.6 / 57.5 | 18.3 / 18.5 / 18.1 | 65.7 | 65.4 | −6.9% | | home.first_page (ms) | 2.68 / 3.22 / 3.15 | 2.35 / 2.94 / 2.99 | 4.66 | 3.12 | −6.5% | | home.page (ms) | 1.04 / 0.973 / 0.99 | 1.08 / 1.15 / 0.949 | 2.48 | 2.78 | +9.2% | | home.page_tag (ms) | 5.75 / 6.24 / 6.36 | 6.79 / 6.71 / 6.29 | 8.3 | 9.55 | +7.5% | | home.page_big (ms) | 18.2 / 19.7 / 19.9 | 19.5 / 19.6 / 20.8 | 22.6 | 22.7 | −0.8% | | search.query (ms) | 1.78 / 2.11 / 1.86 | 1.9 / 2 / 1.98 | 4.49 | 4.78 | +6.6% | | search.first_query (ms) | 5.27 / 5.93 / 10.4 | 6.32 / 5.53 / 8.47 | 32.8 | 31.6 | +6.6% | | cold.index (s) | 1.63 / 1.78 / 1.77 | 1.95 / 1.79 / 1.77 | 1.79 | 1.82 | +0.9% | | cold.first_page (ms) | 305 / 343 / 324 | 411 / 352 / 324 | 349 | 352 | +8.6% | | cold.mem_peak (MB) | 143 / 142 / 143 | 143 / 143 / 142 | 143 | 143 | −0.1% | | sync.save_ack (ms) | 3.8 / 4.16 / 4.13 | 4.01 / 4.08 / 3.98 | 7.01 | 8.98 | −2.9% | | sync.durable_write (ms) | 26.3 / 28 / 26.6 | 25.9 / 26.7 / 25.6 | 45.3 | 31.4 | −3.0% | | storm.save_ack (ms) | 7.5 / 10.3 / 7.8 | 7.72 / 7.71 / 7.87 | 18.6 | 16.8 | −1.1% | | uploads.save_ack (ms) | 5.08 / 5.65 / 5.61 | 5.38 / 5.46 / 5.84 | 12.4 | 14 | −2.7% | | journal.page (ms) | 2.78 / 3.32 / 3.26 | 3.09 / 3.38 / 3.46 | 3.84 | 4.14 | +3.6% | | trash.delete (ms) | 79.9 / 76.8 / 85 | 89.2 / 86.4 / 82.6 | 120 | 136 | +8.2% | | trash.restore (ms) | 76.4 / 75.5 / 73.5 | 77.5 / 78.9 / 72.8 | 129 | 135 | +2.7% | | mem.idle (MB) | 18.8 / 19.3 / 19 | 19.3 / 18.4 / 18.6 | 19.5 | 19.5 | −2.1% | | mem.peak (MB) | 48.9 / 47.6 / 46.9 | 47.8 / 47.9 / 48.5 | 74.7 | 74.5 | +0.7% | Every PR range overlaps dev's own run-to-run range. None of these paths calls the changed code: the executor runs only when `write-executor/` exists, which it doesn't in this suite, and `row_versions`/`stamps_of` have no caller on dev. **But under this contention I cannot show them flat.** `home.page`, `home.page_tag`, `search.query` and `trash.delete` read +6–9% at the median. They need a rerun in a quiet window before this merges. ### Tests (dev-01, Nix shell, `~/bin/slot`, at 1b81f852) `cargo clippy --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-engine --all-targets -- -D warnings` ✓ · `cargo test --locked -j4 -p calternal-notes -p calternal-tasks -p calternal-engine` ✓: notes 549 (includes `write_executor/tests/crash.rs` and the `io_failure_at_every_durability_boundary` sweep), tasks 69 (includes the B1 Home crash children), engine 20. New tests: `same_length_record_edit_with_restored_mtime_still_gates_writes`, `owner_digest_ignores_unrelated_index_writes_and_refuses_new_owners`, the `owners` / `no-probe` refusal variants, `file_tokens_write_and_go_stale_on_any_change`, `tokens_are_negative_stable_and_json_exact`. Not run: the full workspace and the Playwright suite (that's for trains).
kayg changed title from WIP: perf: executor write path costs O(touched notes), not O(journal) to WIP: perf: executor writes cost what they touch (5-20x faster; 40 ms gate not yet met) 2026-11-03 18:03:48 +00:00
kayg commented 2026-11-03 20:06:34 +00:00
Author
Owner
Copy link

Independent review: data safety + quiet perf rerun (head 1b81f852)

I read the plan (77003), the results (77110), docs/implementation/note-write-executor*.md and the full diff against dev 41f2efa3. I changed no code.

Counts: P1 0 · P2 1 · P3 6. Verdict at the bottom.

P2

P2-1. Rolling back past this PR takes whole Homes offline (no data loss). An older binary can't deserialize Requirement::ExpectedOwners (an externally tagged enum, so the variant is unknown). Home records are never compacted, so the plan's suggested mitigation, "drain the journal first", isn't possible: one committed set-status record is enough to block the downgrade forever.
Scenario: deploy #898 → a user ticks one task → prod is rolled back to the previous build → journal::read fails in recover_note_writes → Notes::home shuts the Home down and returns the error → that user's whole Home (all notes, not just Tasks) fails to open until you roll forward again. Every user who wrote through the executor after the deploy is affected.
Fix (process, not code): in the deploy issue (#684) and in the plan's failure-table row, write "roll forward only past #898". Alternatively, ship a tolerant reader in a separate earlier deploy first. Either way, it must be in place before prod.

P3

P3-1. Batched placement drops "creations durable before removals" across folders on power loss. Latent: no current plan has both. The old loop synced each target's folder before placing the next note, and the sort put creations and replacements before removals. Now every placement happens first, then one sync_parents.
Scenario (future plan that moves a note from folder X to Y): power cut after both placements and before sync_parents → ext4 keeps X's rename (target → evidence) but not Y's link → after reboot the note is at neither public path. Startup recovery fixes it before the Home is served: the Prepared record holds the post bytes, and check classifies each note on its own, so any subset of placements is handled. But an outside reader of Notes/ that starts before calternal (a sync tool or backup) sees a deletion. The crash suite uses process exits, where the page cache survives, so no test covers this ordering.
Today's plans (set-status/move = replacements only, delete = one removal, duplicate = one creation) never mix the two. Fix when a mixed plan appears: sync the creation folders before the first removal. That's one extra folder sync, only for such plans.

P3-2. executor_swept isn't cleared on every failure path. It's cleared in run_note_write's error arms. It isn't cleared in executor_undo errors, or when spawn_blocking returns a JoinError.
Scenario: an executor_undo fails after finish persisted a terminal state, and cleanup_staging errors → .calternal-tmp-executor-*.staging stays in the user's note folder and counts against the 1 GB budget until restart.
Impact is limited: any non-terminal record sets open and forces the sweep anyway, and every restart sweeps. Fix: a fail-safe flag, cleared at the start of a write and set only after success.

P3-3. index::view caches before the caller's folder barrier succeeds. view() inserts the refreshed cache. Then NoteWriteExecutor::view runs sync_dir when unconfirmed is set. If that sync_dir fails once (EIO), the next view sees equal stamps and hashes, so unconfirmed = false and the barrier is skipped for that entry.
Mitigated: a non-terminal record forces recover(), whose full load() always syncs. Only terminal records are affected: an idempotent retry can get a Committed receipt before the entry is confirmed durable, and after power loss that record is Prepared again, so recovery rolls the same plan forward. Fix: keep unconfirmed in the cache until a sync succeeds, or sync inside index::view before all.insert.

P3-4. Damaged old records are now found at startup or on the next full read, not on every write. The plan's table accepts this; I agree it's acceptable. Two follow-ups:

  • docs/implementation/note-write-executor.md, row "corrupt/unknown journal → recovery withheld", should say "at startup and on any full read of that record".
  • The non-Home execute() idempotent path now answers from the cached summary (policy::receipt). A retried key whose record later rots on disk returns Committed instead of refusing. That's acceptable, because the operation was committed when it was cached.

P3-5. Stat trust assumes a local POSIX filesystem. On ext4, the stamp's ctime (ns), the 2 s window and rename-only replacement (always a new inode) make a stale cache entry practically impossible:

  • the clock stepping back gives a future ctime, which is always re-read;
  • the clock stepping forward is harmless, because the executor never edits a record in place;
  • restart starts cold;
  • another process's write changes the stamp and forces the folder sync.

On network or FUSE filesystems with attribute caching (NFS actimeo) this doesn't hold. Add one line to the module doc: Homes on local filesystems only.

P3-6. Fewer checks between placements, and a narrower owner guard.

  • Between placements: read dependencies and other notes' evidence are now checked only before COMMIT, not before each placement. An outside edit to a dependency during placement is found after every note is placed. The action becomes Conflicted with all notes placed instead of some; both need manual resolution.
  • Owner guard: ExpectedOwners covers each target's task_lines and links rows, but not the task's own tasks row (path, broken) that update::target reads. A task whose properties broke between plan and admission is still stopped by the exact-byte pre check and MetadataValidation. Only the error message changes; no wrong write is possible.

Answers to the specific questions

  • Owners changed → write succeeds? No. The probe re-reads the same rows under meta_update on the Home's own meta connection, which is the one the task index writes through. Every owner note is also a participant with exact pre bytes. Owner edits that haven't been indexed yet are invisible to both the old epoch and the new digest, the same as before.
  • Stale token accepted after an outside edit? No. The token is a 52-bit hash of the v2 stamp (size, mtime, ctime, dev, inode, path), and any outside write changes ctime. It also requires history = exactly one seed row equal to the room's version. A collision has a chance of about 2⁻⁵². Remaining race (pre-existing, also true for real versions): row_versions reads in a separate snapshot from the row, so a client can hold a token newer than the row it shows. Writes are still computed from current bytes, so nothing is lost.
  • Recovery only when needed: any process crash means a restart, and every start sweeps fully. A crash that isn't visible as in flight only matters inside a live process: P3-2 and P3-3.
  • Crash points (forward, k notes), new order:
    1. PREPARE persist (unchanged).
    2. Stage writes, all notes, before any sync: only .staging files, removed by the sweep.
    3. Parallel fsync done, before links: same as 2.
    4. Links made, before the folder sync: on power loss evidence names may vanish; they're re-staged from the journal.
    5. Between placements, and before sync_parents: a process crash is fine; on power loss any subset survives, and per-note classification handles it (P3-1 for mixed plans).
    6. After sync_parents: placements are durable, and skipping the final barrier for notes placed in this call is sound. Image data was synced before its link, and evidence lives in the target's folder, so one folder sync covers both.
    7. COMMIT and publication (unchanged; flushed is recorded only after sync_dir returns Ok).
    8. open() syncs the folder once per process and re-creates it if !is_dir(): fine.

Quiet perf rerun (perf-01, perf-lock on perf-01 then perf-02, held together)

Locks held from 18:28 to 20:03 UTC. dsync probe (200 × 4 KiB): 4.2 / 4.2 / 5.7 / 4.3 / 6.0 ms per write at start, mid, after gates 1–4, after the n=200 runs, and at the end. That's quiet; the earlier run measured 63 ms. Builds were done on dev-01 and relayed through the control host, with sha256 checked on both ends:

  • calternal dev b4b76071…, PR 6110b01a…
  • calternal-perf dev 7c879bed…, PR 8a700757…
  • tasks-write-bench #894 cac156dd a166c2b3…, #894 + this PR (scratch/executor-on-894 08057dbd) 57974c5a…

Write actions. Command: tasks-write-bench --base data/base-10000 --state st --n 100, #894 vs #894 + this PR, alternating A B A B A B, fresh state each run. Dev has no write bench, so this is the base. Cells are the three runs. Delete only gets samples when n > 100, so it comes from --n 200 run A B B A.

action base p50 PR p50 base p95 PR p95 p50 speed-up
tasks.set-status 976 / 1033 / 998 60.8 / 62.1 / 62.0 1236 / 1205 / 1198 77.8 / 81.6 / 80.6 16×
tasks.move When 169 / 173 / 164 55.7 / 57.2 / 53.6 257 / 265 / 256 73.8 / 70.8 / 73.7 3.0×
tasks.move Someday → day 418 / 428 / 423 59.6 / 57.7 / 62.8 558 / 561 / 567 77.8 / 75.1 / 83.2 7.1×
tasks.undo 294 / 302 / 297 51.9 / 51.7 / 51.2 616 / 620 / 615 62.6 / 67.3 / 64.8 5.8×
move-many 50 1674 / 1806 / 1672 224 / 218 / 244 – – 7.5×
move-many 50 undo 1687 / 1811 / 1698 172 / 162 / 175 – – 9.9×
tasks.delete (n=200) 1917 / 1925 86.4 / 120 2006 / 1991 101 / 146 16–22×
tasks.reorder (control, no executor) 11.5 / 11.5 / 11.5 12.0 / 11.3 / 11.6 12.0 / 13.8 / 15.7 13.8 / 13.1 / 12.4 flat
refetched versions / inventory retries 200 / 100 every run 0 / 0 every run

Gates are still not met: p95 is 62–83 ms against the 40 ms budget, and move-many 50 takes 218–244 ms against 200 ms. That matches 77110. Of the two PR n=200 runs, the second was about 30% slower on every action, including duplicate, which doesn't use the executor; that's environment noise, and it's still 16× faster than base. With 200 records instead of 100, PR cost is roughly the same (set-status 68 vs 61 ms) while base grows from 1000 to 1700 ms: the O(R) term is gone.

Other gates. Command: calternal-perf --sizes 10k --runs 3 --cold-runs 2 --idle-secs 0, dev vs PR, 8 pairs alternating (D P × 4, then P D × 4). dev run 8 is excluded: I released the perf-02 lock about a minute before it finished, and it reads slower on most metrics. So the table has 7 dev runs against 8 PR runs. "dev control" = median of the even dev runs vs the odd dev runs.

metric dev p50 runs PR p50 runs Δ median dev control
home.page (ms) 0.717 / 0.824 / 0.844 / 0.752 / 0.834 / 0.921 / 0.801 0.695 / 0.701 / 0.732 / 0.79 / 0.767 / 0.67 / 0.771 / 0.713 −12.4% +0.9%
home.page_tag (ms) 4.88 / 5.63 / 4.67 / 5.05 / 5.51 / 6.01 / 5.39 5.11 / 4.86 / 5.56 / 4.91 / 5.16 / 5.34 / 5.24 / 5.4 −3.6% +9.6%
search.query (ms) 1.56 / 1.81 / 1.58 / 1.49 / 1.99 / 1.6 / 1.48 1.6 / 1.58 / 1.6 / 1.71 / 1.48 / 1.55 / 1.63 / 1.6 +0.9% +1.8%
trash.delete (ms) 42.1 / 65.7 / 73.1 / 52.3 / 42.9 / 52.8 / 52.8 54.3 / 53.8 / 52 / 71 / 51.4 / 50.6 / 53.6 / 59.2 +1.7% +10.4%
open.note (ms) median 0.711 median 0.727 +2.2% +1.2%
cold.startup (ms) 13.3 / 13.4 / 17.8 / 17.5 / 13.3 / 13.4 / 13.3 14.8 / 17.9 / 15.6 / 20.2 / 12.5 / 13.5 / 13.4 / 15.6 +13.5% +0.8%
cold.first_page (ms) 244 / 231 / 293 / 290 / 233 / 245 / 247 249 / 294 / 281 / 308 / 239 / 250 / 233 / 274 +7.0% −0.2%
cold.index (s) 1.38 / 1.36 / 1.52 / 1.54 / 1.30 / 1.38 / 1.38 1.38 / 1.56 / 1.51 / 2.05 / 1.37 / 1.38 / 1.36 / 1.51 +4.7% −0.3%

The four metrics flagged in 77110 are flat or better in a quiet window; the earlier +6–9% was contention.

The cold.* medians lean against the PR, so I checked them separately:

  • Each run has only 2 cold samples.
  • In the quietest stretch (pairs 5–7, back to back) they're flat: startup 12.5 / 13.5 / 13.4 vs 13.3 / 13.4 / 13.3 ms; first page 239 / 250 / 233 vs 233 / 245 / 247 ms; index 1.37 / 1.38 / 1.36 vs 1.30 / 1.38 / 1.38 s.
  • The PR's code doesn't run on the cold path: recover_note_writes returns early without write-executor/, stamps_of and row_versions have no caller on this branch, and the only other change there is one AtomicBool.

I read them as noise. If you want certainty, run a cold-only A/B with more cold runs (--cold-runs 6).

All 46 metrics (dev runs 1–7, PR runs 1–8)
metric unit dev p50 runs PR p50 runs dev med PR med Δ med dev-odd vs dev-even Δ dev p95 med PR p95 med
cold.disk_written MB 150 / 150 / 150 / 150 / 150 / 150 / 150 150 / 150 / 150 / 151 / 150 / 150 / 150 / 150 150 150 -0.2% -0.2% 150 150
cold.first_page ms 244 / 231 / 293 / 290 / 233 / 245 / 247 249 / 294 / 281 / 308 / 239 / 250 / 233 / 274 245 262 +7.0% -0.2% 264 276
cold.index s 1.38 / 1.36 / 1.52 / 1.54 / 1.3 / 1.38 / 1.38 1.38 / 1.56 / 1.51 / 2.05 / 1.37 / 1.38 / 1.36 / 1.51 1.38 1.44 +4.7% -0.3% 1.47 1.5
cold.mem_peak MB 141 / 143 / 143 / 143 / 143 / 143 / 143 143 / 143 / 143 / 144 / 143 / 143 / 144 / 144 143 143 +0.4% -0.0% 143 144
cold.notes_per_s notes/s 6.78e+03 / 7.35e+03 / 6.52e+03 / 4.7e+03 / 7.27e+03 / 6.68e+03 / 7.15e+03 7.15e+03 / 6.14e+03 / 6.59e+03 / 4.54e+03 / 7.28e+03 / 6.72e+03 / 7.26e+03 / 6.53e+03 6.78e+03 6.66e+03 -1.9% -4.1% 7.25e+03 6.94e+03
cold.startup ms 13.3 / 13.4 / 17.8 / 17.5 / 13.3 / 13.4 / 13.3 14.8 / 17.9 / 15.6 / 20.2 / 12.5 / 13.5 / 13.4 / 15.6 13.4 15.2 +13.5% +0.8% 15.8 18.8
home.first_page ms 2.69 / 2.55 / 2.3 / 2.61 / 2.05 / 3.69 / 2.75 2.01 / 3.33 / 2.35 / 2.37 / 2.75 / 2.03 / 2.45 / 2.89 2.61 2.41 -7.7% +4.6% 2.99 3.79
home.page ms 0.717 / 0.824 / 0.844 / 0.752 / 0.834 / 0.921 / 0.801 0.695 / 0.701 / 0.732 / 0.79 / 0.767 / 0.67 / 0.771 / 0.713 0.824 0.722 -12.4% +0.9% 1.81 1.25
home.page_big ms 16.1 / 19.5 / 17.6 / 15.8 / 17.2 / 17 / 17.1 17.1 / 16.9 / 18.8 / 17.3 / 15.8 / 17.1 / 16.9 / 16.2 17.1 17 -0.7% -1.2% 20.8 23.3
home.page_next ms 0.694 / 0.836 / 0.746 / 0.846 / 0.734 / 0.885 / 0.803 0.735 / 0.739 / 0.757 / 0.678 / 0.671 / 0.745 / 0.675 / 0.828 0.803 0.737 -8.2% +14.3% 1.12 0.981
home.page_tag ms 4.88 / 5.63 / 4.67 / 5.05 / 5.51 / 6.01 / 5.39 5.11 / 4.86 / 5.56 / 4.91 / 5.16 / 5.34 / 5.24 / 5.4 5.39 5.2 -3.6% +9.6% 6.7 7.02
home.page_title ms 0.649 / 0.654 / 0.715 / 0.641 / 0.623 / 0.672 / 0.647 0.612 / 0.645 / 0.684 / 0.602 / 0.661 / 0.652 / 0.629 / 0.965 0.649 0.648 -0.1% +0.9% 0.829 0.83
journal.catch_up ms 27.3 / 30.9 / 28.9 / 28.9 / 29.8 / 31.2 / 30.4 27.7 / 28.6 / 30.7 / 28.8 / 27.9 / 29.5 / 31.2 / 27.7 29.8 28.7 -3.5% +5.2% 31.5 31.4
journal.entries count 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 1.02e+04 1.02e+04 +0.0% +0.0% 1.04e+04 1.04e+04
journal.page ms 2.35 / 2.83 / 2.47 / 2.44 / 2.52 / 2.7 / 2.58 2.34 / 2.41 / 2.51 / 2.43 / 2.34 / 2.45 / 2.64 / 2.44 2.52 2.44 -3.2% +8.4% 2.99 2.98
mem.after_first_page MB 21.4 / 21.6 / 21.2 / 21.4 / 21.5 / 22 / 21.3 21.3 / 21.8 / 21.8 / 21.4 / 21.5 / 21.5 / 21.5 / 21.8 21.4 21.5 +0.4% +0.9% 21.6 22
mem.after_opens MB 40.9 / 39.7 / 41.9 / 41 / 40.2 / 40.7 / 39.9 40.5 / 40.2 / 40.1 / 41.2 / 40.2 / 39.1 / 40.5 / 40.2 40.7 40.2 -1.1% +0.4% 43.9 42.4
mem.after_search MB 47.5 / 48.3 / 46.8 / 46.7 / 47.3 / 46.2 / 48.4 49.5 / 46.2 / 48.2 / 51.4 / 49.8 / 46.6 / 46.9 / 49 47.3 48.6 +2.6% -1.6% 74.9 75.2
mem.idle MB 18.8 / 19.6 / 18.6 / 19.4 / 19.5 / 18.7 / 19.1 19.3 / 19.7 / 19.7 / 18.8 / 18.8 / 19.3 / 18.8 / 19 19.1 19.2 +0.5% +2.5% 19.4 19.7
mem.peak MB 47.2 / 48.1 / 47 / 46.3 / 47 / 46.2 / 48 49 / 45.7 / 47.8 / 51.2 / 49.2 / 46.4 / 46.4 / 48.8 47 48.3 +2.7% -1.6% 74.4 74.8
open.api_get ms 2.57 / 3.26 / 4.89 / 3.51 / 2.69 / 2.99 / 3.56 2.66 / 2.59 / 3.65 / 5.16 / 2.54 / 2.55 / 2.67 / 2.65 3.26 2.65 -18.6% +4.1% 6.74 5.6
open.long_note ms 15.1 / 17.8 / 47.5 / 17.3 / 18.1 / 16.2 / 16.1 37.4 / 25.9 / 17.9 / 15.9 / 16.3 / 17.3 / 15.9 / 45.1 17.3 17.6 +1.7% +1.3% 51.1 45.8
open.note ms 0.679 / 0.714 / 0.723 / 0.711 / 0.703 / 0.739 / 0.707 0.741 / 0.744 / 0.664 / 0.656 / 0.663 / 0.72 / 0.801 / 0.733 0.711 0.727 +2.2% +1.2% 1.33 1.4
search.catch_up s 2.01 / 2.01 / 2.01 / 2.01 / 2.01 / 2.01 / 2.01 2.01 / 2.01 / 2.01 / 2.01 / 2.01 / 2 / 2.01 / 2.01 2.01 2.01 +0.0% +0.0% 4.01 4.01
search.first_query ms 3.62 / 4.88 / 3.96 / 3.74 / 7.56 / 5.65 / 4.47 7.03 / 4.37 / 4.44 / 3.82 / 4.49 / 5.24 / 4.38 / 5.27 4.47 4.46 -0.2% +15.8% 19.5 22.4
search.query ms 1.56 / 1.81 / 1.58 / 1.49 / 1.99 / 1.6 / 1.48 1.6 / 1.58 / 1.6 / 1.71 / 1.48 / 1.55 / 1.63 / 1.6 1.58 1.6 +0.9% +1.8% 3.94 4.05
shutdown ms 64.2 / 64.2 / 64.1 / 64 / 64.3 / 43.3 / 64.2 43 / 64.1 / 43.1 / 64.2 / 43.1 / 64.4 / 64.2 / 64.2 64.2 64.1 -0.0% -0.3% 85.4 64.4
startup.healthz ms 4.9 / 7.04 / 5.15 / 5.06 / 6.11 / 5.55 / 5.15 6.54 / 5.82 / 4.89 / 4.87 / 5.05 / 5.48 / 5.01 / 4.85 5.15 5.03 -2.4% +7.7% 7.88 7.23
storm.converge ms 169 / 173 / 173 / 174 / 175 / 175 / 175 172 / 175 / 175 / 173 / 175 / 176 / 176 / 174 174 175 +0.5% +0.3% 176 177
storm.disk_written MB 41.5 / 41.6 / 41.4 / 42.8 / 40.9 / 42 / 41.5 41.3 / 41.2 / 41 / 41.9 / 41.4 / 41.3 / 42.1 / 41 41.5 41.3 -0.5% +1.3% 43.6 42.9
storm.dropped_clients count 0 / 0 / 0 / 0 / 0 / 0 / 0 0 / 0 / 0 / 0 / 0 / 0 / 0 / 0 0 0 +0.0% +nan% 0 0
storm.edits_per_s edits/s 5.68e+03 / 5.41e+03 / 4.15e+03 / 4.08e+03 / 5.63e+03 / 5.71e+03 / 4.48e+03 5.67e+03 / 5.72e+03 / 4.06e+03 / 4.22e+03 / 5.77e+03 / 5.64e+03 / 5.53e+03 / 5.65e+03 5.41e+03 5.64e+03 +4.4% +7.0% 5.71e+03 5.77e+03
storm.outside_edits count 4 / 4 / 5 / 5 / 4 / 4 / 5 4 / 4 / 5 / 5 / 4 / 4 / 4 / 4 4 4 +0.0% -11.1% 5 4.5
storm.save_ack ms 4.59 / 4.87 / 6.26 / 6.29 / 4.56 / 4.5 / 5.95 4.57 / 4.68 / 6.47 / 6.42 / 4.54 / 4.55 / 4.7 / 4.59 4.87 4.64 -4.7% -7.6% 10.5 9.56
storm.typing s 1.76 / 1.85 / 2.41 / 2.45 / 1.78 / 1.75 / 2.23 1.76 / 1.75 / 2.46 / 2.37 / 1.73 / 1.77 / 1.81 / 1.77 1.85 1.77 -4.2% -7.7% 2.3 2.05
sync.durable_write ms 15.7 / 16.7 / 20.8 / 20.1 / 15.6 / 15.5 / 16.5 15.4 / 15.7 / 20.5 / 21.3 / 15.4 / 15.7 / 15.5 / 15.7 16.5 15.7 -4.8% +3.9% 24 24.1
sync.round_trip ms 0.196 / 0.211 / 0.214 / 0.215 / 0.206 / 0.204 / 0.227 0.213 / 0.209 / 0.224 / 0.208 / 0.202 / 0.227 / 0.222 / 0.225 0.211 0.217 +2.9% +0.5% 0.331 0.336
sync.save_ack ms 2.44 / 3.07 / 3.27 / 3.03 / 2.64 / 2.64 / 3.18 2.51 / 2.43 / 3.24 / 3.32 / 2.4 / 2.53 / 2.49 / 2.42 3.03 2.5 -17.4% +4.0% 4.53 4.21
trash.delete ms 42.1 / 65.7 / 73.1 / 52.3 / 42.9 / 52.8 / 52.8 54.3 / 53.8 / 52 / 71 / 51.4 / 50.6 / 53.6 / 59.2 52.8 53.7 +1.7% +10.4% 87.9 82.5
trash.forget ms 7.49 / 9.58 / 10.5 / 9.55 / 7.59 / 7.68 / 9.32 7.58 / 7.58 / 10.1 / 9.95 / 7.57 / 7.54 / 7.64 / 7.76 9.32 7.61 -18.3% +13.0% 14.5 14.4
trash.restore ms 54.5 / 55.8 / 60.9 / 65.6 / 54.3 / 55.1 / 58.4 54.8 / 57.7 / 57.9 / 60.4 / 54.4 / 56 / 61.9 / 60.1 55.8 57.8 +3.6% -1.2% 98 98.1
uploads.response ms 102 / 130 / 153 / 143 / 102 / 101 / 139 102 / 109 / 146 / 149 / 103 / 103 / 108 / 102 130 106 -19.1% +8.2% 152 116
uploads.round_trip ms 0.956 / 1.04 / 1.03 / 1.05 / 1.03 / 0.996 / 1.06 1 / 1.06 / 1.03 / 1.03 / 0.99 / 1 / 1.04 / 0.991 1.03 1.02 -1.8% +1.0% 1.35 1.32
uploads.save_ack ms 3.2 / 3.92 / 4.35 / 4.24 / 3.26 / 3.19 / 3.96 3.24 / 3.62 / 4.15 / 4.25 / 3.2 / 3.23 / 3.34 / 3.2 3.92 3.29 -16.2% +8.8% 10.2 6.2

Verdict

Safe to train. I found no path that loses or corrupts data, or accepts a write it should refuse, under process crashes, restarts, outside edits or concurrent processes. Power-loss ordering is weaker only for mixed create + remove plans, which don't exist yet (P3-1).

Before prod: the "roll forward only past #898" deploy note (P2-1) is required. P3-2 and P3-3 are small follow-ups worth folding in. Perf: the targets are 3–22× faster, and no non-target metric regresses in the quiet rerun. The 40 ms and 200 ms gates are still not met, as the PR already says.

🤖 Generated with Claude Code

## Independent review: data safety + quiet perf rerun (head 1b81f852) I read the plan (77003), the results (77110), `docs/implementation/note-write-executor*.md` and the full diff against dev 41f2efa3. I changed no code. **Counts: P1 0 · P2 1 · P3 6.** Verdict at the bottom. ### P2 **P2-1. Rolling back past this PR takes whole Homes offline (no data loss).** An older binary can't deserialize `Requirement::ExpectedOwners` (an externally tagged enum, so the variant is unknown). Home records are never compacted, so the plan's suggested mitigation, "drain the journal first", isn't possible: one committed set-status record is enough to block the downgrade forever. *Scenario:* deploy #898 → a user ticks one task → prod is rolled back to the previous build → `journal::read` fails in `recover_note_writes` → `Notes::home` shuts the Home down and returns the error → that user's whole Home (all notes, not just Tasks) fails to open until you roll forward again. Every user who wrote through the executor after the deploy is affected. *Fix (process, not code):* in the deploy issue (#684) and in the plan's failure-table row, write **"roll forward only past #898"**. Alternatively, ship a tolerant reader in a separate earlier deploy first. Either way, it must be in place before prod. ### P3 **P3-1. Batched placement drops "creations durable before removals" across folders on power loss. Latent: no current plan has both.** The old loop synced each target's folder before placing the next note, and the sort put creations and replacements before removals. Now every placement happens first, then one `sync_parents`. *Scenario (future plan that moves a note from folder X to Y):* power cut after both placements and before `sync_parents` → ext4 keeps X's rename (target → evidence) but not Y's link → after reboot the note is at neither public path. Startup recovery fixes it before the Home is served: the Prepared record holds the post bytes, and `check` classifies each note on its own, so any subset of placements is handled. But an outside reader of `Notes/` that starts before calternal (a sync tool or backup) sees a deletion. The crash suite uses process exits, where the page cache survives, so no test covers this ordering. Today's plans (set-status/move = replacements only, delete = one removal, duplicate = one creation) never mix the two. *Fix when a mixed plan appears:* sync the creation folders before the first removal. That's one extra folder sync, only for such plans. **P3-2. `executor_swept` isn't cleared on every failure path.** It's cleared in `run_note_write`'s error arms. It isn't cleared in `executor_undo` errors, or when `spawn_blocking` returns a `JoinError`. *Scenario:* an `executor_undo` fails after `finish` persisted a terminal state, and `cleanup_staging` errors → `.calternal-tmp-executor-*.staging` stays in the user's note folder and counts against the 1 GB budget until restart. Impact is limited: any non-terminal record sets `open` and forces the sweep anyway, and every restart sweeps. *Fix:* a fail-safe flag, cleared at the start of a write and set only after success. **P3-3. `index::view` caches before the caller's folder barrier succeeds.** `view()` inserts the refreshed cache. Then `NoteWriteExecutor::view` runs `sync_dir` when `unconfirmed` is set. If that `sync_dir` fails once (EIO), the next view sees equal stamps and hashes, so `unconfirmed = false` and the barrier is skipped for that entry. Mitigated: a non-terminal record forces `recover()`, whose full `load()` always syncs. Only terminal records are affected: an idempotent retry can get a Committed receipt before the entry is confirmed durable, and after power loss that record is Prepared again, so recovery rolls the same plan forward. *Fix:* keep `unconfirmed` in the cache until a sync succeeds, or sync inside `index::view` before `all.insert`. **P3-4. Damaged old records are now found at startup or on the next full read, not on every write.** The plan's table accepts this; I agree it's acceptable. Two follow-ups: - `docs/implementation/note-write-executor.md`, row "corrupt/unknown journal → recovery withheld", should say "at startup and on any full read of that record". - The non-Home `execute()` idempotent path now answers from the cached summary (`policy::receipt`). A retried key whose record later rots on disk returns Committed instead of refusing. That's acceptable, because the operation was committed when it was cached. **P3-5. Stat trust assumes a local POSIX filesystem.** On ext4, the stamp's ctime (ns), the 2 s window and rename-only replacement (always a new inode) make a stale cache entry practically impossible: - the clock stepping back gives a future ctime, which is always re-read; - the clock stepping forward is harmless, because the executor never edits a record in place; - restart starts cold; - another process's write changes the stamp and forces the folder sync. On network or FUSE filesystems with attribute caching (NFS `actimeo`) this doesn't hold. Add one line to the module doc: Homes on local filesystems only. **P3-6. Fewer checks between placements, and a narrower owner guard.** - *Between placements:* read dependencies and other notes' evidence are now checked only before COMMIT, not before each placement. An outside edit to a dependency during placement is found after every note is placed. The action becomes Conflicted with all notes placed instead of some; both need manual resolution. - *Owner guard:* `ExpectedOwners` covers each target's `task_lines` and `links` rows, but not the task's own `tasks` row (path, `broken`) that `update::target` reads. A task whose properties broke between plan and admission is still stopped by the exact-byte pre check and `MetadataValidation`. Only the error message changes; no wrong write is possible. ### Answers to the specific questions - **Owners changed → write succeeds?** No. The probe re-reads the same rows under `meta_update` on the Home's own `meta` connection, which is the one the task index writes through. Every owner note is also a participant with exact pre bytes. Owner edits that haven't been indexed yet are invisible to both the old epoch and the new digest, the same as before. - **Stale token accepted after an outside edit?** No. The token is a 52-bit hash of the v2 stamp (size, mtime, ctime, dev, inode, path), and any outside write changes ctime. It also requires history = exactly one `seed` row equal to the room's version. A collision has a chance of about 2⁻⁵². Remaining race (pre-existing, also true for real versions): `row_versions` reads in a separate snapshot from the row, so a client can hold a token newer than the row it shows. Writes are still computed from current bytes, so nothing is lost. - **Recovery only when needed:** any process crash means a restart, and every start sweeps fully. A crash that isn't visible as in flight only matters inside a live process: P3-2 and P3-3. - **Crash points (forward, k notes), new order:** 1. PREPARE persist (unchanged). 2. Stage writes, all notes, before any sync: only `.staging` files, removed by the sweep. 3. Parallel fsync done, before links: same as 2. 4. Links made, before the folder sync: on power loss evidence names may vanish; they're re-staged from the journal. 5. Between placements, and before `sync_parents`: a process crash is fine; on power loss any subset survives, and per-note classification handles it (P3-1 for mixed plans). 6. After `sync_parents`: placements are durable, and skipping the final barrier for notes placed in this call is sound. Image data was synced before its link, and evidence lives in the target's folder, so one folder sync covers both. 7. COMMIT and publication (unchanged; `flushed` is recorded only after `sync_dir` returns Ok). 8. `open()` syncs the folder once per process and re-creates it if `!is_dir()`: fine. ### Quiet perf rerun (perf-01, `perf-lock` on perf-01 then perf-02, held together) Locks held from 18:28 to 20:03 UTC. dsync probe (200 × 4 KiB): **4.2 / 4.2 / 5.7 / 4.3 / 6.0 ms** per write at start, mid, after gates 1–4, after the n=200 runs, and at the end. That's quiet; the earlier run measured 63 ms. Builds were done on dev-01 and relayed through the control host, with sha256 checked on both ends: - `calternal` dev `b4b76071…`, PR `6110b01a…` - `calternal-perf` dev `7c879bed…`, PR `8a700757…` - `tasks-write-bench` #894 cac156dd `a166c2b3…`, #894 + this PR (`scratch/executor-on-894` 08057dbd) `57974c5a…` **Write actions.** Command: `tasks-write-bench --base data/base-10000 --state st --n 100`, #894 vs #894 + this PR, alternating A B A B A B, fresh state each run. Dev has no write bench, so this is the base. Cells are the three runs. Delete only gets samples when n > 100, so it comes from `--n 200` run A B B A. | action | base p50 | PR p50 | base p95 | PR p95 | p50 speed-up | |---|---|---|---|---|---| | `tasks.set-status` | 976 / 1033 / 998 | 60.8 / 62.1 / 62.0 | 1236 / 1205 / 1198 | 77.8 / 81.6 / 80.6 | 16× | | `tasks.move` When | 169 / 173 / 164 | 55.7 / 57.2 / 53.6 | 257 / 265 / 256 | 73.8 / 70.8 / 73.7 | 3.0× | | `tasks.move` Someday → day | 418 / 428 / 423 | 59.6 / 57.7 / 62.8 | 558 / 561 / 567 | 77.8 / 75.1 / 83.2 | 7.1× | | `tasks.undo` | 294 / 302 / 297 | 51.9 / 51.7 / 51.2 | 616 / 620 / 615 | 62.6 / 67.3 / 64.8 | 5.8× | | `move-many` 50 | 1674 / 1806 / 1672 | 224 / 218 / 244 | – | – | 7.5× | | `move-many` 50 undo | 1687 / 1811 / 1698 | 172 / 162 / 175 | – | – | 9.9× | | `tasks.delete` (n=200) | 1917 / 1925 | 86.4 / 120 | 2006 / 1991 | 101 / 146 | 16–22× | | `tasks.reorder` (control, no executor) | 11.5 / 11.5 / 11.5 | 12.0 / 11.3 / 11.6 | 12.0 / 13.8 / 15.7 | 13.8 / 13.1 / 12.4 | flat | | refetched versions / inventory retries | 200 / 100 every run | **0 / 0** every run | | | | Gates are still not met: p95 is 62–83 ms against the 40 ms budget, and move-many 50 takes 218–244 ms against 200 ms. That matches 77110. Of the two PR n=200 runs, the second was about 30% slower on every action, including `duplicate`, which doesn't use the executor; that's environment noise, and it's still 16× faster than base. With 200 records instead of 100, PR cost is roughly the same (set-status 68 vs 61 ms) while base grows from 1000 to 1700 ms: the O(R) term is gone. **Other gates.** Command: `calternal-perf --sizes 10k --runs 3 --cold-runs 2 --idle-secs 0`, dev vs PR, 8 pairs alternating (D P × 4, then P D × 4). **dev run 8 is excluded**: I released the perf-02 lock about a minute before it finished, and it reads slower on most metrics. So the table has 7 dev runs against 8 PR runs. "dev control" = median of the even dev runs vs the odd dev runs. | metric | dev p50 runs | PR p50 runs | Δ median | dev control | |---|---|---|---|---| | **home.page** (ms) | 0.717 / 0.824 / 0.844 / 0.752 / 0.834 / 0.921 / 0.801 | 0.695 / 0.701 / 0.732 / 0.79 / 0.767 / 0.67 / 0.771 / 0.713 | **−12.4%** | +0.9% | | **home.page_tag** (ms) | 4.88 / 5.63 / 4.67 / 5.05 / 5.51 / 6.01 / 5.39 | 5.11 / 4.86 / 5.56 / 4.91 / 5.16 / 5.34 / 5.24 / 5.4 | **−3.6%** | +9.6% | | **search.query** (ms) | 1.56 / 1.81 / 1.58 / 1.49 / 1.99 / 1.6 / 1.48 | 1.6 / 1.58 / 1.6 / 1.71 / 1.48 / 1.55 / 1.63 / 1.6 | **+0.9%** | +1.8% | | **trash.delete** (ms) | 42.1 / 65.7 / 73.1 / 52.3 / 42.9 / 52.8 / 52.8 | 54.3 / 53.8 / 52 / 71 / 51.4 / 50.6 / 53.6 / 59.2 | **+1.7%** | +10.4% | | open.note (ms) | median 0.711 | median 0.727 | +2.2% | +1.2% | | cold.startup (ms) | 13.3 / 13.4 / 17.8 / 17.5 / 13.3 / 13.4 / 13.3 | 14.8 / 17.9 / 15.6 / 20.2 / 12.5 / 13.5 / 13.4 / 15.6 | +13.5% | +0.8% | | cold.first_page (ms) | 244 / 231 / 293 / 290 / 233 / 245 / 247 | 249 / 294 / 281 / 308 / 239 / 250 / 233 / 274 | +7.0% | −0.2% | | cold.index (s) | 1.38 / 1.36 / 1.52 / 1.54 / 1.30 / 1.38 / 1.38 | 1.38 / 1.56 / 1.51 / 2.05 / 1.37 / 1.38 / 1.36 / 1.51 | +4.7% | −0.3% | The four metrics flagged in 77110 are flat or better in a quiet window; the earlier +6–9% was contention. The `cold.*` medians lean against the PR, so I checked them separately: - Each run has only 2 cold samples. - In the quietest stretch (pairs 5–7, back to back) they're flat: startup 12.5 / 13.5 / 13.4 vs 13.3 / 13.4 / 13.3 ms; first page 239 / 250 / 233 vs 233 / 245 / 247 ms; index 1.37 / 1.38 / 1.36 vs 1.30 / 1.38 / 1.38 s. - The PR's code doesn't run on the cold path: `recover_note_writes` returns early without `write-executor/`, `stamps_of` and `row_versions` have no caller on this branch, and the only other change there is one `AtomicBool`. I read them as noise. If you want certainty, run a cold-only A/B with more cold runs (`--cold-runs 6`). <details><summary>All 46 metrics (dev runs 1–7, PR runs 1–8)</summary> | metric | unit | dev p50 runs | PR p50 runs | dev med | PR med | Δ med | dev-odd vs dev-even Δ | dev p95 med | PR p95 med | |---|---|---|---|---|---|---|---|---|---| | cold.disk_written | MB | 150 / 150 / 150 / 150 / 150 / 150 / 150 | 150 / 150 / 150 / 151 / 150 / 150 / 150 / 150 | 150 | 150 | -0.2% | -0.2% | 150 | 150 | | cold.first_page | ms | 244 / 231 / 293 / 290 / 233 / 245 / 247 | 249 / 294 / 281 / 308 / 239 / 250 / 233 / 274 | 245 | 262 | +7.0% | -0.2% | 264 | 276 | | cold.index | s | 1.38 / 1.36 / 1.52 / 1.54 / 1.3 / 1.38 / 1.38 | 1.38 / 1.56 / 1.51 / 2.05 / 1.37 / 1.38 / 1.36 / 1.51 | 1.38 | 1.44 | +4.7% | -0.3% | 1.47 | 1.5 | | cold.mem_peak | MB | 141 / 143 / 143 / 143 / 143 / 143 / 143 | 143 / 143 / 143 / 144 / 143 / 143 / 144 / 144 | 143 | 143 | +0.4% | -0.0% | 143 | 144 | | cold.notes_per_s | notes/s | 6.78e+03 / 7.35e+03 / 6.52e+03 / 4.7e+03 / 7.27e+03 / 6.68e+03 / 7.15e+03 | 7.15e+03 / 6.14e+03 / 6.59e+03 / 4.54e+03 / 7.28e+03 / 6.72e+03 / 7.26e+03 / 6.53e+03 | 6.78e+03 | 6.66e+03 | -1.9% | -4.1% | 7.25e+03 | 6.94e+03 | | cold.startup | ms | 13.3 / 13.4 / 17.8 / 17.5 / 13.3 / 13.4 / 13.3 | 14.8 / 17.9 / 15.6 / 20.2 / 12.5 / 13.5 / 13.4 / 15.6 | 13.4 | 15.2 | +13.5% | +0.8% | 15.8 | 18.8 | | home.first_page | ms | 2.69 / 2.55 / 2.3 / 2.61 / 2.05 / 3.69 / 2.75 | 2.01 / 3.33 / 2.35 / 2.37 / 2.75 / 2.03 / 2.45 / 2.89 | 2.61 | 2.41 | -7.7% | +4.6% | 2.99 | 3.79 | | **home.page** | ms | 0.717 / 0.824 / 0.844 / 0.752 / 0.834 / 0.921 / 0.801 | 0.695 / 0.701 / 0.732 / 0.79 / 0.767 / 0.67 / 0.771 / 0.713 | 0.824 | 0.722 | -12.4% | +0.9% | 1.81 | 1.25 | | home.page_big | ms | 16.1 / 19.5 / 17.6 / 15.8 / 17.2 / 17 / 17.1 | 17.1 / 16.9 / 18.8 / 17.3 / 15.8 / 17.1 / 16.9 / 16.2 | 17.1 | 17 | -0.7% | -1.2% | 20.8 | 23.3 | | home.page_next | ms | 0.694 / 0.836 / 0.746 / 0.846 / 0.734 / 0.885 / 0.803 | 0.735 / 0.739 / 0.757 / 0.678 / 0.671 / 0.745 / 0.675 / 0.828 | 0.803 | 0.737 | -8.2% | +14.3% | 1.12 | 0.981 | | **home.page_tag** | ms | 4.88 / 5.63 / 4.67 / 5.05 / 5.51 / 6.01 / 5.39 | 5.11 / 4.86 / 5.56 / 4.91 / 5.16 / 5.34 / 5.24 / 5.4 | 5.39 | 5.2 | -3.6% | +9.6% | 6.7 | 7.02 | | home.page_title | ms | 0.649 / 0.654 / 0.715 / 0.641 / 0.623 / 0.672 / 0.647 | 0.612 / 0.645 / 0.684 / 0.602 / 0.661 / 0.652 / 0.629 / 0.965 | 0.649 | 0.648 | -0.1% | +0.9% | 0.829 | 0.83 | | journal.catch_up | ms | 27.3 / 30.9 / 28.9 / 28.9 / 29.8 / 31.2 / 30.4 | 27.7 / 28.6 / 30.7 / 28.8 / 27.9 / 29.5 / 31.2 / 27.7 | 29.8 | 28.7 | -3.5% | +5.2% | 31.5 | 31.4 | | journal.entries | count | 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 | 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 / 1.02e+04 | 1.02e+04 | 1.02e+04 | +0.0% | +0.0% | 1.04e+04 | 1.04e+04 | | journal.page | ms | 2.35 / 2.83 / 2.47 / 2.44 / 2.52 / 2.7 / 2.58 | 2.34 / 2.41 / 2.51 / 2.43 / 2.34 / 2.45 / 2.64 / 2.44 | 2.52 | 2.44 | -3.2% | +8.4% | 2.99 | 2.98 | | mem.after_first_page | MB | 21.4 / 21.6 / 21.2 / 21.4 / 21.5 / 22 / 21.3 | 21.3 / 21.8 / 21.8 / 21.4 / 21.5 / 21.5 / 21.5 / 21.8 | 21.4 | 21.5 | +0.4% | +0.9% | 21.6 | 22 | | mem.after_opens | MB | 40.9 / 39.7 / 41.9 / 41 / 40.2 / 40.7 / 39.9 | 40.5 / 40.2 / 40.1 / 41.2 / 40.2 / 39.1 / 40.5 / 40.2 | 40.7 | 40.2 | -1.1% | +0.4% | 43.9 | 42.4 | | mem.after_search | MB | 47.5 / 48.3 / 46.8 / 46.7 / 47.3 / 46.2 / 48.4 | 49.5 / 46.2 / 48.2 / 51.4 / 49.8 / 46.6 / 46.9 / 49 | 47.3 | 48.6 | +2.6% | -1.6% | 74.9 | 75.2 | | mem.idle | MB | 18.8 / 19.6 / 18.6 / 19.4 / 19.5 / 18.7 / 19.1 | 19.3 / 19.7 / 19.7 / 18.8 / 18.8 / 19.3 / 18.8 / 19 | 19.1 | 19.2 | +0.5% | +2.5% | 19.4 | 19.7 | | mem.peak | MB | 47.2 / 48.1 / 47 / 46.3 / 47 / 46.2 / 48 | 49 / 45.7 / 47.8 / 51.2 / 49.2 / 46.4 / 46.4 / 48.8 | 47 | 48.3 | +2.7% | -1.6% | 74.4 | 74.8 | | open.api_get | ms | 2.57 / 3.26 / 4.89 / 3.51 / 2.69 / 2.99 / 3.56 | 2.66 / 2.59 / 3.65 / 5.16 / 2.54 / 2.55 / 2.67 / 2.65 | 3.26 | 2.65 | -18.6% | +4.1% | 6.74 | 5.6 | | open.long_note | ms | 15.1 / 17.8 / 47.5 / 17.3 / 18.1 / 16.2 / 16.1 | 37.4 / 25.9 / 17.9 / 15.9 / 16.3 / 17.3 / 15.9 / 45.1 | 17.3 | 17.6 | +1.7% | +1.3% | 51.1 | 45.8 | | open.note | ms | 0.679 / 0.714 / 0.723 / 0.711 / 0.703 / 0.739 / 0.707 | 0.741 / 0.744 / 0.664 / 0.656 / 0.663 / 0.72 / 0.801 / 0.733 | 0.711 | 0.727 | +2.2% | +1.2% | 1.33 | 1.4 | | search.catch_up | s | 2.01 / 2.01 / 2.01 / 2.01 / 2.01 / 2.01 / 2.01 | 2.01 / 2.01 / 2.01 / 2.01 / 2.01 / 2 / 2.01 / 2.01 | 2.01 | 2.01 | +0.0% | +0.0% | 4.01 | 4.01 | | search.first_query | ms | 3.62 / 4.88 / 3.96 / 3.74 / 7.56 / 5.65 / 4.47 | 7.03 / 4.37 / 4.44 / 3.82 / 4.49 / 5.24 / 4.38 / 5.27 | 4.47 | 4.46 | -0.2% | +15.8% | 19.5 | 22.4 | | **search.query** | ms | 1.56 / 1.81 / 1.58 / 1.49 / 1.99 / 1.6 / 1.48 | 1.6 / 1.58 / 1.6 / 1.71 / 1.48 / 1.55 / 1.63 / 1.6 | 1.58 | 1.6 | +0.9% | +1.8% | 3.94 | 4.05 | | shutdown | ms | 64.2 / 64.2 / 64.1 / 64 / 64.3 / 43.3 / 64.2 | 43 / 64.1 / 43.1 / 64.2 / 43.1 / 64.4 / 64.2 / 64.2 | 64.2 | 64.1 | -0.0% | -0.3% | 85.4 | 64.4 | | startup.healthz | ms | 4.9 / 7.04 / 5.15 / 5.06 / 6.11 / 5.55 / 5.15 | 6.54 / 5.82 / 4.89 / 4.87 / 5.05 / 5.48 / 5.01 / 4.85 | 5.15 | 5.03 | -2.4% | +7.7% | 7.88 | 7.23 | | storm.converge | ms | 169 / 173 / 173 / 174 / 175 / 175 / 175 | 172 / 175 / 175 / 173 / 175 / 176 / 176 / 174 | 174 | 175 | +0.5% | +0.3% | 176 | 177 | | storm.disk_written | MB | 41.5 / 41.6 / 41.4 / 42.8 / 40.9 / 42 / 41.5 | 41.3 / 41.2 / 41 / 41.9 / 41.4 / 41.3 / 42.1 / 41 | 41.5 | 41.3 | -0.5% | +1.3% | 43.6 | 42.9 | | storm.dropped_clients | count | 0 / 0 / 0 / 0 / 0 / 0 / 0 | 0 / 0 / 0 / 0 / 0 / 0 / 0 / 0 | 0 | 0 | +0.0% | +nan% | 0 | 0 | | storm.edits_per_s | edits/s | 5.68e+03 / 5.41e+03 / 4.15e+03 / 4.08e+03 / 5.63e+03 / 5.71e+03 / 4.48e+03 | 5.67e+03 / 5.72e+03 / 4.06e+03 / 4.22e+03 / 5.77e+03 / 5.64e+03 / 5.53e+03 / 5.65e+03 | 5.41e+03 | 5.64e+03 | +4.4% | +7.0% | 5.71e+03 | 5.77e+03 | | storm.outside_edits | count | 4 / 4 / 5 / 5 / 4 / 4 / 5 | 4 / 4 / 5 / 5 / 4 / 4 / 4 / 4 | 4 | 4 | +0.0% | -11.1% | 5 | 4.5 | | storm.save_ack | ms | 4.59 / 4.87 / 6.26 / 6.29 / 4.56 / 4.5 / 5.95 | 4.57 / 4.68 / 6.47 / 6.42 / 4.54 / 4.55 / 4.7 / 4.59 | 4.87 | 4.64 | -4.7% | -7.6% | 10.5 | 9.56 | | storm.typing | s | 1.76 / 1.85 / 2.41 / 2.45 / 1.78 / 1.75 / 2.23 | 1.76 / 1.75 / 2.46 / 2.37 / 1.73 / 1.77 / 1.81 / 1.77 | 1.85 | 1.77 | -4.2% | -7.7% | 2.3 | 2.05 | | sync.durable_write | ms | 15.7 / 16.7 / 20.8 / 20.1 / 15.6 / 15.5 / 16.5 | 15.4 / 15.7 / 20.5 / 21.3 / 15.4 / 15.7 / 15.5 / 15.7 | 16.5 | 15.7 | -4.8% | +3.9% | 24 | 24.1 | | sync.round_trip | ms | 0.196 / 0.211 / 0.214 / 0.215 / 0.206 / 0.204 / 0.227 | 0.213 / 0.209 / 0.224 / 0.208 / 0.202 / 0.227 / 0.222 / 0.225 | 0.211 | 0.217 | +2.9% | +0.5% | 0.331 | 0.336 | | sync.save_ack | ms | 2.44 / 3.07 / 3.27 / 3.03 / 2.64 / 2.64 / 3.18 | 2.51 / 2.43 / 3.24 / 3.32 / 2.4 / 2.53 / 2.49 / 2.42 | 3.03 | 2.5 | -17.4% | +4.0% | 4.53 | 4.21 | | **trash.delete** | ms | 42.1 / 65.7 / 73.1 / 52.3 / 42.9 / 52.8 / 52.8 | 54.3 / 53.8 / 52 / 71 / 51.4 / 50.6 / 53.6 / 59.2 | 52.8 | 53.7 | +1.7% | +10.4% | 87.9 | 82.5 | | trash.forget | ms | 7.49 / 9.58 / 10.5 / 9.55 / 7.59 / 7.68 / 9.32 | 7.58 / 7.58 / 10.1 / 9.95 / 7.57 / 7.54 / 7.64 / 7.76 | 9.32 | 7.61 | -18.3% | +13.0% | 14.5 | 14.4 | | trash.restore | ms | 54.5 / 55.8 / 60.9 / 65.6 / 54.3 / 55.1 / 58.4 | 54.8 / 57.7 / 57.9 / 60.4 / 54.4 / 56 / 61.9 / 60.1 | 55.8 | 57.8 | +3.6% | -1.2% | 98 | 98.1 | | uploads.response | ms | 102 / 130 / 153 / 143 / 102 / 101 / 139 | 102 / 109 / 146 / 149 / 103 / 103 / 108 / 102 | 130 | 106 | -19.1% | +8.2% | 152 | 116 | | uploads.round_trip | ms | 0.956 / 1.04 / 1.03 / 1.05 / 1.03 / 0.996 / 1.06 | 1 / 1.06 / 1.03 / 1.03 / 0.99 / 1 / 1.04 / 0.991 | 1.03 | 1.02 | -1.8% | +1.0% | 1.35 | 1.32 | | uploads.save_ack | ms | 3.2 / 3.92 / 4.35 / 4.24 / 3.26 / 3.19 / 3.96 | 3.24 / 3.62 / 4.15 / 4.25 / 3.2 / 3.23 / 3.34 / 3.2 | 3.92 | 3.29 | -16.2% | +8.8% | 10.2 | 6.2 | </details> ### Verdict **Safe to train.** I found no path that loses or corrupts data, or accepts a write it should refuse, under process crashes, restarts, outside edits or concurrent processes. Power-loss ordering is weaker only for mixed create + remove plans, which don't exist yet (P3-1). Before prod: the "roll forward only past #898" deploy note (P2-1) is required. P3-2 and P3-3 are small follow-ups worth folding in. Perf: the targets are 3–22× faster, and no non-target metric regresses in the quiet rerun. The 40 ms and 200 ms gates are still not met, as the PR already says. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kayg commented 2026-11-03 21:44:07 +00:00
Author
Owner
Copy link

Review fixes for 77166 (head 2432b5a4)

Rollback safety is in #905 (fix/journal-tolerant-reader, head 8973b7e6): the reader accepts ExpectedOwners, no write behaviour changes. It must ship in a train BEFORE this PR. Plan (77003) failure-table row, docs/implementation/note-write-executor.md and the PR body now say rollback past this PR is only safe to a build with #905.

item change
P2-1 #905 + test that reads/recovers/compacts a record written by this PR's head 1b81f852
P3-1 creations/replacements are folder-synced before the first removal (recovery.rs); one extra sync only for mixed plans; order test
P3-2 sweep flag is down for the whole write (and undo), restored only on success: error, undo failure, panic all leave it down
P3-3 journal cache dropped when its folder sync fails; test
P3-4, P3-5 rows added to the executor doc failure table (rot found at startup/full read; Homes on local filesystems only)
P3-6 note only, no change

Also one cargo fmt commit: the earlier head was not fmt-clean.

Gates (dev-01, ~/bin/slot, nix develop)

  • cargo fmt --all -- --check: OK
  • cargo clippy -j4 -p calternal-notes -p calternal-tasks -p calternal-engine --all-targets -- -D warnings: OK
  • cargo test -j4 --no-fail-fast -p calternal-notes -p calternal-tasks -p calternal-engine: engine 20 passed; notes 551 passed, 0 failed, 6 ignored (includes the crash suites); tasks 69 passed.

Perf (perf-01, perf-lock held on perf-02 and perf-01 together; 11-03 ~21:35 to 21:44 UTC)

tasks-write-bench --base data/base-10000 --n 100, release builds on dev-01, sha256 checked: previous head (scratch/executor-on-894 08057dbd, 12d22600…) = old; same + these fixes (3b715ca2…) = new. Order old, new, new, old, old, new. Run 1 (old) is the first run after the fixture and ran in a noisy disk window (dsync probe 94 ms/write at start, 6-7 ms at the end): discarded. Cells p50 / p95 ms:

action old (runs 4, 5) new (runs 2, 3, 6)
set-status 85.7/117 , 92.1/170 84.5/111 , 84.1/111 , 89.4/120
move When 86.9/122 , 88.5/139 76.5/109 , 75.9/117 , 83.3/117
move Someday to day 83.5/123 , 81.8/115 87.4/117 , 84.6/115 , 82.8/113
undo 75.3/93 , 75.9/120 74.0/94 , 71.9/94 , 73.9/91
move-many 50 307 , 300 295 , 327 , 306
move-many 50 undo 236 , 206 208 , 225 , 230
reorder (control, no executor) 16.8 , 16.2 16.3 , 15.7 , 15.9
refetched versions 0 0

Flat: new is equal or slightly better than old on every action.

Caveat, not hidden: this window was slower in absolute terms than comment 77166 (set-status p50 about 85 ms against about 61 ms, reorder control 16 ms against 11.5 ms). The control moved by the same factor (about 1.4x), so it is the shared disk, not this change. Numbers against 77166 are therefore not directly comparable; the same-session old-vs-new pair is the valid comparison. If you want an absolute rerun, wait for a quiet disk window (the perf VMs were busy with other jobs for about 1 h before this).

Cleaned up: perf work dirs, bench worktree targets.

🤖 Generated with Claude Code

## Review fixes for 77166 (head 2432b5a4) Rollback safety is in **#905 (`fix/journal-tolerant-reader`, head 8973b7e6)**: the reader accepts `ExpectedOwners`, no write behaviour changes. It must ship in a train BEFORE this PR. Plan (77003) failure-table row, `docs/implementation/note-write-executor.md` and the PR body now say rollback past this PR is only safe to a build with #905. | item | change | |---|---| | P2-1 | #905 + test that reads/recovers/compacts a record written by this PR's head 1b81f852 | | P3-1 | creations/replacements are folder-synced before the first removal (`recovery.rs`); one extra sync only for mixed plans; order test | | P3-2 | sweep flag is down for the whole write (and undo), restored only on success: error, undo failure, panic all leave it down | | P3-3 | journal cache dropped when its folder sync fails; test | | P3-4, P3-5 | rows added to the executor doc failure table (rot found at startup/full read; Homes on local filesystems only) | | P3-6 | note only, no change | Also one `cargo fmt` commit: the earlier head was not fmt-clean. ### Gates (dev-01, `~/bin/slot`, nix develop) - `cargo fmt --all -- --check`: OK - `cargo clippy -j4 -p calternal-notes -p calternal-tasks -p calternal-engine --all-targets -- -D warnings`: OK - `cargo test -j4 --no-fail-fast -p calternal-notes -p calternal-tasks -p calternal-engine`: engine 20 passed; notes 551 passed, 0 failed, 6 ignored (includes the crash suites); tasks 69 passed. ### Perf (perf-01, `perf-lock` held on perf-02 and perf-01 together; 11-03 ~21:35 to 21:44 UTC) `tasks-write-bench --base data/base-10000 --n 100`, release builds on dev-01, sha256 checked: previous head (scratch/executor-on-894 08057dbd, `12d22600…`) = old; same + these fixes (`3b715ca2…`) = new. Order old, new, new, old, old, new. Run 1 (old) is the first run after the fixture and ran in a noisy disk window (dsync probe 94 ms/write at start, 6-7 ms at the end): discarded. Cells p50 / p95 ms: | action | old (runs 4, 5) | new (runs 2, 3, 6) | |---|---|---| | set-status | 85.7/117 , 92.1/170 | 84.5/111 , 84.1/111 , 89.4/120 | | move When | 86.9/122 , 88.5/139 | 76.5/109 , 75.9/117 , 83.3/117 | | move Someday to day | 83.5/123 , 81.8/115 | 87.4/117 , 84.6/115 , 82.8/113 | | undo | 75.3/93 , 75.9/120 | 74.0/94 , 71.9/94 , 73.9/91 | | move-many 50 | 307 , 300 | 295 , 327 , 306 | | move-many 50 undo | 236 , 206 | 208 , 225 , 230 | | reorder (control, no executor) | 16.8 , 16.2 | 16.3 , 15.7 , 15.9 | | refetched versions | 0 | 0 | **Flat: new is equal or slightly better than old on every action.** **Caveat, not hidden:** this window was slower in absolute terms than comment 77166 (set-status p50 about 85 ms against about 61 ms, reorder control 16 ms against 11.5 ms). The control moved by the same factor (about 1.4x), so it is the shared disk, not this change. Numbers against 77166 are therefore not directly comparable; the same-session old-vs-new pair is the valid comparison. If you want an absolute rerun, wait for a quiet disk window (the perf VMs were busy with other jobs for about 1 h before this). Cleaned up: perf work dirs, bench worktree targets. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kayg commented 2026-11-07 18:21:39 +00:00
Author
Owner
Copy link

Housekeeping #1009 status: audited head 2432b5a472f83401eb0164ccd2ea92b70a45e39d against dev cd7f84432feb. git cherry origin/dev <head> reports 17 unmatched commits and 0 equivalent patches. Separate stable patch-id indexing and key-file checks do not establish full landing. Absent from dev: crates/notes/src/history/row_versions.rs, crates/notes/src/row_version.rs, crates/notes/src/write_executor/index.rs. Left unchanged as WIP, per kayg’s instruction; no rebase, close, tests, or merge. The #1009 audit records the unmatched commit list.

Housekeeping #1009 status: audited head `2432b5a472f83401eb0164ccd2ea92b70a45e39d` against dev `cd7f84432feb`. `git cherry origin/dev <head>` reports 17 unmatched commits and 0 equivalent patches. Separate stable patch-id indexing and key-file checks do not establish full landing. Absent from dev: `crates/notes/src/history/row_versions.rs`, `crates/notes/src/row_version.rs`, `crates/notes/src/write_executor/index.rs`. Left unchanged as WIP, per kayg’s instruction; no rebase, close, tests, or merge. The #1009 audit records the unmatched commit list.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin perf/executor-hot-path:perf/executor-hot-path
git switch perf/executor-hot-path

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff perf/executor-hot-path
git switch perf/executor-hot-path
git rebase dev
git switch dev
git merge --ff-only perf/executor-hot-path
git switch perf/executor-hot-path
git rebase dev
git switch dev
git merge --no-ff perf/executor-hot-path
git switch dev
git merge --squash perf/executor-hot-path
git switch dev
git merge --ff-only perf/executor-hot-path
git switch dev
git merge perf/executor-hot-path
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!898
No description provided.