kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

shared: restore detection (#918) + per-field stale check (#919) #928

Closed
kayg wants to merge 12 commits from shared/restore-and-stale-918-919 into dev
pull from: shared/restore-and-stale-918-919
merge into: kayg:dev
Conversation 25 Commits 12 Files changed 31 +2409 -116
kayg commented 2026-11-04 10:46:45 +00:00
Owner
Copy link

Simplest design that meets #918/#919 (updated 2026-11-09)

Evidence and open measurements

Comment #86388 measured candidate 95af4df2: startup_1m_file_evicted was +35.75% (+0.188 ms) against a −2.59% control. Cause: restored_at_open validated a same-epoch witness behind the database head and then called write_now, which rewrote and synced the witness during open. Commit f71379a1c leaves a valid older mark durable and has the existing idle/staleness worker advance it. First-use initialization still makes its witness durable before setting the initialized flag or returning signed tokens, and epoch rotation still waits for its witness before publishing the new epoch. At merged head 8357e925a (dev 2fa9e18fb), comments 88632, 88778 and the room-rerun comment show startup_1m_file_evicted flat against its control at 20 obs/arm, with commit, position and catch-up faster. Open: small p50 shifts in startup_healthz (first-install key and witness creation), sync_round_trip and save_ack (per-entry signing), and a higher soak max. These need an owner decision under the no-regression rule.

Design

  • The journal writer publishes the new head with one existing Relaxed atomic store. It does not read a clock, inspect dirty age, wake the flusher, stat a path, or write the witness. The flush task polls that head at no more than 100 ms intervals and records changes itself, so direct commits are covered without save-path coordination.
  • At startup, a valid same-epoch mark behind the database head is checked with one indexed journal lookup, left durable in place, and marked dirty for the flush worker. Startup does not rewrite it; first-deploy initialization and epoch rotation still require a durable witness before returning or publishing.
  • One background flush task opens and closes the configured witness path for each flush. It reads one 4 KiB page, then overwrites the inactive 64-byte slot with pwrite and calls fdatasync. Two alternating checksummed records carry sequence numbers so a torn write leaves the prior slot available. First initialization writes a valid slot before syncing the parent directory; the steady path does not rename a temporary file or sync the directory. A fresh open adds one open/read/close per flush, at most once per 30 seconds during continuous saves; that bounded metadata and descriptor work is negligible beside the sync and stays off the save and commit paths.
  • A flush never starts while a save is in flight. Saves are notes actions, which hold a scope inside the engine, and room typing saves, which take an EventRecorder::save_scope from the engine for an already-open store and never open one. Flush after one quiet second with no save in flight, or when the oldest unflushed high-water reaches the 30-second maximum age and no save is in flight: the save's end wakes the flusher, so the witness sync lands in the gap before the next save. A save stream that never pauses forces the flush at twice the maximum age (60 seconds). Also flush after the writer drains at clean shutdown. A crash can leave up to 30 seconds, plus the save in flight, of committed journal tail newer than the external witness (60 seconds worst case); a later signed cursor beyond the restored head, at a missing entry, or at a reused entry triggers the safe restore check. Storage failures extend the window until retry succeeds.
  • The flusher takes the head's entry check from the signed token the writer already cached for that entry (epoch-seq-c-m), so a flush opens SQLite only when every cache misses. The commit path is unchanged.
  • First-install key creation and the first durable witness happen on the first store open, which is lazy (first request for that user), never before /healthz. The store is not returned until both are durable, so no signed token can go out earlier.
  • Position is O(1) for exact cached proofs and O(log n) through the journal index on a cache miss. Startup reads one bounded page. Catch-up and reconnect do no extra per-entry witness work.
  • Every flush checks (dev, ino) and a linked descriptor before writing and again after sync. If the configured path was replaced, the worker repairs the last durable mark through the slow atomic rewrite, returns Internal, and publishes no epoch until the witness is durable at the configured path. Identity checks stay on the flush worker; a fresh open does not remove the race between open() and fdatasync().

Failure table

Failure Effect Detection Recovery Test
Crash during a slot write The new slot may be complete or torn; the other slot is untouched Validate bounds, sequence, and checksum on startup Use the newest valid slot; if neither is valid, treat the witness as unknown and run the safe restore check a_crash_mid_flush_keeps_the_previous_valid_slot
Torn slot The new slot is rejected Checksum mismatch Fall back to the previous slot; if it is also invalid, run the safe restore check a_torn_slot_falls_back_to_the_previous_valid_slot
Both slots bad There is no trusted high-water mark Neither slot validates Treat the witness as unknown and run the safe restore check both_invalid_highwater_slots_trigger_the_safe_restore_check
Witness path replaced while the server runs A flush could otherwise reach an unlinked or renamed inode Compare descriptor and configured path identity before writing and after sync; require nlink > 0 Repair the last durable mark through the slow atomic rewrite, return Internal, and retry; never publish an epoch from the stale inode Replacement between flushes, unlink, rename-and-replace, directory replacement, and during-flush tests
Crash before first slot is durable No initial witness is trusted The initialization flag is written only after the slot is durable Retry first initialization; if the store was already initialized, treat the witness as unknown and run the safe restore check Deferred first-page initialization test
Witness write or sync fails The previous mark remains authoritative and the staleness interval can grow The write or sync returns an error Keep retryable state and retry after one second or new save activity Failed-witness-write tests
Valid witness lags the journal at startup The old mark remains authoritative while the new tail is unwitnessed Check the old mark at its indexed entry and mark the head dirty in memory Flush after the quiet window, at 30 seconds maximum age, or on clean shutdown; a rollback inside the remaining window is exposed by a later signed cursor proof opening_a_lagging_witness_does_not_sync_during_startup plus idle, maximum-age, and shutdown tests
30-second maximum staleness during continuous saves (60 seconds if saves never pause) A crash can leave up to 30 seconds plus the in-flight save (60 seconds worst case) of committed journal tail beyond the durable witness; a rollback inside that window may not be visible at startup while the restored head stays at or above the witness The flush worker tracks the oldest unwitnessed head and flushes at 30 seconds once no save is in flight, or at 60 seconds regardless, when storage succeeds; a later signed cursor for a missing or changed entry exposes an unwitnessed rollback Reject the cursor, rotate to a new epoch, and reconcile from the surviving journal; failed witness I/O retries and extends the window maximum_age_flush_waits_for_the_save_in_flight, a_save_stream_that_never_pauses_forces_the_flush_at_twice_the_maximum_age, maximum_age_flush_waits_for_the_save_in_flight_to_end, maximum_staleness_flushes_during_a_continuous_save_stream

Gates

Run cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace --no-fail-fast. Run save/sync Playwright specs on Chromium and WebKit at 1440 and 390. With both perf locks held, interleave dev controls and candidate arms in ABAB order for the signed commit bisect (×3 per stage), then the no-idle full matrix (ABAB ×5; startup rows at least 20 samples per arm) and a three-minute continuous-save soak at 200 ms cadence. Report every path, including notes_save p95 and p99. Every metric must be flat or better beyond the control spread. No merge.

Implements #918 and #919 as approved in the Mail plan v4 review (#920, comments 77534/77555/77574/77580), plus the durability note on #918 (77762). Shared machinery only: no mail code. Notes behaviour is unchanged (no notes action opts in; stale_patch_keeps_newer_typing proves a stale patch still merges).

#918: restore detection

  • Token = epoch-seq-c-m. c = 32 bits of blake3 over the entry at seq (seq ‖ at_ms ‖ kind ‖ item ‖ request_id); m = 32 bits keyed with <secrets>/users/<u>/token.key (0600, outside backups). Written for every entry in the writer and every read (since), plus token_at() for ready/replay/empty-page tokens.
  • UserStore::position (now async): bad m → "invalid token", never a rotation; other epoch → expired; valid m with seq > head, a missing entry or a different c → proof: rotate, then expired. Plain epoch-seq tokens still work while not past the head. A device at the head costs no read (cached c of the newest entry); otherwise one primary-key read.
  • High-water file <secrets>/users/<u>/head is a preallocated one-page file with two alternating checksummed slots. It is flushed after a one-second save quiet window, at 30 seconds maximum age, and after the writer drains at clean shutdown. Steady writes use one inactive-slot pwrite and fdatasync; startup reads one bounded page.
  • At open: missing + meta.highwater=1 → restored; missing, no flag → first deploy (file + flag, nobody rotates); other epoch, or same epoch with file ahead of head or c ≠ entry → restored.
  • Rotation (writer thread, Cmd::Rotate(observed_epoch), so concurrent proofs rotate once): epoch, epoch_head, and highwater in one FULL commit, then the new witness slot is made durable, then the new epoch is published. Ordinary commits stay NORMAL; no fsync is added to the journal writer.
  • Plugins: Engine::restored(user) returns a watch counter (>0 at start = restored), Engine::epoch_head(user), ActionCtx::journal_info(). Live sockets end with sync_token_expired on rotation so no session keeps issuing new-epoch tokens to a client that has not re-read.
  • --secrets DIR / CALTERNAL_SECRETS on serve. Without it: plain tokens, no detection, one warning at start (tests and one-off tools). Deploy needs the flag set to a folder outside the backup set (docs/backup.md updated).
  • Decision: "invalid token" is sync_token_expired with that detail (clients already re-read on it); no new error code, so contracts do not change.
  • Not in this PR: the topic column/index on the save log. Nothing here needs it; it stays with the sync-topics/mail PR, so there is no migration to time. Cold-start/open cost at 1M entries is measured instead.

#919: opt-in per-field check

  • calternal_core::stale: Guard {base, after} (flatten into an action input), Tracked<V> (value, epoch, seq, req; set stamps only on a real change; rebuilt takes the head; with_pending folds accepted-but-unapplied actions), check_field → Apply | NoChange | Stale{epoch_only}.
  • Rule: equal value → success; else stale only if a base is given, the field changed after it, and the last change's op is not in after. A base from an older epoch is stale with epoch_only; marks stamped before a rotation count as epoch_head.
  • Decision: comparison is strict (field.seq > base.seq). The issue text says ≥, but a token already includes the entry at its seq, and ≥ would make a rebuilt row (seq = head) refuse the replay of a device that re-read at that very head, forever.
  • Device side apps/web/src/lib/action-outbox.ts: merges only unsent ops per (target, field); merged op keeps the oldest base; a merge back to the original is dropped only when nothing was sent; after(); restamp(token) re-stamps and re-queues every unconfirmed op after a rotation; replay:false ops (delete forever) are handed back, never replayed. No existing screen uses it yet (prefs keep their own outbox).

Idempotency after a power cut (77762)

Idempotency records in index.db are NORMAL, so an action that replied can run again after a power cut. No fsync added to the shared writer. Audit of every plugin with mutating actions:

Plugin / action Re-run after a lost record Why
notes.* safe id derived from the key; receipts in history.db (FULL)
daily.open and log lines safe key forwarded to the Notes patch receipt
tasks.create / set-status safe key becomes the Notes job id (receipts in history.db)
contacts.create/update/import/delete/restore safe key → job rows in history.db (jobs::attempt)
groups.create, people.invites safe id derived from the key, insert ON CONFLICT DO NOTHING in auth.db (FULL); the reply carries only the id
people.role/quota, groups.rename/delete/member, sharing.grant/revoke, account.revoke, prefs.set safe set-to-state
attachments.add follow-up, low risk the staged upload is consumed by the first run; a re-run answers "no upload arrived with this key" although the file is placed (bytes are content-addressed, a re-upload dedupes). Fix: answer from the placed file's hash
search.visit harmless a ranking nudge

Tests

  • store/src/tests/restore.rs: signed tokens everywhere; garbled/hand-edited token (invalid, no rotation); other-epoch token; first deploy; backup after the last start; backup from before an earlier rotation; host snapshot rollback (found by the first newer token); reused seq with a different entry; lost log tail at open; every file back to its last fsync, head passes the device (77762); zero-length/torn file; ahead file; rotation durable before new tokens + power loss; two proofs rotate once; idle/max-age/shutdown flush; torn-slot fallback; both-slots-bad safe restore; key mode.
  • core/src/stale/tests.rs + engine/src/tests/restore.rs: flag racing read, two devices mark read, lost reply then opposite toggle, stale base racing a pending action, two tabs + retry after the prune, rebuild then late retry, rebuild then unchanged value, online adapter without base, restore → stale_epoch → re-stamp → applies.
  • apps/web/src/lib/action-outbox.test.ts.

Gate and perf tables follow as comments.

🤖 Generated with Claude Code

## Simplest design that meets #918/#919 (updated 2026-11-09) ### Evidence and open measurements Comment #86388 measured candidate `95af4df2`: `startup_1m_file_evicted` was +35.75% (+0.188 ms) against a −2.59% control. Cause: `restored_at_open` validated a same-epoch witness behind the database head and then called `write_now`, which rewrote and synced the witness during open. Commit `f71379a1c` leaves a valid older mark durable and has the existing idle/staleness worker advance it. First-use initialization still makes its witness durable before setting the initialized flag or returning signed tokens, and epoch rotation still waits for its witness before publishing the new epoch. At merged head `8357e925a` (dev `2fa9e18fb`), comments 88632, 88778 and the room-rerun comment show `startup_1m_file_evicted` flat against its control at 20 obs/arm, with commit, position and catch-up faster. Open: small p50 shifts in `startup_healthz` (first-install key and witness creation), `sync_round_trip` and `save_ack` (per-entry signing), and a higher soak max. These need an owner decision under the no-regression rule. ### Design - The journal writer publishes the new head with one existing `Relaxed` atomic store. It does not read a clock, inspect dirty age, wake the flusher, stat a path, or write the witness. The flush task polls that head at no more than 100 ms intervals and records changes itself, so direct commits are covered without save-path coordination. - At startup, a valid same-epoch mark behind the database head is checked with one indexed journal lookup, left durable in place, and marked dirty for the flush worker. Startup does not rewrite it; first-deploy initialization and epoch rotation still require a durable witness before returning or publishing. - One background flush task opens and closes the configured witness path for each flush. It reads one 4 KiB page, then overwrites the inactive 64-byte slot with `pwrite` and calls `fdatasync`. Two alternating checksummed records carry sequence numbers so a torn write leaves the prior slot available. First initialization writes a valid slot before syncing the parent directory; the steady path does not rename a temporary file or sync the directory. A fresh open adds one open/read/close per flush, at most once per 30 seconds during continuous saves; that bounded metadata and descriptor work is negligible beside the sync and stays off the save and commit paths. - A flush never starts while a save is in flight. Saves are notes actions, which hold a scope inside the engine, and room typing saves, which take an `EventRecorder::save_scope` from the engine for an already-open store and never open one. Flush after one quiet second with no save in flight, or when the oldest unflushed high-water reaches the 30-second maximum age and no save is in flight: the save's end wakes the flusher, so the witness sync lands in the gap before the next save. A save stream that never pauses forces the flush at twice the maximum age (60 seconds). Also flush after the writer drains at clean shutdown. A crash can leave up to 30 seconds, plus the save in flight, of committed journal tail newer than the external witness (60 seconds worst case); a later signed cursor beyond the restored head, at a missing entry, or at a reused entry triggers the safe restore check. Storage failures extend the window until retry succeeds. - The flusher takes the head's entry check from the signed token the writer already cached for that entry (`epoch-seq-c-m`), so a flush opens SQLite only when every cache misses. The commit path is unchanged. - First-install key creation and the first durable witness happen on the first store open, which is lazy (first request for that user), never before `/healthz`. The store is not returned until both are durable, so no signed token can go out earlier. - Position is O(1) for exact cached proofs and O(log n) through the journal index on a cache miss. Startup reads one bounded page. Catch-up and reconnect do no extra per-entry witness work. - Every flush checks `(dev, ino)` and a linked descriptor before writing and again after sync. If the configured path was replaced, the worker repairs the last durable mark through the slow atomic rewrite, returns `Internal`, and publishes no epoch until the witness is durable at the configured path. Identity checks stay on the flush worker; a fresh open does not remove the race between `open()` and `fdatasync()`. ### Failure table | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | Crash during a slot write | The new slot may be complete or torn; the other slot is untouched | Validate bounds, sequence, and checksum on startup | Use the newest valid slot; if neither is valid, treat the witness as unknown and run the safe restore check | `a_crash_mid_flush_keeps_the_previous_valid_slot` | | Torn slot | The new slot is rejected | Checksum mismatch | Fall back to the previous slot; if it is also invalid, run the safe restore check | `a_torn_slot_falls_back_to_the_previous_valid_slot` | | Both slots bad | There is no trusted high-water mark | Neither slot validates | Treat the witness as unknown and run the safe restore check | `both_invalid_highwater_slots_trigger_the_safe_restore_check` | | Witness path replaced while the server runs | A flush could otherwise reach an unlinked or renamed inode | Compare descriptor and configured path identity before writing and after sync; require `nlink > 0` | Repair the last durable mark through the slow atomic rewrite, return `Internal`, and retry; never publish an epoch from the stale inode | Replacement between flushes, unlink, rename-and-replace, directory replacement, and during-flush tests | | Crash before first slot is durable | No initial witness is trusted | The initialization flag is written only after the slot is durable | Retry first initialization; if the store was already initialized, treat the witness as unknown and run the safe restore check | Deferred first-page initialization test | | Witness write or sync fails | The previous mark remains authoritative and the staleness interval can grow | The write or sync returns an error | Keep retryable state and retry after one second or new save activity | Failed-witness-write tests | | Valid witness lags the journal at startup | The old mark remains authoritative while the new tail is unwitnessed | Check the old mark at its indexed entry and mark the head dirty in memory | Flush after the quiet window, at 30 seconds maximum age, or on clean shutdown; a rollback inside the remaining window is exposed by a later signed cursor proof | `opening_a_lagging_witness_does_not_sync_during_startup` plus idle, maximum-age, and shutdown tests | | 30-second maximum staleness during continuous saves (60 seconds if saves never pause) | A crash can leave up to 30 seconds plus the in-flight save (60 seconds worst case) of committed journal tail beyond the durable witness; a rollback inside that window may not be visible at startup while the restored head stays at or above the witness | The flush worker tracks the oldest unwitnessed head and flushes at 30 seconds once no save is in flight, or at 60 seconds regardless, when storage succeeds; a later signed cursor for a missing or changed entry exposes an unwitnessed rollback | Reject the cursor, rotate to a new epoch, and reconcile from the surviving journal; failed witness I/O retries and extends the window | `maximum_age_flush_waits_for_the_save_in_flight`, `a_save_stream_that_never_pauses_forces_the_flush_at_twice_the_maximum_age`, `maximum_age_flush_waits_for_the_save_in_flight_to_end`, `maximum_staleness_flushes_during_a_continuous_save_stream` | ### Gates Run `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, and `cargo test --workspace --no-fail-fast`. Run save/sync Playwright specs on Chromium and WebKit at 1440 and 390. With both perf locks held, interleave dev controls and candidate arms in ABAB order for the signed commit bisect (×3 per stage), then the no-idle full matrix (ABAB ×5; startup rows at least 20 samples per arm) and a three-minute continuous-save soak at 200 ms cadence. Report every path, including `notes_save` p95 and p99. Every metric must be flat or better beyond the control spread. No merge. --- Implements #918 and #919 as approved in the Mail plan v4 review (#920, comments 77534/77555/77574/77580), plus the durability note on #918 (77762). Shared machinery only: no mail code. **Notes behaviour is unchanged** (no notes action opts in; `stale_patch_keeps_newer_typing` proves a stale patch still merges). ## #918: restore detection - Token = `epoch-seq-c-m`. `c` = 32 bits of blake3 over the entry at `seq` (`seq ‖ at_ms ‖ kind ‖ item ‖ request_id`); `m` = 32 bits keyed with `<secrets>/users/<u>/token.key` (0600, outside backups). Written for every entry in the writer and every read (`since`), plus `token_at()` for ready/replay/empty-page tokens. - `UserStore::position` (now async): bad `m` → "invalid token", never a rotation; other epoch → expired; valid `m` with `seq > head`, a missing entry or a different `c` → proof: rotate, then expired. Plain `epoch-seq` tokens still work while not past the head. A device at the head costs no read (cached `c` of the newest entry); otherwise one primary-key read. - High-water file `<secrets>/users/<u>/head` is a preallocated one-page file with two alternating checksummed slots. It is flushed after a one-second save quiet window, at 30 seconds maximum age, and after the writer drains at clean shutdown. Steady writes use one inactive-slot `pwrite` and `fdatasync`; startup reads one bounded page. - At open: missing + `meta.highwater=1` → restored; missing, no flag → first deploy (file + flag, **nobody rotates**); other epoch, or same epoch with file ahead of head or `c` ≠ entry → restored. - Rotation (writer thread, `Cmd::Rotate(observed_epoch)`, so concurrent proofs rotate once): `epoch`, `epoch_head`, and `highwater` in one FULL commit, then the new witness slot is made durable, then the new epoch is published. Ordinary commits stay NORMAL; no fsync is added to the journal writer. - Plugins: `Engine::restored(user)` returns a `watch` counter (>0 at start = restored), `Engine::epoch_head(user)`, `ActionCtx::journal_info()`. Live sockets end with `sync_token_expired` on rotation so no session keeps issuing new-epoch tokens to a client that has not re-read. - `--secrets DIR` / `CALTERNAL_SECRETS` on `serve`. Without it: plain tokens, no detection, one warning at start (tests and one-off tools). Deploy needs the flag set to a folder outside the backup set (`docs/backup.md` updated). - Decision: "invalid token" is `sync_token_expired` with that detail (clients already re-read on it); no new error code, so contracts do not change. - **Not in this PR:** the `topic` column/index on the save log. Nothing here needs it; it stays with the sync-topics/mail PR, so there is no migration to time. Cold-start/open cost at 1M entries is measured instead. ## #919: opt-in per-field check - `calternal_core::stale`: `Guard {base, after}` (flatten into an action input), `Tracked<V>` (value, epoch, seq, req; `set` stamps only on a real change; `rebuilt` takes the head; `with_pending` folds accepted-but-unapplied actions), `check_field` → `Apply | NoChange | Stale{epoch_only}`. - Rule: equal value → success; else stale only if a `base` is given, the field changed **after** it, and the last change's op is not in `after`. A base from an older epoch is stale with `epoch_only`; marks stamped before a rotation count as `epoch_head`. - Decision: comparison is strict (`field.seq > base.seq`). The issue text says `≥`, but a token already includes the entry at its seq, and `≥` would make a rebuilt row (seq = head) refuse the replay of a device that re-read at that very head, forever. - Device side `apps/web/src/lib/action-outbox.ts`: merges only unsent ops per (target, field); merged op keeps the oldest base; a merge back to the original is dropped only when nothing was sent; `after()`; `restamp(token)` re-stamps and re-queues every unconfirmed op after a rotation; `replay:false` ops (delete forever) are handed back, never replayed. No existing screen uses it yet (prefs keep their own outbox). ## Idempotency after a power cut (77762) Idempotency records in `index.db` are NORMAL, so an action that replied can run again after a power cut. No fsync added to the shared writer. Audit of every plugin with mutating actions: | Plugin / action | Re-run after a lost record | Why | |---|---|---| | notes.* | safe | id derived from the key; receipts in `history.db` (FULL) | | daily.open and log lines | safe | key forwarded to the Notes patch receipt | | tasks.create / set-status | safe | key becomes the Notes job id (receipts in `history.db`) | | contacts.create/update/import/delete/restore | safe | key → job rows in `history.db` (`jobs::attempt`) | | groups.create, people.invites | safe | id derived from the key, insert `ON CONFLICT DO NOTHING` in `auth.db` (FULL); the reply carries only the id | | people.role/quota, groups.rename/delete/member, sharing.grant/revoke, account.revoke, prefs.set | safe | set-to-state | | attachments.add | **follow-up, low risk** | the staged upload is consumed by the first run; a re-run answers "no upload arrived with this key" although the file is placed (bytes are content-addressed, a re-upload dedupes). Fix: answer from the placed file's hash | | search.visit | harmless | a ranking nudge | ## Tests - `store/src/tests/restore.rs`: signed tokens everywhere; garbled/hand-edited token (invalid, no rotation); other-epoch token; first deploy; backup after the last start; backup from before an earlier rotation; host snapshot rollback (found by the first newer token); reused seq with a different entry; lost log tail at open; **every file back to its last fsync, head passes the device** (77762); zero-length/torn file; ahead file; rotation durable before new tokens + power loss; two proofs rotate once; idle/max-age/shutdown flush; torn-slot fallback; both-slots-bad safe restore; key mode. - `core/src/stale/tests.rs` + `engine/src/tests/restore.rs`: flag racing read, two devices mark read, lost reply then opposite toggle, stale base racing a pending action, two tabs + retry after the prune, rebuild then late retry, rebuild then unchanged value, online adapter without base, restore → `stale_epoch` → re-stamp → applies. - `apps/web/src/lib/action-outbox.test.ts`. Gate and perf tables follow as comments. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
p0b-initial-evidence.tar.gz
33 KiB
p0b-final-evidence.tar.gz
143 KiB
kayg added 12 commits 2026-11-04 10:46:45 +00:00 
Parsing only; nothing issues signed tokens yet. Old two-part tokens
still parse.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Tracked fields, pending-aware current value, own-op exemption through
`after`, older-epoch bases. No action uses it yet; notes unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
store: test the power-cut drill (every file back to its last fsync, head passes the device)
Some checks failed
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
d88a4c4417 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
kayg commented 2026-11-05 05:46:03 +00:00
Author
Owner
Copy link

resuming test/perf qualification. dev merged and pushed in fa1253fb; current test head bb260905. no merge will be performed.

coverage audit: existing tests cover first deploy (no rotation), backup after start, backup before rotation, snapshot rollback, reused seq, lost WAL tail, garbled tokens without rotation, torn files, concurrent proofs, timer, and durable rotation. #919 tests cover independent fields, equal values, own unconfirmed ops, pending actions, two tabs/late retry, rebuild, old epoch replay, and unchanged notes merging. added tests for a missing entry below head, a same-epoch witness with changed entry, a deleted initialized witness, and failed witness rewrite.

the last added test targets an uncovered requirement: restore::rotate publishes the epoch even if highwater::write returns an error. red verification is queued through slot on dev-01. i will report its actual result. also corrected the existing outbox replay-id assertion to snapshot the old ids before restamp mutates the objects.

scope discrepancy: the approved mail plan §2/§18 includes sync topics, topic SQL index and refill collapse in P0b, but this PR explicitly defers them. LiveQuery currently accepts only since; schema has no topic column/index. a “sync with topics” perf pass cannot be claimed against this implementation. it will be reported as missing, alongside measured current catch-up costs.

workspace gates and Chromium are queued on dev-01 under Nix/slot; WebKit build/specs are queued on dev-02. the active train is respected. results and exact commands will follow.

resuming test/perf qualification. dev merged and pushed in fa1253fb; current test head bb260905. no merge will be performed. coverage audit: existing tests cover first deploy (no rotation), backup after start, backup before rotation, snapshot rollback, reused seq, lost WAL tail, garbled tokens without rotation, torn files, concurrent proofs, timer, and durable rotation. #919 tests cover independent fields, equal values, own unconfirmed ops, pending actions, two tabs/late retry, rebuild, old epoch replay, and unchanged notes merging. added tests for a missing entry below head, a same-epoch witness with changed entry, a deleted initialized witness, and failed witness rewrite. the last added test targets an uncovered requirement: restore::rotate publishes the epoch even if highwater::write returns an error. red verification is queued through slot on dev-01. i will report its actual result. also corrected the existing outbox replay-id assertion to snapshot the old ids before restamp mutates the objects. scope discrepancy: the approved mail plan §2/§18 includes sync topics, topic SQL index and refill collapse in P0b, but this PR explicitly defers them. LiveQuery currently accepts only since; schema has no topic column/index. a “sync with topics” perf pass cannot be claimed against this implementation. it will be reported as missing, alongside measured current catch-up costs. workspace gates and Chromium are queued on dev-01 under Nix/slot; WebKit build/specs are queued on dev-02. the active train is respected. results and exact commands will follow.
kayg commented 2026-11-05 05:51:56 +00:00
Author
Owner
Copy link

additional coverage gap: Engine::execute returns store.head().to_string() for read-only actions, so a non-empty snapshot/list reply still carries an unsigned token even with --secrets enabled. added read_only_action_replies_carry_signed_tokens in d117982b to require the entry proof; its first execution is queued. this matters because the snapshot token is the device’s reconnect cursor.

qualification is waiting behind the live bm14e train locks on both dev hosts. queued jobs: dev-01 p0b-final-chromium (workspace gates, store regression, outbox, signed-server Chromium); dev-02 p0b-final-webkit (signed-server WebKit); dev-01 p0b-release (candidate/dev release servers and disposable probe). all heavy commands use Nix and ~/bin/slot, four cargo jobs and one Playwright worker. no perf locks taken while builds wait.

additional coverage gap: Engine::execute returns store.head().to_string() for read-only actions, so a non-empty snapshot/list reply still carries an unsigned token even with --secrets enabled. added read_only_action_replies_carry_signed_tokens in d117982b to require the entry proof; its first execution is queued. this matters because the snapshot token is the device’s reconnect cursor. qualification is waiting behind the live bm14e train locks on both dev hosts. queued jobs: dev-01 p0b-final-chromium (workspace gates, store regression, outbox, signed-server Chromium); dev-02 p0b-final-webkit (signed-server WebKit); dev-01 p0b-release (candidate/dev release servers and disposable probe). all heavy commands use Nix and ~/bin/slot, four cargo jobs and one Playwright worker. no perf locks taken while builds wait.
kayg commented 2026-11-05 07:05:51 +00:00
Author
Owner
Copy link

red verification results on b4a45605 (format-only follow-up ac4f6f29):

  • a_failed_highwater_rewrite_does_not_publish_the_new_epoch: fails because the published epoch changes despite the witness write returning an internal error. this violates the required publish-after-durable-witness ordering.
  • a_failed_first_deploy_can_retry_without_rotating: fails because the first failed witness write leaves the highwater flag set; retry rotates an existing user. this violates first-deploy initialization without rotation.
  • read_only_action_replies_carry_signed_tokens: fails with 2 token parts instead of 4. snapshot replies omit the entry proof.
  • the three other added store cases pass. in a disposable worktree, removing each corresponding check makes its test fail at its assertion: missing entry, changed same-epoch witness entry, missing initialized witness. mutations were restored; no production mutation was committed.

fresh clippy rejects std::fs::rename in highwater.rs:54 and token.rs:44 under the repository’s disallowed-methods rule. no lint was suppressed. rustfmt’s reported changes are pushed in ac4f6f29; its rerun follows after active jobs settle.

workspace tests, browser gates and comparative builds are still underway. no merge, no perf pass claimed.

red verification results on b4a45605 (format-only follow-up ac4f6f29): - `a_failed_highwater_rewrite_does_not_publish_the_new_epoch`: fails because the published epoch changes despite the witness write returning an internal error. this violates the required publish-after-durable-witness ordering. - `a_failed_first_deploy_can_retry_without_rotating`: fails because the first failed witness write leaves the highwater flag set; retry rotates an existing user. this violates first-deploy initialization without rotation. - `read_only_action_replies_carry_signed_tokens`: fails with 2 token parts instead of 4. snapshot replies omit the entry proof. - the three other added store cases pass. in a disposable worktree, removing each corresponding check makes its test fail at its assertion: missing entry, changed same-epoch witness entry, missing initialized witness. mutations were restored; no production mutation was committed. fresh clippy rejects `std::fs::rename` in `highwater.rs:54` and `token.rs:44` under the repository’s disallowed-methods rule. no lint was suppressed. rustfmt’s reported changes are pushed in ac4f6f29; its rerun follows after active jobs settle. workspace tests, browser gates and comparative builds are still underway. no merge, no perf pass claimed.
kayg commented 2026-11-05 07:57:41 +00:00
Author
Owner
Copy link

qualification checkpoint, 13:27 ist. current head dd16a8c93a644f49df1b633bef91f491c2ff20cb; dev merged at 9377d711217532a44973edcf63c2af0b7f7969d8. never merged.

completed first-run gates on b4a45605 (before test/format-only follow-ups):

gate command, with CARGO_BUILD_JOBS=4 under Nix and ~/bin/slot result
format cargo fmt --check failed; reported formatting pushed in ac4f6f29 and 4969da0f; final-head check queued
clippy cargo clippy --workspace --all-targets -- -D warnings failed: disallowed std::fs::rename in highwater.rs and token.rs
workspace cargo test --workspace failed: unsigned read-only action token; stopped at engine
store failure branches cargo test -j4 -p calternal-store tests::restore::failures -- --nocapture 3 pass, 2 fail: witness-write publication and first-deploy retry
outbox bunx vitest run src/lib/action-outbox.test.ts pass on both dev hosts
chromium, dev-01 CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts e2e/lazy-offline.spec.ts e2e/served.spec.ts --project=chromium-1440 --workers=1 --output="$HOME/p0b-output-chromium" --reporter=line 24 pass
webkit, dev-02 same specs and wrapper, --project=webkit-1440 --workers=1 --output="$HOME/p0b-output-webkit" --reporter=line 20 pass, 3 skip, 1 timeout

webkit trace: the timeout is in the page fixture (BrowserContext.newPage), before the test body or navigation. no flake exemption yet: candidate and dev --repeat-each=10 checks are queued. no timeout or assertion was relaxed.

failure-table audit: the shared rollback row is covered by backup-after-start, backup-before-rotation, whole-host snapshot, lost WAL tail/reused seq, missing entry, missing/torn/changed/ahead witness, first deploy and failed first deploy, failed durable witness rewrite, concurrent proofs, timer, and durable rotation tests. the stale-action row is covered by independent fields, equal values, own unconfirmed op ids, accepted pending actions, two tabs/late retry, rebuild, epoch replay, and unchanged notes patch merging. added a two-session real-websocket snapshot-rollback test for the previously untested requirement that an active session must end with sync_token_expired; its disabled-notification red check and production check are queued. mail-specific provider/sender failure rows belong to later mail implementation, not these two shared helpers.

final-head runs are queued behind the live train locks: fmt, clippy, workspace --no-fail-fast, expanded save/sync browser cases (large pastes, deploy reload, offline prefs), and the new live regression. release candidate/dev servers and disposable probes have built successfully and their relayed SHA-256 hashes match. perf is unmeasured: another qualification holds the pair; p0b waits on perf-01 before acquiring perf-02.

perf coverage current state
save/ack, existing catch-up/reconnect, notes.save ABAB ×5 and dev-vs-dev control prepared; not run
startup + external high-water read at 1M entries same matrix prepared; file-level fsync/fadvise cache eviction, not a whole-machine cache drop
sync with topics / topic-index migration missing implementation: PR defers it although the approved P0b plan includes it; cannot claim coverage

blocking functional findings remain the three regressions recorded above. no performance pass and no shipping approval are claimed.

first-run raw evidence, mutation logs, webkit trace, probe sources and binary hashes. latest test-only follow-up dd16a8c9 accepts a close frame or transport closure after the required structured expiry error, while rejecting any later application message. its red/green run remains queued; no new pass is claimed.

qualification checkpoint, 13:27 ist. current head `dd16a8c93a644f49df1b633bef91f491c2ff20cb`; dev merged at `9377d711217532a44973edcf63c2af0b7f7969d8`. never merged. completed first-run gates on `b4a45605` (before test/format-only follow-ups): | gate | command, with `CARGO_BUILD_JOBS=4` under Nix and `~/bin/slot` | result | |---|---|---| | format | `cargo fmt --check` | failed; reported formatting pushed in ac4f6f29 and 4969da0f; final-head check queued | | clippy | `cargo clippy --workspace --all-targets -- -D warnings` | failed: disallowed `std::fs::rename` in highwater.rs and token.rs | | workspace | `cargo test --workspace` | failed: unsigned read-only action token; stopped at engine | | store failure branches | `cargo test -j4 -p calternal-store tests::restore::failures -- --nocapture` | 3 pass, 2 fail: witness-write publication and first-deploy retry | | outbox | `bunx vitest run src/lib/action-outbox.test.ts` | pass on both dev hosts | | chromium, dev-01 | `CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts e2e/lazy-offline.spec.ts e2e/served.spec.ts --project=chromium-1440 --workers=1 --output="$HOME/p0b-output-chromium" --reporter=line` | 24 pass | | webkit, dev-02 | same specs and wrapper, `--project=webkit-1440 --workers=1 --output="$HOME/p0b-output-webkit" --reporter=line` | 20 pass, 3 skip, 1 timeout | webkit trace: the timeout is in the page fixture (`BrowserContext.newPage`), before the test body or navigation. no flake exemption yet: candidate and dev `--repeat-each=10` checks are queued. no timeout or assertion was relaxed. failure-table audit: the shared rollback row is covered by backup-after-start, backup-before-rotation, whole-host snapshot, lost WAL tail/reused seq, missing entry, missing/torn/changed/ahead witness, first deploy and failed first deploy, failed durable witness rewrite, concurrent proofs, timer, and durable rotation tests. the stale-action row is covered by independent fields, equal values, own unconfirmed op ids, accepted pending actions, two tabs/late retry, rebuild, epoch replay, and unchanged notes patch merging. added a two-session real-websocket snapshot-rollback test for the previously untested requirement that an active session must end with `sync_token_expired`; its disabled-notification red check and production check are queued. mail-specific provider/sender failure rows belong to later mail implementation, not these two shared helpers. final-head runs are queued behind the live train locks: fmt, clippy, workspace `--no-fail-fast`, expanded save/sync browser cases (large pastes, deploy reload, offline prefs), and the new live regression. release candidate/dev servers and disposable probes have built successfully and their relayed SHA-256 hashes match. perf is unmeasured: another qualification holds the pair; p0b waits on perf-01 before acquiring perf-02. | perf coverage | current state | |---|---| | save/ack, existing catch-up/reconnect, notes.save | ABAB ×5 and dev-vs-dev control prepared; not run | | startup + external high-water read at 1M entries | same matrix prepared; file-level fsync/fadvise cache eviction, not a whole-machine cache drop | | sync with topics / topic-index migration | missing implementation: PR defers it although the approved P0b plan includes it; cannot claim coverage | blocking functional findings remain the three regressions recorded above. no performance pass and no shipping approval are claimed. [first-run raw evidence, mutation logs, webkit trace, probe sources and binary hashes](https://git.kayg.org/attachments/8b1fb6ae-524b-4aab-8c30-40e0b8636abc). latest test-only follow-up dd16a8c9 accepts a close frame or transport closure after the required structured expiry error, while rejecting any later application message. its red/green run remains queued; no new pass is claimed.
kayg commented 2026-11-05 09:09:35 +00:00
Author
Owner
Copy link

qualification complete: does not qualify to ship. functional regressions, clippy errors, a reproduced webkit failure, and perf regressions remain. no gate or budget was loosened. PR remains open and unmerged; no approved code review is recorded.

head dd16a8c93a644f49df1b633bef91f491c2ff20cb; dev baseline 9377d711217532a44973edcf63c2af0b7f7969d8. origin/dev was merged and pushed in fa1253fb. a final fetch confirms the branch is not behind dev. all follow-ups after perf build b4a45605bb92f283844b16e14fb2511ffceba8f6 change tests or formatting only; measured application code is identical to the final head. binaries were built on dev-01 and SHA-256 verified on dev-01, control and perf-01. raw logs, all samples, scripts, lock evidence, traces and cleanup records.

all heavy gates ran in Nix through ordinary ~/bin/slot, with CARGO_BUILD_JOBS=4. Rust commands ran in ~/Developer/Worktrees/p0b on dev-01; browser commands ran from apps/web on the specified host. the signed-server wrapper adds a unique --secrets path outside each fixture's data directory.

gate exact command result
fmt, final head cargo fmt --check pass
clippy, final head cargo clippy --workspace --all-targets -- -D warnings fail: direct std::fs::rename in highwater.rs:54 and token.rs:44 is disallowed
requested workspace run, b4a45605 cargo test --workspace fail at the engine snapshot-token regression
complete workspace run, final head cargo test --workspace --no-fail-fast fail: 3 cases in 2 targets; engine 27 pass / 1 fail / 1 ignored, store 28 pass / 2 fail; all other targets completed
added store branches, b4a45605 cargo test -j4 -p calternal-store tests::restore::failures -- --nocapture 3 pass, 2 fail
live-session restore, final head cargo test -j4 -p calternal-server --test restore -- --nocapture disabled-notification mutation fails at the expected missing-response assertion; production branch restored, 1 pass
web outbox, both dev hosts, b4a45605 bunx vitest run src/lib/action-outbox.test.ts 10 pass on each host; web source unchanged in later commits
chromium, dev-01, final head CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts e2e/lazy-offline.spec.ts e2e/served.spec.ts e2e/big-paste.spec.ts e2e/pwa-update.spec.ts --project=chromium-1440 --workers=1 --output="$HOME/p0b-head-output-chromium" --reporter=line 26 pass, 4 configured skips (webkit-only update tests)
webkit, dev-02, final head CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts e2e/lazy-offline.spec.ts e2e/served.spec.ts e2e/big-paste.spec.ts e2e/pwa-update.spec.ts --project=webkit-1440 --workers=1 --output="$HOME/p0b-head-output-webkit" --reporter=line 26 pass, 4 configured skips
offline prefs, dev-01 CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/prefs-offline.spec.ts --project=prefs-chromium --workers=1 --output="$HOME/p0b-head-output-prefs" --reporter=line 3 pass
offline prefs, dev-02 CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/prefs-offline.spec.ts --project=prefs-webkit --no-deps --workers=1 --output="$HOME/p0b-head-output-prefs" --reporter=line 3 pass
candidate timeout repeat, dev-02 CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=10 --output="$HOME/p0b-repeat-head" --reporter=line 9 pass, 1 fail
dev timeout repeat, first attempt CALTERNAL_BIN="$HOME/Developer/Worktrees/p0b-webkit-dev/target/debug/calternal" bunx playwright test e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=10 --output="$HOME/p0b-repeat-dev" --reporter=line fixture server startup timed out at unchanged 60 s; no tests ran
dev timeout repeat, retry DEBUG=pw:webserver CALTERNAL_BIN="$HOME/p0b-baseline-server" bunx playwright test e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=10 --output="$HOME/p0b-repeat-dev2" --reporter=line 10 pass; baseline wrapper only retains backend logs

webkit's first four-spec run was 20 pass / 3 skip / 1 timeout. both its original trace and the candidate repeat-6 trace stall at BrowserContext.newPage, before navigation or test assertions. the candidate failure reproduced, so the brief's non-reproducing-flake exemption does not apply. neither the 30 s test deadline nor the 60 s server deadline was increased. the expanded webkit skips are the existing Layout Instability API case, CDP-only IME, Chromium-only offline SW inspection, and the existing signed-out SW routing fixme.

blocking code findings:

  • restore.rs:108–116 publishes the new epoch and restored counter even when durable witness rewrite fails. a_failed_highwater_rewrite_does_not_publish_the_new_epoch fails.
  • restore.rs:61 persists the first-deploy flag before writing the witness. an unavailable witness path followed by a repaired retry rotates an existing user. a_failed_first_deploy_can_retry_without_rotating fails.
  • engine/execute.rs:70 returns an unsigned head for read-only action replies. read_only_action_replies_carry_signed_tokens fails with 2 parts instead of 4.
  • test builds also emit warnings for the unused SyncToken import in execute.rs, unused wire, and overwritten head in storage-bench.rs. clippy currently stops earlier at the two disallowed renames.

failure-table coverage: the shared rollback row has tests for backup after start, backup before an earlier rotation, whole-host snapshot, lost WAL tail/reused seq, missing entry, missing/torn/changed/ahead witness, first deploy and failed first-deploy retry, failed witness rewrite, concurrent proofs, timer, durable rotation, signed snapshot replies and expiry of both active live sessions. the shared stale-action row covers independent fields, equal values, own unconfirmed op ids, pending actions, two tabs/late retry, rebuild, older-epoch replay and unchanged notes merging. stale_patch_keeps_newer_typing passes in the complete workspace run. three added already-implemented store branches were mutation checked: removing each check makes its test fail at its assertion; final production coverage passes. the live-session test was run against disabled notifications first, then restored production. the three uncovered code defects fail on their first execution. mail-specific provider/sender rows remain later mail milestones, not tests of these shared helpers.

perf ran on perf-01 while both ~/bin/perf-lock locks were held, acquired perf-01 then perf-02. each mode runs dev/dev ABAB ×5, then dev/head ABAB ×5: 10 windows per arm, 120 completed invocations. both holders were stopped immediately after matrix completion. the full matrix began at 14:03:39 ist. ordinary commits remain NORMAL; no budget was changed.

an initial attempt completed journal/open windows but failed at room server startup: the existing perf helper created state/server.log before serve --create, so storage pairing correctly refused a nonempty state root. its 80 partial rows and diagnostic log are retained. the full matrix was restarted from the beginning with an identical wrapper for both binaries that moves the preopened log outside state; this preserves the storage gate. process-to-healthz timings include that common wrapper. the following tables use only the complete 120-invocation matrix, with every window retained.

sampling per window: 20k journal preload, 4k timed commits, 2k checked positions at head and mid-log, 300 pages of 100; store open with 1m journal entries; 200 real room edits/acks and peer convergence, 30 notes.save calls, 50 reconnects and process-to-healthz. non-startup cells are medians of 10 per-window quantiles. startup cells are nearest-rank quantiles over 10 observations, so their p95 and p99 coincide. raw samples retain all counts. existing-store open includes the external witness read/rewrite; process healthz uses a fresh lazy-open server.

positive deltas are slower. no positive result is waived as under budget. the save/ack tail is noisy relative to its control; that does not establish flatness. the other large store/open regressions and the notes.save tail prevent a perf pass.

metric candidate Δ p50 / p95 / p99 dev/dev control Δ p50 / p95 / p99
save_ack +0.46% / +0.79% / +8.58% +0.12% / -2.82% / +11.41%
sync_round_trip +2.54% / +6.33% / +3.91% +1.91% / -0.24% / -0.12%
notes_save -0.15% / -0.04% / +17.49% -0.49% / -1.98% / -6.61%
reconnect_ready +7.15% / +4.83% / +13.60% +4.60% / +2.10% / +4.01%
catch_up_page +32.53% / +30.91% / +33.34% -1.82% / -1.23% / +0.64%
commit +4.03% / +2.45% / +5.02% +0.06% / -0.53% / -1.83%
position_head +272.86% / +281.69% / +338.41% +0.00% / +0.00% / +0.00%
position_mid +157224.43% / +184316.20% / +213095.77% +0.00% / +0.00% / +0.00%
startup_1m_file_evicted +1569.13% / +1593.79% / +1593.79% +27.55% / +10.53% / +10.53%
startup_healthz +0.44% / -0.19% / -0.19% -5.27% / -25.90% / -25.90%
all absolute values and deltas (milliseconds)
metric q dev head Δ ms Δ % control A control B control Δ ms control Δ %
save_ack p50 4.429015 4.449169 +0.020154 +0.46% 4.439029 4.444219 +0.005191 +0.12%
save_ack p95 4.900253 4.939155 +0.038902 +0.79% 4.992431 4.851883 -0.140548 -2.82%
save_ack p99 7.782545 8.450226 +0.667682 +8.58% 7.857957 8.754577 +0.896620 +11.41%
sync_round_trip p50 0.203080 0.208235 +0.005155 +2.54% 0.207023 0.210975 +0.003952 +1.91%
sync_round_trip p95 0.316714 0.336771 +0.020057 +6.33% 0.330955 0.330159 -0.000796 -0.24%
sync_round_trip p99 0.371367 0.385879 +0.014512 +3.91% 0.371255 0.370826 -0.000430 -0.12%
notes_save p50 12.756626 12.737967 -0.018660 -0.15% 12.817726 12.754357 -0.063369 -0.49%
notes_save p95 17.219546 17.213294 -0.006252 -0.04% 17.468592 17.122704 -0.345888 -1.98%
notes_save p99 17.592429 20.669982 +3.077552 +17.49% 21.650390 20.219853 -1.430537 -6.61%
reconnect_ready p50 0.286973 0.307482 +0.020509 +7.15% 0.292724 0.306194 +0.013469 +4.60%
reconnect_ready p95 0.347697 0.364509 +0.016811 +4.83% 0.363672 0.371316 +0.007644 +2.10%
reconnect_ready p99 0.399429 0.453765 +0.054335 +13.60% 0.406011 0.422272 +0.016261 +4.01%
catch_up_page p50 0.176045 0.233308 +0.057262 +32.53% 0.175854 0.172653 -0.003201 -1.82%
catch_up_page p95 0.204819 0.268127 +0.063308 +30.91% 0.205270 0.202750 -0.002520 -1.23%
catch_up_page p99 0.228453 0.304611 +0.076158 +33.34% 0.231112 0.232586 +0.001474 +0.64%
commit p50 0.033949 0.035316 +0.001367 +4.03% 0.034635 0.034655 +0.000020 +0.06%
commit p95 0.050760 0.052003 +0.001243 +2.45% 0.051086 0.050815 -0.000271 -0.53%
commit p99 0.071173 0.074745 +0.003572 +5.02% 0.073148 0.071809 -0.001338 -1.83%
position_head p50 0.000070 0.000261 +0.000191 +272.86% 0.000070 0.000070 +0.000000 +0.00%
position_head p95 0.000071 0.000271 +0.000200 +281.69% 0.000071 0.000071 +0.000000 +0.00%
position_head p99 0.000076 0.000331 +0.000255 +338.41% 0.000071 0.000071 +0.000000 +0.00%
position_mid p50 0.000066 0.103048 +0.102982 +157224.43% 0.000070 0.000070 +0.000000 +0.00%
position_mid p95 0.000071 0.130935 +0.130864 +184316.20% 0.000071 0.000071 +0.000000 +0.00%
position_mid p99 0.000071 0.151369 +0.151298 +213095.77% 0.000071 0.000071 +0.000000 +0.00%
startup_1m_file_evicted p50 0.539753 9.009203 +8.469450 +1569.13% 0.581980 0.742331 +0.160351 +27.55%
startup_1m_file_evicted p95 0.737211 12.486781 +11.749570 +1593.79% 17.386093 19.217205 +1.831112 +10.53%
startup_1m_file_evicted p99 0.737211 12.486781 +11.749570 +1593.79% 17.386093 19.217205 +1.831112 +10.53%
startup_healthz p50 56.437006 56.685823 +0.248817 +0.44% 60.704554 57.508285 -3.196269 -5.27%
startup_healthz p95 80.148163 79.996067 -0.152096 -0.19% 82.425130 61.073976 -21.351154 -25.90%
startup_healthz p99 80.148163 79.996067 -0.152096 -0.19% 82.425130 61.073976 -21.351154 -25.90%

open risks and gaps:

  • sync with topics is unqualified because it is absent. the approved P0b plan includes topics, the SQL topic index and refill collapse; this PR explicitly defers them. existing room sync and unfiltered catch-up measurements do not prove topic behavior or index/migration cost.
  • whole-machine cold start is unqualified. no root cache-drop capability was available. the 1m-store result uses fsync plus per-file POSIX_FADV_DONTNEED; it includes high-water I/O and is labeled accordingly. it is not a dropped-machine-page-cache claim. startup tail estimates have only 10 observations per arm.
  • the three contract regressions, clippy failure, candidate WebKit reproduction and measured perf regressions must be resolved and requalified. the median notes.save result does not cancel its +17.49% p99 regression.
  • deployment still requires --secrets outside backups. the PR's declared attachments.add retry-after-lost-idempotency-record follow-up remains; no production deployment or settings were changed.

all owned measurement/gate jobs are finished. both perf holders are inactive. owned targets, large build outputs, perf binaries and seeded fixtures were removed; recorded browser fixture paths were cleaned or confirmed gone. cleanup records and final free-space values are attached. no PR merge was performed.

qualification complete: **does not qualify to ship**. functional regressions, clippy errors, a reproduced webkit failure, and perf regressions remain. no gate or budget was loosened. PR remains open and unmerged; no approved code review is recorded. head `dd16a8c93a644f49df1b633bef91f491c2ff20cb`; dev baseline `9377d711217532a44973edcf63c2af0b7f7969d8`. origin/dev was merged and pushed in `fa1253fb`. a final fetch confirms the branch is not behind dev. all follow-ups after perf build `b4a45605bb92f283844b16e14fb2511ffceba8f6` change tests or formatting only; measured application code is identical to the final head. binaries were built on dev-01 and SHA-256 verified on dev-01, control and perf-01. [raw logs, all samples, scripts, lock evidence, traces and cleanup records](https://git.kayg.org/attachments/1e0edd83-f843-4df7-b863-c3ac5ff75a63). all heavy gates ran in Nix through ordinary `~/bin/slot`, with `CARGO_BUILD_JOBS=4`. Rust commands ran in `~/Developer/Worktrees/p0b` on dev-01; browser commands ran from `apps/web` on the specified host. the signed-server wrapper adds a unique `--secrets` path outside each fixture's data directory. | gate | exact command | result | |---|---|---| | fmt, final head | `cargo fmt --check` | pass | | clippy, final head | `cargo clippy --workspace --all-targets -- -D warnings` | fail: direct `std::fs::rename` in highwater.rs:54 and token.rs:44 is disallowed | | requested workspace run, b4a45605 | `cargo test --workspace` | fail at the engine snapshot-token regression | | complete workspace run, final head | `cargo test --workspace --no-fail-fast` | fail: 3 cases in 2 targets; engine 27 pass / 1 fail / 1 ignored, store 28 pass / 2 fail; all other targets completed | | added store branches, b4a45605 | `cargo test -j4 -p calternal-store tests::restore::failures -- --nocapture` | 3 pass, 2 fail | | live-session restore, final head | `cargo test -j4 -p calternal-server --test restore -- --nocapture` | disabled-notification mutation fails at the expected missing-response assertion; production branch restored, 1 pass | | web outbox, both dev hosts, b4a45605 | `bunx vitest run src/lib/action-outbox.test.ts` | 10 pass on each host; web source unchanged in later commits | | chromium, dev-01, final head | `CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts e2e/lazy-offline.spec.ts e2e/served.spec.ts e2e/big-paste.spec.ts e2e/pwa-update.spec.ts --project=chromium-1440 --workers=1 --output="$HOME/p0b-head-output-chromium" --reporter=line` | 26 pass, 4 configured skips (webkit-only update tests) | | webkit, dev-02, final head | `CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts e2e/lazy-offline.spec.ts e2e/served.spec.ts e2e/big-paste.spec.ts e2e/pwa-update.spec.ts --project=webkit-1440 --workers=1 --output="$HOME/p0b-head-output-webkit" --reporter=line` | 26 pass, 4 configured skips | | offline prefs, dev-01 | `CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/prefs-offline.spec.ts --project=prefs-chromium --workers=1 --output="$HOME/p0b-head-output-prefs" --reporter=line` | 3 pass | | offline prefs, dev-02 | `CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/prefs-offline.spec.ts --project=prefs-webkit --no-deps --workers=1 --output="$HOME/p0b-head-output-prefs" --reporter=line` | 3 pass | | candidate timeout repeat, dev-02 | `CALTERNAL_BIN="$HOME/p0b-signed-server" bunx playwright test e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=10 --output="$HOME/p0b-repeat-head" --reporter=line` | **9 pass, 1 fail** | | dev timeout repeat, first attempt | `CALTERNAL_BIN="$HOME/Developer/Worktrees/p0b-webkit-dev/target/debug/calternal" bunx playwright test e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=10 --output="$HOME/p0b-repeat-dev" --reporter=line` | fixture server startup timed out at unchanged 60 s; no tests ran | | dev timeout repeat, retry | `DEBUG=pw:webserver CALTERNAL_BIN="$HOME/p0b-baseline-server" bunx playwright test e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=10 --output="$HOME/p0b-repeat-dev2" --reporter=line` | 10 pass; baseline wrapper only retains backend logs | webkit's first four-spec run was 20 pass / 3 skip / 1 timeout. both its original trace and the candidate repeat-6 trace stall at `BrowserContext.newPage`, before navigation or test assertions. the candidate failure **reproduced**, so the brief's non-reproducing-flake exemption does not apply. neither the 30 s test deadline nor the 60 s server deadline was increased. the expanded webkit skips are the existing Layout Instability API case, CDP-only IME, Chromium-only offline SW inspection, and the existing signed-out SW routing fixme. blocking code findings: - `restore.rs:108–116` publishes the new epoch and restored counter even when durable witness rewrite fails. `a_failed_highwater_rewrite_does_not_publish_the_new_epoch` fails. - `restore.rs:61` persists the first-deploy flag before writing the witness. an unavailable witness path followed by a repaired retry rotates an existing user. `a_failed_first_deploy_can_retry_without_rotating` fails. - `engine/execute.rs:70` returns an unsigned head for read-only action replies. `read_only_action_replies_carry_signed_tokens` fails with 2 parts instead of 4. - test builds also emit warnings for the unused `SyncToken` import in execute.rs, unused `wire`, and overwritten `head` in storage-bench.rs. clippy currently stops earlier at the two disallowed renames. failure-table coverage: the shared rollback row has tests for backup after start, backup before an earlier rotation, whole-host snapshot, lost WAL tail/reused seq, missing entry, missing/torn/changed/ahead witness, first deploy and failed first-deploy retry, failed witness rewrite, concurrent proofs, timer, durable rotation, signed snapshot replies and expiry of both active live sessions. the shared stale-action row covers independent fields, equal values, own unconfirmed op ids, pending actions, two tabs/late retry, rebuild, older-epoch replay and unchanged notes merging. `stale_patch_keeps_newer_typing` passes in the complete workspace run. three added already-implemented store branches were mutation checked: removing each check makes its test fail at its assertion; final production coverage passes. the live-session test was run against disabled notifications first, then restored production. the three uncovered code defects fail on their first execution. mail-specific provider/sender rows remain later mail milestones, not tests of these shared helpers. perf ran on perf-01 while **both** `~/bin/perf-lock` locks were held, acquired perf-01 then perf-02. each mode runs dev/dev ABAB ×5, then dev/head ABAB ×5: 10 windows per arm, **120 completed invocations**. both holders were stopped immediately after matrix completion. the full matrix began at 14:03:39 ist. ordinary commits remain NORMAL; no budget was changed. an initial attempt completed journal/open windows but failed at room server startup: the existing perf helper created `state/server.log` before `serve --create`, so storage pairing correctly refused a nonempty state root. its 80 partial rows and diagnostic log are retained. the full matrix was restarted from the beginning with an identical wrapper for both binaries that moves the preopened log outside state; this preserves the storage gate. process-to-healthz timings include that common wrapper. the following tables use only the complete 120-invocation matrix, with every window retained. sampling per window: 20k journal preload, 4k timed commits, 2k checked positions at head and mid-log, 300 pages of 100; store open with 1m journal entries; 200 real room edits/acks and peer convergence, 30 `notes.save` calls, 50 reconnects and process-to-healthz. non-startup cells are medians of 10 per-window quantiles. startup cells are nearest-rank quantiles over 10 observations, so their p95 and p99 coincide. raw samples retain all counts. existing-store open includes the external witness read/rewrite; process healthz uses a fresh lazy-open server. positive deltas are slower. no positive result is waived as under budget. the save/ack tail is noisy relative to its control; that does not establish flatness. the other large store/open regressions and the notes.save tail prevent a perf pass. | metric | candidate Δ p50 / p95 / p99 | dev/dev control Δ p50 / p95 / p99 | |---|---|---| | save_ack | +0.46% / +0.79% / +8.58% | +0.12% / -2.82% / +11.41% | | sync_round_trip | +2.54% / +6.33% / +3.91% | +1.91% / -0.24% / -0.12% | | notes_save | -0.15% / -0.04% / +17.49% | -0.49% / -1.98% / -6.61% | | reconnect_ready | +7.15% / +4.83% / +13.60% | +4.60% / +2.10% / +4.01% | | catch_up_page | +32.53% / +30.91% / +33.34% | -1.82% / -1.23% / +0.64% | | commit | +4.03% / +2.45% / +5.02% | +0.06% / -0.53% / -1.83% | | position_head | +272.86% / +281.69% / +338.41% | +0.00% / +0.00% / +0.00% | | position_mid | +157224.43% / +184316.20% / +213095.77% | +0.00% / +0.00% / +0.00% | | startup_1m_file_evicted | +1569.13% / +1593.79% / +1593.79% | +27.55% / +10.53% / +10.53% | | startup_healthz | +0.44% / -0.19% / -0.19% | -5.27% / -25.90% / -25.90% | <details> <summary>all absolute values and deltas (milliseconds)</summary> | metric | q | dev | head | Δ ms | Δ % | control A | control B | control Δ ms | control Δ % | |---|---|---|---|---|---|---|---|---|---| | save_ack | p50 | 4.429015 | 4.449169 | +0.020154 | +0.46% | 4.439029 | 4.444219 | +0.005191 | +0.12% | | save_ack | p95 | 4.900253 | 4.939155 | +0.038902 | +0.79% | 4.992431 | 4.851883 | -0.140548 | -2.82% | | save_ack | p99 | 7.782545 | 8.450226 | +0.667682 | +8.58% | 7.857957 | 8.754577 | +0.896620 | +11.41% | | sync_round_trip | p50 | 0.203080 | 0.208235 | +0.005155 | +2.54% | 0.207023 | 0.210975 | +0.003952 | +1.91% | | sync_round_trip | p95 | 0.316714 | 0.336771 | +0.020057 | +6.33% | 0.330955 | 0.330159 | -0.000796 | -0.24% | | sync_round_trip | p99 | 0.371367 | 0.385879 | +0.014512 | +3.91% | 0.371255 | 0.370826 | -0.000430 | -0.12% | | notes_save | p50 | 12.756626 | 12.737967 | -0.018660 | -0.15% | 12.817726 | 12.754357 | -0.063369 | -0.49% | | notes_save | p95 | 17.219546 | 17.213294 | -0.006252 | -0.04% | 17.468592 | 17.122704 | -0.345888 | -1.98% | | notes_save | p99 | 17.592429 | 20.669982 | +3.077552 | +17.49% | 21.650390 | 20.219853 | -1.430537 | -6.61% | | reconnect_ready | p50 | 0.286973 | 0.307482 | +0.020509 | +7.15% | 0.292724 | 0.306194 | +0.013469 | +4.60% | | reconnect_ready | p95 | 0.347697 | 0.364509 | +0.016811 | +4.83% | 0.363672 | 0.371316 | +0.007644 | +2.10% | | reconnect_ready | p99 | 0.399429 | 0.453765 | +0.054335 | +13.60% | 0.406011 | 0.422272 | +0.016261 | +4.01% | | catch_up_page | p50 | 0.176045 | 0.233308 | +0.057262 | +32.53% | 0.175854 | 0.172653 | -0.003201 | -1.82% | | catch_up_page | p95 | 0.204819 | 0.268127 | +0.063308 | +30.91% | 0.205270 | 0.202750 | -0.002520 | -1.23% | | catch_up_page | p99 | 0.228453 | 0.304611 | +0.076158 | +33.34% | 0.231112 | 0.232586 | +0.001474 | +0.64% | | commit | p50 | 0.033949 | 0.035316 | +0.001367 | +4.03% | 0.034635 | 0.034655 | +0.000020 | +0.06% | | commit | p95 | 0.050760 | 0.052003 | +0.001243 | +2.45% | 0.051086 | 0.050815 | -0.000271 | -0.53% | | commit | p99 | 0.071173 | 0.074745 | +0.003572 | +5.02% | 0.073148 | 0.071809 | -0.001338 | -1.83% | | position_head | p50 | 0.000070 | 0.000261 | +0.000191 | +272.86% | 0.000070 | 0.000070 | +0.000000 | +0.00% | | position_head | p95 | 0.000071 | 0.000271 | +0.000200 | +281.69% | 0.000071 | 0.000071 | +0.000000 | +0.00% | | position_head | p99 | 0.000076 | 0.000331 | +0.000255 | +338.41% | 0.000071 | 0.000071 | +0.000000 | +0.00% | | position_mid | p50 | 0.000066 | 0.103048 | +0.102982 | +157224.43% | 0.000070 | 0.000070 | +0.000000 | +0.00% | | position_mid | p95 | 0.000071 | 0.130935 | +0.130864 | +184316.20% | 0.000071 | 0.000071 | +0.000000 | +0.00% | | position_mid | p99 | 0.000071 | 0.151369 | +0.151298 | +213095.77% | 0.000071 | 0.000071 | +0.000000 | +0.00% | | startup_1m_file_evicted | p50 | 0.539753 | 9.009203 | +8.469450 | +1569.13% | 0.581980 | 0.742331 | +0.160351 | +27.55% | | startup_1m_file_evicted | p95 | 0.737211 | 12.486781 | +11.749570 | +1593.79% | 17.386093 | 19.217205 | +1.831112 | +10.53% | | startup_1m_file_evicted | p99 | 0.737211 | 12.486781 | +11.749570 | +1593.79% | 17.386093 | 19.217205 | +1.831112 | +10.53% | | startup_healthz | p50 | 56.437006 | 56.685823 | +0.248817 | +0.44% | 60.704554 | 57.508285 | -3.196269 | -5.27% | | startup_healthz | p95 | 80.148163 | 79.996067 | -0.152096 | -0.19% | 82.425130 | 61.073976 | -21.351154 | -25.90% | | startup_healthz | p99 | 80.148163 | 79.996067 | -0.152096 | -0.19% | 82.425130 | 61.073976 | -21.351154 | -25.90% | </details> open risks and gaps: - **sync with topics is unqualified because it is absent.** the approved P0b plan includes topics, the SQL topic index and refill collapse; this PR explicitly defers them. existing room sync and unfiltered catch-up measurements do not prove topic behavior or index/migration cost. - **whole-machine cold start is unqualified.** no root cache-drop capability was available. the 1m-store result uses fsync plus per-file `POSIX_FADV_DONTNEED`; it includes high-water I/O and is labeled accordingly. it is not a dropped-machine-page-cache claim. startup tail estimates have only 10 observations per arm. - the three contract regressions, clippy failure, candidate WebKit reproduction and measured perf regressions must be resolved and requalified. the median notes.save result does not cancel its +17.49% p99 regression. - deployment still requires `--secrets` outside backups. the PR's declared `attachments.add` retry-after-lost-idempotency-record follow-up remains; no production deployment or settings were changed. all owned measurement/gate jobs are finished. both perf holders are inactive. owned targets, large build outputs, perf binaries and seeded fixtures were removed; recorded browser fixture paths were cleaned or confirmed gone. cleanup records and final free-space values are attached. no PR merge was performed.
kayg commented 2026-11-05 09:11:17 +00:00
Author
Owner
Copy link

Orchestrator: #928 is RED, so it doesn't ship as-is.

Blockers:

  1. Correctness. Three tests the new work added now fail:

    • the epoch is published after a witness-write failure;
    • a first-deploy retry rotates the epoch;
    • read-only reply tokens are unsigned.

  2. Clippy. Two disallowed filesystem renames. Use the fsio helper.

  3. Perf. Two paths are orders of magnitude slower than dev:

    • position mid-log +157,000 %, at head +270 %;
    • 1M-entry store open +1,570 %.

    That can only come from scanning the log. The position check and the startup check must be O(1), or a single bounded read of the high-water file, never a walk of the log. Catch-up page (+31 %) and reconnect (+5–14 %) also move.

  4. WebKit. The timeout reproduces at 1/10 on the candidate and 0/10 on dev.

Next: one fix round on Codex, targeting O(1) checks and the three correctness bugs. If that round still finds new problems, the rethink rule applies: write a simpler design here first.

Orchestrator: **#928 is RED**, so it doesn't ship as-is. **Blockers:** 1. **Correctness.** Three tests the new work added now fail: - the epoch is published after a witness-write failure; - a first-deploy retry rotates the epoch; - read-only reply tokens are unsigned. 2. **Clippy.** Two disallowed filesystem renames. Use the fsio helper. 3. **Perf.** Two paths are orders of magnitude slower than dev: - position mid-log +157,000 %, at head +270 %; - 1M-entry store open +1,570 %. That can only come from scanning the log. The position check and the startup check must be O(1), or a single bounded read of the high-water file, never a walk of the log. Catch-up page (+31 %) and reconnect (+5–14 %) also move. 4. **WebKit.** The timeout reproduces at 1/10 on the candidate and 0/10 on dev. **Next:** one fix round on Codex, targeting O(1) checks and the three correctness bugs. If that round still finds new problems, the rethink rule applies: write a simpler design here first.
kayg commented 2026-11-06 04:14:38 +00:00
Author
Owner
Copy link

Status update:

  • WebKit hides markdown syntax investigation: pending: std VMs paused.
  • Perf requalification: start a fresh CPU-pinned ABAB ×5 session with both shared locks and a dev-vs-dev control after pinning is complete. Pre-pin results will stay separate.
Status update: - WebKit hides markdown syntax investigation: pending: std VMs paused. - Perf requalification: start a fresh CPU-pinned ABAB ×5 session with both shared locks and a dev-vs-dev control after pinning is complete. Pre-pin results will stay separate.
kayg commented 2026-11-06 06:12:31 +00:00
Author
Owner
Copy link

Qualification at head 2127c26fbd

Gates

Rust gates ran on dev-01 in the PR worktree via ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash /home/claude/p928-rust-gates-inner-2127.sh with CARGO_BUILD_JOBS=4.

Gate Exact command Result
Formatting cargo fmt --check PASS
Clippy cargo clippy --workspace --all-targets -- -D warnings PASS
Workspace tests cargo test --workspace --no-fail-fast PASS
Chromium save/sync at 1440 and 390 cd apps/web && CALTERNAL_BIN=/home/claude/p928-2127-save-sync-signed bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1 --output=/home/claude/p928-2127-save-sync-chromium --reporter=line PASS: 36 passed, 8 skipped
WebKit save/sync at 1440 and 390 e2e/editor.spec.ts e2e/list-catch-up.spec.ts, projects webkit-1440 and webkit-390 pending: std VMs paused
WebKit hides markdown syntax, candidate and dev, --repeat-each=20 Not run pending: std VMs paused

Post-pin perf session

Fresh CPU-pinned ABAB x5 session, with both shared perf locks acquired in order (perf-01 then perf-02) and held for the whole session. It completed 120 measurements including a dev-vs-dev control; both locks were released afterward. The corrected room setup used candidate secrets and 200 save samples per room run. The earlier exploratory room run with invalid setup was excluded.

Positive percentages mean slower. Non-startup cells are medians of 10 per-window quantiles. Startup cells use 10 observations; p50 is the median and p95/p99 are nearest-rank (the maximum here). Values are candidate vs dev and dev-vs-dev control, respectively.

Metric Candidate vs dev, p50 / p95 / p99 Dev-vs-dev control, p50 / p95 / p99
catch_up_page +3.13% / +1.68% / +4.48% -0.12% / -2.19% / -0.75%
commit -20.32% / -22.49% / -21.78% +0.06% / +1.58% / +0.85%
notes_save -0.17% / +2.82% / -3.50% -0.06% / -3.19% / -5.55%
position_head -42.86% / -42.25% / -39.74% 0.00% / 0.00% / 0.00%
position_mid -42.86% / -45.70% / -46.05% 0.00% / -11.25% / -21.11%
reconnect_ready -4.63% / -6.26% / -9.16% +3.84% / +2.95% / +2.25%
save_ack -0.02% / +13.79% / +8.00% -0.66% / -11.21% / -20.97%
startup_1m_file_evicted -0.31% / -93.14% / -93.14% +5.02% / -0.20% / -0.20%
startup_healthz -8.10% / +52.56% / +52.56% -0.35% / -1.89% / -1.89%
sync_round_trip -0.17% / -0.42% / +8.08% +0.70% / -0.90% / +3.46%

Perf is not a pass under the flat-or-better gate: catch_up_page is positive at all three quantiles and notes_save p95 is +2.82%. The complete table also shows positive tails for save_ack, startup_healthz, and sync_round_trip; they are not rounded away or waived. No merge performed.

## Qualification at head 2127c26fbddea64fdbbde2a376802612351ab160 ### Gates Rust gates ran on dev-01 in the PR worktree via `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash /home/claude/p928-rust-gates-inner-2127.sh` with `CARGO_BUILD_JOBS=4`. | Gate | Exact command | Result | |---|---|---| | Formatting | `cargo fmt --check` | PASS | | Clippy | `cargo clippy --workspace --all-targets -- -D warnings` | PASS | | Workspace tests | `cargo test --workspace --no-fail-fast` | PASS | | Chromium save/sync at 1440 and 390 | `cd apps/web && CALTERNAL_BIN=/home/claude/p928-2127-save-sync-signed bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1 --output=/home/claude/p928-2127-save-sync-chromium --reporter=line` | PASS: 36 passed, 8 skipped | | WebKit save/sync at 1440 and 390 | `e2e/editor.spec.ts e2e/list-catch-up.spec.ts`, projects `webkit-1440` and `webkit-390` | pending: std VMs paused | | WebKit `hides markdown syntax`, candidate and dev, `--repeat-each=20` | Not run | pending: std VMs paused | ### Post-pin perf session Fresh CPU-pinned ABAB x5 session, with both shared perf locks acquired in order (perf-01 then perf-02) and held for the whole session. It completed 120 measurements including a dev-vs-dev control; both locks were released afterward. The corrected room setup used candidate secrets and 200 save samples per room run. The earlier exploratory room run with invalid setup was excluded. Positive percentages mean slower. Non-startup cells are medians of 10 per-window quantiles. Startup cells use 10 observations; p50 is the median and p95/p99 are nearest-rank (the maximum here). Values are candidate vs dev and dev-vs-dev control, respectively. | Metric | Candidate vs dev, p50 / p95 / p99 | Dev-vs-dev control, p50 / p95 / p99 | |---|---|---| | `catch_up_page` | +3.13% / +1.68% / +4.48% | -0.12% / -2.19% / -0.75% | | `commit` | -20.32% / -22.49% / -21.78% | +0.06% / +1.58% / +0.85% | | `notes_save` | -0.17% / +2.82% / -3.50% | -0.06% / -3.19% / -5.55% | | `position_head` | -42.86% / -42.25% / -39.74% | 0.00% / 0.00% / 0.00% | | `position_mid` | -42.86% / -45.70% / -46.05% | 0.00% / -11.25% / -21.11% | | `reconnect_ready` | -4.63% / -6.26% / -9.16% | +3.84% / +2.95% / +2.25% | | `save_ack` | -0.02% / +13.79% / +8.00% | -0.66% / -11.21% / -20.97% | | `startup_1m_file_evicted` | -0.31% / -93.14% / -93.14% | +5.02% / -0.20% / -0.20% | | `startup_healthz` | -8.10% / +52.56% / +52.56% | -0.35% / -1.89% / -1.89% | | `sync_round_trip` | -0.17% / -0.42% / +8.08% | +0.70% / -0.90% / +3.46% | Perf is not a pass under the flat-or-better gate: `catch_up_page` is positive at all three quantiles and `notes_save` p95 is +2.82%. The complete table also shows positive tails for `save_ack`, `startup_healthz`, and `sync_round_trip`; they are not rounded away or waived. No merge performed.
kayg commented 2026-11-06 08:24:39 +00:00
Author
Owner
Copy link

Current-head gates at 95c42ddbe1af2e3228f812a4a0e449cbd1b7d6f3

Gate Exact command Result
Formatting cargo fmt --check PASS
Clippy cargo clippy --workspace --all-targets -- -D warnings PASS
Workspace tests cargo test --workspace --no-fail-fast PASS; command exit 0, doctests included
Chromium save/sync at 1440 and 390 CALTERNAL_BIN="$HOME/p928-95c42-signed-server" P928_SECRETS_ROOT="$HOME/p928-95c42-web-secrets.o1gNyn" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1 --output="$HOME/p928-95c42-save-sync-chromium" --reporter=line PASS: 36 passed, 8 skipped
WebKit save/sync at 1440 and 390 e2e/editor.spec.ts e2e/list-catch-up.spec.ts, projects webkit-1440 and webkit-390 pending: std VMs paused
hides markdown syntax, candidate and dev, repeat 20 e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=20 pending: std VMs paused
Perf ABAB ×5 and dev-vs-dev control all paths from comment 79701 pending: maintainer holds both perf locks while CPU pinning; start a fresh session after release

Rust gates ran on dev-01 in the PR worktree through ~/bin/slot and Nix with CARGO_BUILD_JOBS=4. The first setup attempt hit the full /tmp inode table before cargo started; the successful retry used a private TMPDIR on /.

The earlier perf matrix at 2127c26 predates this head and stays separate. No pre-pin samples are included in the new post-pin session.

## Current-head gates at `95c42ddbe1af2e3228f812a4a0e449cbd1b7d6f3` | Gate | Exact command | Result | |---|---|---| | Formatting | `cargo fmt --check` | PASS | | Clippy | `cargo clippy --workspace --all-targets -- -D warnings` | PASS | | Workspace tests | `cargo test --workspace --no-fail-fast` | PASS; command exit 0, doctests included | | Chromium save/sync at 1440 and 390 | `CALTERNAL_BIN="$HOME/p928-95c42-signed-server" P928_SECRETS_ROOT="$HOME/p928-95c42-web-secrets.o1gNyn" bunx playwright test e2e/editor.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1 --output="$HOME/p928-95c42-save-sync-chromium" --reporter=line` | PASS: 36 passed, 8 skipped | | WebKit save/sync at 1440 and 390 | `e2e/editor.spec.ts e2e/list-catch-up.spec.ts`, projects `webkit-1440` and `webkit-390` | pending: std VMs paused | | `hides markdown syntax`, candidate and dev, repeat 20 | `e2e/editor.spec.ts --grep 'hides markdown syntax' --project=webkit-1440 --workers=1 --repeat-each=20` | pending: std VMs paused | | Perf ABAB ×5 and dev-vs-dev control | all paths from comment 79701 | pending: maintainer holds both perf locks while CPU pinning; start a fresh session after release | Rust gates ran on dev-01 in the PR worktree through `~/bin/slot` and Nix with `CARGO_BUILD_JOBS=4`. The first setup attempt hit the full `/tmp` inode table before cargo started; the successful retry used a private `TMPDIR` on `/`. The earlier perf matrix at `2127c26` predates this head and stays separate. No pre-pin samples are included in the new post-pin session.
kayg commented 2026-11-06 09:20:46 +00:00
Author
Owner
Copy link

Post-pin perf session at 95c42ddbe1af2e3228f812a4a0e449cbd1b7d6f3

Candidate: 95c42ddbe1af2e3228f812a4a0e449cbd1b7d6f3; dev baseline: 7a9bd54eb8f385364c712198af45b31d706493fd.

The run started after the perf VMs returned from the CPU pin reboot. The shared lock was acquired on perf-01, then perf-02, and both were held through the ABAB ×5 and the same-session dev-vs-dev control. Both were released after completion. The result file contains 60 ABAB rows and 60 control rows; each arm has 10 window observations per metric. Positive deltas mean slower. Non-startup cells are medians of 10 per-window p50/p95/p99 values. Startup uses 10 observations with median p50 and nearest-rank p95/p99.

Metric Candidate vs dev p50 / p95 / p99 Dev-vs-dev control p50 / p95 / p99
catch_up_page +5.41% / +9.67% / +12.85% -0.08% / +7.99% / +7.00%
commit -20.93% / -19.40% / -13.60% +0.84% / +3.04% / -1.45%
position_head -42.86% / -42.25% / -42.25% +6.87% / 0.00% / 0.00%
position_mid -42.86% / -42.25% / -42.25% +6.87% / 0.00% / 0.00%
startup_1m_file_evicted +0.55% / -96.18% / -96.18% -2.00% / -7.97% / -7.97%
notes_save -0.02% / +0.74% / -4.31% -0.63% / -1.08% / -5.07%
reconnect_ready +0.14% / +0.67% / +6.69% +0.45% / +5.57% / -5.94%
save_ack +0.80% / +4.69% / +3.19% -0.01% / +5.09% / +8.13%
startup_healthz -9.46% / -9.46% / -9.46% +3.02% / +3.02% / +3.02%
sync_round_trip +0.62% / -3.98% / -3.69% -0.56% / -3.70% / -4.51%

This does not qualify yet: catch_up_page is slower at all three quantiles, reconnect_ready p99 is slower, and startup p50 and notes_save p95 are slightly slower. notes_save p99 is 4.31% faster. The control itself shows catch-up tail drift, so I am tracing the hot-page path before the next session. This result is separate from the earlier 2127c26 matrix.

Exact measurement command on perf-01:
systemd-run --user --unit=p928-95c42-postpin-run-1 --collect -p RuntimeMaxSec=7200 /bin/bash /home/claude/p928-perf-95c42-postpin-20261106-run1/run-all.sh

Lock commands, run in order:
systemd-run --user --unit=p928-95c42-postpin-lock-01 --collect -p RuntimeMaxSec=3600 /home/claude/bin/perf-lock sleep 3600
systemd-run --user --unit=p928-95c42-postpin-lock-02 --collect -p RuntimeMaxSec=3600 /home/claude/bin/perf-lock sleep 3600

## Post-pin perf session at `95c42ddbe1af2e3228f812a4a0e449cbd1b7d6f3` Candidate: `95c42ddbe1af2e3228f812a4a0e449cbd1b7d6f3`; dev baseline: `7a9bd54eb8f385364c712198af45b31d706493fd`. The run started after the perf VMs returned from the CPU pin reboot. The shared lock was acquired on perf-01, then perf-02, and both were held through the ABAB ×5 and the same-session dev-vs-dev control. Both were released after completion. The result file contains 60 ABAB rows and 60 control rows; each arm has 10 window observations per metric. Positive deltas mean slower. Non-startup cells are medians of 10 per-window p50/p95/p99 values. Startup uses 10 observations with median p50 and nearest-rank p95/p99. | Metric | Candidate vs dev p50 / p95 / p99 | Dev-vs-dev control p50 / p95 / p99 | |---|---:|---:| | `catch_up_page` | +5.41% / +9.67% / +12.85% | -0.08% / +7.99% / +7.00% | | `commit` | -20.93% / -19.40% / -13.60% | +0.84% / +3.04% / -1.45% | | `position_head` | -42.86% / -42.25% / -42.25% | +6.87% / 0.00% / 0.00% | | `position_mid` | -42.86% / -42.25% / -42.25% | +6.87% / 0.00% / 0.00% | | `startup_1m_file_evicted` | +0.55% / -96.18% / -96.18% | -2.00% / -7.97% / -7.97% | | `notes_save` | -0.02% / +0.74% / -4.31% | -0.63% / -1.08% / -5.07% | | `reconnect_ready` | +0.14% / +0.67% / +6.69% | +0.45% / +5.57% / -5.94% | | `save_ack` | +0.80% / +4.69% / +3.19% | -0.01% / +5.09% / +8.13% | | `startup_healthz` | -9.46% / -9.46% / -9.46% | +3.02% / +3.02% / +3.02% | | `sync_round_trip` | +0.62% / -3.98% / -3.69% | -0.56% / -3.70% / -4.51% | This does not qualify yet: `catch_up_page` is slower at all three quantiles, `reconnect_ready` p99 is slower, and startup p50 and `notes_save` p95 are slightly slower. `notes_save` p99 is 4.31% faster. The control itself shows catch-up tail drift, so I am tracing the hot-page path before the next session. This result is separate from the earlier `2127c26` matrix. Exact measurement command on perf-01: `systemd-run --user --unit=p928-95c42-postpin-run-1 --collect -p RuntimeMaxSec=7200 /bin/bash /home/claude/p928-perf-95c42-postpin-20261106-run1/run-all.sh` Lock commands, run in order: `systemd-run --user --unit=p928-95c42-postpin-lock-01 --collect -p RuntimeMaxSec=3600 /home/claude/bin/perf-lock sleep 3600` `systemd-run --user --unit=p928-95c42-postpin-lock-02 --collect -p RuntimeMaxSec=3600 /home/claude/bin/perf-lock sleep 3600`
kayg commented 2026-11-06 11:08:43 +00:00
Author
Owner
Copy link

Qualification status

  • WebKit save/sync and the hides markdown syntax investigation: pending: std VMs paused. No jobs will start on dev-02 or the Mac while paused.
  • Perf requalification: the maintainer is applying CPU pinning while holding both shared perf locks. Wait for maintenance to finish, then run a fresh ABAB ×5 against current dev with a same-session dev-vs-dev control. Keep all earlier sessions separate; none of their samples will be reused.
  • No merge.
## Qualification status - WebKit save/sync and the `hides markdown syntax` investigation: **pending: std VMs paused**. No jobs will start on dev-02 or the Mac while paused. - Perf requalification: the maintainer is applying CPU pinning while holding both shared perf locks. Wait for maintenance to finish, then run a fresh ABAB ×5 against current dev with a same-session dev-vs-dev control. Keep all earlier sessions separate; none of their samples will be reused. - No merge.
kayg commented 2026-11-07 01:21:55 +00:00
Author
Owner
Copy link

P1b is now open at #995 with its design at the top. It will store the cursor with search documents, validate it through Engine::position, and page catch-up through Engine::journal. P1b does not add a token format or read/copy the high-water value; your P0b implementation remains the sole owner of signed proofs and restore detection. An expired cursor triggers a paged full reconcile. The implementation will rebase onto #928 if it merges first and use the merged engine contract.

P1b is now open at https://git.kayg.org/kayg/calternal/pulls/995 with its design at the top. It will store the cursor with search documents, validate it through `Engine::position`, and page catch-up through `Engine::journal`. P1b does not add a token format or read/copy the high-water value; your P0b implementation remains the sole owner of signed proofs and restore detection. An expired cursor triggers a paged full reconcile. The implementation will rebase onto #928 if it merges first and use the merged engine contract.
kayg commented 2026-11-07 03:57:57 +00:00
Author
Owner
Copy link

Qualification at head 5f7b057f2527d9c09f7aab1ff50bf4d7f07bc0f5

Rust and save/sync browser gates pass. Performance qualification does not pass. Positive percentages mean slower; non-startup rows are medians of 10 per-window p50/p95/p99 values. Startup uses 10 observations (median p50, nearest-rank p95/p99).

Rust and browser gates

Rust ran on dev-01 in the candidate worktree with CARGO_BUILD_JOBS=4, through ~/bin/slot and Nix.

Gate Exact command Result
Format CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo fmt --check' PASS
Clippy CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo clippy --workspace --all-targets -- -D warnings' PASS
Workspace tests CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo test --workspace --no-fail-fast' PASS

Save/sync browser specs were e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts, one worker:

Browser/project Exact command Result
Chromium 1440 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/p928-final-3c824/apps/web && CALTERNAL_BIN=/home/claude/Developer/Worktrees/p928-final-3c824/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --workers=1' PASS: 26 passed
Chromium 390 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/p928-final-3c824/apps/web && CALTERNAL_BIN=/home/claude/Developer/Worktrees/p928-final-3c824/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=chromium-390 --workers=1' PASS: 18 passed, 8 configured skips
WebKit 1440 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-webkit-candidate/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-webkit-candidate/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=webkit-1440 --workers=1' PASS: 22 passed, 4 configured skips
WebKit 390 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-webkit-candidate/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-webkit-candidate/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=webkit-390 --workers=1' PASS: 14 passed, 12 configured skips

The WebKit hides markdown syntax case was repeated on quiet dev-02 with --repeat-each=20. Candidate command: DEBUG=pw:webserver CALTERNAL_BIN=/home/cua/p928-codex-5f7-webkit-candidate-signed bunx playwright test e2e/editor.spec.ts --grep "hides markdown syntax" --project=webkit-1440 --workers=1 --repeat-each=20 --output=/home/cua/p928-quiet-repeat-candidate-5f7 --reporter=line. Dev used the same command with CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-dev-current-20261106/target/debug/calternal and output /home/cua/p928-quiet-repeat-dev-935fd-5f7. Both passed 20/20; the initial candidate-only failure did not reproduce. Tracking issue: #996.

Full post-pin perf matrix

Candidate 5f7b057f vs dev 935fdcacd9cf2837d313817d6786d8a810b3bdaa; perf epoch 2026-11-06 14:16 IST. Both shared locks were held throughout, perf-01 then perf-02, and the same-session dev-vs-dev control ran before the candidate matrix. The 120 cells finished at 2026-11-07 00:16 UTC. Measurement wrapper: /bin/bash /home/claude/p928-perf-5f7-20261106-run1/run-matrix.sh, which runs /home/claude/bin/slot /home/claude/.nix-profile/bin/python3 run.py.

Metric Candidate vs dev p50 / p95 / p99 Dev-vs-dev control p50 / p95 / p99
catch_up_page +4.63% / -1.37% / -0.25% +2.62% / +2.04% / +2.95%
commit -21.55% / -23.59% / -13.09% +1.19% / +5.66% / +4.24%
position_head -42.86% / -42.25% / -42.25% 0.00% / 0.00% / 0.00%
position_mid -42.86% / -42.25% / -42.25% 0.00% / +7.04% / +13.38%
startup_1m_file_evicted -39.88% / -95.72% / -95.72% -6.03% / +11.51% / +11.51%
notes_save -0.51% / +8.90% / +17.02% -0.44% / -1.10% / -0.62%
reconnect_ready +1.23% / -7.68% / -21.05% +2.03% / +6.82% / +0.37%
save_ack +0.32% / +1.17% / -2.72% -0.56% / -3.71% / -11.93%
startup_healthz +0.24% / -42.64% / -42.64% +0.56% / -16.68% / -16.68%
sync_round_trip +2.32% / +1.52% / +9.34% -3.24% / -2.07% / -8.43%

This matrix fails the flat-or-better gate: catch_up_page p50, reconnect_ready p50, and notes_save p95/p99 are positive; other positive deltas are shown above.

Eight-second-idle diagnostic

On the same perf epoch and under both shared locks, a 40-cell ABAB ×5 diagnostic inserted eight seconds before the 30 measured notes_save calls. It started at 2026-11-07 03:41:36 UTC and completed at 03:50:53 UTC. Exact launch: systemd-run --user --unit=p928-5f7-idle-diagnostic --collect -p RuntimeMaxSec=7200 /bin/bash /home/claude/p928-perf-5f7-delay-20261106/run-matrix.sh. All four probe/server hashes matched the dev-01 relay copies.

Metric Candidate vs dev p50 / p95 / p99 Dev-vs-dev control p50 / p95 / p99
notes_save -0.55% / -2.86% / -12.32% +0.31% / +2.03% / -0.58%
reconnect_ready -0.74% / -0.51% / +5.03% +2.94% / +2.18% / +9.42%
save_ack +0.08% / +5.57% / +9.73% +0.32% / +6.09% / -4.40%
startup_healthz -0.06% / +5.15% / +5.15% +1.58% / +5.08% / +5.08%
sync_round_trip +1.35% / +0.83% / -1.27% +0.21% / +2.37% / +2.58%

The idle shifts the notes_save p99 from +17.02% to -12.32% while the control stays near flat (-0.62% to -0.58%). This is timing-dependent evidence that the five-second durable witness flush overlaps the save batch. The full matrix remains red; the diagnostic idle is not a production fix. This new design finding is recorded at the top of the PR. No merge performed.

## Qualification at head `5f7b057f2527d9c09f7aab1ff50bf4d7f07bc0f5` Rust and save/sync browser gates pass. Performance qualification does not pass. Positive percentages mean slower; non-startup rows are medians of 10 per-window p50/p95/p99 values. Startup uses 10 observations (median p50, nearest-rank p95/p99). ### Rust and browser gates Rust ran on dev-01 in the candidate worktree with `CARGO_BUILD_JOBS=4`, through `~/bin/slot` and Nix. | Gate | Exact command | Result | |---|---|---| | Format | `CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo fmt --check'` | PASS | | Clippy | `CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo clippy --workspace --all-targets -- -D warnings'` | PASS | | Workspace tests | `CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo test --workspace --no-fail-fast'` | PASS | Save/sync browser specs were `e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts`, one worker: | Browser/project | Exact command | Result | |---|---|---| | Chromium 1440 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/p928-final-3c824/apps/web && CALTERNAL_BIN=/home/claude/Developer/Worktrees/p928-final-3c824/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --workers=1'` | PASS: 26 passed | | Chromium 390 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/p928-final-3c824/apps/web && CALTERNAL_BIN=/home/claude/Developer/Worktrees/p928-final-3c824/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=chromium-390 --workers=1'` | PASS: 18 passed, 8 configured skips | | WebKit 1440 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-webkit-candidate/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-webkit-candidate/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=webkit-1440 --workers=1'` | PASS: 22 passed, 4 configured skips | | WebKit 390 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-webkit-candidate/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-webkit-candidate/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=webkit-390 --workers=1'` | PASS: 14 passed, 12 configured skips | The WebKit `hides markdown syntax` case was repeated on quiet dev-02 with `--repeat-each=20`. Candidate command: `DEBUG=pw:webserver CALTERNAL_BIN=/home/cua/p928-codex-5f7-webkit-candidate-signed bunx playwright test e2e/editor.spec.ts --grep "hides markdown syntax" --project=webkit-1440 --workers=1 --repeat-each=20 --output=/home/cua/p928-quiet-repeat-candidate-5f7 --reporter=line`. Dev used the same command with `CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-dev-current-20261106/target/debug/calternal` and output `/home/cua/p928-quiet-repeat-dev-935fd-5f7`. Both passed 20/20; the initial candidate-only failure did not reproduce. Tracking issue: #996. ### Full post-pin perf matrix Candidate `5f7b057f` vs dev `935fdcacd9cf2837d313817d6786d8a810b3bdaa`; perf epoch `2026-11-06 14:16 IST`. Both shared locks were held throughout, perf-01 then perf-02, and the same-session dev-vs-dev control ran before the candidate matrix. The 120 cells finished at 2026-11-07 00:16 UTC. Measurement wrapper: `/bin/bash /home/claude/p928-perf-5f7-20261106-run1/run-matrix.sh`, which runs `/home/claude/bin/slot /home/claude/.nix-profile/bin/python3 run.py`. | Metric | Candidate vs dev p50 / p95 / p99 | Dev-vs-dev control p50 / p95 / p99 | |---|---:|---:| | `catch_up_page` | +4.63% / -1.37% / -0.25% | +2.62% / +2.04% / +2.95% | | `commit` | -21.55% / -23.59% / -13.09% | +1.19% / +5.66% / +4.24% | | `position_head` | -42.86% / -42.25% / -42.25% | 0.00% / 0.00% / 0.00% | | `position_mid` | -42.86% / -42.25% / -42.25% | 0.00% / +7.04% / +13.38% | | `startup_1m_file_evicted` | -39.88% / -95.72% / -95.72% | -6.03% / +11.51% / +11.51% | | `notes_save` | -0.51% / +8.90% / +17.02% | -0.44% / -1.10% / -0.62% | | `reconnect_ready` | +1.23% / -7.68% / -21.05% | +2.03% / +6.82% / +0.37% | | `save_ack` | +0.32% / +1.17% / -2.72% | -0.56% / -3.71% / -11.93% | | `startup_healthz` | +0.24% / -42.64% / -42.64% | +0.56% / -16.68% / -16.68% | | `sync_round_trip` | +2.32% / +1.52% / +9.34% | -3.24% / -2.07% / -8.43% | This matrix fails the flat-or-better gate: `catch_up_page` p50, `reconnect_ready` p50, and `notes_save` p95/p99 are positive; other positive deltas are shown above. ### Eight-second-idle diagnostic On the same perf epoch and under both shared locks, a 40-cell ABAB ×5 diagnostic inserted eight seconds before the 30 measured `notes_save` calls. It started at 2026-11-07 03:41:36 UTC and completed at 03:50:53 UTC. Exact launch: `systemd-run --user --unit=p928-5f7-idle-diagnostic --collect -p RuntimeMaxSec=7200 /bin/bash /home/claude/p928-perf-5f7-delay-20261106/run-matrix.sh`. All four probe/server hashes matched the dev-01 relay copies. | Metric | Candidate vs dev p50 / p95 / p99 | Dev-vs-dev control p50 / p95 / p99 | |---|---:|---:| | `notes_save` | -0.55% / -2.86% / -12.32% | +0.31% / +2.03% / -0.58% | | `reconnect_ready` | -0.74% / -0.51% / +5.03% | +2.94% / +2.18% / +9.42% | | `save_ack` | +0.08% / +5.57% / +9.73% | +0.32% / +6.09% / -4.40% | | `startup_healthz` | -0.06% / +5.15% / +5.15% | +1.58% / +5.08% / +5.08% | | `sync_round_trip` | +1.35% / +0.83% / -1.27% | +0.21% / +2.37% / +2.58% | The idle shifts the `notes_save` p99 from +17.02% to -12.32% while the control stays near flat (-0.62% to -0.58%). This is timing-dependent evidence that the five-second durable witness flush overlaps the save batch. The full matrix remains red; the diagnostic idle is not a production fix. This new design finding is recorded at the top of the PR. No merge performed.
kayg commented 2026-11-07 11:39:26 +00:00
Author
Owner
Copy link

Qualification at 7b36b0d463f11e0138b15a6b629b4589c309ae38

Gates

Gate Exact command Result
Rust formatting, clippy, workspace tests CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast' PASS: 1,672 passed, 22 ignored
Chromium save/sync, 1440 and 390 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/p928-perf-fix-5fe8/apps/web && CALTERNAL_BIN=/home/claude/p928-ca27-signed-server bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1 --output=/home/claude/p928-7b36-save-sync-chromium --reporter=line' PASS: 44 passed, 8 skipped
WebKit save/sync, 1440 and 390 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-path-identity-20261107/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-path-identity-20261107/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=webkit-1440 --project=webkit-390 --workers=1 --output=/home/cua/p928-7b36-save-sync-webkit --reporter=line' PASS: 36 passed, 16 skipped
WebKit hides markdown syntax, candidate 20 repeats ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-path-identity-20261107/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-path-identity-20261107/target/debug/calternal bunx playwright test e2e/editor.spec.ts --grep "hides markdown syntax" --project=webkit-1440 --workers=1 --repeat-each=20 --output=/home/cua/p928-7b36-repeat-candidate --reporter=line' PASS: 20/20
WebKit hides markdown syntax, dev 20 repeats ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-dev-current-20261106/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-dev-current-20261106/target/debug/calternal bunx playwright test e2e/editor.spec.ts --grep "hides markdown syntax" --project=webkit-1440 --workers=1 --repeat-each=20 --output=/home/cua/p928-792b-repeat-dev --reporter=line' PASS: 20/20

Post-pin no-idle ABAB x5, with same-session dev-vs-dev control

Candidate 7b36b0d; dev 792b33536. Both perf locks were held on perf-01 and perf-02 for the full run. The post-pin epoch and boot IDs were unchanged. The run completed 120 cells with exit 0. Probe hashes were matched to the release build on both arms; server binaries were both debug builds. The room mode had no idle insert. Startup uses fsync plus per-file POSIX_FADV_DONTNEED, not a whole-machine cache drop. Positive percentages mean slower. Non-startup values are medians of 10 per-window p50/p95/p99 values; startup uses 10 observations, nearest-rank p95/p99.

Metric Candidate vs dev p50 / p95 / p99 Dev-vs-dev control p50 / p95 / p99
catch_up_page -5.00% / -3.71% / -0.69% -1.97% / -2.98% / -5.85%
commit +6.33% / +14.04% / +10.69% +1.71% / +6.59% / +3.48%
position_head -42.86% / -42.25% / -45.70% 0.00% / -11.25% / -11.70%
position_mid -42.86% / -42.25% / -42.25% 0.00% / 0.00% / 0.00%
startup_1m_file_evicted +18.58% / +1393.06% / +1393.06% +40.37% / +2112.70% / +2112.70%
notes_save -0.14% / -0.74% / -0.04% -0.33% / -1.79% / +9.73%
reconnect_ready +1.36% / -1.82% / +1.78% -0.44% / +0.57% / -7.93%
save_ack +0.53% / +0.85% / -22.59% +0.20% / +3.19% / +13.00%
startup_healthz -0.19% / +92.25% / +92.25% -1.60% / -22.83% / -22.83%
sync_round_trip +1.83% / +1.29% / +1.59% -2.19% / +1.23% / +1.57%

This fails the unchanged all-path gate. commit is slower at all three quantiles beyond the control; reconnect_ready p50/p99, startup_healthz p95/p99, and sync_round_trip p50 are also outside the control. save_ack p50 is +0.53% vs dev and +0.20% in the control. The startup_healthz p95/p99 is one 143.93 ms sample among 10 candidate observations (the other nine candidate observations were at or below 73.57 ms), so it is shown without treating it as a broad shift. notes_save is flat or better in this matrix.

Continuous-save soak, 200 ms cadence, 180 seconds per arm

Each arm completed 900 saves with zero retries. Candidate values sit within the two dev controls' variation; candidate max is lower than both controls.

Arm notes_save p95 p99 Max Saves / retries
Dev A 18.740 ms 28.425 ms 50.713 ms 900 / 0
Candidate 18.217 ms 28.881 ms 35.619 ms 900 / 0
Dev B 17.314 ms 25.649 ms 46.449 ms 900 / 0

The p99 candidate value is 0.46 ms above dev A, within the 2.78 ms dev-vs-dev p99 spread. Save starts remained at a 200 ms median interval.

Current result

The idle-gated bounded flush fixed the measured notes_save tail, but the fresh matrix has new performance regressions outside the control, especially commit. The design is not qualified under the all-path gate and needs a performance rethink before this PR proceeds. No merge.

## Qualification at `7b36b0d463f11e0138b15a6b629b4589c309ae38` ### Gates | Gate | Exact command | Result | |---|---|---| | Rust formatting, clippy, workspace tests | `CARGO_BUILD_JOBS=4 ~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast'` | PASS: 1,672 passed, 22 ignored | | Chromium save/sync, 1440 and 390 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/claude/Developer/Worktrees/p928-perf-fix-5fe8/apps/web && CALTERNAL_BIN=/home/claude/p928-ca27-signed-server bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=chromium-1440 --project=chromium-390 --workers=1 --output=/home/claude/p928-7b36-save-sync-chromium --reporter=line'` | PASS: 44 passed, 8 skipped | | WebKit save/sync, 1440 and 390 | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-path-identity-20261107/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-path-identity-20261107/target/debug/calternal bunx playwright test e2e/editor.spec.ts e2e/big-paste.spec.ts e2e/traps.spec.ts e2e/list-catch-up.spec.ts --project=webkit-1440 --project=webkit-390 --workers=1 --output=/home/cua/p928-7b36-save-sync-webkit --reporter=line'` | PASS: 36 passed, 16 skipped | | WebKit `hides markdown syntax`, candidate 20 repeats | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-path-identity-20261107/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-path-identity-20261107/target/debug/calternal bunx playwright test e2e/editor.spec.ts --grep "hides markdown syntax" --project=webkit-1440 --workers=1 --repeat-each=20 --output=/home/cua/p928-7b36-repeat-candidate --reporter=line'` | PASS: 20/20 | | WebKit `hides markdown syntax`, dev 20 repeats | `~/bin/slot /nix/var/nix/profiles/default/bin/nix develop -c bash -c 'cd /home/cua/Developer/Worktrees/p928-dev-current-20261106/apps/web && CALTERNAL_BIN=/home/cua/Developer/Worktrees/p928-dev-current-20261106/target/debug/calternal bunx playwright test e2e/editor.spec.ts --grep "hides markdown syntax" --project=webkit-1440 --workers=1 --repeat-each=20 --output=/home/cua/p928-792b-repeat-dev --reporter=line'` | PASS: 20/20 | ### Post-pin no-idle ABAB x5, with same-session dev-vs-dev control Candidate `7b36b0d`; dev `792b33536`. Both perf locks were held on perf-01 and perf-02 for the full run. The post-pin epoch and boot IDs were unchanged. The run completed 120 cells with exit 0. Probe hashes were matched to the release build on both arms; server binaries were both debug builds. The room mode had no idle insert. Startup uses fsync plus per-file `POSIX_FADV_DONTNEED`, not a whole-machine cache drop. Positive percentages mean slower. Non-startup values are medians of 10 per-window p50/p95/p99 values; startup uses 10 observations, nearest-rank p95/p99. | Metric | Candidate vs dev p50 / p95 / p99 | Dev-vs-dev control p50 / p95 / p99 | |---|---:|---:| | `catch_up_page` | -5.00% / -3.71% / -0.69% | -1.97% / -2.98% / -5.85% | | `commit` | +6.33% / +14.04% / +10.69% | +1.71% / +6.59% / +3.48% | | `position_head` | -42.86% / -42.25% / -45.70% | 0.00% / -11.25% / -11.70% | | `position_mid` | -42.86% / -42.25% / -42.25% | 0.00% / 0.00% / 0.00% | | `startup_1m_file_evicted` | +18.58% / +1393.06% / +1393.06% | +40.37% / +2112.70% / +2112.70% | | `notes_save` | -0.14% / -0.74% / -0.04% | -0.33% / -1.79% / +9.73% | | `reconnect_ready` | +1.36% / -1.82% / +1.78% | -0.44% / +0.57% / -7.93% | | `save_ack` | +0.53% / +0.85% / -22.59% | +0.20% / +3.19% / +13.00% | | `startup_healthz` | -0.19% / +92.25% / +92.25% | -1.60% / -22.83% / -22.83% | | `sync_round_trip` | +1.83% / +1.29% / +1.59% | -2.19% / +1.23% / +1.57% | This fails the unchanged all-path gate. `commit` is slower at all three quantiles beyond the control; `reconnect_ready` p50/p99, `startup_healthz` p95/p99, and `sync_round_trip` p50 are also outside the control. `save_ack` p50 is +0.53% vs dev and +0.20% in the control. The `startup_healthz` p95/p99 is one 143.93 ms sample among 10 candidate observations (the other nine candidate observations were at or below 73.57 ms), so it is shown without treating it as a broad shift. `notes_save` is flat or better in this matrix. ### Continuous-save soak, 200 ms cadence, 180 seconds per arm Each arm completed 900 saves with zero retries. Candidate values sit within the two dev controls' variation; candidate max is lower than both controls. | Arm | `notes_save` p95 | p99 | Max | Saves / retries | |---|---:|---:|---:|---:| | Dev A | 18.740 ms | 28.425 ms | 50.713 ms | 900 / 0 | | Candidate | 18.217 ms | 28.881 ms | 35.619 ms | 900 / 0 | | Dev B | 17.314 ms | 25.649 ms | 46.449 ms | 900 / 0 | The p99 candidate value is 0.46 ms above dev A, within the 2.78 ms dev-vs-dev p99 spread. Save starts remained at a 200 ms median interval. ### Current result The idle-gated bounded flush fixed the measured `notes_save` tail, but the fresh matrix has new performance regressions outside the control, especially `commit`. The design is not qualified under the all-path gate and needs a performance rethink before this PR proceeds. No merge.
kayg commented 2026-11-07 12:28:42 +00:00
Author
Owner
Copy link

P1b #995 now adds a Notes source-generation fence. It reads the accepted version from each notes.saved entry, waits on the existing Notes publication watch when the pinned source is behind, and rechecks the saved cursor tail on restart. The search side continues to use the engine position and journal contract. It does not inspect token fields, read the high-water file, or duplicate #918 restore detection; P1b does not need the high-water value. If #928 merges first, #995 will rebase and use its interface.

P1b #995 now adds a Notes source-generation fence. It reads the accepted version from each `notes.saved` entry, waits on the existing Notes publication watch when the pinned source is behind, and rechecks the saved cursor tail on restart. The search side continues to use the engine `position` and `journal` contract. It does not inspect token fields, read the high-water file, or duplicate #918 restore detection; P1b does not need the high-water value. If #928 merges first, #995 will rebase and use its interface.
kayg commented 2026-11-07 15:41:39 +00:00
Author
Owner
Copy link

Startup rerun and commit bisect

Startup metrics: 20 observations per arm

Candidate 7b36b0d463f11e0138b15a6b629b4589c309ae38; dev 792b33536bc9743904bf7a5602386027183d0776. The run had 160 samples, ABAB ×10 per phase for each metric, with a same-session dev-vs-dev phase followed by dev-vs-candidate. The perf-01 and perf-02 lock holders bracketed the run; both boot IDs match the post-pin epoch. Results below are milliseconds. Each arm shows median, IQR, and min–max. Paired deltas compare adjacent A/B observations within cycles.

Metric Phase Arm A Arm B Paired B/A median (IQR)
startup_healthz dev control 59.461 (58.597–60.122; 57.584–81.572) 58.771 (58.429–61.726; 57.763–89.932) -0.02% (-2.42%–+2.44%)
startup_healthz dev vs candidate 82.412 (75.769–84.178; 58.629–92.114) 79.151 (75.053–83.766; 58.858–110.134) +0.24% (-5.75%–+2.62%)
startup_1m_file_evicted dev control 0.688 (0.640–0.790; 0.526–19.862) 0.679 (0.648–0.746; 0.597–17.592) +1.18% (-21.80%–+14.65%)
startup_1m_file_evicted dev vs candidate 0.686 (0.645–0.758; 0.585–19.621) 0.760 (0.689–0.854; 0.562–0.979) +11.48% (-13.51%–+24.11%)

startup_healthz is flat within the paired control spread despite a phase-to-phase shift in absolute times. startup_1m_file_evicted has a small positive median shift of 0.073 ms; its paired spread is wide and overlaps the control, with 12/20 candidate pairs slower. The p95/p99 tail outliers are dev-side cache-eviction samples (up to 19.862 ms); the candidate arm's p95/p99 are 0.931/0.979 ms. This does not show a broad startup regression, but the small p50 shift remains visible and will be checked again in the final matrix.

Commit micro-bisect

The 60-row micro-bisect used three ABAB cycles per stage, six 4,000-commit observations per arm. Positive deltas mean slower.

Stage Candidate vs dev p50 / p95 / p99
Before flush work -20.84% / -17.07% / -13.96%
Idle-gated flush + slot design +5.57% / -4.79% / +3.43%
Path identity checks +3.94% / -2.84% / +21.44%
Current 7b36b0d -3.46% / -8.01% / +5.19%

The final-head p99 is inside the broad same-session dev control IQR; the full matrix's commit regression remains a gate failure. Source inspection identifies the per-call work to remove: UserStore::commit/commit_once activity guards, the writer's dirty-age clock/CAS, and commit_inner's seven-atomic witness snapshot publication. fstat/stat is confined to the flush worker and does not run on commit. The selected simpler design is now at the top of the PR description; no code has been patched yet.

## Startup rerun and commit bisect ### Startup metrics: 20 observations per arm Candidate `7b36b0d463f11e0138b15a6b629b4589c309ae38`; dev `792b33536bc9743904bf7a5602386027183d0776`. The run had 160 samples, ABAB ×10 per phase for each metric, with a same-session dev-vs-dev phase followed by dev-vs-candidate. The perf-01 and perf-02 lock holders bracketed the run; both boot IDs match the post-pin epoch. Results below are milliseconds. Each arm shows median, IQR, and min–max. Paired deltas compare adjacent A/B observations within cycles. | Metric | Phase | Arm A | Arm B | Paired B/A median (IQR) | |---|---|---:|---:|---:| | `startup_healthz` | dev control | 59.461 (58.597–60.122; 57.584–81.572) | 58.771 (58.429–61.726; 57.763–89.932) | -0.02% (-2.42%–+2.44%) | | `startup_healthz` | dev vs candidate | 82.412 (75.769–84.178; 58.629–92.114) | 79.151 (75.053–83.766; 58.858–110.134) | +0.24% (-5.75%–+2.62%) | | `startup_1m_file_evicted` | dev control | 0.688 (0.640–0.790; 0.526–19.862) | 0.679 (0.648–0.746; 0.597–17.592) | +1.18% (-21.80%–+14.65%) | | `startup_1m_file_evicted` | dev vs candidate | 0.686 (0.645–0.758; 0.585–19.621) | 0.760 (0.689–0.854; 0.562–0.979) | +11.48% (-13.51%–+24.11%) | `startup_healthz` is flat within the paired control spread despite a phase-to-phase shift in absolute times. `startup_1m_file_evicted` has a small positive median shift of 0.073 ms; its paired spread is wide and overlaps the control, with 12/20 candidate pairs slower. The p95/p99 tail outliers are dev-side cache-eviction samples (up to 19.862 ms); the candidate arm's p95/p99 are 0.931/0.979 ms. This does not show a broad startup regression, but the small p50 shift remains visible and will be checked again in the final matrix. ### Commit micro-bisect The 60-row micro-bisect used three ABAB cycles per stage, six 4,000-commit observations per arm. Positive deltas mean slower. | Stage | Candidate vs dev p50 / p95 / p99 | |---|---:| | Before flush work | -20.84% / -17.07% / -13.96% | | Idle-gated flush + slot design | +5.57% / -4.79% / +3.43% | | Path identity checks | +3.94% / -2.84% / +21.44% | | Current `7b36b0d` | -3.46% / -8.01% / +5.19% | The final-head p99 is inside the broad same-session dev control IQR; the full matrix's commit regression remains a gate failure. Source inspection identifies the per-call work to remove: `UserStore::commit/commit_once` activity guards, the writer's dirty-age clock/CAS, and `commit_inner`'s seven-atomic witness snapshot publication. `fstat/stat` is confined to the flush worker and does not run on commit. The selected simpler design is now at the top of the PR description; no code has been patched yet.
kayg commented 2026-11-08 06:23:23 +00:00
Author
Owner
Copy link

Startup rerun at 95af4df2cc0add210d70850c2c092a23cebe4e5d

Candidate was 95af4df2cc0add210d70850c2c092a23cebe4e5d; dev was 792b33536bc9743904bf7a5602386027183d0776. The post-pin epoch and both perf VM boot IDs match the session metadata. Both shared perf locks were held for the full run. Startup and room scenarios ran ABAB ×10 per phase, giving 20 observations per arm. Values are milliseconds: median of the per-invocation p50, Tukey IQR, and min–max.

Metric Phase / arm Median (IQR; min–max)
startup_healthz control A / dev 57.059 (56.434–59.486; 55.050–85.801)
startup_healthz control B / dev 56.584 (55.335–58.138; 54.083–80.343)
startup_healthz candidate A / dev 75.530 (65.568–82.002; 54.585–129.396)
startup_healthz candidate B / head 76.349 (69.958–79.299; 55.261–97.161)
startup_1m_file_evicted control A / dev 0.703 (0.600–3.014; 0.528–18.332)
startup_1m_file_evicted control B / dev 0.654 (0.591–0.730; 0.541–3.474)
startup_1m_file_evicted candidate A / dev 0.631 (0.590–0.664; 0.538–1.892)
startup_1m_file_evicted candidate B / head 0.819 (0.694–11.578; 0.498–17.328)

Paired B/A median was −0.30% for startup_healthz (9/20 candidate pairs slower; control −3.48%, 7/20 slower), so there is no paired shift beyond the phase noise. For startup_1m_file_evicted, candidate paired median was +35.75% (16/20 slower) versus control −2.59% (10/20 slower). The absolute paired median difference was 0.188 ms. The wide tails remain noisy, but the 1M-entry median shift is visible in the paired observations and is not explained by the p95/p99 sample count alone.

The current startup path creates and syncs a blank page before the initial valid witness slot is ready, then syncs the slot separately. That is the leading cause to test by deferring first-page allocation until the valid mark is written.

Runner: /home/claude/p928-final-95af-perf-20261108-run1/run-all.sh (startup data in samples.jsonl; run metadata in metadata.json). No idle insert was used.

## Startup rerun at `95af4df2cc0add210d70850c2c092a23cebe4e5d` Candidate was `95af4df2cc0add210d70850c2c092a23cebe4e5d`; dev was `792b33536bc9743904bf7a5602386027183d0776`. The post-pin epoch and both perf VM boot IDs match the session metadata. Both shared perf locks were held for the full run. Startup and room scenarios ran ABAB ×10 per phase, giving 20 observations per arm. Values are milliseconds: median of the per-invocation p50, Tukey IQR, and min–max. | Metric | Phase / arm | Median (IQR; min–max) | |---|---|---:| | `startup_healthz` | control A / dev | 57.059 (56.434–59.486; 55.050–85.801) | | `startup_healthz` | control B / dev | 56.584 (55.335–58.138; 54.083–80.343) | | `startup_healthz` | candidate A / dev | 75.530 (65.568–82.002; 54.585–129.396) | | `startup_healthz` | candidate B / head | 76.349 (69.958–79.299; 55.261–97.161) | | `startup_1m_file_evicted` | control A / dev | 0.703 (0.600–3.014; 0.528–18.332) | | `startup_1m_file_evicted` | control B / dev | 0.654 (0.591–0.730; 0.541–3.474) | | `startup_1m_file_evicted` | candidate A / dev | 0.631 (0.590–0.664; 0.538–1.892) | | `startup_1m_file_evicted` | candidate B / head | 0.819 (0.694–11.578; 0.498–17.328) | Paired B/A median was −0.30% for `startup_healthz` (9/20 candidate pairs slower; control −3.48%, 7/20 slower), so there is no paired shift beyond the phase noise. For `startup_1m_file_evicted`, candidate paired median was +35.75% (16/20 slower) versus control −2.59% (10/20 slower). The absolute paired median difference was 0.188 ms. The wide tails remain noisy, but the 1M-entry median shift is visible in the paired observations and is not explained by the p95/p99 sample count alone. The current startup path creates and syncs a blank page before the initial valid witness slot is ready, then syncs the slot separately. That is the leading cause to test by deferring first-page allocation until the valid mark is written. Runner: `/home/claude/p928-final-95af-perf-20261108-run1/run-all.sh` (startup data in `samples.jsonl`; run metadata in `metadata.json`). No idle insert was used.
kayg commented 2026-11-09 10:09:38 +00:00
Author
Owner
Copy link

Startup diagnostic at 8357e925a (merged with dev 2fa9e18fb)

Takeover of #928 after the previous agent stopped. 8357e925a merges current dev; the one conflict (crates/calternal/tests/notes_crash_typing.rs) takes dev's fix for the same flake, so the branch's own test-only fix drops out of the diff.

Rust gates (dev-01, merged head)

CARGO_BUILD_JOBS=4 ~/bin/slot nix develop -c bash -c "cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast" in p928-takeover-cand: PASS (exit 0).

startup_1m_file_evicted, 20 observations per arm

Same session, both perf locks held (perf-01, then perf-02), 2026-11-09 08:12–08:18 UTC, post-pin epoch. Release probes, with sha256 checked on dev-01 and on perf-01. Each cycle runs a dev-vs-dev pair, a dev/candidate ABAB, then another dev-vs-dev pair. Positive means slower.

Case Dev median Candidate median Candidate paired B/A median (slower pairs) Control paired median
Fully evicted (the #86388 case) 0.792 ms 0.757 ms −4.38% (9/20) −0.89% (10/20)
Store evicted, secrets dir warm 0.698 ms 0.805 ms +14.68% (13/20) −8.49% (8/20)
Variant: key and witness read on separate threads 0.756 ms 0.696 ms −1.52% (8/20) −1.92% (9/20)

Cause: the +35.75% at 95af4df2 came from restored_at_open calling write_now on a valid older same-epoch mark: a slot rewrite plus fdatasync during open. f71379a1c leaves that mark in place and has the flush worker advance it. With that change the fully evicted case is flat against its control. Path-identity checks are unchanged and stay on the flush worker.

The "secrets warm" row is a diagnostic. Its paired IQR (−10%…+67%) overlaps the control's (−91%…+27%). The parallel-read variant shows no gain, so it is not committed. The full ABAB matrix and soak come next, in a FIFO perf turn.

## Startup diagnostic at `8357e925a` (merged with dev `2fa9e18fb`) Takeover of #928 after the previous agent stopped. `8357e925a` merges current dev; the one conflict (`crates/calternal/tests/notes_crash_typing.rs`) takes dev's fix for the same flake, so the branch's own test-only fix drops out of the diff. ### Rust gates (dev-01, merged head) `CARGO_BUILD_JOBS=4 ~/bin/slot nix develop -c bash -c "cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast"` in `p928-takeover-cand`: **PASS** (exit 0). ### `startup_1m_file_evicted`, 20 observations per arm Same session, both perf locks held (perf-01, then perf-02), 2026-11-09 08:12–08:18 UTC, post-pin epoch. Release probes, with sha256 checked on dev-01 and on perf-01. Each cycle runs a dev-vs-dev pair, a dev/candidate ABAB, then another dev-vs-dev pair. Positive means slower. | Case | Dev median | Candidate median | Candidate paired B/A median (slower pairs) | Control paired median | |---|---:|---:|---:|---:| | Fully evicted (the #86388 case) | 0.792 ms | 0.757 ms | −4.38% (9/20) | −0.89% (10/20) | | Store evicted, secrets dir warm | 0.698 ms | 0.805 ms | +14.68% (13/20) | −8.49% (8/20) | | Variant: key and witness read on separate threads | 0.756 ms | 0.696 ms | −1.52% (8/20) | −1.92% (9/20) | **Cause:** the +35.75% at `95af4df2` came from `restored_at_open` calling `write_now` on a valid older same-epoch mark: a slot rewrite plus `fdatasync` during open. `f71379a1c` leaves that mark in place and has the flush worker advance it. With that change the fully evicted case is flat against its control. Path-identity checks are unchanged and stay on the flush worker. The "secrets warm" row is a diagnostic. Its paired IQR (−10%…+67%) overlaps the control's (−91%…+27%). The parallel-read variant shows no gain, so it is not committed. The full ABAB matrix and soak come next, in a FIFO perf turn.
kayg commented 2026-11-09 12:29:57 +00:00
Author
Owner
Copy link

Final ABAB matrix and soak at 8357e925a vs dev 2fa9e18fb

FIFO perf turn (~/bin/perf-turn p928), perf-01 then perf-02 locks held 12:06–12:29 UTC 2026-11-09, post-pin epoch. Release probes and release servers (server-cand 746b4fdd…, server-dev 883e18bd…), sha256 checked on perf-01 at start. Each cycle: dev-vs-dev control pair, dev/candidate ABAB, control pair. Journal ×3 cycles (6 obs/arm, kept small for the 90-min turn), startup ×10 and room ×10 (20 obs/arm). No idle insert. Positive = slower. Non-startup values are medians of per-run p50/p95/p99; startup uses per-run times with nearest-rank p95/p99.

Metric Candidate vs dev p50 / p95 / p99 Dev-vs-dev control p50 / p95 / p99 Paired median, candidate (control) p50 / p95 / p99
commit -20.55% / -18.55% / -15.84% +2.89% / +5.03% / +7.21% -20.44% (+2.75%) / -17.97% (+3.67%) / -16.09% (+6.25%)
position_head -42.86% / -48.75% / -49.38% +0.00% / +0.00% / +0.00% -42.86% (+0.00%) / -48.75% (+0.00%) / -49.38% (+0.00%)
position_mid -42.86% / -48.75% / -49.38% +0.00% / -5.88% / -5.81% -42.86% (+0.00%) / -48.75% (-5.56%) / -49.38% (-5.49%)
catch_up_page -2.24% / -1.50% / -9.00% +2.62% / +3.60% / +7.62% -2.46% (+2.96%) / -1.73% (+3.53%) / -4.31% (+8.22%)
startup_1m_file_evicted -65.65% / -94.17% / -94.21% -70.34% / -94.89% / -94.88% -50.02% (-52.32%) / -50.02% (-52.32%) / -50.02% (-52.32%)
startup_healthz +2.17% / +10.88% / +21.12% +1.35% / -3.17% / -19.73% +2.91% (-0.59%) / +2.91% (-0.59%) / +2.91% (-0.59%)
notes_save -1.61% / +2.48% / +9.30% +1.93% / +2.25% / -2.86% +0.22% (+0.35%) / +4.96% (+2.56%) / -0.22% (+0.06%)
save_ack +0.30% / +4.00% / +5.51% -0.16% / -3.22% / -8.94% +0.21% (+0.12%) / +1.03% (-3.16%) / -0.08% (-6.17%)
sync_round_trip +1.02% / +0.69% / +5.69% +0.80% / -1.26% / -3.13% +0.54% (+1.17%) / +1.72% (+3.31%) / -1.74% (-3.62%)
reconnect_ready -0.56% / +2.24% / +1.31% +1.55% / +2.19% / -1.59% -0.17% (+0.55%) / +3.13% (-3.06%) / +3.46% (-3.15%)

startup_1m_file_evicted matches its control. Both phases show the same first-of-pair order effect (paired −50.0% vs control −52.3%; candidate arm median 0.746 ms vs control B 0.622 ms, dev and control A medians inflated by eviction outliers). The +35.75% from #86388 is gone.

Soak: 200 ms cadence, 180 s per arm, 900 saves, 0 retries each

Arm notes_save p50 p95 p99 max
Dev A 20.948 ms 28.289 ms 37.875 ms 209.405 ms
Candidate 20.948 ms 29.497 ms 42.429 ms 285.562 ms
Dev B 21.010 ms 28.713 ms 43.426 ms 178.499 ms

Open: room-scenario tails

save_ack p95/p99 (+4.00%/+5.51% vs control −3.22%/−8.94%), startup_healthz paired p50 (+2.91% vs −0.59%), notes_save p99, and soak p95/max sit above dev. Their paired IQRs overlap the controls (for example startup_healthz [−1.6%, +5.6%] vs [−4.1%, +6.1%], notes_save p95 [−1.2%, +9.5%] vs [−1.9%, +7.8%]), but 20 observations cannot show they are flat. A room-only rerun with 40 obs/arm plus a second soak is queued in the FIFO line.

## Final ABAB matrix and soak at `8357e925a` vs dev `2fa9e18fb` FIFO perf turn (`~/bin/perf-turn p928`), perf-01 then perf-02 locks held 12:06–12:29 UTC 2026-11-09, post-pin epoch. Release probes and release servers (`server-cand` 746b4fdd…, `server-dev` 883e18bd…), sha256 checked on perf-01 at start. Each cycle: dev-vs-dev control pair, dev/candidate ABAB, control pair. Journal ×3 cycles (6 obs/arm, kept small for the 90-min turn), startup ×10 and room ×10 (20 obs/arm). No idle insert. Positive = slower. Non-startup values are medians of per-run p50/p95/p99; startup uses per-run times with nearest-rank p95/p99. | Metric | Candidate vs dev p50 / p95 / p99 | Dev-vs-dev control p50 / p95 / p99 | Paired median, candidate (control) p50 / p95 / p99 | |---|---:|---:|---:| | `commit` | -20.55% / -18.55% / -15.84% | +2.89% / +5.03% / +7.21% | -20.44% (+2.75%) / -17.97% (+3.67%) / -16.09% (+6.25%) | | `position_head` | -42.86% / -48.75% / -49.38% | +0.00% / +0.00% / +0.00% | -42.86% (+0.00%) / -48.75% (+0.00%) / -49.38% (+0.00%) | | `position_mid` | -42.86% / -48.75% / -49.38% | +0.00% / -5.88% / -5.81% | -42.86% (+0.00%) / -48.75% (-5.56%) / -49.38% (-5.49%) | | `catch_up_page` | -2.24% / -1.50% / -9.00% | +2.62% / +3.60% / +7.62% | -2.46% (+2.96%) / -1.73% (+3.53%) / -4.31% (+8.22%) | | `startup_1m_file_evicted` | -65.65% / -94.17% / -94.21% | -70.34% / -94.89% / -94.88% | -50.02% (-52.32%) / -50.02% (-52.32%) / -50.02% (-52.32%) | | `startup_healthz` | +2.17% / +10.88% / +21.12% | +1.35% / -3.17% / -19.73% | +2.91% (-0.59%) / +2.91% (-0.59%) / +2.91% (-0.59%) | | `notes_save` | -1.61% / +2.48% / +9.30% | +1.93% / +2.25% / -2.86% | +0.22% (+0.35%) / +4.96% (+2.56%) / -0.22% (+0.06%) | | `save_ack` | +0.30% / +4.00% / +5.51% | -0.16% / -3.22% / -8.94% | +0.21% (+0.12%) / +1.03% (-3.16%) / -0.08% (-6.17%) | | `sync_round_trip` | +1.02% / +0.69% / +5.69% | +0.80% / -1.26% / -3.13% | +0.54% (+1.17%) / +1.72% (+3.31%) / -1.74% (-3.62%) | | `reconnect_ready` | -0.56% / +2.24% / +1.31% | +1.55% / +2.19% / -1.59% | -0.17% (+0.55%) / +3.13% (-3.06%) / +3.46% (-3.15%) | `startup_1m_file_evicted` matches its control. Both phases show the same first-of-pair order effect (paired −50.0% vs control −52.3%; candidate arm median 0.746 ms vs control B 0.622 ms, dev and control A medians inflated by eviction outliers). The +35.75% from #86388 is gone. ### Soak: 200 ms cadence, 180 s per arm, 900 saves, 0 retries each | Arm | `notes_save` p50 | p95 | p99 | max | |---|---:|---:|---:|---:| | Dev A | 20.948 ms | 28.289 ms | 37.875 ms | 209.405 ms | | Candidate | 20.948 ms | 29.497 ms | 42.429 ms | 285.562 ms | | Dev B | 21.010 ms | 28.713 ms | 43.426 ms | 178.499 ms | ### Open: room-scenario tails `save_ack` p95/p99 (+4.00%/+5.51% vs control −3.22%/−8.94%), `startup_healthz` paired p50 (+2.91% vs −0.59%), `notes_save` p99, and soak p95/max sit above dev. Their paired IQRs overlap the controls (for example `startup_healthz` [−1.6%, +5.6%] vs [−4.1%, +6.1%], `notes_save` p95 [−1.2%, +9.5%] vs [−1.9%, +7.8%]), but 20 observations cannot show they are flat. A room-only rerun with 40 obs/arm plus a second soak is queued in the FIFO line.
kayg commented 2026-11-09 13:25:08 +00:00
Author
Owner
Copy link

Room rerun (40 obs/arm) and second soak at 8357e925a vs dev 2fa9e18fb

FIFO turn p928, perf-01 then perf-02 locks held 12:56:39–13:24:33 UTC 2026-11-09, same binaries and hashes as comment 88778. Room ×20 ABAB cycles with control pairs around each cycle. Positive = slower.

Metric Candidate vs dev p50 / p95 / p99 Control p50 / p95 / p99 Paired median p50, candidate (control)
notes_save -0.65% / -1.66% / -12.12% -0.31% / +2.30% / +0.82% -0.61% (-0.48%)
save_ack +0.93% / -0.66% / +4.32% -0.59% / -0.73% / -2.88% +0.59% (-0.61%)
sync_round_trip +2.37% / +0.54% / +3.49% +0.36% / +0.54% / +2.12% +3.03% (+0.57%)
reconnect_ready -1.25% / +1.09% / +0.70% +3.00% / +2.76% / +3.74% +1.97% (+3.61%)
startup_healthz +2.15% / -1.16% / +12.24% +0.56% / -0.73% / -8.81% +2.35% (+1.51%)

Soak 2 (200 ms cadence, 180 s per arm, 900 saves, 0 retries each), notes_save p50 / p95 / p99 / max: dev A 21.040 / 27.421 / 37.557 / 58.461 ms; candidate 20.928 / 27.121 / 38.256 / 171.097 ms; dev B 21.015 / 28.036 / 37.250 / 65.754 ms.

Verdict across both sessions

  • Fixed: startup_1m_file_evicted is flat against its control (cause: restored_at_open used write_now on a valid older mark; f71379a1c defers that to the flush worker; the path-identity checks stay on the flush worker).
  • Better: commit −20…−16%, position −43…−49%, catch-up −2…−9%; notes_save p50/p95 flat or better at 40 obs/arm.
  • Still above dev, beyond control, in both sessions (p50):
    • startup_healthz +2.15% (control +0.56%), about +1.9 ms. Every room run is a fresh --create install with a fresh --secrets dir, so this times first-ever startup: durable key creation plus the first durable witness write. The design requires both before signed tokens go out. It is not a per-restart cost, and the store-level reopen (startup_1m_file_evicted) is flat.
    • sync_round_trip +2.37% (control +0.36%, ≈ +5 µs) and save_ack +0.93% (control −0.59%, ≈ +0.07 ms): per-entry signed tokens on the live path. Dev does not sign at all.
    • Soak max: the candidate is the highest arm in both soaks (285 ms and 171 ms vs dev 58–209 ms), and its p99 is +0.7 ms in soak 2. This is consistent with the 30 s maximum-age witness fdatasync landing next to a save during continuous typing.

These are new work required by #918/#919, not accidental cost, but under the no-regression rule they block. Needs an owner decision: accept them as the price of signed tokens and restore detection, or ask for further work (for example moving first-use key and witness creation off the /healthz path, or aligning the max-age flush away from in-flight saves).

## Room rerun (40 obs/arm) and second soak at `8357e925a` vs dev `2fa9e18fb` FIFO turn `p928`, perf-01 then perf-02 locks held 12:56:39–13:24:33 UTC 2026-11-09, same binaries and hashes as comment 88778. Room ×20 ABAB cycles with control pairs around each cycle. Positive = slower. | Metric | Candidate vs dev p50 / p95 / p99 | Control p50 / p95 / p99 | Paired median p50, candidate (control) | |---|---:|---:|---:| | `notes_save` | -0.65% / -1.66% / -12.12% | -0.31% / +2.30% / +0.82% | -0.61% (-0.48%) | | `save_ack` | +0.93% / -0.66% / +4.32% | -0.59% / -0.73% / -2.88% | +0.59% (-0.61%) | | `sync_round_trip` | +2.37% / +0.54% / +3.49% | +0.36% / +0.54% / +2.12% | +3.03% (+0.57%) | | `reconnect_ready` | -1.25% / +1.09% / +0.70% | +3.00% / +2.76% / +3.74% | +1.97% (+3.61%) | | `startup_healthz` | +2.15% / -1.16% / +12.24% | +0.56% / -0.73% / -8.81% | +2.35% (+1.51%) | Soak 2 (200 ms cadence, 180 s per arm, 900 saves, 0 retries each), `notes_save` p50 / p95 / p99 / max: dev A 21.040 / 27.421 / 37.557 / 58.461 ms; **candidate 20.928 / 27.121 / 38.256 / 171.097 ms**; dev B 21.015 / 28.036 / 37.250 / 65.754 ms. ### Verdict across both sessions - **Fixed:** `startup_1m_file_evicted` is flat against its control (cause: `restored_at_open` used `write_now` on a valid older mark; `f71379a1c` defers that to the flush worker; the path-identity checks stay on the flush worker). - **Better:** commit −20…−16%, position −43…−49%, catch-up −2…−9%; notes_save p50/p95 flat or better at 40 obs/arm. - **Still above dev, beyond control, in both sessions (p50):** - `startup_healthz` +2.15% (control +0.56%), about +1.9 ms. Every room run is a fresh `--create` install with a fresh `--secrets` dir, so this times first-ever startup: durable key creation plus the first durable witness write. The design requires both before signed tokens go out. It is not a per-restart cost, and the store-level reopen (`startup_1m_file_evicted`) is flat. - `sync_round_trip` +2.37% (control +0.36%, ≈ +5 µs) and `save_ack` +0.93% (control −0.59%, ≈ +0.07 ms): per-entry signed tokens on the live path. Dev does not sign at all. - Soak max: the candidate is the highest arm in both soaks (285 ms and 171 ms vs dev 58–209 ms), and its p99 is +0.7 ms in soak 2. This is consistent with the 30 s maximum-age witness `fdatasync` landing next to a save during continuous typing. These are new work required by #918/#919, not accidental cost, but under the no-regression rule they block. Needs an owner decision: accept them as the price of signed tokens and restore detection, or ask for further work (for example moving first-use key and witness creation off the `/healthz` path, or aligning the max-age flush away from in-flight saves).
kayg commented 2026-11-10 00:52:37 +00:00
Author
Owner
Copy link

Final matrix at ee2bc32dc vs dev 2fa9e18fb

One FIFO turn (perf-turn p928), perf-01 then perf-02 locks held 00:18–00:51 UTC 2026-11-10. Release builds, sha256 checked on dev-01 and perf-01 (probe-cand 38a3fdf4…, server-cand df8b1a1e…, probe-dev e0ce08c9…, server-dev 883e18bd…). ABAB with dev-vs-dev control pairs around every cycle: journal ×5 (10 observations/arm), startup ×10 (20/arm), room ×20 and boot ×20 (40/arm). Positive = slower.

Changes since comment 88839

Commit Change Why
b82586428 Witness flush never starts while a save is in flight; forced at 2× maximum age soak max / save_ack tails
3d1eaa7ba Flush reads the head check from the cached signed token, not a fresh SQLite reader callgrind: ~222K Ir + file I/O per flush
a949fa358 EventRecorder::save_scope: room typing saves mark a save in flight (never opens a store) typing saves were invisible to the flush scheduler
1f55329ff One-shot blake3 over a stack buffer and direct token text; byte-identical values (tests pin them) callgrind: minting was 112.5M of 256.9M Ir in since()
ee2bc32dc DB opens on a pool thread while the two tiny secret files are read inline a second pool thread cost +0.2 ms on warm reopen; serial reads cost +29% cold

/healthz never waited for first-install durability: strace shows no secrets-folder access before or within 2 s after listen. Key and witness are made durable on the first store open, before any signed token is returned; that cost is now visible as first_token_fresh.

Matrix

Metric Candidate vs dev p50 / p95 / p99 Control p50 / p95 / p99 Paired p50, candidate (control)
commit -22.14% / -25.75% / -25.66% +0.01% / +0.19% / +2.89% -23.04% (+0.13%)
position_head -42.86% / -42.25% / -45.70% +0.00% / +0.00% / +6.34% -42.86% (+0.00%)
position_mid -42.86% / -42.25% / -39.74% +0.00% / +6.34% / +13.38% -42.86% (+0.00%)
catch_up_page -3.29% / -2.33% / +7.22% +2.04% / +6.89% / +3.49% -3.23% (+1.15%)
startup_1m_file_evicted -7.55% / -95.58% / -93.47% -30.69% / -95.89% / -95.75% -9.14% (-37.88%)
startup_healthz -0.31% / +8.70% / +7.37% +0.06% / -2.36% / -3.47% -1.29% (-0.13%)
startup_healthz_existing +0.45% / -6.22% / +13.11% +0.64% / +3.81% / +1.03% +0.13% (+1.06%)
first_token_existing +5.42% / +5.42% / +5.42% +9.75% / +9.75% / +9.75% +8.13% (+2.58%)
first_token_fresh +50.07% / +50.07% / +50.07% +0.06% / +0.06% / +0.06% +49.61% (+0.66%)
notes_save -0.45% / -2.93% / -1.74% +0.58% / -0.75% / +5.32% -0.37% (+0.77%)
save_ack -0.03% / -0.93% / -1.77% +0.76% / -0.05% / -4.41% +0.10% (+0.17%)
sync_round_trip +1.15% / +0.06% / +3.06% +0.64% / +1.46% / +0.49% +1.42% (+2.36%)
reconnect_ready +1.74% / +0.66% / +2.20% -0.90% / -0.00% / -7.28% +0.76% (+0.13%)

Boot (startup only, 40 obs/arm; candidate server with and without --secrets)

Stage Metric Paired p50, candidate (control)
boot_cand first_token_existing -0.89% (-2.44%)
boot_cand first_token_fresh +50.92% (+0.94%)
boot_cand startup_healthz +1.30% (-1.21%)
boot_cand startup_healthz_existing +0.25% (+0.54%)
boot_nosecrets first_token_existing +2.68% (-0.31%)
boot_nosecrets first_token_fresh +2.68% (-0.49%)
boot_nosecrets startup_healthz -0.73% (+0.19%)
boot_nosecrets startup_healthz_existing +0.42% (-0.68%)

Soak (200 ms cadence, 180 s per arm, 900 saves, 0 retries each)

Arm p50 p95 p99 max
Dev A 20.822 26.841 34.528 45.725 ms
Candidate 20.884 27.450 38.146 88.438 ms
Dev B 20.730 28.219 36.947 52.013 ms

Gates at ee2bc32dc

  • Rust (dev-01): cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast: PASS, 1,691 passed, 22 ignored.
  • Chromium 1440 + 390 save/sync specs (editor, big-paste, traps, list-catch-up): PASS, 44 passed, 8 skipped.
  • WebKit 1440 + 390, same specs (dev-02): PASS, 36 passed, 16 skipped.

Open

  • Soak max: the candidate is still the highest arm in every quiet session (88.4 ms vs dev 45.7/52.0; p99 +1.2 ms over the slower dev arm; p50/p95 between the dev arms). One sample in 900, but consistent. Not yet explained.
  • first_token_fresh +50% (about +20 ms) is the required key and witness durability, moved from startup to the first token.
  • Fresh /healthz: the room phase is flat (−1.29% vs −0.13%); the boot phase reads +1.30% vs −1.21% with --secrets and flat without. No syscall difference was found, so this looks like noise, but it is reported as measured.
## Final matrix at `ee2bc32dc` vs dev `2fa9e18fb` One FIFO turn (`perf-turn p928`), perf-01 then perf-02 locks held 00:18–00:51 UTC 2026-11-10. Release builds, sha256 checked on dev-01 and perf-01 (`probe-cand` 38a3fdf4…, `server-cand` df8b1a1e…, `probe-dev` e0ce08c9…, `server-dev` 883e18bd…). ABAB with dev-vs-dev control pairs around every cycle: journal ×5 (10 observations/arm), startup ×10 (20/arm), room ×20 and boot ×20 (40/arm). Positive = slower. ### Changes since comment 88839 | Commit | Change | Why | |---|---|---| | `b82586428` | Witness flush never starts while a save is in flight; forced at 2× maximum age | soak max / save_ack tails | | `3d1eaa7ba` | Flush reads the head check from the cached signed token, not a fresh SQLite reader | callgrind: ~222K Ir + file I/O per flush | | `a949fa358` | `EventRecorder::save_scope`: room typing saves mark a save in flight (never opens a store) | typing saves were invisible to the flush scheduler | | `1f55329ff` | One-shot blake3 over a stack buffer and direct token text; byte-identical values (tests pin them) | callgrind: minting was 112.5M of 256.9M Ir in `since()` | | `ee2bc32dc` | DB opens on a pool thread while the two tiny secret files are read inline | a second pool thread cost +0.2 ms on warm reopen; serial reads cost +29% cold | `/healthz` never waited for first-install durability: strace shows no secrets-folder access before or within 2 s after `listen`. Key and witness are made durable on the first store open, before any signed token is returned; that cost is now visible as `first_token_fresh`. ### Matrix | Metric | Candidate vs dev p50 / p95 / p99 | Control p50 / p95 / p99 | Paired p50, candidate (control) | |---|---:|---:|---:| | `commit` | -22.14% / -25.75% / -25.66% | +0.01% / +0.19% / +2.89% | -23.04% (+0.13%) | | `position_head` | -42.86% / -42.25% / -45.70% | +0.00% / +0.00% / +6.34% | -42.86% (+0.00%) | | `position_mid` | -42.86% / -42.25% / -39.74% | +0.00% / +6.34% / +13.38% | -42.86% (+0.00%) | | `catch_up_page` | -3.29% / -2.33% / +7.22% | +2.04% / +6.89% / +3.49% | -3.23% (+1.15%) | | `startup_1m_file_evicted` | -7.55% / -95.58% / -93.47% | -30.69% / -95.89% / -95.75% | -9.14% (-37.88%) | | `startup_healthz` | -0.31% / +8.70% / +7.37% | +0.06% / -2.36% / -3.47% | -1.29% (-0.13%) | | `startup_healthz_existing` | +0.45% / -6.22% / +13.11% | +0.64% / +3.81% / +1.03% | +0.13% (+1.06%) | | `first_token_existing` | +5.42% / +5.42% / +5.42% | +9.75% / +9.75% / +9.75% | +8.13% (+2.58%) | | `first_token_fresh` | +50.07% / +50.07% / +50.07% | +0.06% / +0.06% / +0.06% | +49.61% (+0.66%) | | `notes_save` | -0.45% / -2.93% / -1.74% | +0.58% / -0.75% / +5.32% | -0.37% (+0.77%) | | `save_ack` | -0.03% / -0.93% / -1.77% | +0.76% / -0.05% / -4.41% | +0.10% (+0.17%) | | `sync_round_trip` | +1.15% / +0.06% / +3.06% | +0.64% / +1.46% / +0.49% | +1.42% (+2.36%) | | `reconnect_ready` | +1.74% / +0.66% / +2.20% | -0.90% / -0.00% / -7.28% | +0.76% (+0.13%) | ### Boot (startup only, 40 obs/arm; candidate server with and without `--secrets`) | Stage | Metric | Paired p50, candidate (control) | |---|---|---:| | boot_cand | `first_token_existing` | -0.89% (-2.44%) | | boot_cand | `first_token_fresh` | +50.92% (+0.94%) | | boot_cand | `startup_healthz` | +1.30% (-1.21%) | | boot_cand | `startup_healthz_existing` | +0.25% (+0.54%) | | boot_nosecrets | `first_token_existing` | +2.68% (-0.31%) | | boot_nosecrets | `first_token_fresh` | +2.68% (-0.49%) | | boot_nosecrets | `startup_healthz` | -0.73% (+0.19%) | | boot_nosecrets | `startup_healthz_existing` | +0.42% (-0.68%) | ### Soak (200 ms cadence, 180 s per arm, 900 saves, 0 retries each) | Arm | p50 | p95 | p99 | max | |---|---:|---:|---:|---:| | Dev A | 20.822 | 26.841 | 34.528 | 45.725 ms | | Candidate | 20.884 | 27.450 | 38.146 | 88.438 ms | | Dev B | 20.730 | 28.219 | 36.947 | 52.013 ms | ### Gates at `ee2bc32dc` - Rust (dev-01): `cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast`: PASS, 1,691 passed, 22 ignored. - Chromium 1440 + 390 save/sync specs (`editor`, `big-paste`, `traps`, `list-catch-up`): PASS, 44 passed, 8 skipped. - WebKit 1440 + 390, same specs (dev-02): PASS, 36 passed, 16 skipped. ### Open - **Soak max:** the candidate is still the highest arm in every quiet session (88.4 ms vs dev 45.7/52.0; p99 +1.2 ms over the slower dev arm; p50/p95 between the dev arms). One sample in 900, but consistent. Not yet explained. - `first_token_fresh` +50% (about +20 ms) is the required key and witness durability, moved from startup to the first token. - Fresh `/healthz`: the room phase is flat (−1.29% vs −0.13%); the boot phase reads +1.30% vs −1.21% with `--secrets` and flat without. No syscall difference was found, so this looks like noise, but it is reported as measured.
kayg commented 2026-11-10 05:02:51 +00:00
Author
Owner
Copy link

Soak max: cause and plan

Diagnosis only; no code changed for this. Head ee2bc32dc, dev 2fa9e18fb.

How it was measured

One FIFO turn (perf-turn p928), perf-01 then perf-02 locks held 04:43–05:01 UTC 2026-11-10. Four 180 s soak arms, interleaved dev A, candidate 1, dev B, candidate 2, at a 200 ms cadence (900 saves per arm, 0 retries). Same release servers as comment 89376 (server-cand df8b1a1e…, server-dev 883e18bd…).

  • Probe (disposable, not in the repo): per save it logs the typing-ack time, notes.save start and end (Unix µs), and the size of every *-wal file under the state folder.
  • Server: every arm runs under a static strace --seccomp-bpf -f -ttt -T -y -e trace=fsync,fdatasync,openat. Only those calls stop the process, and -y names the file behind each fd. The overhead is the same for both builds (p50 about 22 ms in every arm, vs 20.8 ms untraced).
  • Join: a script lines up each save window with every sync that ran inside it, and each witness sync (secrets/users/<u>/head and its first-install files) with the saves around it.

Finding

Arm Saves p50 max Witness syncs Witness syncs overlapping a save
Dev A 900 22.33 ms 89.36 ms 0 n/a
Candidate 1 900 21.94 ms 50.26 ms 10 0
Dev B 900 22.39 ms 86.82 ms 0 n/a
Candidate 2 900 22.50 ms 66.71 ms 10 0

  1. #928 adds no sync to any save. Each candidate arm has 10 witness syncs: 4 at first install (key tmp file, secrets folder, head, folder again) and 6 maximum-age flushes about 30 s apart. Each takes 3.3–7.1 ms. None overlaps a save window. The next save starts 75.5–267.5 ms after a witness sync ends. Every other sync class has the same count in both builds (for example 2,731 history WAL and 904 note tmpfile syncs per arm), and the journal WAL synced 3 times per arm in both.

  2. What the slowest save waits on, in every arm, is dev's own save chain on a shared-disk burst. Every save runs four syncs in series:

    • history WAL fdatasync;
    • note tmpfile fsync;
    • Notes/ folder fsync;
    • history WAL fdatasync.

    Normally that's about 3.4 + 9.6 + 6.6 + 3.4 ms. The worst saves are the same chain with each sync stretched by perf-01/perf-02's shared disk. Dev B save 100 took 86.8 ms: tmpfile 41.9 ms, folder 21.0 ms, history WAL 21.1 + 21.1 ms. Dev A save 765 took 89.4 ms: 33.1 + 20.9 + 33.0 ms. Candidate 2's worst (save 329, 66.7 ms) is the same chain at 31.4 + 21.6 + 10.7 ms.

  3. There is also one deterministic slow save per run, on dev and candidate alike. Save 93 in all four arms runs a meta.db WAL checkpoint inside the save. meta.db-wal is at 4,128,272 bytes (SQLite's 1000-page auto-checkpoint), so the save adds a meta.db-wal fdatasync (14.7–17.8 ms) and a meta.db fdatasync (6.6–10.0 ms), giving 48–59 ms. This is outside #928 and present on dev.

  4. The earlier candidate-only max (88.4 ms vs dev 45.7/52.0 ms in comment 89376) was this same shared-disk burst landing on the candidate arm in that session. With one 180 s arm per side, a single burst decides which arm "has the max". In this session two bursts landed on the dev arms.

Proposed change (one, not implemented)

Change the soak gate, not #928's code: judge notes_save max per arm against the dev arms' range over at least three interleaved arms per side (ABABAB), and keep p50/p95/p99 as the main gate. When a candidate arm exceeds the dev range, run this strace join and accept only if the candidate adds a sync inside a save window.

  • Expected effect: the gate stops failing on one host-disk burst. On today's data the candidate is inside the dev range (50.3/66.7 ms vs dev 86.8/89.4 ms) with 0 overlapping witness syncs, so #928's soak passes. Real regressions, such as a flush overlapping saves or an extra sync per save, still fail, because the join counts syncs inside saves directly.
  • Risk: a real but rare stall could hide inside a wide dev range. The join check covers the mechanism we care about (extra or overlapping syncs) but not CPU-only stalls; those show in p99 and in save_ack, which stay gated. Cost: three soak arms per side add about 9 minutes per turn.

A separate, dev-wide follow-up (its own issue, not #928): the deterministic in-save meta.db checkpoint (finding 3) could move to a background passive checkpoint below the 4 MB threshold, taking about 20–25 ms off one save in every ~90 at this cadence.

first_token_fresh +50% (+20 ms): accept or decline

  • What it is: on a fresh install, the first request that opens a user's store waits for that user's sync-token key and high-water witness to be written durably. Paired median: dev 40.6 ms, candidate 60.9 ms; boot phase +50.9% (control +0.9%). It is one token.key atomic write with fsyncs plus the first witness slot write, fdatasync and folder sync, the four first-install witness-class syncs above at about 6.5 ms each.
  • When it happens: once per user, on the first store open after install, never again. Restarts and later opens are flat: startup_healthz_existing +0.13% (control +1.06%), first_token_existing −0.89% (control −2.44%) in the boot phase. /healthz does not wait for it (strace: no secrets-folder access before or within 2 s after listen).
  • Why it can't be removed: #918 requires that no signed token leaves the server before its key and witness are durable. Otherwise a crash right after the first token could leave a client holding a token that a fresh key cannot verify, or let a restore go undetected.
  • The question for kayg: accept a one-time +20 ms on each user's very first request after a fresh install, or decline. Declining means signed tokens can't ship as designed. The only alternative is to hand out unsigned tokens until the first background flush, which weakens #918 for that window.
## Soak max: cause and plan Diagnosis only; no code changed for this. Head `ee2bc32dc`, dev `2fa9e18fb`. ### How it was measured One FIFO turn (`perf-turn p928`), perf-01 then perf-02 locks held 04:43–05:01 UTC 2026-11-10. Four 180 s soak arms, interleaved dev A, candidate 1, dev B, candidate 2, at a 200 ms cadence (900 saves per arm, 0 retries). Same release servers as comment 89376 (`server-cand` df8b1a1e…, `server-dev` 883e18bd…). - **Probe (disposable, not in the repo):** per save it logs the typing-ack time, `notes.save` start and end (Unix µs), and the size of every `*-wal` file under the state folder. - **Server:** every arm runs under a static `strace --seccomp-bpf -f -ttt -T -y -e trace=fsync,fdatasync,openat`. Only those calls stop the process, and `-y` names the file behind each fd. The overhead is the same for both builds (p50 about 22 ms in every arm, vs 20.8 ms untraced). - **Join:** a script lines up each save window with every sync that ran inside it, and each witness sync (`secrets/users/<u>/head` and its first-install files) with the saves around it. ### Finding | Arm | Saves | p50 | max | Witness syncs | Witness syncs overlapping a save | |---|---:|---:|---:|---:|---:| | Dev A | 900 | 22.33 ms | **89.36 ms** | 0 | n/a | | Candidate 1 | 900 | 21.94 ms | **50.26 ms** | 10 | **0** | | Dev B | 900 | 22.39 ms | **86.82 ms** | 0 | n/a | | Candidate 2 | 900 | 22.50 ms | **66.71 ms** | 10 | **0** | 1. **#928 adds no sync to any save.** Each candidate arm has 10 witness syncs: 4 at first install (key tmp file, secrets folder, `head`, folder again) and 6 maximum-age flushes about 30 s apart. Each takes 3.3–7.1 ms. None overlaps a save window. The next save starts 75.5–267.5 ms after a witness sync ends. Every other sync class has the same count in both builds (for example 2,731 history WAL and 904 note tmpfile syncs per arm), and the journal WAL synced 3 times per arm in both. 2. **What the slowest save waits on, in every arm, is dev's own save chain on a shared-disk burst.** Every save runs four syncs in series: - history WAL `fdatasync`; - note tmpfile `fsync`; - `Notes/` folder `fsync`; - history WAL `fdatasync`. Normally that's about 3.4 + 9.6 + 6.6 + 3.4 ms. The worst saves are the same chain with each sync stretched by perf-01/perf-02's shared disk. Dev B save 100 took 86.8 ms: tmpfile 41.9 ms, folder 21.0 ms, history WAL 21.1 + 21.1 ms. Dev A save 765 took 89.4 ms: 33.1 + 20.9 + 33.0 ms. Candidate 2's worst (save 329, 66.7 ms) is the same chain at 31.4 + 21.6 + 10.7 ms. 3. **There is also one deterministic slow save per run, on dev and candidate alike.** Save 93 in all four arms runs a `meta.db` WAL checkpoint inside the save. `meta.db-wal` is at 4,128,272 bytes (SQLite's 1000-page auto-checkpoint), so the save adds a `meta.db-wal` `fdatasync` (14.7–17.8 ms) and a `meta.db` `fdatasync` (6.6–10.0 ms), giving 48–59 ms. This is outside #928 and present on dev. 4. **The earlier candidate-only max** (88.4 ms vs dev 45.7/52.0 ms in comment 89376) was this same shared-disk burst landing on the candidate arm in that session. With one 180 s arm per side, a single burst decides which arm "has the max". In this session two bursts landed on the dev arms. ### Proposed change (one, not implemented) **Change the soak gate, not #928's code:** judge `notes_save` max per arm against the dev arms' range over at least three interleaved arms per side (ABABAB), and keep p50/p95/p99 as the main gate. When a candidate arm exceeds the dev range, run this strace join and accept only if the candidate adds a sync inside a save window. - **Expected effect:** the gate stops failing on one host-disk burst. On today's data the candidate is inside the dev range (50.3/66.7 ms vs dev 86.8/89.4 ms) with 0 overlapping witness syncs, so #928's soak passes. Real regressions, such as a flush overlapping saves or an extra sync per save, still fail, because the join counts syncs inside saves directly. - **Risk:** a real but rare stall could hide inside a wide dev range. The join check covers the mechanism we care about (extra or overlapping syncs) but not CPU-only stalls; those show in p99 and in `save_ack`, which stay gated. Cost: three soak arms per side add about 9 minutes per turn. A separate, dev-wide follow-up (its own issue, not #928): the deterministic in-save `meta.db` checkpoint (finding 3) could move to a background passive checkpoint below the 4 MB threshold, taking about 20–25 ms off one save in every ~90 at this cadence. ## first_token_fresh +50% (+20 ms): accept or decline - **What it is:** on a fresh install, the first request that opens a user's store waits for that user's sync-token key and high-water witness to be written durably. Paired median: dev 40.6 ms, candidate 60.9 ms; boot phase +50.9% (control +0.9%). It is one `token.key` atomic write with fsyncs plus the first witness slot write, `fdatasync` and folder sync, the four first-install witness-class syncs above at about 6.5 ms each. - **When it happens:** once per user, on the first store open after install, never again. Restarts and later opens are flat: `startup_healthz_existing` +0.13% (control +1.06%), `first_token_existing` −0.89% (control −2.44%) in the boot phase. `/healthz` does not wait for it (strace: no secrets-folder access before or within 2 s after `listen`). - **Why it can't be removed:** #918 requires that no signed token leaves the server before its key and witness are durable. Otherwise a crash right after the first token could leave a client holding a token that a fresh key cannot verify, or let a restore go undetected. - **The question for kayg:** accept a one-time +20 ms on each user's very first request after a fresh install, or decline. Declining means signed tokens can't ship as designed. The only alternative is to hand out unsigned tokens until the first background flush, which weakens #918 for that window.
kayg commented 2026-11-10 05:12:16 +00:00
Author
Owner
Copy link

READY pending kayg's first_token_fresh decision

Head ee2bc32dc. The remaining perf item is closed:

  • Soak max. The orchestrator accepted the measurement change, written down in #1050: the max is judged against the dev arms' range over ≥3 interleaved arms per side, with the sync-overlap check above that range. By that rule #928 passes. Comment 89399: candidate maxima 50.3/66.7 ms inside dev's 86.8/89.4 ms, 0 witness syncs inside any save.
  • Gates at ee2bc32dc (comment 89376): Rust fmt, clippy and workspace tests pass (1,691 passed); Chromium 1440/390 save/sync 44 passed; WebKit 1440/390 36 passed. Everything else in the matrix is flat or better than dev.

One decision is open, for kayg: first_token_fresh +50% (about +20 ms), once per user on the first store open after a fresh install, to make the token key and witness durable before the first signed token (#918). The details are in comment 89399. Not merged; waiting on that answer.

The in-save SQLite checkpoints found during the diagnosis are dev-wide and tracked separately in #1051.

## READY pending kayg's `first_token_fresh` decision Head `ee2bc32dc`. The remaining perf item is closed: - **Soak max.** The orchestrator accepted the measurement change, written down in #1050: the max is judged against the dev arms' range over ≥3 interleaved arms per side, with the sync-overlap check above that range. By that rule #928 passes. Comment 89399: candidate maxima 50.3/66.7 ms inside dev's 86.8/89.4 ms, 0 witness syncs inside any save. - **Gates at `ee2bc32dc`** (comment 89376): Rust fmt, clippy and workspace tests pass (1,691 passed); Chromium 1440/390 save/sync 44 passed; WebKit 1440/390 36 passed. Everything else in the matrix is flat or better than dev. One decision is open, for kayg: `first_token_fresh` +50% (about +20 ms), once per user on the first store open after a fresh install, to make the token key and witness durable before the first signed token (#918). The details are in comment 89399. Not merged; waiting on that answer. The in-save SQLite checkpoints found during the diagnosis are dev-wide and tracked separately in #1051.
kayg commented 2026-11-10 09:43:25 +00:00
Author
Owner
Copy link

READY: head e5559455f, rebased onto dev fe2efbf11

kayg accepted first_token_fresh (+20 ms once per user on a fresh install, 11-10). The soak gate is decided (#1050). Nothing open.

Rebase: 104 commits replayed onto fe2efbf11. crates/server/src/live.rs conflicted with dev's credential revalidation during catch-up (#1048 line). Resolution: catch_up keeps dev's auth argument and signed-out early return, and also returns the signed page token, as Result<Option<(i64, String)>>. The ready message sends that token; the lag path keeps the position only. One branch commit (5e68c649e, a crash-typing test fix) was dropped because dev's own fix supersedes it; that file now equals dev's. The PR diff is 39 files, the same as before the rebase. The branch merges cleanly into dev.

Gates at e5559455f:

Gate Host Result
cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast dev-01 (x86_64) PASS: 1,731 passed, 0 failed, 22 ignored
Save/sync specs (editor, big-paste, traps, list-catch-up), chromium-1440 + chromium-390 dev-01 PASS: 44 passed, 8 skipped
admin-sharing.spec.ts, admin-chromium-1440 + admin-chromium-390, --repeat-each=2 dev-01 PASS: 21 passed
Save/sync specs, webkit-1440 + webkit-390 dev-02 (aarch64) PASS: 36 passed, 16 skipped
admin-sharing.spec.ts, admin-webkit-1440 + admin-webkit-390, --repeat-each=2 dev-02 PASS: 21 passed

Perf: no code changed in the rebase apart from the live.rs merge, which only combines the two return values. The last matrix (comment 89376) and the soak diagnosis (comment 89399) stand.

## READY: head `e5559455f`, rebased onto dev `fe2efbf11` kayg accepted `first_token_fresh` (+20 ms once per user on a fresh install, 11-10). The soak gate is decided (#1050). Nothing open. **Rebase:** 104 commits replayed onto `fe2efbf11`. `crates/server/src/live.rs` conflicted with dev's credential revalidation during catch-up (#1048 line). Resolution: `catch_up` keeps dev's `auth` argument and signed-out early return, and also returns the signed page token, as `Result<Option<(i64, String)>>`. The ready message sends that token; the lag path keeps the position only. One branch commit (`5e68c649e`, a crash-typing test fix) was dropped because dev's own fix supersedes it; that file now equals dev's. The PR diff is 39 files, the same as before the rebase. The branch merges cleanly into dev. **Gates at `e5559455f`:** | Gate | Host | Result | |---|---|---| | `cargo fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace --no-fail-fast` | dev-01 (x86_64) | PASS: 1,731 passed, 0 failed, 22 ignored | | Save/sync specs (`editor`, `big-paste`, `traps`, `list-catch-up`), `chromium-1440` + `chromium-390` | dev-01 | PASS: 44 passed, 8 skipped | | `admin-sharing.spec.ts`, `admin-chromium-1440` + `admin-chromium-390`, `--repeat-each=2` | dev-01 | PASS: 21 passed | | Save/sync specs, `webkit-1440` + `webkit-390` | dev-02 (aarch64) | PASS: 36 passed, 16 skipped | | `admin-sharing.spec.ts`, `admin-webkit-1440` + `admin-webkit-390`, `--repeat-each=2` | dev-02 | PASS: 21 passed | Perf: no code changed in the rebase apart from the `live.rs` merge, which only combines the two return values. The last matrix (comment 89376) and the soak diagnosis (comment 89399) stand.
kayg commented 2026-11-10 10:09:22 +00:00
Author
Owner
Copy link

Merged into dev (4274dafb4, integration merge with #1043/#928/#917/#934; clippy -D warnings + workspace tests green on dev-01). Closing.

Merged into dev (4274dafb4, integration merge with #1043/#928/#917/#934; clippy -D warnings + workspace tests green on dev-01). Closing.
kayg closed this pull request 2026-11-10 10:09:23 +00:00
Some checks failed
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!928
No description provided.