kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

test(notes): make long-note crash coverage independent of throughput #844

Closed
kayg wants to merge 0 commits from fix/notes-crash-deterministic into dev
pull from: fix/notes-crash-deterministic
merge into: kayg:dev
Conversation 3 Commits - Files changed -
kayg commented 2026-11-02 08:57:03 +00:00
Owner
Copy link

The long-note crash test could fail after 12 random kill windows despite preserving every confirmed append (96 under load). Keep at least 12 kill/restart rounds and continue until at least 100 appends are confirmed, with a 180-second stalled-progress cap and count diagnostics. Every retry, no-loss, no-double and saved-file assertion is retained. The test is unchanged on dev, so this is a separate source PR. No functional speed budget is introduced.

Validation: BM8 validation at scratch b9dac07806a398f53f8a9acaf1506d4b6f718a30: cargo fmt --all -- --check and cargo test --workspace -j4 passed on dev-01 in the standard Nix/slot shell, 1,483 passed and 22 ignored, with no skips or test-thread override. Four report selectors passed 22/22 on Chromium/WebKit, zero retries. Final production code passed 30 fresh Chromium composer invocations (90 tests, zero retries) during one ordinary slot build. Exact commands, head provenance, append samples, retained failures and attached raw evidence: #844 (comment). No merge or train change; owned jobs finished and VM targets/temp Homes cleaned.

The long-note crash test could fail after 12 random kill windows despite preserving every confirmed append (96 under load). Keep at least 12 kill/restart rounds and continue until at least 100 appends are confirmed, with a 180-second stalled-progress cap and count diagnostics. Every retry, no-loss, no-double and saved-file assertion is retained. The test is unchanged on dev, so this is a separate source PR. No functional speed budget is introduced. Validation: BM8 validation at scratch `b9dac07806a398f53f8a9acaf1506d4b6f718a30`: `cargo fmt --all -- --check` and `cargo test --workspace -j4` passed on dev-01 in the standard Nix/slot shell, 1,483 passed and 22 ignored, with no skips or test-thread override. Four report selectors passed 22/22 on Chromium/WebKit, zero retries. Final production code passed 30 fresh Chromium composer invocations (90 tests, zero retries) during one ordinary slot build. Exact commands, head provenance, append samples, retained failures and attached raw evidence: https://git.kayg.org/kayg/calternal/pulls/844#issuecomment-74911. No merge or train change; owned jobs finished and VM targets/temp Homes cleaned.
test(notes): run crash rounds until confirmed append coverage
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
50cce9dcce
kayg commented 2026-11-02 13:21:15 +00:00
Author
Owner
Copy link

bm8 preparation

The manual gates pass. Final scratch scratch/bm8-prep is b9dac07806a398f53f8a9acaf1506d4b6f718a30, in ~/Developer/Worktrees/bm8-prep. It reconstructs all eight BM7 heads in original order, preserving the prior Cargo.lock/execute.rs merge resolutions, then includes the source fixes below. The scratch is unpushed. No PR was merged and the train was not edited; #838 remains 57f5c6bbd72cabd1c166d7f147ccc591eb7428f4.

PR included head change
#824 f1c1aa961e169ef99e132063a97ccbd9a89a5def unchanged
#827 91a19dcf0a9cf3844861e0e6acf4318618f3ea33 prior capability fix retained
#829 d455842942731aabc05474c7929597d57090a7cc unchanged
#835 19071be8455d718822ba98f6b9a402c5fd345242 unchanged
#828 5d7b9b36cd008de78851809724c90a85d92b40a7 placed stamps and owned initialization
#833 ccaece3dfecbe2048634fc792d7d4a2c6e31f324 explicit indexed fixture dates
#836 5f7d1a59776411e7f3353d0023817b43affae51e unchanged
#831 316b43e9578c3e4d2944df8df43caebc611356f7 unchanged; standard shell now supplies strace
#844 50cce9dcceb87f23a9f346300b3336e315ff6c0c new: deterministic crash coverage, into dev
#847 40448327a15730271d56226b898e48bcc87b2a4d new: daily capture/history race, into dev
#855 c38c3083d51f4adcf5cd76a70f1f6960b427413e new: transient Undo fixture, into dev
#857 144e717145e35f70356f593b7c04a527e00f7b17 new: idle upload completion fixture, into dev
#779 7b97e4d14231be07f6a80cb9ca172d02442c487e shared shell overlay: Linux-only strace

#779's flake files are a shared, uncommitted VM overlay, following BRIEF-COMMON.md. They are not merged into the scratch. All source changes are committed and pushed to their own branches. New PRs are #844, #847, #855 and #857, all into dev.

source decisions

Crash coverage (#844). The test is unchanged on dev. It now runs at least 12 kill/restart rounds and continues until at least 100 confirmed appends, retaining all no-loss, no-double, retry and saved-file assertions. A 180-second stalled-progress cap reports rounds and confirmed appends. It no longer treats work completed in random kill windows as a throughput requirement.

Standard shell (#779). pkgs.strace is added only to the Linux devShell packages. The standard shell resolves strace on both Linux architectures, and the existing attachment flush syscall test passes. #831 requires no source change.

Upcoming fixture (#833). The tasks writes v2 plan (docs/plans/tasks-v2-set-status.md) has no kept-first rule. ADR 0066 and the implementation use normal date/time/priority/created/id sorting. The fixture now supplies distinct note date: values, which feed indexed created_ms, and asserts both indexed timestamps. The earlier created: fixture property was the wrong field and could still pass by random id order; that iteration and its failed full run are preserved. Product sorting is unchanged.

Daily capture race (#847). The original 30-run Chromium batch reproduced the line-58 done:1 failure twice, during ordinary build load. A concurrent reader can create history for an unwritten daily placeholder between capture's history check and private placement, causing the second seed to fail with “already has history.” This path is unchanged on dev. Capture now opens the placeholder's existing room and uses its existing history through daily_seed. A deterministic paused-reader regression fails on dev and passes with the fix, checking the same lineage, both completed steps, one captured line and exact saved bytes. The 13 task-create regressions also pass.

List initialization cancellation (#828). The stronger timeout test observes the original owned initializer rather than issuing a replacement request. Under the full suite it exposed note list: database is locked: cancellation of a request-owned OnceCell initializer released its permit while its blocking SQLite open continued, allowing a second open. List initialization now belongs to Notes' existing initialization task set; callers own only result observers. Shutdown closes admission before draining that set, and failed opens remain retryable. A deterministic aborted-caller regression fails before the fix with two schema opens and passes afterward with exactly one and the same cached Arc. All 14 restart-listing tests pass. The prior ctime/placed-stamp fix is retained. Plan and failure table: #828 (comment).

Undo fixture (#855). WebKit traces showed Playwright waiting over seven seconds for spring stability before hover, outliving the eight-second Undo toast. These functional creation/Undo tests now select the supported reduced-motion mode, then hover to hold the normal clock while server assertions run. Real clicks and every content/search/snippet/Undo assertion remain. Product motion, lifetime and retry settings are unchanged.

Idle upload fixture (#857). An unchanged baseline auth test slept 31 seconds after observing only two pending files, then assumed every admitted upload's idle timer had finished. The full run found four still pending. It now observes all upload tasks completing with a 90-second failure cap and preserves response statuses, per-person admission, cleanup, sign-in and worker checks. The worker probe uses a 30-second liveness cap instead of a 100 ms speed assertion. Product upload code/timeouts are unchanged.

gates and results

All heavy commands ran on the VMs through ~/bin/slot and Nix, with CI=1 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0. No build, test, install or Rust tooling ran on the control host.

# dev-01, exact final scratch; standard shared Nix shell
~/bin/slot nix develop -c bash ~/bm8-prep-report/workspace-clean.sh
# script executes cargo test; formatting checked separately in the same shell:
cargo fmt --all -- --check
cargo test --workspace -j4

Full workspace: exit 0, 1,483 passed, 22 ignored, no skips or test-thread override supplied. The crash, attachment strace, owned-initialization and indexed Upcoming regressions are included. Existing ignored performance/platform tests remain ignored by their own declarations.

# dev-01, apps/web; CALTERNAL_BIN points to the final production binary
# 30 separate invocations, each with fresh servers/Homes:
bunx playwright test e2e/composer-tasks.spec.ts   --project=chromium-1440 --workers=1 --retries=0 --reporter=line,json
# one ordinary background job, in a separate owned target tree:
~/bin/slot cargo build --workspace --all-targets --release -j4

# requested four selectors:
bunx playwright test e2e/composer-tasks.spec.ts e2e/empty.spec.ts   e2e/glass.spec.ts:341 e2e/link-suggest.spec.ts   --project=chromium-1440 --project=webkit-1440   --workers=1 --retries=0 --reporter=line,json

Composer: 30/30 invocations, 90 tests passed, zero retries. The ordinary build was active throughout every invocation, verified from individual start/finish records. Its interval was 17:04:03–17:26:38 IST on November 2. It was cancelled through its own systemd unit after reproduction and the accompanying matrix; no successful release-build gate is claimed.

Four-selector matrix: 22/22 passed, zero retries, Chromium 1440 and WebKit 1440. The passing matrix uses #855's final reduced-motion fixture.

Provenance: the benchmark and 30-run batch used 72149532, with final production code. The passing matrix used 74aa02f7; its web/package trees are identical to final scratch. Changes from 72149532 to final scratch are confined to the composer e2e fixture and Rust test fixtures. The Rust gate itself ran at exact final head b9dac078.

Focused source commands also passed:

cargo test -p calternal-notes -j4 restart_listing -- --nocapture
cargo test -p calternal-notes -j4 daily_capture_keeps_history_opened_by_a_concurrent_reader -- --nocapture
cargo test -p calternal-tasks -j4 create -- --nocapture
cargo test -p calternal-tasks -j4 upcoming_uses_when_else_deadline_and_orders_days_before_months -- --nocapture
cargo test -p calternal-auth --test reliability_round3 -j4 attachment_slow_bodies_must_not_hold_all_general_blocking_workers -- --nocapture

200-append comparison on dev-01

Three alternating rounds per revision, 200 sequential acknowledged HTTP appends, fresh 1,000-line note, 15 ms save quiet period and identical debug profiles. Startup, note creation and final verification are excluded. Every appended token is independently checked exactly once. Script and binary hashes are in append-attribution-final.py and append-attribution-final.json.

revision round 1 (s) round 2 (s) round 3 (s) median (s)
baseline 55f35fcd 2.291 1.557 2.008 2.008
#831 316b43e9 2.267 1.525 1.904 1.904
#828 5d7b9b36 1.948 1.820 1.944 1.944
final production tree 2.427 2.001 2.269 2.269

The combined scratch median is 13.0% higher than baseline. Neither isolated #828 nor #831 has a higher median in this final batch. These shared-VM diagnostics do not establish which change caused the combined difference, and are not performance qualification. Earlier samples and their varying paired differences are retained in the history and raw archives.

retained failures and practical limits

The latest full command is green; earlier attempts are not discarded or relabelled. They include the initial #828 completion helper, baseline upload admission, markdown's 50 ms parse success assertion, baseline tag-rename/shutdown settlement, the confirmed list-open cancellation race, the first incorrect Upcoming fixture, the idle-upload fixed sleep and an unchanged 80 ms session-expiry fixture. That session test expired before its first expected-200 assertion, then passed in the identical-head full rerun. A second dev-02 attempt reported the unchanged delete-event/read and shutdown-settlement 30-second watchdogs, then stalled in the save/crash test's restarted runtime. Native stacks and buffered panic excerpts were preserved before stopping that owned unit. The cause of these intermittent waits is unconfirmed; focused follow-up results are retained separately: exact outside_delete passed in 14.77 seconds and exact save/crash passed in 0.43 seconds on the same dev-02 tree. These failures remain an observed risk; this task does not claim to eliminate them all.

Browser history retains the two original daily-capture failures, one pre-test port collision in an earlier batch (with its replacement), and WebKit Undo expiry failures before #855 was complete. The final 30-run batch had no failed invocation or port collision. Details and per-attempt commands are in history.md.

evidence and closeout

  • dev01-final.tar.gz: final append logs/hashes, 30 individual composer JSON/log/exit records, exact build-load interval, passing browser JSON, all earlier browser traces and forensic capture data.
  • dev01-list-race-evidence.tar.gz: deterministic list-open red/green evidence and the failed full-suite list-open error.
  • dev01-rust-final.tar.gz: successful final full Rust stdout, exit/head and formatting result.
  • dev02-final.tar.gz: retained earlier full/focused runs, stopped save/crash wait, native stacks and buffered panic excerpts.
  • Earlier snapshots remain in dev01-before-final.tar.gz, dev01-browser-final.tar.gz, dev01-intermediate.tar.gz and dev02.tar.gz.

Owned VM targets, copied binaries, bundles, temporary Homes, dist/results and installed node_modules were removed after archiving. Both VMs were checked for running BM8 units; none remain. Control worktrees and small evidence remain. Source heads and the untouched train were verified through Forgejo. Manual results above are separate from Forgejo CI/review state:

  • #779: checks no commit status reported; reviews none.
  • #828: checks no commit status reported; reviews none.
  • #833: checks no commit status reported; reviews none.
  • #844: checks pending; reviews none.
  • #847: checks no commit status reported; reviews none.
  • #855: checks no commit status reported; reviews none.
  • #857: checks pending; reviews none.
# bm8 preparation The manual gates pass. Final scratch `scratch/bm8-prep` is `b9dac07806a398f53f8a9acaf1506d4b6f718a30`, in `~/Developer/Worktrees/bm8-prep`. It reconstructs all eight BM7 heads in original order, preserving the prior Cargo.lock/execute.rs merge resolutions, then includes the source fixes below. The scratch is unpushed. No PR was merged and the train was not edited; #838 remains `57f5c6bbd72cabd1c166d7f147ccc591eb7428f4`. | PR | included head | change | |---|---|---| | [#824](https://git.kayg.org/kayg/calternal/pulls/824) | `f1c1aa961e169ef99e132063a97ccbd9a89a5def` | unchanged | | [#827](https://git.kayg.org/kayg/calternal/pulls/827) | `91a19dcf0a9cf3844861e0e6acf4318618f3ea33` | prior capability fix retained | | [#829](https://git.kayg.org/kayg/calternal/pulls/829) | `d455842942731aabc05474c7929597d57090a7cc` | unchanged | | [#835](https://git.kayg.org/kayg/calternal/pulls/835) | `19071be8455d718822ba98f6b9a402c5fd345242` | unchanged | | [#828](https://git.kayg.org/kayg/calternal/pulls/828) | `5d7b9b36cd008de78851809724c90a85d92b40a7` | placed stamps and owned initialization | | [#833](https://git.kayg.org/kayg/calternal/pulls/833) | `ccaece3dfecbe2048634fc792d7d4a2c6e31f324` | explicit indexed fixture dates | | [#836](https://git.kayg.org/kayg/calternal/pulls/836) | `5f7d1a59776411e7f3353d0023817b43affae51e` | unchanged | | [#831](https://git.kayg.org/kayg/calternal/pulls/831) | `316b43e9578c3e4d2944df8df43caebc611356f7` | unchanged; standard shell now supplies strace | | [#844](https://git.kayg.org/kayg/calternal/pulls/844) | `50cce9dcceb87f23a9f346300b3336e315ff6c0c` | new: deterministic crash coverage, into dev | | [#847](https://git.kayg.org/kayg/calternal/pulls/847) | `40448327a15730271d56226b898e48bcc87b2a4d` | new: daily capture/history race, into dev | | [#855](https://git.kayg.org/kayg/calternal/pulls/855) | `c38c3083d51f4adcf5cd76a70f1f6960b427413e` | new: transient Undo fixture, into dev | | [#857](https://git.kayg.org/kayg/calternal/pulls/857) | `144e717145e35f70356f593b7c04a527e00f7b17` | new: idle upload completion fixture, into dev | | [#779](https://git.kayg.org/kayg/calternal/pulls/779) | `7b97e4d14231be07f6a80cb9ca172d02442c487e` | shared shell overlay: Linux-only strace | #779's flake files are a shared, uncommitted VM overlay, following BRIEF-COMMON.md. They are not merged into the scratch. All source changes are committed and pushed to their own branches. New PRs are #844, #847, #855 and #857, all into dev. ## source decisions **Crash coverage (#844).** The test is unchanged on dev. It now runs at least 12 kill/restart rounds and continues until at least 100 confirmed appends, retaining all no-loss, no-double, retry and saved-file assertions. A 180-second stalled-progress cap reports rounds and confirmed appends. It no longer treats work completed in random kill windows as a throughput requirement. **Standard shell (#779).** `pkgs.strace` is added only to the Linux devShell packages. The standard shell resolves strace on both Linux architectures, and the existing attachment flush syscall test passes. #831 requires no source change. **Upcoming fixture (#833).** The tasks writes v2 plan (`docs/plans/tasks-v2-set-status.md`) has no kept-first rule. ADR 0066 and the implementation use normal date/time/priority/created/id sorting. The fixture now supplies distinct note `date:` values, which feed indexed `created_ms`, and asserts both indexed timestamps. The earlier `created:` fixture property was the wrong field and could still pass by random id order; that iteration and its failed full run are preserved. Product sorting is unchanged. **Daily capture race (#847).** The original 30-run Chromium batch reproduced the line-58 `done:1` failure twice, during ordinary build load. A concurrent reader can create history for an unwritten daily placeholder between capture's history check and private placement, causing the second seed to fail with “already has history.” This path is unchanged on dev. Capture now opens the placeholder's existing room and uses its existing history through `daily_seed`. A deterministic paused-reader regression fails on dev and passes with the fix, checking the same lineage, both completed steps, one captured line and exact saved bytes. The 13 task-create regressions also pass. **List initialization cancellation (#828).** The stronger timeout test observes the original owned initializer rather than issuing a replacement request. Under the full suite it exposed `note list: database is locked`: cancellation of a request-owned OnceCell initializer released its permit while its blocking SQLite open continued, allowing a second open. List initialization now belongs to Notes' existing initialization task set; callers own only result observers. Shutdown closes admission before draining that set, and failed opens remain retryable. A deterministic aborted-caller regression fails before the fix with two schema opens and passes afterward with exactly one and the same cached Arc. All 14 restart-listing tests pass. The prior ctime/placed-stamp fix is retained. Plan and failure table: https://git.kayg.org/kayg/calternal/pulls/828#issuecomment-74665. **Undo fixture (#855).** WebKit traces showed Playwright waiting over seven seconds for spring stability before hover, outliving the eight-second Undo toast. These functional creation/Undo tests now select the supported reduced-motion mode, then hover to hold the normal clock while server assertions run. Real clicks and every content/search/snippet/Undo assertion remain. Product motion, lifetime and retry settings are unchanged. **Idle upload fixture (#857).** An unchanged baseline auth test slept 31 seconds after observing only two pending files, then assumed every admitted upload's idle timer had finished. The full run found four still pending. It now observes all upload tasks completing with a 90-second failure cap and preserves response statuses, per-person admission, cleanup, sign-in and worker checks. The worker probe uses a 30-second liveness cap instead of a 100 ms speed assertion. Product upload code/timeouts are unchanged. ## gates and results All heavy commands ran on the VMs through `~/bin/slot` and Nix, with `CI=1 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0`. No build, test, install or Rust tooling ran on the control host. ```sh # dev-01, exact final scratch; standard shared Nix shell ~/bin/slot nix develop -c bash ~/bm8-prep-report/workspace-clean.sh # script executes cargo test; formatting checked separately in the same shell: cargo fmt --all -- --check cargo test --workspace -j4 ``` **Full workspace: exit 0, 1,483 passed, 22 ignored**, no skips or test-thread override supplied. The crash, attachment strace, owned-initialization and indexed Upcoming regressions are included. Existing ignored performance/platform tests remain ignored by their own declarations. ```sh # dev-01, apps/web; CALTERNAL_BIN points to the final production binary # 30 separate invocations, each with fresh servers/Homes: bunx playwright test e2e/composer-tasks.spec.ts --project=chromium-1440 --workers=1 --retries=0 --reporter=line,json # one ordinary background job, in a separate owned target tree: ~/bin/slot cargo build --workspace --all-targets --release -j4 # requested four selectors: bunx playwright test e2e/composer-tasks.spec.ts e2e/empty.spec.ts e2e/glass.spec.ts:341 e2e/link-suggest.spec.ts --project=chromium-1440 --project=webkit-1440 --workers=1 --retries=0 --reporter=line,json ``` **Composer: 30/30 invocations, 90 tests passed, zero retries.** The ordinary build was active throughout every invocation, verified from individual start/finish records. Its interval was 17:04:03–17:26:38 IST on November 2. It was cancelled through its own systemd unit after reproduction and the accompanying matrix; no successful release-build gate is claimed. **Four-selector matrix: 22/22 passed, zero retries**, Chromium 1440 and WebKit 1440. The passing matrix uses #855's final reduced-motion fixture. Provenance: the benchmark and 30-run batch used `72149532`, with final production code. The passing matrix used `74aa02f7`; its web/package trees are identical to final scratch. Changes from `72149532` to final scratch are confined to the composer e2e fixture and Rust test fixtures. The Rust gate itself ran at exact final head `b9dac078`. Focused source commands also passed: ```sh cargo test -p calternal-notes -j4 restart_listing -- --nocapture cargo test -p calternal-notes -j4 daily_capture_keeps_history_opened_by_a_concurrent_reader -- --nocapture cargo test -p calternal-tasks -j4 create -- --nocapture cargo test -p calternal-tasks -j4 upcoming_uses_when_else_deadline_and_orders_days_before_months -- --nocapture cargo test -p calternal-auth --test reliability_round3 -j4 attachment_slow_bodies_must_not_hold_all_general_blocking_workers -- --nocapture ``` ## 200-append comparison on dev-01 Three alternating rounds per revision, 200 sequential acknowledged HTTP appends, fresh 1,000-line note, 15 ms save quiet period and identical debug profiles. Startup, note creation and final verification are excluded. Every appended token is independently checked exactly once. Script and binary hashes are in `append-attribution-final.py` and `append-attribution-final.json`. | revision | round 1 (s) | round 2 (s) | round 3 (s) | median (s) | |---|---:|---:|---:|---:| | baseline `55f35fcd` | 2.291 | 1.557 | 2.008 | 2.008 | | #831 `316b43e9` | 2.267 | 1.525 | 1.904 | 1.904 | | #828 `5d7b9b36` | 1.948 | 1.820 | 1.944 | 1.944 | | final production tree | 2.427 | 2.001 | 2.269 | 2.269 | The combined scratch median is 13.0% higher than baseline. Neither isolated #828 nor #831 has a higher median in this final batch. These shared-VM diagnostics do not establish which change caused the combined difference, and are not performance qualification. Earlier samples and their varying paired differences are retained in the history and raw archives. ## retained failures and practical limits The latest full command is green; earlier attempts are not discarded or relabelled. They include the initial #828 completion helper, baseline upload admission, markdown's 50 ms parse success assertion, baseline tag-rename/shutdown settlement, the confirmed list-open cancellation race, the first incorrect Upcoming fixture, the idle-upload fixed sleep and an unchanged 80 ms session-expiry fixture. That session test expired before its first expected-200 assertion, then passed in the identical-head full rerun. A second dev-02 attempt reported the unchanged delete-event/read and shutdown-settlement 30-second watchdogs, then stalled in the save/crash test's restarted runtime. Native stacks and buffered panic excerpts were preserved before stopping that owned unit. The cause of these intermittent waits is unconfirmed; focused follow-up results are retained separately: exact outside_delete passed in 14.77 seconds and exact save/crash passed in 0.43 seconds on the same dev-02 tree. These failures remain an observed risk; this task does not claim to eliminate them all. Browser history retains the two original daily-capture failures, one pre-test port collision in an earlier batch (with its replacement), and WebKit Undo expiry failures before #855 was complete. The final 30-run batch had no failed invocation or port collision. Details and per-attempt commands are in `history.md`. ## evidence and closeout - `dev01-final.tar.gz`: final append logs/hashes, 30 individual composer JSON/log/exit records, exact build-load interval, passing browser JSON, all earlier browser traces and forensic capture data. - `dev01-list-race-evidence.tar.gz`: deterministic list-open red/green evidence and the failed full-suite list-open error. - `dev01-rust-final.tar.gz`: successful final full Rust stdout, exit/head and formatting result. - `dev02-final.tar.gz`: retained earlier full/focused runs, stopped save/crash wait, native stacks and buffered panic excerpts. - Earlier snapshots remain in `dev01-before-final.tar.gz`, `dev01-browser-final.tar.gz`, `dev01-intermediate.tar.gz` and `dev02.tar.gz`. Owned VM targets, copied binaries, bundles, temporary Homes, dist/results and installed node_modules were removed after archiving. Both VMs were checked for running BM8 units; none remain. Control worktrees and small evidence remain. Source heads and the untouched train were verified through Forgejo. Manual results above are separate from Forgejo CI/review state: - #779: checks no commit status reported; reviews none. - #828: checks no commit status reported; reviews none. - #833: checks no commit status reported; reviews none. - #844: checks pending; reviews none. - #847: checks no commit status reported; reviews none. - #855: checks no commit status reported; reviews none. - #857: checks pending; reviews none.
dev01-final.tar.gz
14 MiB
dev01-list-race-evidence.tar.gz
2.4 MiB
dev01-rust-final.tar.gz
337 KiB
dev02-final.tar.gz
127 KiB
append-attribution-final.json
3.7 KiB
report.md
13 KiB
kayg commented 2026-11-02 13:44:54 +00:00
Author
Owner
Copy link

bm8 cumulative append attribution on the perf VMs

first confirmed >3% step: #827 (91a19dcf), merged cumulatively at 199a938b. It adds 5.74% on perf-01 and 5.02% on perf-02 versus the preceding #824 step in the full matrix. A separate 20-run interleaved confirmation gives +3.48% and +4.75%. The 3% limit is unchanged; this boundary does not pass it.

The final scratch versus dev is +6.10% on perf-01 and +2.42% on perf-02 in the full matrix. The previously observed shared-dev +13% is not reproduced at that magnitude. This experiment attributes a repeatable incremental cost to #827, not the entire original 13% to one change.

workload and provenance

Recovered ~/bm8-prep-report/append-attribution-final.py from the original job. Preserved its 200 sequential acknowledged HTTP appends, fresh 1,000-line note, fresh Home per run, CALTERNAL_NOTES_SAVE_MS=15, --dev-user, HTTP client and exact-once token checks. Startup, note creation, verification and hashing are outside the timed interval. Ten runs per revision per VM; each round rotates case order and alternates direction. p90 is nearest-rank over the ten 200-append totals, not request latency. Every primary and confirmation sample passed the content checks.

Builds ran only on dev-01 through ~/bin/slot, with the shared Nix shell and the original unoptimized profile:

~/bin/slot nix develop -c bash -c 'CI=1 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0 cargo build -p calternal --bin calternal -j4'

All twelve binaries were relayed dev-01 → control host → each perf VM. sha256sum -c SHA256SUMS passed for every binary on both VMs, and each measured row independently records its matching binary hash. No builds or tests ran on the control host or perf VMs. The perf tool was fetched from Nix's binary cache with --max-jobs 0; no compilation was allowed there.

origin/dev remained 55f35fcde3bc91ca3781055577b93e93819f6cd4. The first nine cumulative steps reuse the original scratch's actual merge commits, including its conflict resolutions and the requested #828 fe46caf5. #833 uses the original step with head 158af347; later fixture fixes remain confined to the separate final scratch. #847 was merged directly on the #844 cumulative commit, producing 185dba5910eb2e09e3320d0996501019cc0299c1, without the later unrelated initialization follow-up. No existing branch was edited or pushed. PR #844 head remains 50cce9dcceb87f23a9f346300b3336e315ff6c0c.

Both VMs were idle before measurement (load approximately 0.00–0.02, no competing benchmark). Every measurement held the one shared lock:

# claude@calternal-perf-01-backup01 and claude@calternal-perf-02-backup01
~/bin/perf-lock bash ~/bm8-perf-attribution/run.sh
# run.sh checks disk, records uptime/processes, verifies SHA256SUMS, then:
python3 ~/bm8-perf-attribution/measure.py
# separate confirmation, also under the shared lock:
~/bin/perf-lock python3 ~/bm8-perf-attribution/followup.py

All primary and confirmation jobs exited 0: 240 primary runs plus 160 confirmation runs. These are performance-harness measurements, not functional-test wall-clock assertions. Disk headroom remained above the required thresholds.

full cumulative matrix

Each entry is seconds for 200 appends. Step delta compares the preceding row; the final scratch row compares the #847 cumulative row.

cumulative step head perf-01 median / p90 (s) step Δ perf-02 median / p90 (s) step Δ
base 55f35fcd 0.7443 / 0.7558 — 0.7777 / 0.8012 —
#824 53269288 0.7378 / 0.8195 -0.87% 0.7376 / 0.7529 -5.16%
#827 199a938b 0.7801 / 0.8153 +5.74% 0.7746 / 0.7871 +5.02%
#829 b42ee47d 0.7704 / 0.7818 -1.24% 0.7778 / 0.8267 +0.41%
#835 aa4c34a1 0.7598 / 0.7965 -1.38% 0.7781 / 0.7844 +0.03%
#828 9c1a3523 0.7824 / 0.7922 +2.98% 0.7749 / 0.7913 -0.40%
#833 4dc11d76 0.7626 / 0.7782 -2.54% 0.7689 / 0.7868 -0.79%
#836 b79c17ff 0.7723 / 0.7820 +1.28% 0.7809 / 0.8142 +1.56%
#831 42fbcd3a 0.7847 / 0.8048 +1.61% 0.7820 / 0.8091 +0.15%
#844 d49db9e2 0.7902 / 0.8141 +0.71% 0.7773 / 0.8107 -0.61%
#847 185dba59 0.7911 / 0.8621 +0.11% 0.8007 / 0.8663 +3.01%
scratch b9dac078 0.7897 / 0.8551 -0.18% 0.7966 / 0.8116 -0.51%

#836 and its preceding #833 step have identical binary hashes; #844 and #831 also have identical hashes. Their small observed differences show timing noise. #847 exceeds 3% only on perf-02 (+3.01%, versus +0.11% on perf-01), so this batch does not establish an additional repeatable regression there. #828 is +2.98% on perf-01 and -0.40% on perf-02; no confirmed >3% boundary at #828.

separate 20-run confirmation

revision perf-01 median / p90 (s) perf-02 median / p90 (s)
dev 0.7233 / 0.7437 0.7699 / 0.7845
cumulative #824 0.7349 / 0.7514 0.7348 / 0.7562
cumulative #827 0.7605 / 0.7773 0.7697 / 0.7912
final scratch 0.7766 / 0.7880 0.7782 / 0.7975

syscall profile and code cause

On perf-01 under the same shared lock, three fresh-Home runs each for dev, cumulative #824, cumulative #827 and final scratch used:

~/bin/perf-lock python3 ~/bm8-perf-attribution/profile.py
# server launch prefix inside profile.py:
strace -f -c -o strace-<round>-<revision>.txt <binary> serve ...
# one additional path-resolved run each for #824 and #827:
~/bin/perf-lock python3 ~/bm8-perf-attribution/paths.py
strace -f -yy -e trace=fcntl,fsync,fdatasync -o paths-0-<revision>.txt <binary> serve ...

Profiled timings are diagnostic only and are excluded from the table. Syscall counts below cover server startup, creation, 200 appends, verification and shutdown; all profile runs used the same lifecycle. All profile jobs exited 0.

cumulative revision fcntl calls, three runs futex calls, three runs fdatasync fsync
#824 7695 / 7693 / 7695 9433 / 9453 / 9741 262 each 10 each
#827 8499 / 8501 / 8501 11548 / 11601 / 11695 262 each 10 each
final scratch 8291 each 11683 / 11808 / 11659 262 each 10 each

The independent -yy trace localizes the increase: auth.db-related fcntl calls rise from 63 at #824 to 869 at #827. The trace and source support extra serialized auth-database work and worker handoffs on the private-note append path. There is no added fsync/fdatasync count in this workload; these traces do not support attributing the cost to extra durable flushes. Summed multithreaded futex time is not treated as elapsed time.

Code locations at final scratch b9dac078:

  • crates/notes/src/room/edit.rs:44: every API edit first awaits write authorization, before checking a cached receipt.
  • crates/notes/src/room/edit.rs:72: a fresh edit then awaits admission, which repeats authority resolution.
  • crates/auth/src/sharing.rs:984 and :1012: authorization dispatches resource resolution; admission takes the note writer fence, dispatches another DB job, opens an IMMEDIATE transaction, rechecks authority and queries note_shares, then commits even when the private note has no shares.
  • crates/auth/src/db.rs:337: each DB operation crosses the bounded dedicated-worker queue and awaits its response. The equivalent introduced #827 paths are sharing.rs:398–477 and room/edit.rs:42–85 at cumulative 199a938b.

perf stat -e task-clock true was unavailable because the VM has perf_event_paranoid=3; passwordless sudo is unavailable. No kernel settings or access limits were changed. Used the requested strace fallback; no CPU flamegraph claim is made.

proposed fix, not implemented

Remove the redundant fresh-edit authorization round trip while retaining every authority check: keep the cheap scope guard first; look up a receipt internally; if a receipt/replay exists, authorize before returning any result; otherwise use the fenced admission's existing in-transaction authorization as the authority check before mutating the room/history. This preserves revoked-credential and replay checks and avoids checking the same fresh write twice.

The remaining owner/private-note admission transaction should be evaluated separately. If needed, introduce a bounded per-note sharing-state fast path that is maintained under the same writer fence as first grant/revoke and invalidated before those transitions become visible. It must retain app-password liveness/scope checks and shared-note sequencing. Do not simply skip authorization for owners or weaken SQLite durability.

Re-run this exact two-VM matrix after any fix with the same 3% limit, plus replay-after-revocation, first-grant versus append, shared admission/order and crash-durability gates. No fix was applied or claimed validated in this task.

cleanup and scope

Owned binaries on dev-01 and both perf VMs, the throwaway dev target/worktree and the relayed binary bundle on the control host were removed. Temporary Homes were automatically removed; a final check found no bm8-append-* directories. No owned units remain running. Only small scripts, logs, JSON and report archives remain, attached here. No evidence or source changes were committed. Conclusions apply to the requested unoptimized, dev-user, private-note append workload; authenticated/shared-note and release-build performance were not measured.

## bm8 cumulative append attribution on the perf VMs **first confirmed >3% step: #827 (`91a19dcf`), merged cumulatively at `199a938b`.** It adds **5.74% on perf-01 and 5.02% on perf-02** versus the preceding #824 step in the full matrix. A separate 20-run interleaved confirmation gives **+3.48% and +4.75%**. The 3% limit is unchanged; this boundary does not pass it. The final scratch versus dev is +6.10% on perf-01 and +2.42% on perf-02 in the full matrix. The previously observed shared-dev +13% is not reproduced at that magnitude. This experiment attributes a repeatable incremental cost to #827, not the entire original 13% to one change. ### workload and provenance Recovered `~/bm8-prep-report/append-attribution-final.py` from the original job. Preserved its 200 sequential acknowledged HTTP appends, fresh 1,000-line note, fresh Home per run, `CALTERNAL_NOTES_SAVE_MS=15`, `--dev-user`, HTTP client and exact-once token checks. Startup, note creation, verification and hashing are outside the timed interval. Ten runs per revision per VM; each round rotates case order and alternates direction. p90 is nearest-rank over the ten **200-append totals**, not request latency. Every primary and confirmation sample passed the content checks. Builds ran only on dev-01 through `~/bin/slot`, with the shared Nix shell and the original unoptimized profile: ```sh ~/bin/slot nix develop -c bash -c 'CI=1 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0 cargo build -p calternal --bin calternal -j4' ``` All twelve binaries were relayed dev-01 → control host → each perf VM. `sha256sum -c SHA256SUMS` passed for every binary on both VMs, and each measured row independently records its matching binary hash. No builds or tests ran on the control host or perf VMs. The perf tool was fetched from Nix's binary cache with `--max-jobs 0`; no compilation was allowed there. `origin/dev` remained `55f35fcde3bc91ca3781055577b93e93819f6cd4`. The first nine cumulative steps reuse the original scratch's actual merge commits, including its conflict resolutions and the requested #828 `fe46caf5`. #833 uses the original step with head `158af347`; later fixture fixes remain confined to the separate final scratch. #847 was merged directly on the #844 cumulative commit, producing `185dba5910eb2e09e3320d0996501019cc0299c1`, without the later unrelated initialization follow-up. No existing branch was edited or pushed. PR #844 head remains `50cce9dcceb87f23a9f346300b3336e315ff6c0c`. Both VMs were idle before measurement (load approximately 0.00–0.02, no competing benchmark). Every measurement held the one shared lock: ```sh # claude@calternal-perf-01-backup01 and claude@calternal-perf-02-backup01 ~/bin/perf-lock bash ~/bm8-perf-attribution/run.sh # run.sh checks disk, records uptime/processes, verifies SHA256SUMS, then: python3 ~/bm8-perf-attribution/measure.py # separate confirmation, also under the shared lock: ~/bin/perf-lock python3 ~/bm8-perf-attribution/followup.py ``` All primary and confirmation jobs exited 0: **240 primary runs plus 160 confirmation runs**. These are performance-harness measurements, not functional-test wall-clock assertions. Disk headroom remained above the required thresholds. ### full cumulative matrix Each entry is seconds for 200 appends. Step delta compares the preceding row; the final scratch row compares the #847 cumulative row. | cumulative step | head | perf-01 median / p90 (s) | step Δ | perf-02 median / p90 (s) | step Δ | |---|---|---:|---:|---:|---:| | base | `55f35fcd` | 0.7443 / 0.7558 | — | 0.7777 / 0.8012 | — | | #824 | `53269288` | 0.7378 / 0.8195 | -0.87% | 0.7376 / 0.7529 | -5.16% | | #827 | `199a938b` | 0.7801 / 0.8153 | +5.74% | 0.7746 / 0.7871 | +5.02% | | #829 | `b42ee47d` | 0.7704 / 0.7818 | -1.24% | 0.7778 / 0.8267 | +0.41% | | #835 | `aa4c34a1` | 0.7598 / 0.7965 | -1.38% | 0.7781 / 0.7844 | +0.03% | | #828 | `9c1a3523` | 0.7824 / 0.7922 | +2.98% | 0.7749 / 0.7913 | -0.40% | | #833 | `4dc11d76` | 0.7626 / 0.7782 | -2.54% | 0.7689 / 0.7868 | -0.79% | | #836 | `b79c17ff` | 0.7723 / 0.7820 | +1.28% | 0.7809 / 0.8142 | +1.56% | | #831 | `42fbcd3a` | 0.7847 / 0.8048 | +1.61% | 0.7820 / 0.8091 | +0.15% | | #844 | `d49db9e2` | 0.7902 / 0.8141 | +0.71% | 0.7773 / 0.8107 | -0.61% | | #847 | `185dba59` | 0.7911 / 0.8621 | +0.11% | 0.8007 / 0.8663 | +3.01% | | scratch | `b9dac078` | 0.7897 / 0.8551 | -0.18% | 0.7966 / 0.8116 | -0.51% | #836 and its preceding #833 step have identical binary hashes; #844 and #831 also have identical hashes. Their small observed differences show timing noise. #847 exceeds 3% only on perf-02 (+3.01%, versus +0.11% on perf-01), so this batch does not establish an additional repeatable regression there. #828 is +2.98% on perf-01 and -0.40% on perf-02; no confirmed >3% boundary at #828. ### separate 20-run confirmation | revision | perf-01 median / p90 (s) | perf-02 median / p90 (s) | |---|---:|---:| | dev | 0.7233 / 0.7437 | 0.7699 / 0.7845 | | cumulative #824 | 0.7349 / 0.7514 | 0.7348 / 0.7562 | | cumulative #827 | 0.7605 / 0.7773 | 0.7697 / 0.7912 | | final scratch | 0.7766 / 0.7880 | 0.7782 / 0.7975 | ### syscall profile and code cause On perf-01 under the same shared lock, three fresh-Home runs each for dev, cumulative #824, cumulative #827 and final scratch used: ```sh ~/bin/perf-lock python3 ~/bm8-perf-attribution/profile.py # server launch prefix inside profile.py: strace -f -c -o strace-<round>-<revision>.txt <binary> serve ... # one additional path-resolved run each for #824 and #827: ~/bin/perf-lock python3 ~/bm8-perf-attribution/paths.py strace -f -yy -e trace=fcntl,fsync,fdatasync -o paths-0-<revision>.txt <binary> serve ... ``` Profiled timings are diagnostic only and are excluded from the table. Syscall counts below cover server startup, creation, 200 appends, verification and shutdown; all profile runs used the same lifecycle. All profile jobs exited 0. | cumulative revision | fcntl calls, three runs | futex calls, three runs | fdatasync | fsync | |---|---|---|---|---| | #824 | 7695 / 7693 / 7695 | 9433 / 9453 / 9741 | 262 each | 10 each | | #827 | 8499 / 8501 / 8501 | 11548 / 11601 / 11695 | 262 each | 10 each | | final scratch | 8291 each | 11683 / 11808 / 11659 | 262 each | 10 each | The independent `-yy` trace localizes the increase: **auth.db-related fcntl calls rise from 63 at #824 to 869 at #827**. The trace and source support extra serialized auth-database work and worker handoffs on the private-note append path. There is no added fsync/fdatasync count in this workload; these traces do not support attributing the cost to extra durable flushes. Summed multithreaded futex time is not treated as elapsed time. Code locations at final scratch `b9dac078`: - [`crates/notes/src/room/edit.rs:44`](https://git.kayg.org/kayg/calternal/src/commit/b9dac07806a398f53f8a9acaf1506d4b6f718a30/crates/notes/src/room/edit.rs#L44): every API edit first awaits write authorization, before checking a cached receipt. - [`crates/notes/src/room/edit.rs:72`](https://git.kayg.org/kayg/calternal/src/commit/b9dac07806a398f53f8a9acaf1506d4b6f718a30/crates/notes/src/room/edit.rs#L72): a fresh edit then awaits admission, which repeats authority resolution. - [`crates/auth/src/sharing.rs:984`](https://git.kayg.org/kayg/calternal/src/commit/b9dac07806a398f53f8a9acaf1506d4b6f718a30/crates/auth/src/sharing.rs#L984) and [`:1012`](https://git.kayg.org/kayg/calternal/src/commit/b9dac07806a398f53f8a9acaf1506d4b6f718a30/crates/auth/src/sharing.rs#L1012): authorization dispatches resource resolution; admission takes the note writer fence, dispatches another DB job, opens an IMMEDIATE transaction, rechecks authority and queries note_shares, then commits even when the private note has no shares. - [`crates/auth/src/db.rs:337`](https://git.kayg.org/kayg/calternal/src/commit/b9dac07806a398f53f8a9acaf1506d4b6f718a30/crates/auth/src/db.rs#L337): each DB operation crosses the bounded dedicated-worker queue and awaits its response. The equivalent introduced #827 paths are `sharing.rs:398–477` and `room/edit.rs:42–85` at cumulative `199a938b`. `perf stat -e task-clock true` was unavailable because the VM has `perf_event_paranoid=3`; passwordless sudo is unavailable. No kernel settings or access limits were changed. Used the requested strace fallback; no CPU flamegraph claim is made. ### proposed fix, not implemented Remove the redundant fresh-edit authorization round trip while retaining every authority check: keep the cheap scope guard first; look up a receipt internally; if a receipt/replay exists, authorize before returning any result; otherwise use the fenced admission's existing in-transaction authorization as the authority check before mutating the room/history. This preserves revoked-credential and replay checks and avoids checking the same fresh write twice. The remaining owner/private-note admission transaction should be evaluated separately. If needed, introduce a bounded per-note sharing-state fast path that is maintained under the same writer fence as first grant/revoke and invalidated before those transitions become visible. It must retain app-password liveness/scope checks and shared-note sequencing. Do not simply skip authorization for owners or weaken SQLite durability. Re-run this exact two-VM matrix after any fix with the **same 3% limit**, plus replay-after-revocation, first-grant versus append, shared admission/order and crash-durability gates. No fix was applied or claimed validated in this task. ### cleanup and scope Owned binaries on dev-01 and both perf VMs, the throwaway dev target/worktree and the relayed binary bundle on the control host were removed. Temporary Homes were automatically removed; a final check found no `bm8-append-*` directories. No owned units remain running. Only small scripts, logs, JSON and report archives remain, attached here. No evidence or source changes were committed. Conclusions apply to the requested unoptimized, dev-user, private-note append workload; authenticated/shared-note and release-build performance were not measured.
dev01-evidence.tar.gz
5.7 KiB
perf01-evidence.tar.gz
442 KiB
perf02-evidence.tar.gz
358 KiB
kayg commented 2026-11-04 13:04:58 +00:00
Author
Owner
Copy link

Closing: head 50cce9dc is already in dev (reached via 995baf64); nothing left to merge. Branch kept.

Closing: head 50cce9dc is already in dev (reached via 995baf64); nothing left to merge. Branch kept.
kayg closed this pull request 2026-11-04 13:04:58 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!844
No description provided.