kayg/calternal-v0

Archived

Watch

Fork 0

Money cleared-mark writes regress at 50k transactions #1496

Open

opened 2026-10-09 03:13:07 +00:00 by kayg · 2 comments

kayg commented

2026-10-09 03:13:07 +00:00

Owner

Evidence

An interleaved perf-VM profile used a synthetic 50,000-transaction Home, with three pairs in the order current/origin-dev, origin-dev/current, current/origin-dev. The load average at the start was 0.19, 1.29, 0.93. Each side used the same current register client and the same imported fixture.

The five single-write samples in each run returned HTTP 200. Their p95 values were:

Pair	Current branch	origin/dev
1	26,553.9 ms	3,962.5 ms
2	33,081.8 ms	2,940.7 ms
3	4,929.8 ms	4,125.4 ms
Median	26,553.9 ms	3,962.5 ms

The current median is 6.7 times the matched origin/dev median. The same run measured first-row p95 medians of 1,733.13 ms current and 8,952.14 ms origin/dev; scroll-frame p95 medians were 100 ms and 116.6 ms. Four-read burst peak RSS medians were 723,107,840 bytes and 2,468,065,280 bytes. The regression is specific to the cleared-mark write path in this profile.

Current server binary SHA-256: 6c271a456c102e6faee599698b669e5ec7d4e112c61f151e22a76557a331dad0; origin/dev binary SHA-256: 6c25a4023cebe0e6baf1230270af5d864c9931b8bda9eec9d4bca77178357fb2. Full samples are recorded under metrics.money_register_688.interleaved_qualification_2026_10_09 in docs/perf/baseline.json on the #688 branch.

Expected

Cleared-mark writes should not regress by multiple seconds against the matched origin/dev profile. Trace the write and projection paths and restore single-write latency while preserving scoped projection correctness.

The fixture was synthetic. No transaction contents or values were retained.

## Evidence An interleaved perf-VM profile used a synthetic 50,000-transaction Home, with three pairs in the order current/origin-dev, origin-dev/current, current/origin-dev. The load average at the start was 0.19, 1.29, 0.93. Each side used the same current register client and the same imported fixture. The five single-write samples in each run returned HTTP 200. Their p95 values were: | Pair | Current branch | origin/dev | | --- | ---: | ---: | | 1 | 26,553.9 ms | 3,962.5 ms | | 2 | 33,081.8 ms | 2,940.7 ms | | 3 | 4,929.8 ms | 4,125.4 ms | | Median | 26,553.9 ms | 3,962.5 ms | The current median is 6.7 times the matched origin/dev median. The same run measured first-row p95 medians of 1,733.13 ms current and 8,952.14 ms origin/dev; scroll-frame p95 medians were 100 ms and 116.6 ms. Four-read burst peak RSS medians were 723,107,840 bytes and 2,468,065,280 bytes. The regression is specific to the cleared-mark write path in this profile. Current server binary SHA-256: `6c271a456c102e6faee599698b669e5ec7d4e112c61f151e22a76557a331dad0`; origin/dev binary SHA-256: `6c25a4023cebe0e6baf1230270af5d864c9931b8bda9eec9d4bca77178357fb2`. Full samples are recorded under `metrics.money_register_688.interleaved_qualification_2026_10_09` in `docs/perf/baseline.json` on the #688 branch. ## Expected Cleared-mark writes should not regress by multiple seconds against the matched origin/dev profile. Trace the write and projection paths and restore single-write latency while preserving scoped projection correctness. The fixture was synthetic. No transaction contents or values were retained.

kayg referenced this issue

2026-10-09 03:16:07 +00:00

PERF: Money rule 2 — bound lists by rows and bytes with signed keyset pages (#663) #688

kayg referenced this issue

2026-10-09 06:14:53 +00:00

PERF: Money rule 2 — bound lists by rows and bytes with signed keyset pages (#663) #688

kayg commented

2026-10-09 06:14:54 +00:00

Author

Owner

The first complete 50k qualification (before the final projection ownership repair) retains three pairs in baseline.json: current median first-row p95 2,014 ms vs origin/dev 11,719 ms; mark p95 4,873 vs 3,690 ms; amount p95 5,880 vs 3,203 ms; burst RSS 1.07 vs 2.47 GB. Every mark returned 200. This is an improvement from #1496's 26,554 ms, but it still fails the 300 ms target and the no-regression comparison. It is retained under qualification_before_projection_ownership_2026_10_09, not claimed as the final result.

Root cause: full Notes scans and Files/live adoption accept foreign Money Markdown, despite the Notes UI hiding those Plugin-owned paths. Each Month produces thousands of link rows and repeats that work on a mark. The final fix supplies the existing ownership registry before Notes subscriptions start and uses one immutable process policy in scans and live adoption. It skips Money before reading/parsing. It preserves Notes-owned Tasks and unclaimed Markdown. Reference batching is no longer needed for these foreign sources.

The isolated-process regression passes with 2000 foreign checkbox/link rows: full reconcile, direct adoption and the live Files event produce zero Note, Task or link rows; real Notes remain indexed; Money bytes and fingerprint stay unchanged. Final clippy passes. Final Notes tests: 319 passed / 5 ignored plus 2 integration tests. Web check has zero performance-lint violations and unchanged exception count. Final server checks are running. Runtime code is 3ea25dc4b. The fresh final 50k/200k paired qualification is queued under one perf-run lock with the replacement release build. An obsolete pending 200k run was stopped before measuring the replacement code; only the fresh run will qualify it.

The loaded/restarted 50k trace changes the earlier conclusion about Notes projection. Post-import had zero Note rows, but after restart and the real Account screen/scroll workload the perf VM had 14 Note items and 58,340 references from Money Markdown. Local reproduction had 16 items and 66,672 references. Money totals/compare were 0 ms and selected-row SQL 2–6 ms; the local marks instead spent 0.7–1.7 s before source durability and up to 1.3 s waiting for the projection writer. The VM traced mark p95 was 5,739 ms. The first complete 50k qualification (before the final projection ownership repair) retains three pairs in baseline.json: current median first-row p95 2,014 ms vs origin/dev 11,719 ms; mark p95 4,873 vs 3,690 ms; amount p95 5,880 vs 3,203 ms; burst RSS 1.07 vs 2.47 GB. Every mark returned 200. This is an improvement from #1496's 26,554 ms, but it still fails the 300 ms target and the no-regression comparison. It is retained under qualification_before_projection_ownership_2026_10_09, not claimed as the final result. Root cause: full Notes scans and Files/live adoption accept foreign Money Markdown, despite the Notes UI hiding those Plugin-owned paths. Each Month produces thousands of link rows and repeats that work on a mark. The final fix supplies the existing ownership registry before Notes subscriptions start and uses one immutable process policy in scans and live adoption. It skips Money before reading/parsing. It preserves Notes-owned Tasks and unclaimed Markdown. Reference batching is no longer needed for these foreign sources. The isolated-process regression passes with 2000 foreign checkbox/link rows: full reconcile, direct adoption and the live Files event produce zero Note, Task or link rows; real Notes remain indexed; Money bytes and fingerprint stay unchanged. Final clippy passes. Final Notes tests: 319 passed / 5 ignored plus 2 integration tests. Web check has zero performance-lint violations and unchanged exception count. Final server checks are running. Runtime code is 3ea25dc4b. The fresh final 50k/200k paired qualification is queued under one perf-run lock with the replacement release build. An obsolete pending 200k run was stopped before measuring the replacement code; only the fresh run will qualify it.

kayg commented

2026-10-09 07:01:41 +00:00

Author

Owner

money688h final: head a5aaf55009, measured runtime 3ea25dc4b. Full report and verbatim gates posted on #688.

The original 200k HTTP 500 was SQLite BUSY at BEGIN IMMEDIATE. The fix retains durable intents on retryable contention, removes mark-triggered Budget replay, and excludes foreign Money sources from Note date migration, scans and live adoption. Selected-row SQL uses the transaction index. Constant-row tests, incremental/full replay properties and SIGKILL recovery pass.

Three 50k default-allocator pairs: median candidate/origin cleared p95 299.8/3228.5 ms; amount p95 1192.7/3277.2 ms. Candidate individual cleared p95s 340.0,299.8,277.3 ms: the first exceeds 300 ms. Every measured write succeeded.

Default 200k candidate run succeeded, cleared p95 992.8 ms; origin/dev was OOM-killed, filed #1502. This attempt has zero complete pairs. A separately labelled comparison applies MALLOC_ARENA_MAX=2 and MALLOC_TRIM_THRESHOLD_=131072 equally to both servers, harness and Chromium: three complete 200k pairs, all writes successful, median cleared p95 963.3/14401.6 ms and amount p95 12942.8/13707.0 ms. Raw statuses, retries, CPU/RSS, load5 and hashes are committed in baseline.json. This does not erase the default-settings crash.

Full server gates had one SLOW-only Calendar budget failure; its isolated parent retry and both integration targets passed. No expectation or latency budget changed. Production UI browser exceptions remain filed as #1498; real-Mac timing was not run. No push, deploy or issue closure. Cargo output and local web build output cleaned.

money688h final: head a5aaf55009efafc6a77239124375a74769fc8620, measured runtime 3ea25dc4b. Full report and verbatim gates posted on #688. The original 200k HTTP 500 was SQLite BUSY at BEGIN IMMEDIATE. The fix retains durable intents on retryable contention, removes mark-triggered Budget replay, and excludes foreign Money sources from Note date migration, scans and live adoption. Selected-row SQL uses the transaction index. Constant-row tests, incremental/full replay properties and SIGKILL recovery pass. Three 50k default-allocator pairs: median candidate/origin cleared p95 299.8/3228.5 ms; amount p95 1192.7/3277.2 ms. Candidate individual cleared p95s 340.0,299.8,277.3 ms: the first exceeds 300 ms. Every measured write succeeded. Default 200k candidate run succeeded, cleared p95 992.8 ms; origin/dev was OOM-killed, filed #1502. This attempt has zero complete pairs. A separately labelled comparison applies MALLOC_ARENA_MAX=2 and MALLOC_TRIM_THRESHOLD_=131072 equally to both servers, harness and Chromium: three complete 200k pairs, all writes successful, median cleared p95 963.3/14401.6 ms and amount p95 12942.8/13707.0 ms. Raw statuses, retries, CPU/RSS, load5 and hashes are committed in baseline.json. This does not erase the default-settings crash. Full server gates had one SLOW-only Calendar budget failure; its isolated parent retry and both integration targets passed. No expectation or latency budget changed. Production UI browser exceptions remain filed as #1498; real-Mac timing was not run. No push, deploy or issue closure. Cargo output and local web build output cleaned.

Commenting is not possible because the repository is archived.

No labels

No milestone

No project

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal-v0#1496

No description provided.

Rows
Columns