kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

WIP: Cold growth + fast startup on #764 #767

Draft
kayg wants to merge 30 commits from perf/cold-growth-on-764 into dev
pull from: perf/cold-growth-on-764
merge into: kayg:dev
Conversation 7 Commits 30 Files changed 122 +12667 -1451
kayg commented 2026-10-29 17:54:31 +00:00
Owner
Copy link

Current stacked revision

Head: 8b5b8cdc74385b3bcbf1540d1c612c075952ef06; target: dev.

Stack: #761-A feat/tiering-761-a-migration (5441212fd8b445fb80ff8ed640a4b33c17ae1d07) → #761-B feat/tiering-761-b-reader (58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8) → #764 (75978ddb5ed30152a85fbb1d766a62d5c5a73113) → #767. This change needs A+B only; it is not based on feat/tiering-761-s1 and does not include C or D.

Cold-growth work is stacked on the userdata-only quota accounting. The deterministic startup fence regression fails on race-red with DirectoryNotEmpty and passes on this head.

Verified gates

  • cargo fmt --all -- --check: pass.
  • cargo clippy --workspace --all-targets -j4 -- -D warnings: pass.
  • QUOTA_PROPERTY_CASES=50000 cargo test --workspace -j4: pass, exit 0.
  • cargo test -p calternal-fsio -j4 fence_never_blocks_a_recursive_home_delete -- --nocapture: fails on race-red, passes on this head.
  • CALTERNAL_INODE_REUSE_DIR=/home/claude cargo test -p calternal-fsio -j4 review_754d_reused_directory_inode_must_not_certify_a_replacement -- --nocapture: pass; ext4 recycled the directory inode and the replacement was charged.

Performance qualification

The original same-Home reproduction completed on 86f48a39 with 100k notes and index.db inside the Home. Across three runs, medians were typing CPU 35.483%, idle CPU 0.000%, idle RSS 81.44 MB, and idle wakeups 0.000/s. This reproduces the high typing CPU when the Notes index database is under the Home.

The replacement matrix used perf/vm/typing-idle.sh (SHA256 13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b) through ~/bin/perf-lock, holding /var/lock/calternal-perf.lock on claude@calternal-perf-02-backup01. It completed 13 samples before stopping in quota-4: the warm run's first notes.list page was empty and marked incomplete: true. That failed run has no accepted metrics. The first dev-quota-1 sample is excluded because an external diagnostic file walk overlapped its idle interval; its valid quota-1 partner is excluded to keep the pair matched. Twelve completed samples remain usable, but the six-pair matrix is incomplete and no comparison is claimed. The #759b matrix currently holds the shared lock; continuation will wait for it.

Supplied dev reference (d83e54db, six runs): typing CPU 1.125%, idle CPU 0.647%, idle RSS 106.1 MB, wakeups 0.102/s. Budgets are typing CPU <=2%, idle CPU <=0.1%, RSS <=100 MB, and wakeups <=0.2/s.

Measurement audit

#788 held /var/lock/calternal-perf.lock from 19:26:10Z on perf-01 and 19:28:31Z on perf-02 until 22:28:36Z. Two original-path attempts using the old /tmp/perf.lock at 20:35:12Z and 20:39:41Z are excluded; they produced only fixture/startup output, not typing or idle metrics. No numeric results were discarded for this overlap. The valid same-Home reproduction started at 22:30:01Z, after release.

The later r2 matrix did not overlap #788, but its loop repeated quota-1; these five misattributed samples are excluded from the new medians (typing CPU %, idle CPU %, RSS MB, wakeups/s): dev-quota-1 (1.150, 0.673, 131.98, 0.100), quota-1 (0.467, 0.000, 81.68, 0.000), dev-cold-1 (1.217, 0.550, 112.49, 0.073), cold-1 (0.483, 0.000, 81.05, 0.000), and dev-quota-2 (1.217, 0.723, 126.16, 0.123).

No merge or deployment.

Historical description from earlier head follows.

Port #760's cold-growth index and startup-readiness work from d23e80b9 onto #764 at d0e6d6f6. Stable shared note keys, covering task-membership ranges and unchanged memberships survive field edits; bounded cold scans, shared parsing and checkpoint/reclamation changes come with their correctness tests. The cold-growth harness is carried for later measurements on the perf VMs.

Production quota, usage and native watcher code keeps #764's whole-Home recount and inotify-barrier design. No incremental counter or #760 watch-registration implementation is restored. Startup remains asynchronous; never-counted or provisional Homes must obtain a verified fenced total at admission. The readiness regression blocks a Home recount and proves startup returns, then checks its initial bytes; the HTTP regression enforces quota immediately after healthz.

Reviewer coverage from review/754*, review/755-tests and review/764*-tests is retained. The old 754g retry/index contracts are also expressed against per-Home failure accounting; 755's quota tests use #764's APIs, and review/764c-tests is included without production changes.

Validation on 10by32-02-std (arm64), with heavy commands through /usr/local/bin/build-slot.sh and cargo -j8:

  • cargo fmt --all --check; cargo clippy --workspace --all-targets -j8 -- -D warnings: passed.
  • cargo test --workspace -j8: 1,342 passed, 0 failed, 27 ignored. Tasks abrupt-stop passed on its first run; no #757 rerun needed.
  • Cold-growth/task correctness (notes, tasks, perf and store): 527 passed, 0 failed, 6 ignored. Covers identical cold-fill rows, unchanged tag memberships, membership model/rollback/crash tests and cold-gate statistics.
  • QUOTA_PROPERTY_CASES=50000: whole-Home filesystem, Auth admission, Auth fallback, filesystem fallback, and review-764c mode-switch properties all passed (50,000 cases each). Auth normal admission: 23,398 accepted / 26,602 quota-refused; filesystem fallback: 25,000 outside mutations during recount, all refused retryably.
  • CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse was set for every Rust test command; the required physical directory-inode reuse regression passed.
  • The ignored non-linearizable fallback contract was run explicitly and passed; upload-retry review-764c JavaScript tests: 2 passed.

The initial separate correctness run lacked y-protocols/sync in the fresh worktree. After bun install --frozen-lockfile, its full rerun and the workspace run passed; no source fix was needed. Existing ignored perf/idle observations were left for the perf VMs.

No performance measurements ran here and no performance claims are made for this head. The coordinator will build x86 binaries and measure on the perf-test VMs. This PR targets dev and includes #764 until that base lands. Supersedes #760; refs #686. No merge or deployment performed.

Head: 2df9787047. Forgejo represents draft status with the WIP title prefix.

<!-- current stacked revision: 2026-10-31 --> ## Current stacked revision Head: `8b5b8cdc74385b3bcbf1540d1c612c075952ef06`; target: `dev`. Stack: #761-A `feat/tiering-761-a-migration` (`5441212fd8b445fb80ff8ed640a4b33c17ae1d07`) → #761-B `feat/tiering-761-b-reader` (`58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8`) → #764 (`75978ddb5ed30152a85fbb1d766a62d5c5a73113`) → #767. This change needs A+B only; it is not based on `feat/tiering-761-s1` and does not include C or D. Cold-growth work is stacked on the userdata-only quota accounting. The deterministic startup fence regression fails on `race-red` with `DirectoryNotEmpty` and passes on this head. ### Verified gates - `cargo fmt --all -- --check`: pass. - `cargo clippy --workspace --all-targets -j4 -- -D warnings`: pass. - `QUOTA_PROPERTY_CASES=50000 cargo test --workspace -j4`: pass, exit 0. - `cargo test -p calternal-fsio -j4 fence_never_blocks_a_recursive_home_delete -- --nocapture`: fails on `race-red`, passes on this head. - `CALTERNAL_INODE_REUSE_DIR=/home/claude cargo test -p calternal-fsio -j4 review_754d_reused_directory_inode_must_not_certify_a_replacement -- --nocapture`: pass; ext4 recycled the directory inode and the replacement was charged. ### Performance qualification The original same-Home reproduction completed on `86f48a39` with 100k notes and `index.db` inside the Home. Across three runs, medians were typing CPU 35.483%, idle CPU 0.000%, idle RSS 81.44 MB, and idle wakeups 0.000/s. This reproduces the high typing CPU when the Notes index database is under the Home. The replacement matrix used `perf/vm/typing-idle.sh` (SHA256 `13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b`) through `~/bin/perf-lock`, holding `/var/lock/calternal-perf.lock` on `claude@calternal-perf-02-backup01`. It completed 13 samples before stopping in `quota-4`: the warm run's first `notes.list` page was empty and marked `incomplete: true`. That failed run has no accepted metrics. The first `dev-quota-1` sample is excluded because an external diagnostic file walk overlapped its idle interval; its valid `quota-1` partner is excluded to keep the pair matched. Twelve completed samples remain usable, but the six-pair matrix is incomplete and no comparison is claimed. The #759b matrix currently holds the shared lock; continuation will wait for it. Supplied dev reference (`d83e54db`, six runs): typing CPU 1.125%, idle CPU 0.647%, idle RSS 106.1 MB, wakeups 0.102/s. Budgets are typing CPU <=2%, idle CPU <=0.1%, RSS <=100 MB, and wakeups <=0.2/s. #### Measurement audit #788 held `/var/lock/calternal-perf.lock` from 19:26:10Z on perf-01 and 19:28:31Z on perf-02 until 22:28:36Z. Two original-path attempts using the old `/tmp/perf.lock` at 20:35:12Z and 20:39:41Z are excluded; they produced only fixture/startup output, not typing or idle metrics. No numeric results were discarded for this overlap. The valid same-Home reproduction started at 22:30:01Z, after release. The later r2 matrix did not overlap #788, but its loop repeated `quota-1`; these five misattributed samples are excluded from the new medians (typing CPU %, idle CPU %, RSS MB, wakeups/s): dev-quota-1 `(1.150, 0.673, 131.98, 0.100)`, quota-1 `(0.467, 0.000, 81.68, 0.000)`, dev-cold-1 `(1.217, 0.550, 112.49, 0.073)`, cold-1 `(0.483, 0.000, 81.05, 0.000)`, and dev-quota-2 `(1.217, 0.723, 126.16, 0.123)`. No merge or deployment. <!-- end current stacked revision --> --- Historical description from earlier head follows. Port #760's cold-growth index and startup-readiness work from d23e80b9 onto #764 at d0e6d6f6. Stable shared note keys, covering task-membership ranges and unchanged memberships survive field edits; bounded cold scans, shared parsing and checkpoint/reclamation changes come with their correctness tests. The cold-growth harness is carried for later measurements on the perf VMs. Production quota, usage and native watcher code keeps #764's whole-Home recount and inotify-barrier design. No incremental counter or #760 watch-registration implementation is restored. Startup remains asynchronous; never-counted or provisional Homes must obtain a verified fenced total at admission. The readiness regression blocks a Home recount and proves startup returns, then checks its initial bytes; the HTTP regression enforces quota immediately after healthz. Reviewer coverage from review/754*, review/755-tests and review/764*-tests is retained. The old 754g retry/index contracts are also expressed against per-Home failure accounting; 755's quota tests use #764's APIs, and review/764c-tests is included without production changes. Validation on 10by32-02-std (arm64), with heavy commands through /usr/local/bin/build-slot.sh and cargo -j8: - cargo fmt --all --check; cargo clippy --workspace --all-targets -j8 -- -D warnings: passed. - cargo test --workspace -j8: 1,342 passed, 0 failed, 27 ignored. Tasks abrupt-stop passed on its first run; no #757 rerun needed. - Cold-growth/task correctness (notes, tasks, perf and store): 527 passed, 0 failed, 6 ignored. Covers identical cold-fill rows, unchanged tag memberships, membership model/rollback/crash tests and cold-gate statistics. - QUOTA_PROPERTY_CASES=50000: whole-Home filesystem, Auth admission, Auth fallback, filesystem fallback, and review-764c mode-switch properties all passed (50,000 cases each). Auth normal admission: 23,398 accepted / 26,602 quota-refused; filesystem fallback: 25,000 outside mutations during recount, all refused retryably. - CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse was set for every Rust test command; the required physical directory-inode reuse regression passed. - The ignored non-linearizable fallback contract was run explicitly and passed; upload-retry review-764c JavaScript tests: 2 passed. The initial separate correctness run lacked y-protocols/sync in the fresh worktree. After bun install --frozen-lockfile, its full rerun and the workspace run passed; no source fix was needed. Existing ignored perf/idle observations were left for the perf VMs. No performance measurements ran here and no performance claims are made for this head. The coordinator will build x86 binaries and measure on the perf-test VMs. This PR targets dev and includes #764 until that base lands. Supersedes #760; refs #686. No merge or deployment performed. Head: 2df978704739f7e734862df53841eedf50a0fef2. Forgejo represents draft status with the WIP title prefix.
kayg added 30 commits 2026-10-29 17:54:31 +00:00
fix(fsio): keep usage totals consistent across aliases and moves
Some checks failed
CI / server (pull_request) Successful in 1m11s
Details
CI / rust (pull_request) Successful in 14m1s
Details
CI / web (pull_request) Has been cancelled
Details
7c2e78cc33
fix(fsio): reconcile storage usage by inode and repair uncertain changes
All checks were successful
CI / server (pull_request) Successful in 1m27s
Details
CI / rust (pull_request) Successful in 15m40s
Details
CI / web (pull_request) Successful in 2h46m25s
Details
81e6141a2e 
Process every consumed repair scope and coalesce retry timers within each Home.
Retire broad retries after validated partial scans identify the failing Homes,
retain unmeasurable overlapping writes, and cancel historical timers on deletion.

Use compact directory fingerprints, reuse saved scan identities, and avoid
recording snapshot publication as another map of overlapping writes. Enable
reviewer reproductions and cover partial publication and retry lifecycle edges.
Replace quota repairs with fenced whole-Home recounts
Some checks failed
CI / server (pull_request) Successful in 1m52s
Details
CI / rust (pull_request) Failing after 10m28s
Details
CI / web (pull_request) Failing after 42m18s
Details
798f1bbe91
Port cold growth indexes and fast startup onto whole-Home quotas
Some checks failed
CI / server (pull_request) Successful in 1m32s
Details
CI / rust (pull_request) Failing after 8m55s
Details
CI / web (pull_request) Has been cancelled
Details
2df9787047
kayg commented 2026-10-29 18:03:51 +00:00
Author
Owner
Copy link

Independent review of 2df9787047, 10by32-02-std arm64. Verdict: fix first.

P2 — unchanged memberships are rewritten on renames and partial tag changes. crates/tasks/src/index.rs:295–303 couples tag retention to path equality; a rename with the same stable ID/tags clears and reinserts all memberships. At lines 435–462, adding home to [work/calls] also deletes/reinserts retained work and work/calls. Independent audit triggers observe four writes to unchanged memberships in each case, expected zero. Counts also churn. Logical results are correct, but the explicit write-avoidance contract is violated. Separate path/link tracking from membership and apply tag-set differences, retaining intersections. This is not a measured cold-growth regression or a data-loss finding.

Tests only pushed to review/767-tests, commit d7820b5a. Reproduce there:

/usr/local/bin/build-slot.sh env CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 -p calternal-tasks review_767_rename -- --ignored --nocapture
/usr/local/bin/build-slot.sh env CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 -p calternal-tasks review_767_tag_delta -- --ignored --nocapture

Both fail left: 4, right: 0. These two reproducers are explicitly ignored in default gates and were separately enabled. No production changes.

Independent checks passed: 5,000 random edits/renames/tag/status changes/deletes against independent membership and badge-count models; covering-index reads agree with task-table identities; index columns are exactly tag/note_key/id; process exit after membership write before table write recovers the original WAL transaction with integrity_check=ok; dev-format cache rebuild preserves 13 notes and their bytes/identities. Real HTTP immediately after healthz enforces existing quota, permits an upload with room, refuses a new upload immediately after outside growth, and permits it after moving those outside bytes out, without settling waits.

The earlier non-linearizable fallback reproducer was enabled and passes: WouldBlock, provisional/display=0, actual=1009, minimum Home during transfer=202. Actual directory inode reuse on /srv/inode-reuse passed with st_ino=16 and replacement/later mutation charged. All six client retry tests pass.

Preservation: quota/Auth/watcher/client changes against origin/perf/quota-simple are additions only: no existing quota assertion changed/removed, all 57 reviewer-named base tests byte-for-byte present. Literal preservation of every historical #760/#755 test is false: #764 deliberately adapted scoped scanner tests to whole-Home boundaries. Five #755 fsio tests are moved/adapted in usage/review_755.rs; Auth stale-admission and task random/crash/dev-cache tests remain. Seven older inherited scoped/descriptor test names are absent; plan 0754 and current whole-Home containment/property/failure tests document their dispositions/replacements. Full source inventories/diffs are in the local report. Tasks schema/crash-harness adaptations do not weaken quota tests.

Final gates all exit 0, heavy commands through build-slot.sh at -j8:

cargo fmt --all --check
/usr/local/bin/build-slot.sh cargo clippy --workspace --all-targets -j8 -- -D warnings
/usr/local/bin/build-slot.sh env CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test --workspace -j8

108 successful test-process reports, zero failures; summed results including nested-child repetitions: 1,361 passed / 29 ignored. Tasks abrupt-stop passed, no Tasks rerun. Initial reviewer-fixture issues (deduplicated upload and deliberate outside-write lint allowances) were corrected before final gates; attempts remain logged.

Findings/reproducers, inventories, diffs and logs: /home/agent/codex-jobs/verify-767/findings.md and siblings. No cold-growth measurement, perf-test VM access, merge or deployment. Target cleanup and heartbeat shutdown follow this comment.

Independent review of **2df978704739f7e734862df53841eedf50a0fef2**, 10by32-02-std arm64. Verdict: **fix first**. **P2 — unchanged memberships are rewritten on renames and partial tag changes.** `crates/tasks/src/index.rs:295–303` couples tag retention to path equality; a rename with the same stable ID/tags clears and reinserts all memberships. At lines 435–462, adding `home` to `[work/calls]` also deletes/reinserts retained `work` and `work/calls`. Independent audit triggers observe **four writes to unchanged memberships in each case**, expected zero. Counts also churn. Logical results are correct, but the explicit write-avoidance contract is violated. Separate path/link tracking from membership and apply tag-set differences, retaining intersections. This is not a measured cold-growth regression or a data-loss finding. Tests only pushed to [review/767-tests](https://git.kayg.org/kayg/calternal/src/branch/review/767-tests), commit [d7820b5a](https://git.kayg.org/kayg/calternal/commit/d7820b5a67c05b8422b86226faf269cc5e99b8ab). Reproduce there: ``` /usr/local/bin/build-slot.sh env CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 -p calternal-tasks review_767_rename -- --ignored --nocapture /usr/local/bin/build-slot.sh env CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 -p calternal-tasks review_767_tag_delta -- --ignored --nocapture ``` Both fail `left: 4, right: 0`. These two reproducers are explicitly ignored in default gates and were separately enabled. No production changes. Independent checks passed: 5,000 random edits/renames/tag/status changes/deletes against independent membership and badge-count models; covering-index reads agree with task-table identities; index columns are exactly tag/note_key/id; process exit after membership write before table write recovers the original WAL transaction with integrity_check=ok; dev-format cache rebuild preserves 13 notes and their bytes/identities. Real HTTP immediately after healthz enforces existing quota, permits an upload with room, refuses a new upload immediately after outside growth, and permits it after moving those outside bytes out, without settling waits. The earlier non-linearizable fallback reproducer was enabled and passes: WouldBlock, provisional/display=0, actual=1009, minimum Home during transfer=202. Actual directory inode reuse on `/srv/inode-reuse` passed with st_ino=16 and replacement/later mutation charged. All six client retry tests pass. **Preservation:** quota/Auth/watcher/client changes against `origin/perf/quota-simple` are additions only: no existing quota assertion changed/removed, all 57 reviewer-named base tests byte-for-byte present. Literal preservation of every historical #760/#755 test is false: #764 deliberately adapted scoped scanner tests to whole-Home boundaries. Five #755 fsio tests are moved/adapted in `usage/review_755.rs`; Auth stale-admission and task random/crash/dev-cache tests remain. Seven older inherited scoped/descriptor test names are absent; plan 0754 and current whole-Home containment/property/failure tests document their dispositions/replacements. Full source inventories/diffs are in the local report. Tasks schema/crash-harness adaptations do not weaken quota tests. Final gates all **exit 0**, heavy commands through build-slot.sh at **-j8**: ``` cargo fmt --all --check /usr/local/bin/build-slot.sh cargo clippy --workspace --all-targets -j8 -- -D warnings /usr/local/bin/build-slot.sh env CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test --workspace -j8 ``` 108 successful test-process reports, zero failures; summed results including nested-child repetitions: 1,361 passed / 29 ignored. Tasks abrupt-stop passed, no Tasks rerun. Initial reviewer-fixture issues (deduplicated upload and deliberate outside-write lint allowances) were corrected before final gates; attempts remain logged. Findings/reproducers, inventories, diffs and logs: `/home/agent/codex-jobs/verify-767/findings.md` and siblings. No cold-growth measurement, perf-test VM access, merge or deployment. Target cleanup and heartbeat shutdown follow this comment.
kayg commented 2026-10-29 18:12:52 +00:00
Author
Owner
Copy link

Fixed P2 comment #69795 in 30f2716fac, pushed only to perf/cold-growth-on-764.

Fetched first and fast-forward merged reviewer tests d7820b5a onto the requested 2df97870 head. Both original ignored reproducers failed before the fix with 4 unchanged-membership writes, expected 0. Their assertions are preserved; both are now enabled in the default suite.

Task renames retain tag membership for the same stable identity while updating path/link tracking independently. Tag updates compare sorted old/new membership sets and delete/insert only their differences; only changed memberships update badge counts. Cold fills still defer badge aggregation until after_fill.

Added SQLite trigger counters for membership and badge writes covering rename, add tag, remove child tag, replace child tag with a shared parent, move to Someday, close task, other field edits, and no-op save. Rename, other fields, no-op saves, and live-list moves write zero membership/badge rows; tag additions/removals write only their actual differences. All retained-membership reviewer checks pass.

Gates on 10by32-01-std arm64, all exit 0:

cargo fmt --all --check
/usr/local/bin/build-slot.sh cargo clippy --workspace --all-targets -j8 -- -D warnings
/usr/local/bin/build-slot.sh cargo test --workspace -j8
/usr/local/bin/build-slot.sh cargo test -j8 -p calternal-tasks

Full workspace: 108 successful test-process reports, 1,364 reported passes / 0 failures / 27 ignored, including nested-child repetitions. Includes cold-fill/index equivalence, task correctness/crash recovery, all #767 reviewer tests, 5,000 random edits/renames/tags/deletes against the independent model, dev-cache rebuild, and immediate-after-ready growth/quota behavior. The focused Tasks run passed 61 tests, 0 ignored.

As requested, CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse was omitted: that directory does not exist on this machine, so the dedicated inode-reuse filesystem exercise was not performed here. This run makes no cold-growth performance measurement claim.

Logs: /home/agent/codex-jobs/fix-767/{repro,delta-red,tasks-green,fmt,clippy,workspace}.log. Worktree: /home/agent/wt/cold-growth-fix. target/ deleted and lease 938679b5d8264326ad0ed41039ee99a6 heartbeat stopped. PR remains draft.

Fixed P2 comment #69795 in 30f2716fac7e343780446ba873240eed4d02e2f1, pushed only to `perf/cold-growth-on-764`. Fetched first and fast-forward merged reviewer tests `d7820b5a` onto the requested `2df97870` head. Both original ignored reproducers failed before the fix with **4 unchanged-membership writes, expected 0**. Their assertions are preserved; both are now enabled in the default suite. Task renames retain tag membership for the same stable identity while updating path/link tracking independently. Tag updates compare sorted old/new membership sets and delete/insert only their differences; only changed memberships update badge counts. Cold fills still defer badge aggregation until `after_fill`. Added SQLite trigger counters for membership and badge writes covering rename, add tag, remove child tag, replace child tag with a shared parent, move to Someday, close task, other field edits, and no-op save. Rename, other fields, no-op saves, and live-list moves write zero membership/badge rows; tag additions/removals write only their actual differences. All retained-membership reviewer checks pass. Gates on **10by32-01-std arm64**, all exit 0: ``` cargo fmt --all --check /usr/local/bin/build-slot.sh cargo clippy --workspace --all-targets -j8 -- -D warnings /usr/local/bin/build-slot.sh cargo test --workspace -j8 /usr/local/bin/build-slot.sh cargo test -j8 -p calternal-tasks ``` Full workspace: **108 successful test-process reports, 1,364 reported passes / 0 failures / 27 ignored**, including nested-child repetitions. Includes cold-fill/index equivalence, task correctness/crash recovery, all #767 reviewer tests, 5,000 random edits/renames/tags/deletes against the independent model, dev-cache rebuild, and immediate-after-ready growth/quota behavior. The focused Tasks run passed **61 tests, 0 ignored**. As requested, `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse` was omitted: that directory does not exist on this machine, so the dedicated inode-reuse filesystem exercise was not performed here. This run makes no cold-growth performance measurement claim. Logs: `/home/agent/codex-jobs/fix-767/{repro,delta-red,tasks-green,fmt,clippy,workspace}.log`. Worktree: `/home/agent/wt/cold-growth-fix`. `target/` deleted and lease `938679b5d8264326ad0ed41039ee99a6` heartbeat stopped. PR remains draft.
kayg commented 2026-10-29 18:23:35 +00:00
Author
Owner
Copy link

Independent verification, round 2, exact head 30f2716f (2df97870..30f2716f).

Verdict: ship. Round-1 P2 is resolved. Four independent tests pass: exact delta write counts for rename, add/remove tag, list transitions and no-op; 1,000-task rename/field edits (0 writes) and tag additions (1,000); real room-protocol Yjs undo/redo; 3,000 random operations checked against a model, task identities/list eligibility and badge totals; and subprocess exit inside the production diff after its first delete, followed by WAL recovery and retry.

Earlier #767 reviewer assertions are intact and their two reproducer tests are now unignored. #755 membership/crash/migration reviewer tests remain intact. Against perf/quota-simple, tests gain coverage and adapt schema comparisons; no reviewer case is removed. Older quota tests do deliberately allow bounded retryable accounting failures under churn, with eventual recovery checks; those changes precede this fix.

Gates: fmt PASS; clippy --workspace --all-targets -j8 -- -D warnings PASS; full workspace tests PASS (cargo test --workspace --no-fail-fast -j8, exit 0, including doc tests). Builds/tests ran on 10by32-01-std via /usr/local/bin/build-slot.sh, -j8.

P3, inherited/outside this diff: auth/tests/migration.rs:86 intermittently returns 1 person instead of 2. One workspace run failed; an isolated rerun passed, and the same workspace-built binary failed once in 20 repeats. Likely fixture race: millisecond creation-time ties let the legacy ORDER BY created_at,id choose Member as admin. Keep all assertions and stabilize fixture chronology. Details/reproducer are in the report.

Forgejo mergeability: current git merge-tree --write-tree origin/dev 30f2716f exits 0. Conflicting files: none; no conflict-resolution difficulty to classify. Current dev d83e54db is an ancestor of the PR head. Forgejo says draft=true/mergeable=false while API head is 30f2716f, but refs/pull/767/head remains 2df97870 and there is no advertised merge ref. This is inconsistent/stale PR merge state, not a current file conflict. Git alone cannot establish why Forgejo has not recomputed it.

Tests only: review/767b-tests, fda3803c. Full report and logs: /home/agent/codex-jobs/verify-767b/findings.md. No merges, deployments or perf-test-* changes.

Independent verification, round 2, exact head `30f2716f` (`2df97870..30f2716f`). **Verdict: ship.** Round-1 P2 is resolved. Four independent tests pass: exact delta write counts for rename, add/remove tag, list transitions and no-op; 1,000-task rename/field edits (0 writes) and tag additions (1,000); real room-protocol Yjs undo/redo; 3,000 random operations checked against a model, task identities/list eligibility and badge totals; and subprocess exit inside the production diff after its first delete, followed by WAL recovery and retry. Earlier #767 reviewer assertions are intact and their two reproducer tests are now unignored. #755 membership/crash/migration reviewer tests remain intact. Against `perf/quota-simple`, tests gain coverage and adapt schema comparisons; no reviewer case is removed. Older quota tests do deliberately allow bounded retryable accounting failures under churn, with eventual recovery checks; those changes precede this fix. **Gates:** fmt PASS; clippy `--workspace --all-targets -j8 -- -D warnings` PASS; full workspace tests PASS (`cargo test --workspace --no-fail-fast -j8`, exit 0, including doc tests). Builds/tests ran on 10by32-01-std via `/usr/local/bin/build-slot.sh`, `-j8`. **P3, inherited/outside this diff:** `auth/tests/migration.rs:86` intermittently returns 1 person instead of 2. One workspace run failed; an isolated rerun passed, and the same workspace-built binary failed once in 20 repeats. Likely fixture race: millisecond creation-time ties let the legacy `ORDER BY created_at,id` choose Member as admin. Keep all assertions and stabilize fixture chronology. Details/reproducer are in the report. **Forgejo mergeability:** current `git merge-tree --write-tree origin/dev 30f2716f` exits 0. Conflicting files: **none**; no conflict-resolution difficulty to classify. Current dev `d83e54db` is an ancestor of the PR head. Forgejo says draft=true/mergeable=false while API head is `30f2716f`, but `refs/pull/767/head` remains `2df97870` and there is no advertised merge ref. This is inconsistent/stale PR merge state, not a current file conflict. Git alone cannot establish why Forgejo has not recomputed it. Tests only: [`review/767b-tests`, `fda3803c`](https://git.kayg.org/kayg/calternal/src/branch/review/767b-tests). Full report and logs: `/home/agent/codex-jobs/verify-767b/findings.md`. No merges, deployments or perf-test-* changes.
kayg commented 2026-10-29 21:43:02 +00:00
Author
Owner
Copy link

Updated head: 4481816d4120325c540ada41507e3cc1524d0665.

Merged origin/perf/quota-simple at a506b739 with a normal merge (7edee1eb). Kept #764's quota implementation and the existing reviewer regression test in the sole conflict. Preserved the ported #754/#764 reviewer coverage and added the missing review/767b-tests commit (fda3803c, cherry-picked as 67c6c4ea); both #767 reviewer sets are present.

calternal-perf uploads now replay only explicitly retryable 503 storage_busy and 429 responses. Retries keep the same body and idempotency key, honour numeric/date Retry-After, add full jitter above the server minimum, follow the web client's attempt limits, and share a 30-second deadline including network I/O. Non-retryable errors return immediately. Scenario metrics include uploads.retries and uploads.time_to_success; uploads.response also includes the complete retry wait. Failure messages include elapsed time and retry count.

Gates on 10by32-01-std (arm64), using /usr/local/bin/build-slot.sh and -j8 for cargo build/test/clippy work:

  • fmt; clippy -D warnings: PASS (cargo fmt --all --check; cargo clippy --workspace --all-targets -j8 -- -D warnings).
  • full workspace tests: PASS (cargo test --workspace --no-fail-fast -j8). CALTERNAL_INODE_REUSE_DIR exists only on 10by32-02-std; the physical inode-reuse test was skipped here. Default ignored perf/stress/diagnostic tests remain ignored.
  • cold/task correctness tests: PASS (cargo test -p calternal-tasks --lib -j8: 65 passed, including cold-fill equivalence, migration, membership/model, and real undo/redo checks). Upload HTTP regressions: 5 passed, covering replay identity, refusal classification, Retry-After parsing, attempt exhaustion, and stalled-I/O deadline.
  • reviewer tests: PASS (included in full workspace run plus cargo test --workspace -j8 review -- --nocapture). Web upload retry/reviewer tests: 16 passed, zero skipped with REVIEW_764D_REQUIRE_JITTER=1.

The first workspace attempt exposed a pre-existing migration-fixture race: owner/member creation could share a millisecond and the v2 migration chose its admin by random user-ID order. The fixture now supplies an explicit owner-first chronology; production migration code is unchanged. Its focused rerun passed.

Cold-path inspection: fresh fills already skip old-membership reads, defer secondary indexes, and aggregate tag counts after filling. Remaining work includes per-task tag expansion (BTreeSet allocation followed by collection), folded path keys, narrow membership inserts, and final index builds/membership resolution. I found no further change clearly safe to make without measurement, so cold-path production code is unchanged. No perf measurements were run on this host; the 100k cold-growth margin remains unverified for this head.

Updated head: `4481816d4120325c540ada41507e3cc1524d0665`. Merged `origin/perf/quota-simple` at `a506b739` with a normal merge (`7edee1eb`). Kept #764's quota implementation and the existing reviewer regression test in the sole conflict. Preserved the ported #754/#764 reviewer coverage and added the missing `review/767b-tests` commit (`fda3803c`, cherry-picked as `67c6c4ea`); both #767 reviewer sets are present. `calternal-perf` uploads now replay only explicitly retryable `503 storage_busy` and `429` responses. Retries keep the same body and idempotency key, honour numeric/date `Retry-After`, add full jitter above the server minimum, follow the web client's attempt limits, and share a 30-second deadline including network I/O. Non-retryable errors return immediately. Scenario metrics include `uploads.retries` and `uploads.time_to_success`; `uploads.response` also includes the complete retry wait. Failure messages include elapsed time and retry count. Gates on 10by32-01-std (arm64), using `/usr/local/bin/build-slot.sh` and `-j8` for cargo build/test/clippy work: - fmt; clippy -D warnings: PASS (`cargo fmt --all --check`; `cargo clippy --workspace --all-targets -j8 -- -D warnings`). - full workspace tests: PASS (`cargo test --workspace --no-fail-fast -j8`). `CALTERNAL_INODE_REUSE_DIR` exists only on 10by32-02-std; the physical inode-reuse test was skipped here. Default ignored perf/stress/diagnostic tests remain ignored. - cold/task correctness tests: PASS (`cargo test -p calternal-tasks --lib -j8`: 65 passed, including cold-fill equivalence, migration, membership/model, and real undo/redo checks). Upload HTTP regressions: 5 passed, covering replay identity, refusal classification, Retry-After parsing, attempt exhaustion, and stalled-I/O deadline. - reviewer tests: PASS (included in full workspace run plus `cargo test --workspace -j8 review -- --nocapture`). Web upload retry/reviewer tests: 16 passed, zero skipped with `REVIEW_764D_REQUIRE_JITTER=1`. The first workspace attempt exposed a pre-existing migration-fixture race: owner/member creation could share a millisecond and the v2 migration chose its admin by random user-ID order. The fixture now supplies an explicit owner-first chronology; production migration code is unchanged. Its focused rerun passed. Cold-path inspection: fresh fills already skip old-membership reads, defer secondary indexes, and aggregate tag counts after filling. Remaining work includes per-task tag expansion (`BTreeSet` allocation followed by collection), folded path keys, narrow membership inserts, and final index builds/membership resolution. I found no further change clearly safe to make without measurement, so cold-path production code is unchanged. No perf measurements were run on this host; the 100k cold-growth margin remains unverified for this head.
kayg commented 2026-11-01 11:51:49 +00:00
Author
Owner
Copy link

quota/cold r4 retained results

Budgets: typing CPU <=2%; idle CPU <=0.1%; idle RSS <=100 MB; wakeups <=0.2/s. Six matched pairs per comparison required; matrix stopped at quota-4.

sample typing CPU % idle CPU % RSS MB wakeups/s verdict
cold-1 0.567 0.000 83.953 0.000 PASS
cold-2 0.483 0.000 76.297 0.000 PASS
cold-3 0.583 0.000 84.355 0.000 PASS
dev-cold-1 1.383 0.740 125.262 0.097 FAIL idle CPU and RSS
dev-cold-2 1.333 0.697 129.840 0.073 FAIL idle CPU and RSS
dev-cold-3 1.333 0.720 122.289 0.087 FAIL idle CPU and RSS
dev-quota-1 1.250 0.600 112.441 0.117 EXCLUDED: diagnostic walk overlapped
dev-quota-2 1.283 0.710 114.918 0.117 FAIL idle CPU and RSS
dev-quota-3 1.233 0.687 118.234 0.080 FAIL idle CPU and RSS
dev-quota-4 1.367 0.717 120.625 0.087 FAIL idle CPU and RSS; unpaired
quota-1 0.600 0.000 85.453 0.000 EXCLUDED: paired baseline excluded
quota-2 0.567 0.000 82.914 0.000 PASS
quota-3 0.567 0.000 83.332 0.000 PASS
quota-4 — — — — INVALID: first page empty/incomplete

13 runs completed, one failed run (quota-4). Two completed samples are excluded from matched comparisons. Available accepted pairs: quota rounds 2–3 (2/6); cold rounds 1–3 (3/6). dev-quota-4 is valid but unpaired. No six-pair qualification or cold-growth gate is claimed. Remaining runs: quota-4 retry, dev-cold-4/cold-4, and every arm of rounds 5–6 (11 invocations total).

disposition, diagnosis and reproduction

The r4 script had already exited after quota-4; no quota r4 process or measurement lock remained to stop. typing-idle.sh writes DONE and rc=0 even when the driver records scenario failures (it does not pass --gate). The outer r4 script correctly rejects a nonempty sizes[0].failures; that check caught quota-4. Treating DONE or rc=0 alone as success would be a harness error.

Exact failure: HTTP 200, result={"incomplete":true,"items":[],"next":null,"total":0}. There was no quota refusal or failed HTTP status. The failed fixture's derived notes/meta.db had zero notes, schema 4, stats notes=0, and PRAGMA quick_check returned ok. The retained warm server log contains the development-mode warning, with no recorded scan/disk error. The first-page harness in crates/perf/src/scenarios/home.rs rejects any empty list immediately; server.rs waits only for /healthz. Notes explicitly allow a partial list before Home discovery completes (Notes::page and the_first_page_never_waits_for_the_home_to_open). incomplete:true alone is not evidence of a product defect.

Functional reproduction on dev-01, using a copy of the exact failed quota-4 userdata and appdata and the existing quota binary (SHA256 9c055318e954f79d0309dc6c99637eff98df35fba06c506cb28167730a10078d), under ~/bin/slot, with no build or perf claim:

  1. Start calternal serve --create --listen 127.0.0.1:<free-port> --dev-user 0192f0a0-0000-7000-8000-00000000f1f0 --data <copy>/runs/n100000/indexed --state <copy>/appdata/runs/n100000/indexed, with GLIBC_TUNABLES=glibc.malloc.arena_max=2.
  2. Wait only for healthz, then POST /api/v1/actions/notes.list with {"limit":50}, Content-Type: application/json and X-Requested-With: calternal. At 0.021s: 0 total / 0 items / incomplete=true, reproducing the r4 assertion failure.
  3. Repeat the request once per second. At 10.457s: 32,768 total; at 20.530s: 61,440; at 30.965s: 100,000 but still incomplete; at 32.975s: 100,000 / 50 items / incomplete=false. Stop this exact child with SIGTERM and wait; exit 0.
  4. Restart the same repaired copy. At 0.021s: 100,000 / 50 items / incomplete=true; at 5.031s: incomplete=false. SIGTERM and wait; exit 0.

Verdict: observed failure is a harness readiness/precondition failure on a rebuilding cache, not evidence of a persistent list failure under quota or cold growth. The source of the earlier derived-cache reset is not established. MetaDb::open_with on the measured quota head silently removes/rebuilds the derived DB after any open error, so a storage/open problem is a possible cause; neither corruption, ENOSPC nor a quota-specific regression has been proved. Do not call this root cause closed: a separate clean rerun needs logs of the first open error and cache counts before/after cold-to-warm handoff. Keep a genuine complete-first-page warm-start gate if that is the intended product requirement; otherwise explicitly wait for readiness outside typing/idle measurement, with a deadline and preserved startup timing. Do not silently weaken that gate or count quota-4 as passed.

Measured stack is #764 75978ddb5ed30152a85fbb1d766a62d5c5a73113 and #767 8b5b8cdc74385b3bcbf1540d1c612c075952ef06 (the quota/cold-on-tiering work), using the common driver SHA256 2e3656cd3255e5d6b590705e2c5ea521f8d1dd7642fcb26b7f3aa8eaad94e4fc, and typing script 13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b. Successful retained candidate samples pass all four absolute typing/idle budgets. This is not a completed paired qualification and does not establish the cold-growth <=10% gate.

Small JSON/MD reports, run logs, scripts and reproduction observations are retained on perf-02 at ~/perf-reports-kept/takeover-20261101/quota-r4/ and on control at /home/kayg/perf-reports/takeover-20261101/. Seeded r4 work/Homes are removed after copying the diagnostic fixture; the dev-01 copy is removed after both exact child processes have exited. The previous r2 cleanup was left untouched until it finished; only then were seeded Markdown Homes inadvertently copied into its report directory removed, retaining its real top-level reports. No merge or deployment.

Final cleanup verification

Both 759b units stopped; quota r4 had already exited. All takeover cleanup and reproduction processes have exited. Earlier r2 cleanup completed untouched. Final free: perf-01 root 51G, split Home 58G; perf-02 root 37G, split Home 97G. Shared locks are held by #792 qualification (159697 / 1325428), not the stopped jobs. Cleanup/fixture-copy I/O overlapped the first #792 samples; those samples need overlap exclusion. Retained observations do not establish why quota-4 cache was reset.

Perf-02 cleanup and fixture-copy overlap was approximately 17:13–17:22 IST on 1 November (11:43–11:52 UTC). The earlier r2 file walk also overlapped #792 startup until it completed. #792 first overlapping samples must not be treated as uncontended measurements. No #792 process was stopped.

# quota/cold r4 retained results Budgets: typing CPU <=2%; idle CPU <=0.1%; idle RSS <=100 MB; wakeups <=0.2/s. Six matched pairs per comparison required; matrix stopped at quota-4. | sample | typing CPU % | idle CPU % | RSS MB | wakeups/s | verdict | |---|---:|---:|---:|---:|---| | cold-1 | 0.567 | 0.000 | 83.953 | 0.000 | PASS | | cold-2 | 0.483 | 0.000 | 76.297 | 0.000 | PASS | | cold-3 | 0.583 | 0.000 | 84.355 | 0.000 | PASS | | dev-cold-1 | 1.383 | 0.740 | 125.262 | 0.097 | FAIL idle CPU and RSS | | dev-cold-2 | 1.333 | 0.697 | 129.840 | 0.073 | FAIL idle CPU and RSS | | dev-cold-3 | 1.333 | 0.720 | 122.289 | 0.087 | FAIL idle CPU and RSS | | dev-quota-1 | 1.250 | 0.600 | 112.441 | 0.117 | EXCLUDED: diagnostic walk overlapped | | dev-quota-2 | 1.283 | 0.710 | 114.918 | 0.117 | FAIL idle CPU and RSS | | dev-quota-3 | 1.233 | 0.687 | 118.234 | 0.080 | FAIL idle CPU and RSS | | dev-quota-4 | 1.367 | 0.717 | 120.625 | 0.087 | FAIL idle CPU and RSS; unpaired | | quota-1 | 0.600 | 0.000 | 85.453 | 0.000 | EXCLUDED: paired baseline excluded | | quota-2 | 0.567 | 0.000 | 82.914 | 0.000 | PASS | | quota-3 | 0.567 | 0.000 | 83.332 | 0.000 | PASS | | quota-4 | — | — | — | — | INVALID: first page empty/incomplete | 13 runs completed, one failed run (quota-4). Two completed samples are excluded from matched comparisons. Available accepted pairs: quota rounds 2–3 (2/6); cold rounds 1–3 (3/6). dev-quota-4 is valid but unpaired. No six-pair qualification or cold-growth gate is claimed. Remaining runs: quota-4 retry, dev-cold-4/cold-4, and every arm of rounds 5–6 (11 invocations total). ## disposition, diagnosis and reproduction The r4 script had already exited after quota-4; no quota r4 process or measurement lock remained to stop. `typing-idle.sh` writes DONE and `rc=0` even when the driver records scenario failures (it does not pass `--gate`). The outer r4 script correctly rejects a nonempty `sizes[0].failures`; that check caught quota-4. Treating DONE or rc=0 alone as success would be a harness error. Exact failure: HTTP 200, `result={"incomplete":true,"items":[],"next":null,"total":0}`. There was no quota refusal or failed HTTP status. The failed fixture's derived `notes/meta.db` had zero notes, schema 4, stats `notes=0`, and `PRAGMA quick_check` returned `ok`. The retained warm server log contains the development-mode warning, with no recorded scan/disk error. The first-page harness in `crates/perf/src/scenarios/home.rs` rejects *any* empty list immediately; `server.rs` waits only for `/healthz`. Notes explicitly allow a partial list before Home discovery completes (`Notes::page` and `the_first_page_never_waits_for_the_home_to_open`). `incomplete:true` alone is not evidence of a product defect. Functional reproduction on **dev-01**, using a copy of the exact failed quota-4 userdata and appdata and the existing quota binary (SHA256 `9c055318e954f79d0309dc6c99637eff98df35fba06c506cb28167730a10078d`), under `~/bin/slot`, with no build or perf claim: 1. Start `calternal serve --create --listen 127.0.0.1:<free-port> --dev-user 0192f0a0-0000-7000-8000-00000000f1f0 --data <copy>/runs/n100000/indexed --state <copy>/appdata/runs/n100000/indexed`, with `GLIBC_TUNABLES=glibc.malloc.arena_max=2`. 2. Wait only for healthz, then POST `/api/v1/actions/notes.list` with `{"limit":50}`, `Content-Type: application/json` and `X-Requested-With: calternal`. At 0.021s: **0 total / 0 items / incomplete=true**, reproducing the r4 assertion failure. 3. Repeat the request once per second. At 10.457s: 32,768 total; at 20.530s: 61,440; at 30.965s: 100,000 but still incomplete; at **32.975s: 100,000 / 50 items / incomplete=false**. Stop this exact child with SIGTERM and wait; exit 0. 4. Restart the same repaired copy. At **0.021s: 100,000 / 50 items / incomplete=true**; at **5.031s: incomplete=false**. SIGTERM and wait; exit 0. **Verdict: observed failure is a harness readiness/precondition failure on a rebuilding cache, not evidence of a persistent list failure under quota or cold growth.** The source of the earlier derived-cache reset is not established. `MetaDb::open_with` on the measured quota head silently removes/rebuilds the derived DB after any open error, so a storage/open problem is a possible cause; neither corruption, ENOSPC nor a quota-specific regression has been proved. Do not call this root cause closed: a separate clean rerun needs logs of the first open error and cache counts before/after cold-to-warm handoff. Keep a genuine complete-first-page warm-start gate if that is the intended product requirement; otherwise explicitly wait for readiness outside typing/idle measurement, with a deadline and preserved startup timing. Do not silently weaken that gate or count quota-4 as passed. Measured stack is #764 `75978ddb5ed30152a85fbb1d766a62d5c5a73113` and #767 `8b5b8cdc74385b3bcbf1540d1c612c075952ef06` (the quota/cold-on-tiering work), using the common driver SHA256 `2e3656cd3255e5d6b590705e2c5ea521f8d1dd7642fcb26b7f3aa8eaad94e4fc`, and typing script `13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b`. Successful retained candidate samples pass all four absolute typing/idle budgets. This is not a completed paired qualification and does not establish the cold-growth <=10% gate. Small JSON/MD reports, run logs, scripts and reproduction observations are retained on perf-02 at `~/perf-reports-kept/takeover-20261101/quota-r4/` and on control at `/home/kayg/perf-reports/takeover-20261101/`. Seeded r4 work/Homes are removed after copying the diagnostic fixture; the dev-01 copy is removed after both exact child processes have exited. The previous r2 cleanup was left untouched until it finished; only then were seeded Markdown Homes inadvertently copied into its report directory removed, retaining its real top-level reports. No merge or deployment. ### Final cleanup verification Both 759b units stopped; quota r4 had already exited. All takeover cleanup and reproduction processes have exited. Earlier r2 cleanup completed untouched. Final free: perf-01 root 51G, split Home 58G; perf-02 root 37G, split Home 97G. Shared locks are held by #792 qualification (159697 / 1325428), not the stopped jobs. Cleanup/fixture-copy I/O overlapped the first #792 samples; those samples need overlap exclusion. Retained observations do not establish why quota-4 cache was reset. Perf-02 cleanup and fixture-copy overlap was approximately **17:13–17:22 IST on 1 November** (11:43–11:52 UTC). The earlier r2 file walk also overlapped #792 startup until it completed. #792 first overlapping samples must not be treated as uncontended measurements. No #792 process was stopped.
kayg commented 2026-11-02 03:18:56 +00:00
Author
Owner
Copy link

paired requalification blocked by rebase conflict

Current PR head: 8b5b8cdc74385b3bcbf1540d1c612c075952ef06 (perf/cold-growth-on-764). Target checked: origin/dev 55f35fcde3bc91ca3781055577b93e93819f6cd4.

An isolated test branch ran git rebase origin/dev. It stopped at step 2/74, applying aef9715e (WIP fixes), with content conflicts in docs/storage.md:

  • storage root table: current dev's data/state paths and .calternal/settings.json versus the stacked userdata/appdata paths, prefs and additional task/contact state;
  • migration instructions: current dev's coordinated manual move versus the stacked storage split procedure.

These describe storage behavior and recovery procedures, so I treated them as non-trivial and aborted the rebase, as requested. No conflict resolution, source edit, PR branch push or merge was performed. Both isolated worktrees returned clean to their original heads. Further conflicts beyond this first stop have not been assessed.

qualification gate new evidence verdict
rebase onto current dev conflict above; rebase aborted blocked
bounded nonempty-first-page harness readiness known r4 failure reviewed; fix not implemented or tested pending
complete candidate/dev pairs, same VM 0/6 new pairs not qualified
typing CPU ≤2% no new samples unmeasured
idle CPU ≤0.1% no new samples unmeasured
idle RSS ≤100 MB no new samples unmeasured
idle wakeups ≤0.2/s no new samples unmeasured
cold-growth ≤10% (where applicable) no new samples unmeasured

The r4 template on perf-02 was read; its partial retained results remain historical evidence, not a completed qualification. No builds or performance jobs were started, no performance locks were taken, and no seeded Homes or large artifacts were created. Both VMs remain available. Resume after the storage conflict is resolved by the implementation owner: fix the bounded readiness precondition, build on dev-01, relay and SHA256-check binaries, then run six fresh matched pairs per comparison split across perf-01/perf-02 under ~/bin/perf-lock.

Verdict: qualification FAIL (blocked/unmeasured), not a measured product regression.

## paired requalification blocked by rebase conflict Current PR head: `8b5b8cdc74385b3bcbf1540d1c612c075952ef06` (`perf/cold-growth-on-764`). Target checked: origin/dev `55f35fcde3bc91ca3781055577b93e93819f6cd4`. An isolated test branch ran `git rebase origin/dev`. It stopped at step 2/74, applying `aef9715e` (`WIP fixes`), with content conflicts in `docs/storage.md`: - storage root table: current dev's data/state paths and `.calternal/settings.json` versus the stacked userdata/appdata paths, prefs and additional task/contact state; - migration instructions: current dev's coordinated manual move versus the stacked `storage split` procedure. These describe storage behavior and recovery procedures, so I treated them as non-trivial and aborted the rebase, as requested. No conflict resolution, source edit, PR branch push or merge was performed. Both isolated worktrees returned clean to their original heads. Further conflicts beyond this first stop have not been assessed. | qualification gate | new evidence | verdict | |---|---|---| | rebase onto current dev | conflict above; rebase aborted | blocked | | bounded nonempty-first-page harness readiness | known r4 failure reviewed; fix not implemented or tested | pending | | complete candidate/dev pairs, same VM | 0/6 new pairs | not qualified | | typing CPU ≤2% | no new samples | unmeasured | | idle CPU ≤0.1% | no new samples | unmeasured | | idle RSS ≤100 MB | no new samples | unmeasured | | idle wakeups ≤0.2/s | no new samples | unmeasured | | cold-growth ≤10% (where applicable) | no new samples | unmeasured | The r4 template on perf-02 was read; its partial retained results remain historical evidence, not a completed qualification. No builds or performance jobs were started, no performance locks were taken, and no seeded Homes or large artifacts were created. Both VMs remain available. Resume after the storage conflict is resolved by the implementation owner: fix the bounded readiness precondition, build on dev-01, relay and SHA256-check binaries, then run six fresh matched pairs per comparison split across perf-01/perf-02 under `~/bin/perf-lock`. **Verdict: qualification FAIL (blocked/unmeasured), not a measured product regression.**
kayg commented 2026-11-07 18:21:40 +00:00
Author
Owner
Copy link

Housekeeping #1009 status: audited head 8b5b8cdc74385b3bcbf1540d1c612c075952ef06 against dev cd7f84432feb. git cherry origin/dev <head> reports 74 unmatched commits and 0 equivalent patches. Separate stable patch-id indexing and key-file checks do not establish full landing. Absent from dev: apps/web/src/lib/upload-retry-jitter.test.mjs, apps/web/src/lib/upload-retry-review-764b.test.mjs, apps/web/src/lib/upload-retry-review-764c.test.mjs. Left unchanged as WIP, per kayg’s instruction; no rebase, close, tests, or merge. The #1009 audit records the unmatched commit list.

Housekeeping #1009 status: audited head `8b5b8cdc74385b3bcbf1540d1c612c075952ef06` against dev `cd7f84432feb`. `git cherry origin/dev <head>` reports 74 unmatched commits and 0 equivalent patches. Separate stable patch-id indexing and key-file checks do not establish full landing. Absent from dev: `apps/web/src/lib/upload-retry-jitter.test.mjs`, `apps/web/src/lib/upload-retry-review-764b.test.mjs`, `apps/web/src/lib/upload-retry-review-764c.test.mjs`. Left unchanged as WIP, per kayg’s instruction; no rebase, close, tests, or merge. The #1009 audit records the unmatched commit list.
Some checks failed
CI / server (pull_request) Successful in 1m32s
Details
CI / rust (pull_request) Failing after 8m55s
Details
CI / web (pull_request) Has been cancelled
Details
This pull request has changes conflicting with the target branch.
  • apps/web/src/lib/backdrops.ts
  • crates/attachments/src/actions.rs
  • crates/contacts-actions/src/tests/review3.rs
  • crates/fsio/src/lib.rs
  • crates/fsio/src/swap.rs
  • crates/fsio/src/tests.rs
  • crates/fsio/src/usage.rs
  • crates/notes/src/file_stamp.rs
  • crates/notes/src/meta.rs
  • crates/notes/src/peek.rs
  • crates/notes/src/room/actor.rs
  • crates/notes/src/scan.rs
  • crates/notes/src/search.rs
  • crates/notes/src/tests/listing.rs
  • crates/notes/src/tests/saving.rs
  • crates/notes/src/watch.rs
  • crates/store/src/tests.rs
  • crates/tasks/Cargo.toml
  • crates/tasks/src/counts.rs
  • crates/tasks/src/found.rs
  • crates/tasks/src/found/tests.rs
  • crates/tasks/src/index.rs
  • crates/tasks/src/lib.rs
  • crates/tasks/src/lists.rs
  • crates/tasks/src/tests.rs
  • docs/storage.md
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin perf/cold-growth-on-764:perf/cold-growth-on-764
git switch perf/cold-growth-on-764

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff perf/cold-growth-on-764
git switch perf/cold-growth-on-764
git rebase dev
git switch dev
git merge --ff-only perf/cold-growth-on-764
git switch perf/cold-growth-on-764
git rebase dev
git switch dev
git merge --no-ff perf/cold-growth-on-764
git switch dev
git merge --squash perf/cold-growth-on-764
git switch dev
git merge --ff-only perf/cold-growth-on-764
git switch dev
git merge perf/cold-growth-on-764
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!767
No description provided.