kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

WIP: Cold growth + fast startup on #754 #760

Closed
kayg wants to merge 41 commits from perf/cold-growth-on-754 into perf/idle-honest-2
pull from: perf/cold-growth-on-754
merge into: kayg:perf/idle-honest-2
Conversation 2 Commits 41 Files changed 85 +5676 -1155
kayg commented 2026-10-29 05:55:11 +00:00
Owner
Copy link

The measured integration base is the task’s explicitly requested e8a99a6fee. During this run, the target branch perf/idle-honest-2 advanced to a6b1fa14d2. PR #760 is a draft with conflicts against that newer head in crates/auth/src/quota.rs, crates/fsio/src/usage.rs and crates/fsio/src/usage/watch.rs. The tested source remains d23e80b9 on e8a99a6f. Newer quota retry and admission changes are outside these measurements; integration with a6b1fa14 remains follow-up work. The open-PR branch rule requires a new branch and PR for any further code update.

Uploads made while quota watches are being registered now measure the uploading Home through #754's bounded resolver before deciding. Each admission remeasures outside growth, deletion and cross-person moves until native coverage and its closing recount succeed. Retained storage failures require a full successful recount; fallback also keeps fresh admission checks active. Readiness and reads stay fast.

This combines #755's cold index and covering task-membership work with #754 head e8a99a6f. The resolver, inode accounting and reviewer tests are retained. Native child names remain exact OsStr names with descriptor-relative metadata. Reviewer branch review/755-tests at edba00ec is merged. The watch registry stores tokens and uses the existing event index for selected paths, reducing repeated bookkeeping at 100k entries.

Formatting, workspace clippy with -D warnings, the release build and all 1302 workspace tests pass (21 ignored). The Tasks abrupt-stop test passed on its first run; no #757 rerun was needed. Reviewer assertions remain enabled.

Three fresh full server reports per size ran on perf-test-01, with HDD xfs Homes and SSD ext4 state. Each has five warm starts, three unindexed starts and a 300-second idle window. Each task comparison has 15 alternating cold pairs and includes shutdown, an empty WAL and sync -f. Every requested cache drop succeeded through the approved helper. Budgets are unchanged.

Notes Report healthz p95 ms Cold growth % Child-tag p95 ms Idle CPU % Wakeups/s Idle RSS MB open.note p95 ms
10000 1 78.200889 6.453428 2.899438 0.000000 0.000000 36.218750 6.477991
10000 2 101.096701 6.085047 2.676665 0.000000 0.000000 36.722656 7.261425
10000 3 89.935247 6.356895 2.787623 0.000000 0.000000 37.843750 5.983108
100000 1 115.323829 4.737185 15.911452 0.000000 0.000000 97.484375 19.793982
100000 2 127.534600 4.618282 17.097791 0.000000 0.000000 93.187500 20.753939
100000 3 109.077354 5.505946 16.795405 0.000000 0.000000 96.007812 18.815419

All requested startup, cold growth, task, child-tag and idle gates pass 3/3 at both sizes. Full server gates remain red for the known open.note work in #731. Over-budget open.note p95 values: 100000 notes report 1: 19.793982 ms; 100000 notes report 2: 20.753939 ms; 100000 notes report 3: 18.815419 ms. All original warnings and scenario results are retained in the full reports.

The first attempt at 30f65caf measured 101.2421875 MB idle RSS at 100k, above budget. Its failed report is retained in #755 comment 68984 and #686 comment 68985. The registry correction was followed by the complete new measurement set above.

Full reports, raw samples and verbatim gate output: report 1, report 2, report 3, report 4, report 5, report 6, report 7, report 8, report 9, report 10, report 11, report 12.

Head: d23e80b949. Fixture Homes and state have been deleted. Refs #755 and #686. No pull request was merged and nothing was deployed. Forgejo uses the WIP title prefix for draft status.

The measured integration base is the task’s explicitly requested e8a99a6feef86aac154083973e7f441f65d05d24. During this run, the target branch perf/idle-honest-2 advanced to a6b1fa14d21944b25fa1538615501f414ae9ddbf. PR #760 is a draft with conflicts against that newer head in crates/auth/src/quota.rs, crates/fsio/src/usage.rs and crates/fsio/src/usage/watch.rs. The tested source remains d23e80b9 on e8a99a6f. Newer quota retry and admission changes are outside these measurements; integration with a6b1fa14 remains follow-up work. The open-PR branch rule requires a new branch and PR for any further code update. Uploads made while quota watches are being registered now measure the uploading Home through #754's bounded resolver before deciding. Each admission remeasures outside growth, deletion and cross-person moves until native coverage and its closing recount succeed. Retained storage failures require a full successful recount; fallback also keeps fresh admission checks active. Readiness and reads stay fast. This combines #755's cold index and covering task-membership work with #754 head e8a99a6f. The resolver, inode accounting and reviewer tests are retained. Native child names remain exact OsStr names with descriptor-relative metadata. Reviewer branch review/755-tests at edba00ec is merged. The watch registry stores tokens and uses the existing event index for selected paths, reducing repeated bookkeeping at 100k entries. Formatting, workspace clippy with -D warnings, the release build and all 1302 workspace tests pass (21 ignored). The Tasks abrupt-stop test passed on its first run; no #757 rerun was needed. Reviewer assertions remain enabled. Three fresh full server reports per size ran on perf-test-01, with HDD xfs Homes and SSD ext4 state. Each has five warm starts, three unindexed starts and a 300-second idle window. Each task comparison has 15 alternating cold pairs and includes shutdown, an empty WAL and sync -f. Every requested cache drop succeeded through the approved helper. Budgets are unchanged. | Notes | Report | healthz p95 ms | Cold growth % | Child-tag p95 ms | Idle CPU % | Wakeups/s | Idle RSS MB | open.note p95 ms | |---:|---:|---:|---:|---:|---:|---:|---:|---:| | 10000 | 1 | 78.200889 | 6.453428 | 2.899438 | 0.000000 | 0.000000 | 36.218750 | 6.477991 | | 10000 | 2 | 101.096701 | 6.085047 | 2.676665 | 0.000000 | 0.000000 | 36.722656 | 7.261425 | | 10000 | 3 | 89.935247 | 6.356895 | 2.787623 | 0.000000 | 0.000000 | 37.843750 | 5.983108 | | 100000 | 1 | 115.323829 | 4.737185 | 15.911452 | 0.000000 | 0.000000 | 97.484375 | 19.793982 | | 100000 | 2 | 127.534600 | 4.618282 | 17.097791 | 0.000000 | 0.000000 | 93.187500 | 20.753939 | | 100000 | 3 | 109.077354 | 5.505946 | 16.795405 | 0.000000 | 0.000000 | 96.007812 | 18.815419 | All requested startup, cold growth, task, child-tag and idle gates pass 3/3 at both sizes. Full server gates remain red for the known open.note work in #731. Over-budget open.note p95 values: 100000 notes report 1: 19.793982 ms; 100000 notes report 2: 20.753939 ms; 100000 notes report 3: 18.815419 ms. All original warnings and scenario results are retained in the full reports. The first attempt at 30f65caf measured 101.2421875 MB idle RSS at 100k, above budget. Its failed report is retained in #755 comment 68984 and #686 comment 68985. The registry correction was followed by the complete new measurement set above. Full reports, raw samples and verbatim gate output: [report 1](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69034), [report 2](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69063), [report 3](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69080), [report 4](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69082), [report 5](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69091), [report 6](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69094), [report 7](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69100), [report 8](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69102), [report 9](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69119), [report 10](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69138), [report 11](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69173), [report 12](https://git.kayg.org/kayg/calternal/issues/686#issuecomment-69178). Head: d23e80b9493b2f3da32ac5c4cd15e09695da41af. Fixture Homes and state have been deleted. Refs #755 and #686. No pull request was merged and nothing was deployed. Forgejo uses the WIP title prefix for draft status.
kayg added 41 commits 2026-10-29 05:55:11 +00:00
Run perf gates locally on the Studio VM
Some checks failed
CI / web (pull_request) Failing after 46s
Details
CI / server (pull_request) Successful in 1m5s
Details
CI / rust (pull_request) Successful in 14m24s
Details
31c49e8b60
Cover list boundaries in multiline reference titles
Some checks failed
CI / web (pull_request) Failing after 50s
Details
CI / server (pull_request) Successful in 1m17s
Details
CI / rust (pull_request) Successful in 13m37s
Details
5ed9dcc7e3
Keep unchanged backlinks when refreshing note metadata
Some checks failed
CI / server (pull_request) Successful in 1m44s
Details
CI / rust (pull_request) Successful in 15m53s
Details
CI / web (pull_request) Failing after 54s
Details
21180d0b2a 
This reverts commit 795b4ad20c.
Borrow plain checkbox fields while building task index rows
Some checks failed
CI / server (pull_request) Successful in 1m7s
Details
CI / rust (pull_request) Successful in 16m8s
Details
CI / web (pull_request) Failing after 37s
Details
e837280e03
Settle filesystem writes when measuring cold index growth
Some checks failed
CI / server (pull_request) Successful in 1m20s
Details
CI / rust (pull_request) Successful in 14m36s
Details
CI / web (pull_request) Failing after 36s
Details
7c4b85d08e
Share Markdown structure between backlinks and describers
Some checks failed
CI / server (pull_request) Successful in 1m18s
Details
CI / rust (pull_request) Successful in 13m55s
Details
CI / web (pull_request) Failing after 40s
Details
0ea217c7e4
Count tags through covering ranges and release transient heaps
Some checks failed
CI / server (pull_request) Successful in 1m6s
Details
CI / rust (pull_request) Successful in 12m44s
Details
CI / web (pull_request) Failing after 39s
Details
135a01a9df 
Plan scan and metadata comparisons on blocking workers using bounded snapshots, then recheck identities and live rooms before committing. Append bounded durable scan-state changes and defer automatic allocator reclamation until bulk work finishes.

Store only live tag membership keys and join current task fields. Compact derived path keys and checkbox locations in the tasks cache codec, and rebuild cache version 16. Preserve fast tag counts and all durability barriers.

Cover snapshot races, key reuse, unchanged memberships, codec vectors and torn scan-state tails. Synchronize the crash test child before its next write.
Stopped because this host (Mac-backed VM) caches reads in host RAM, so its
cold-start numbers are not trusted; measurement moves to perf-test VMs.
Not reviewed; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Cj2Q1hCskP8Frsk6X7EHw
Keep child-tag groups on a covering membership range
Some checks failed
CI / server (pull_request) Successful in 1m34s
Details
CI / rust (pull_request) Failing after 21m33s
Details
CI / web (pull_request) Failing after 43s
Details
1d03e06f4a
Keep quota watch paths only in the event index
Some checks failed
CI / server (pull_request) Successful in 1m1s
Details
CI / rust (pull_request) Failing after 8m34s
Details
CI / web (pull_request) Failing after 2m44s
Details
d23e80b949
kayg commented 2026-10-29 17:54:53 +00:00
Author
Owner
Copy link

Superseded by draft #767: #767 (perf/cold-growth-on-764, head 2df9787047). The cold-growth index and startup-readiness work is now based on #764 d0e6d6f6. Production quota/watch code retains #764's fenced whole-Home recount; #760's incremental quota and registration changes are excluded. Reviewer tests from both lines are retained/adapted. Workspace, clippy, cold/task correctness, physical inode-reuse and five 50,000-case quota properties pass. No performance measurement ran on the shared arm64 builder; x86 builds and perf-VM measurements remain with the coordinator. This comment marks #760 superseded; no merge or deployment was performed.

Superseded by draft #767: https://git.kayg.org/kayg/calternal/pulls/767 (perf/cold-growth-on-764, head 2df978704739f7e734862df53841eedf50a0fef2). The cold-growth index and startup-readiness work is now based on #764 d0e6d6f6. Production quota/watch code retains #764's fenced whole-Home recount; #760's incremental quota and registration changes are excluded. Reviewer tests from both lines are retained/adapted. Workspace, clippy, cold/task correctness, physical inode-reuse and five 50,000-case quota properties pass. No performance measurement ran on the shared arm64 builder; x86 builds and perf-VM measurements remain with the coordinator. This comment marks #760 superseded; no merge or deployment was performed.
kayg commented 2026-11-04 13:04:37 +00:00
Author
Owner
Copy link

Closing: superseded by #767 (named in the PR body or comments). Branch kept.

Closing: superseded by #767 (named in the PR body or comments). Branch kept.
kayg closed this pull request 2026-11-04 13:04:37 +00:00
Some checks failed
CI / server (pull_request) Successful in 1m1s
Details
CI / rust (pull_request) Failing after 8m34s
Details
CI / web (pull_request) Failing after 2m44s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!760
No description provided.