kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

WIP: Quota: simple whole-Home recount (replaces #754 counter) #764

Draft
kayg wants to merge 19 commits from perf/quota-simple into dev
pull from: perf/quota-simple
merge into: kayg:dev
Conversation 17 Commits 19 Files changed 42 +4696 -519
kayg commented 2026-10-29 12:54:15 +00:00
Owner
Copy link

Current stacked revision

Head: 75978ddb5ed30152a85fbb1d766a62d5c5a73113; target: dev.

Stack: #761-A feat/tiering-761-a-migration (5441212fd8b445fb80ff8ed640a4b33c17ae1d07) → #761-B feat/tiering-761-b-reader (58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8) → #764. #767 stacks on this PR. This change needs A+B only; it is not based on feat/tiering-761-s1 and does not include C or D.

Quota totals use userdata; appdata is excluded. Startup fence markers are placed beside Homes so a recursive Home deletion cannot leave a marker behind.

Verified gates

  • cargo fmt --all -- --check: pass.
  • cargo clippy --workspace --all-targets -j4 -- -D warnings: pass.
  • QUOTA_PROPERTY_CASES=50000 cargo test --workspace -j4: pass, exit 0.
  • cargo test -p calternal-fsio -j4 fence_never_blocks_a_recursive_home_delete -- --nocapture: fails on race-red with DirectoryNotEmpty, passes on this head.
  • CALTERNAL_INODE_REUSE_DIR=/home/claude cargo test -p calternal-fsio -j4 review_754d_reused_directory_inode_must_not_certify_a_replacement -- --nocapture: pass; ext4 recycled the directory inode and the replacement was charged. /dev/shm did not recycle it within the test's 10,000 attempts.

Performance qualification

The original same-Home reproduction completed on 86f48a39 with 100k notes and index.db inside the Home. Across three runs, medians were typing CPU 35.483%, idle CPU 0.000%, idle RSS 81.44 MB, and idle wakeups 0.000/s. This reproduces the high typing CPU when the Notes index database is under the Home.

The replacement matrix used perf/vm/typing-idle.sh (SHA256 13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b) through ~/bin/perf-lock, holding /var/lock/calternal-perf.lock on claude@calternal-perf-02-backup01. It completed 13 samples before stopping in quota-4: the warm run's first notes.list page was empty and marked incomplete: true. That failed run has no accepted metrics. The first dev-quota-1 sample is excluded because an external diagnostic file walk overlapped its idle interval; its valid quota-1 partner is excluded to keep the pair matched. Twelve completed samples remain usable, but the six-pair matrix is incomplete and no comparison is claimed. The #759b matrix currently holds the shared lock; continuation will wait for it.

Supplied dev reference (d83e54db, six runs): typing CPU 1.125%, idle CPU 0.647%, idle RSS 106.1 MB, wakeups 0.102/s. Budgets are typing CPU <=2%, idle CPU <=0.1%, RSS <=100 MB, and wakeups <=0.2/s.

Measurement audit

#788 held /var/lock/calternal-perf.lock from 19:26:10Z on perf-01 and 19:28:31Z on perf-02 until 22:28:36Z. Two original-path attempts using the old /tmp/perf.lock at 20:35:12Z and 20:39:41Z are excluded; they produced only fixture/startup output, not typing or idle metrics. No numeric results were discarded for this overlap. The valid same-Home reproduction started at 22:30:01Z, after release.

The later r2 matrix did not overlap #788, but its loop repeated quota-1; these five misattributed samples are excluded from the new medians (typing CPU %, idle CPU %, RSS MB, wakeups/s): dev-quota-1 (1.150, 0.673, 131.98, 0.100), quota-1 (0.467, 0.000, 81.68, 0.000), dev-cold-1 (1.217, 0.550, 112.49, 0.073), cold-1 (0.483, 0.000, 81.05, 0.000), and dev-quota-2 (1.217, 0.723, 126.16, 0.123).

No merge or deployment.

Historical description from earlier head follows.

Replace #754's scope and partial-repair counter with a debounced whole-Home recount. Recounts use directory fds, raw names and no-follow metadata. A single inotify reader flushes sentinel barriers before accepting a count or admitting an upload; inode watches cover outside hard-link writes and mountinfo changes invalidate counts. Hard links still charge each name. Startup stays asynchronous, and incomplete coverage is visible in Health and refuses uploads retryably.

The plan was committed first in 3f7eb017. Admission has one bounded deadline across the upload queue, recount, plugin providers and database queue. App publications add provisional bytes, confirmed by recount. The old scope, retry, directory-index and resolver-chain machinery and unused counter dependencies are removed.

Validation: formatting, strict workspace clippy, fsio tests, 50,000 filesystem cases and 50,000 real Auth admission cases, and full workspace tests. Gate output is posted on #754 and #753. Reviewer behavioral checks were retained or ported; the full test inventory is in a PR comment and the issue inventory.

crates/fsio/src/usage* goes from 5,987 to 3,793 lines, including tests. Local dev-build warm recounts took 49.359 ms at 10k files and 522.726 ms at 100k, with about two syscalls per entry. These are not perf VM numbers. Initial 100k watch setup took 2,066.224 ms untraced, so first uploads may need a retry while the background count finishes. Initial 100k traces exceeded the 30-second recount deadline and were excluded. Measurement method and raw warm strace output.

Related: #754 and #753. No merge or deployment. This forge derives draft status from the WIP title prefix, used before the requested title.

<!-- current stacked revision: 2026-10-31 --> ## Current stacked revision Head: `75978ddb5ed30152a85fbb1d766a62d5c5a73113`; target: `dev`. Stack: #761-A `feat/tiering-761-a-migration` (`5441212fd8b445fb80ff8ed640a4b33c17ae1d07`) → #761-B `feat/tiering-761-b-reader` (`58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8`) → #764. #767 stacks on this PR. This change needs A+B only; it is not based on `feat/tiering-761-s1` and does not include C or D. Quota totals use `userdata`; `appdata` is excluded. Startup fence markers are placed beside Homes so a recursive Home deletion cannot leave a marker behind. ### Verified gates - `cargo fmt --all -- --check`: pass. - `cargo clippy --workspace --all-targets -j4 -- -D warnings`: pass. - `QUOTA_PROPERTY_CASES=50000 cargo test --workspace -j4`: pass, exit 0. - `cargo test -p calternal-fsio -j4 fence_never_blocks_a_recursive_home_delete -- --nocapture`: fails on `race-red` with `DirectoryNotEmpty`, passes on this head. - `CALTERNAL_INODE_REUSE_DIR=/home/claude cargo test -p calternal-fsio -j4 review_754d_reused_directory_inode_must_not_certify_a_replacement -- --nocapture`: pass; ext4 recycled the directory inode and the replacement was charged. `/dev/shm` did not recycle it within the test's 10,000 attempts. ### Performance qualification The original same-Home reproduction completed on `86f48a39` with 100k notes and `index.db` inside the Home. Across three runs, medians were typing CPU 35.483%, idle CPU 0.000%, idle RSS 81.44 MB, and idle wakeups 0.000/s. This reproduces the high typing CPU when the Notes index database is under the Home. The replacement matrix used `perf/vm/typing-idle.sh` (SHA256 `13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b`) through `~/bin/perf-lock`, holding `/var/lock/calternal-perf.lock` on `claude@calternal-perf-02-backup01`. It completed 13 samples before stopping in `quota-4`: the warm run's first `notes.list` page was empty and marked `incomplete: true`. That failed run has no accepted metrics. The first `dev-quota-1` sample is excluded because an external diagnostic file walk overlapped its idle interval; its valid `quota-1` partner is excluded to keep the pair matched. Twelve completed samples remain usable, but the six-pair matrix is incomplete and no comparison is claimed. The #759b matrix currently holds the shared lock; continuation will wait for it. Supplied dev reference (`d83e54db`, six runs): typing CPU 1.125%, idle CPU 0.647%, idle RSS 106.1 MB, wakeups 0.102/s. Budgets are typing CPU <=2%, idle CPU <=0.1%, RSS <=100 MB, and wakeups <=0.2/s. #### Measurement audit #788 held `/var/lock/calternal-perf.lock` from 19:26:10Z on perf-01 and 19:28:31Z on perf-02 until 22:28:36Z. Two original-path attempts using the old `/tmp/perf.lock` at 20:35:12Z and 20:39:41Z are excluded; they produced only fixture/startup output, not typing or idle metrics. No numeric results were discarded for this overlap. The valid same-Home reproduction started at 22:30:01Z, after release. The later r2 matrix did not overlap #788, but its loop repeated `quota-1`; these five misattributed samples are excluded from the new medians (typing CPU %, idle CPU %, RSS MB, wakeups/s): dev-quota-1 `(1.150, 0.673, 131.98, 0.100)`, quota-1 `(0.467, 0.000, 81.68, 0.000)`, dev-cold-1 `(1.217, 0.550, 112.49, 0.073)`, cold-1 `(0.483, 0.000, 81.05, 0.000)`, and dev-quota-2 `(1.217, 0.723, 126.16, 0.123)`. No merge or deployment. <!-- end current stacked revision --> --- Historical description from earlier head follows. Replace #754's scope and partial-repair counter with a debounced whole-Home recount. Recounts use directory fds, raw names and no-follow metadata. A single inotify reader flushes sentinel barriers before accepting a count or admitting an upload; inode watches cover outside hard-link writes and mountinfo changes invalidate counts. Hard links still charge each name. Startup stays asynchronous, and incomplete coverage is visible in Health and refuses uploads retryably. The plan was committed first in 3f7eb017. Admission has one bounded deadline across the upload queue, recount, plugin providers and database queue. App publications add provisional bytes, confirmed by recount. The old scope, retry, directory-index and resolver-chain machinery and unused counter dependencies are removed. Validation: formatting, strict workspace clippy, fsio tests, 50,000 filesystem cases and 50,000 real Auth admission cases, and full workspace tests. Gate output is posted on #754 and #753. Reviewer behavioral checks were retained or ported; the full test inventory is in a PR comment and [the issue inventory](https://git.kayg.org/kayg/calternal/pulls/754#issuecomment-69362). crates/fsio/src/usage* goes from 5,987 to 3,793 lines, including tests. Local dev-build warm recounts took 49.359 ms at 10k files and 522.726 ms at 100k, with about two syscalls per entry. These are not perf VM numbers. Initial 100k watch setup took 2,066.224 ms untraced, so first uploads may need a retry while the background count finishes. Initial 100k traces exceeded the 30-second recount deadline and were excluded. [Measurement method and raw warm strace output](https://git.kayg.org/kayg/calternal/pulls/754#issuecomment-69427). Related: #754 and #753. No merge or deployment. This forge derives draft status from the WIP title prefix, used before the requested title.
kayg added 19 commits 2026-10-29 12:54:16 +00:00
fix(fsio): keep usage totals consistent across aliases and moves
Some checks failed
CI / server (pull_request) Successful in 1m11s
Details
CI / rust (pull_request) Successful in 14m1s
Details
CI / web (pull_request) Has been cancelled
Details
7c2e78cc33
fix(fsio): reconcile storage usage by inode and repair uncertain changes
All checks were successful
CI / server (pull_request) Successful in 1m27s
Details
CI / rust (pull_request) Successful in 15m40s
Details
CI / web (pull_request) Successful in 2h46m25s
Details
81e6141a2e 
Process every consumed repair scope and coalesce retry timers within each Home.
Retire broad retries after validated partial scans identify the failing Homes,
retain unmeasurable overlapping writes, and cancel historical timers on deletion.

Use compact directory fingerprints, reuse saved scan identities, and avoid
recording snapshot publication as another map of overlapping writes. Enable
reviewer reproductions and cover partial publication and retry lifecycle edges.
Replace quota repairs with fenced whole-Home recounts
Some checks failed
CI / server (pull_request) Successful in 1m52s
Details
CI / rust (pull_request) Failing after 10m28s
Details
CI / web (pull_request) Failing after 42m18s
Details
798f1bbe91
kayg commented 2026-10-29 12:54:30 +00:00
Author
Owner
Copy link

Reviewer test inventory for the whole-Home replacement. Names below are no longer present at their old internal locations; each line states its replacement or the removed internal mechanism. Existing public mount integration tests remain unchanged from a6b1fa14.

  • missing_openat2_leaves_the_scope_unmeasured_without_fallback: Ported: the openat2 resolver was removed; direct fstatat/openat walks no longer require that syscall. No-follow and mount behavior remains independently tested.
  • unsupported_handles_exhaust_the_pin_budget_without_publishing: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain.
  • review_754d_open_sub_must_keep_its_ancestry: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain.
  • directory_replaced_by_an_application_file_drops_stale_descendants: Ported: whole-Home scan tests plus the accepted-count oracle cover directory/file replacement; no cached descendants remain.
  • every_failed_scope_in_a_batch_is_retried_before_admission: Ported: review_754g_mixed_batches_keep_every_failure_in_all_positions and the real Auth admission property preserve failure isolation and no stale quota decisions.
  • moving_ancestor_invalidates_unchanged_child_descriptor: Ported: review_754c_rename_at_each_depth_rejects_then_rescans_and_accepts_return tests the public acceptance boundary.
  • removed_directory_is_unmeasured_instead_of_successfully_empty: Ported: deletion is retried to a valid empty snapshot by review_754c_deletion_after_last_stat_discards_snapshot_and_recount_succeeds.
  • rename_at_end_of_walk_invalidates_already_observed_metadata: Ported: review_754c_deletion_after_last_stat_discards_snapshot_and_recount_succeeds and rename-at-depth cases cover final fencing.
  • seeded_bind_and_tmpfs_topologies_exclude_mounts_and_recover: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases.
  • directory_moved_after_its_recursion_returns_rejects_the_whole_scope: Ported: review_754c_rename_at_each_depth_rejects_then_rescans_and_accepts_return uses exit checkpoints.
  • expired_scope_deadline_performs_no_walk: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • two_thousand_seeded_topology_changes_match_final_tree_or_retry: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases.
  • directory_records_use_sixteen_bytes_without_path_allocations: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain.
  • inode_listener_stops_under_continuous_outside_writes: Ported: listener_stops_under_continuous_outside_hardlink_writes uses the new shared reader.
  • missing_new_folder_repairs_only_its_home: Ported: review_754g_coalesced_retry_visits_no_healthy_home and independent_folder_and_file_moves cover Home isolation and publication.
  • temporary_rename_paths_do_not_repair_unrelated_homes: Ported: temporary names remain excluded and per-Home admission never walks another Home.
  • ten_thousand_random_operation_sequences_match_full_walk: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases.
  • write_then_unlink_event_name_repairs_surviving_aliases: Ported: independent_hardlink_lifecycle and existing external-write regressions retain per-name charging and surviving-alias checks.
  • reconciliation_wait_is_bounded_by_the_scope_deadline: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • a_failed_full_scan_publishes_healthy_homes_before_scoped_recovery: Ported: failed_home_keeps_a_retry_without_new_events_and_healthy_home_stays_clean preserves independent Home publication.
  • a_root_timeout_narrows_to_the_home_identified_by_its_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • an_unmeasurable_overlap_does_not_discard_other_homes_measurements: Ported: unreadable_home_reports_failure_and_keeps_healthy_home_total preserves healthy Home publication.
  • empty_directories_retain_zero_bytes_per_directory: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain.
  • failed_initial_scan_recovers_without_retrying_healthy_homes: Ported: failed_home_keeps_a_retry_without_new_events_and_healthy_home_stays_clean and review_754g_coalesced_retry_visits_no_healthy_home cover recovery isolation.
  • retry_ancestors_stay_inside_their_own_home_and_keep_exact_health_counts: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • disjoint_failures_and_failed_full_recounts_retain_the_union: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • a_successful_attempt_cannot_clear_a_newer_failure: Ported: review_754g_admission_rejects_a_newer_failure_during_its_walk checks final freshness.
  • admission_does_not_walk_while_background_retry_is_pending: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • admission_state_timeout_marks_the_scope_unmeasured_and_schedules_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • admission_wait_does_not_invalidate_an_active_measurement: Ported: review_754f_admission_during_long_reconciliation_refuses_stale_totals checks a timed-out join followed by successful fresh publication.
  • event_free_retry_recovers_only_the_failed_scope: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • failed_scope_backs_off_and_health_clears_after_recovery: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • mount_discovery_evicts_siblings_and_preserves_other_homes: Ported to the retained mount integration tests and seeded_whole_home_property; complete Home walks evict every hidden sibling and check mount/unmount fences.
  • timeout_after_walk_does_not_block_in_recording_cleanup: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • scoped_failure_must_not_forget_other_retained_unmeasured_subtrees: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754c_mixed_scoped_retries_retain_exact_union_until_full_success: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754c_raw_names_unreadable_sparse_files_and_symlinks: Ported: raw_names_sparse_hardlinks_symlinks_and_unreadable_files_are_charged_by_metadata covers the same byte and traversal contract.
  • review_754d_mount_appearing_during_walk_is_currently_in_scope: Ported to the retained mount integration tests and seeded_whole_home_property; complete Home walks evict every hidden sibling and check mount/unmount fences.
  • review_754d_outer_reconciliation_must_bound_restarts: Ported: repeated_users_replacement_exhausts_three_attempts_without_charging checks the three-attempt bound.
  • review_754d_twenty_thousand_additional_topologies: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases.
  • review_754e_fresh_seeded_topologies: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases.
  • review_754e_state_mutex_wait_must_obey_the_scope_deadline: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754e_events_from_replaced_parent_never_charge_outside_bytes: Ported: review_754d_refresh_must_not_follow_a_replaced_parent_symlink and review_754e_native_events_follow_replaced_homes_without_stale_fd_charges cover replacements.
  • review_754e_refresh_failure_must_schedule_a_full_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754f_deleted_home_cancels_historical_timers_immediately: Ported: review_754g_home_deleted_inside_running_scheduled_retry and deleted_home_releases_watch_index_capacity cover deletion cleanup.
  • review_754f_failure_timers_are_bounded_like_event_queue: Ported: review_754g_failure_boundaries_keep_exact_health_without_path_retry_records tests 1/2/63/64/65/1000 real failures; only one schedule exists per Home.
  • review_754f_gate_timeout_retains_exact_scope_and_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754f_healthy_and_deleted_incremental_scopes_have_no_timers: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754f_mount_churn_evicts_and_restores_cached_siblings: Ported to the retained mount integration tests and seeded_whole_home_property; complete Home walks evict every hidden sibling and check mount/unmount fences.
  • review_754f_same_scope_deduplicates_and_quiet_failures_do_no_work: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests.
  • review_754f_retry_does_not_walk_healthy_homes: Ported: review_754g_coalesced_retry_visits_no_healthy_home checks the per-Home scan boundary.
  • review_754f_scaled_tree_operations_and_pause_points: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases.
  • review_754f_failed_batch_preserves_later_scopes: Ported: review_754g_mixed_batches_keep_every_failure_in_all_positions preserves each healthy/failing position.
  • changes_keep_history_only_during_a_recount_even_after_failure: Ported: the overlap-history map is deleted; there is no write history to retain outside a recount.
  • a_file_removed_before_native_watch_install_is_transient: Ported: registration_swap_cannot_cache_another_inodes_watch and the seeded deletion oracle cover disappearing and replaced watch targets.
  • dropping_usage_wakes_the_idle_event_worker: Ported: background_notices_a_created_and_replaced_users_tree and background_does_not_count_symlink_targets retain idle worker destruction checks; the new listener also has a continuous-writer shutdown test.
  • fallback_recounts_when_due_even_with_queued_events: Ported: the old event-batch gather loop is deleted; watcher_failure_only_reports_provisional_bytes_and_polling_health requires a fallback deadline, independent of queued events.
  • gathering_stops_at_its_deadline_even_with_queued_events: Ported: the old event-batch gather loop is deleted; complete Auth deadlines and the continuous-writer listener shutdown test bound the current interfaces.
  • failed_measurement_with_missing_native_coverage_keeps_fallback: Ported: watcher_failure_only_reports_provisional_bytes_and_polling_health verifies the new fallback.
  • successful_recount_with_restored_watches_turns_polling_off: Ported: restored_inode_coverage_turns_fallback_off verifies the new coverage lifecycle.

754g was not merged into a6b1fa14; its behavioral cases are explicitly carried forward in review_containment and lifecycle_tests. Its remaining internal-only cases are accounted for here:

  • review_754g_coalescing_permutations_keep_all_paths_and_exact_counts: No retry paths or coalescing orders exist; actual failure counts and recovery are tested at 1/2/63/64/65/1000 entries.
  • review_754g_timer_bound_and_health_at_63_64_65_and_1000_failures: Replaced by review_754g_failure_boundaries_keep_exact_health_without_path_retry_records with actual inaccessible entries.
  • review_754g_deletion_during_running_repair_cancels_timers: Ported as review_754g_home_deleted_inside_running_scheduled_retry; no path timers remain.
  • review_754g_occupied_directory_records_compact_after_deletion_walk: The directory index is deleted; deleted_home_releases_watch_index_capacity also checks the new watch maps release their allocations.
  • review_754g_100k_directory_capacity_bytes: The measured directory-record type no longer exists; native watches still necessarily use memory per observed inode.

Other behavioral adaptations, listed rather than claimed unchanged:

  • Unreadable regular-file tests still require exact metadata bytes. They now also require polling Health and retryable admission when inode watches cannot be installed; mode 000 alone cannot authorize a fresh total.
  • Existing tests that read confirmed totals after app writes now call ensure_home before asserting. App writes add a conservative provisional total; only fenced recounts authorize uploads.
  • Concurrent-edit recount tests allow the bounded retryable result under active churn, then require exact recovery once quiescent.
  • Old private scanner and ready-field hooks are replaced with whole-Home entry/exit checkpoints and freshness observations.
  • The shipping upload storm keeps its exact deletion totals and waits for people.list to reach them through a bounded condition loop. Upload success/refusal assertions are unchanged.
  • Auth reliability and contacts correctness fixtures now confirm the owning Home before their same exact byte assertions. The Health fixture places its failing subtree inside the affected person’s Home, and seed uploads retry explicitly retryable errors within a deadline.
  • Contacts Limit fixtures now derive their artificial limits from confirmed bytes. This preserves the existing exact StorageQuota and undo assertions under provisional display accounting.
  • The old 754d mount oracles had already changed from inclusion to exclusion before this branch. This branch preserves those integration files from a6b1fa14 and does not claim byte-for-byte preservation from their earliest review branch.
Reviewer test inventory for the whole-Home replacement. Names below are no longer present at their old internal locations; each line states its replacement or the removed internal mechanism. Existing public mount integration tests remain unchanged from a6b1fa14. - missing_openat2_leaves_the_scope_unmeasured_without_fallback: Ported: the openat2 resolver was removed; direct fstatat/openat walks no longer require that syscall. No-follow and mount behavior remains independently tested. - unsupported_handles_exhaust_the_pin_budget_without_publishing: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain. - review_754d_open_sub_must_keep_its_ancestry: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain. - directory_replaced_by_an_application_file_drops_stale_descendants: Ported: whole-Home scan tests plus the accepted-count oracle cover directory/file replacement; no cached descendants remain. - every_failed_scope_in_a_batch_is_retried_before_admission: Ported: review_754g_mixed_batches_keep_every_failure_in_all_positions and the real Auth admission property preserve failure isolation and no stale quota decisions. - moving_ancestor_invalidates_unchanged_child_descriptor: Ported: review_754c_rename_at_each_depth_rejects_then_rescans_and_accepts_return tests the public acceptance boundary. - removed_directory_is_unmeasured_instead_of_successfully_empty: Ported: deletion is retried to a valid empty snapshot by review_754c_deletion_after_last_stat_discards_snapshot_and_recount_succeeds. - rename_at_end_of_walk_invalidates_already_observed_metadata: Ported: review_754c_deletion_after_last_stat_discards_snapshot_and_recount_succeeds and rename-at-depth cases cover final fencing. - seeded_bind_and_tmpfs_topologies_exclude_mounts_and_recover: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases. - directory_moved_after_its_recursion_returns_rejects_the_whole_scope: Ported: review_754c_rename_at_each_depth_rejects_then_rescans_and_accepts_return uses exit checkpoints. - expired_scope_deadline_performs_no_walk: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - two_thousand_seeded_topology_changes_match_final_tree_or_retry: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases. - directory_records_use_sixteen_bytes_without_path_allocations: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain. - inode_listener_stops_under_continuous_outside_writes: Ported: listener_stops_under_continuous_outside_hardlink_writes uses the new shared reader. - missing_new_folder_repairs_only_its_home: Ported: review_754g_coalesced_retry_visits_no_healthy_home and independent_folder_and_file_moves cover Home isolation and publication. - temporary_rename_paths_do_not_repair_unrelated_homes: Ported: temporary names remain excluded and per-Home admission never walks another Home. - ten_thousand_random_operation_sequences_match_full_walk: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases. - write_then_unlink_event_name_repairs_surviving_aliases: Ported: independent_hardlink_lifecycle and existing external-write regressions retain per-name charging and surviving-alias checks. - reconciliation_wait_is_bounded_by_the_scope_deadline: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - a_failed_full_scan_publishes_healthy_homes_before_scoped_recovery: Ported: failed_home_keeps_a_retry_without_new_events_and_healthy_home_stays_clean preserves independent Home publication. - a_root_timeout_narrows_to_the_home_identified_by_its_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - an_unmeasurable_overlap_does_not_discard_other_homes_measurements: Ported: unreadable_home_reports_failure_and_keeps_healthy_home_total preserves healthy Home publication. - empty_directories_retain_zero_bytes_per_directory: Removed assertion about deleted directory-index, resolver/chain or pin-budget internals; the replacement uses direct fd walks and ordered event fences. Public containment and watch-capacity tests remain. - failed_initial_scan_recovers_without_retrying_healthy_homes: Ported: failed_home_keeps_a_retry_without_new_events_and_healthy_home_stays_clean and review_754g_coalesced_retry_visits_no_healthy_home cover recovery isolation. - retry_ancestors_stay_inside_their_own_home_and_keep_exact_health_counts: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - disjoint_failures_and_failed_full_recounts_retain_the_union: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - a_successful_attempt_cannot_clear_a_newer_failure: Ported: review_754g_admission_rejects_a_newer_failure_during_its_walk checks final freshness. - admission_does_not_walk_while_background_retry_is_pending: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - admission_state_timeout_marks_the_scope_unmeasured_and_schedules_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - admission_wait_does_not_invalidate_an_active_measurement: Ported: review_754f_admission_during_long_reconciliation_refuses_stale_totals checks a timed-out join followed by successful fresh publication. - event_free_retry_recovers_only_the_failed_scope: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - failed_scope_backs_off_and_health_clears_after_recovery: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - mount_discovery_evicts_siblings_and_preserves_other_homes: Ported to the retained mount integration tests and seeded_whole_home_property; complete Home walks evict every hidden sibling and check mount/unmount fences. - timeout_after_walk_does_not_block_in_recording_cleanup: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - scoped_failure_must_not_forget_other_retained_unmeasured_subtrees: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754c_mixed_scoped_retries_retain_exact_union_until_full_success: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754c_raw_names_unreadable_sparse_files_and_symlinks: Ported: raw_names_sparse_hardlinks_symlinks_and_unreadable_files_are_charged_by_metadata covers the same byte and traversal contract. - review_754d_mount_appearing_during_walk_is_currently_in_scope: Ported to the retained mount integration tests and seeded_whole_home_property; complete Home walks evict every hidden sibling and check mount/unmount fences. - review_754d_outer_reconciliation_must_bound_restarts: Ported: repeated_users_replacement_exhausts_three_attempts_without_charging checks the three-attempt bound. - review_754d_twenty_thousand_additional_topologies: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases. - review_754e_fresh_seeded_topologies: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases. - review_754e_state_mutex_wait_must_obey_the_scope_deadline: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754e_events_from_replaced_parent_never_charge_outside_bytes: Ported: review_754d_refresh_must_not_follow_a_replaced_parent_symlink and review_754e_native_events_follow_replaced_homes_without_stale_fd_charges cover replacements. - review_754e_refresh_failure_must_schedule_a_full_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754f_deleted_home_cancels_historical_timers_immediately: Ported: review_754g_home_deleted_inside_running_scheduled_retry and deleted_home_releases_watch_index_capacity cover deletion cleanup. - review_754f_failure_timers_are_bounded_like_event_queue: Ported: review_754g_failure_boundaries_keep_exact_health_without_path_retry_records tests 1/2/63/64/65/1000 real failures; only one schedule exists per Home. - review_754f_gate_timeout_retains_exact_scope_and_retry: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754f_healthy_and_deleted_incremental_scopes_have_no_timers: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754f_mount_churn_evicts_and_restores_cached_siblings: Ported to the retained mount integration tests and seeded_whole_home_property; complete Home walks evict every hidden sibling and check mount/unmount fences. - review_754f_same_scope_deduplicates_and_quiet_failures_do_no_work: Removed internal scope/retry/state-lock assertion: those structures no longer exist. Failure counts, bounded joins, recovery and stale admission are checked at the Home/Auth boundaries in lifecycle_tests, review_containment and quota_tests. - review_754f_retry_does_not_walk_healthy_homes: Ported: review_754g_coalesced_retry_visits_no_healthy_home checks the per-Home scan boundary. - review_754f_scaled_tree_operations_and_pause_points: Ported to seeded_whole_home_property: random tree operations, exit/entry pauses, real exchanges, users/Home moves, delayed delivery and scalable namespace mounts; the gate now runs 50,000 cases. - review_754f_failed_batch_preserves_later_scopes: Ported: review_754g_mixed_batches_keep_every_failure_in_all_positions preserves each healthy/failing position. - changes_keep_history_only_during_a_recount_even_after_failure: Ported: the overlap-history map is deleted; there is no write history to retain outside a recount. - a_file_removed_before_native_watch_install_is_transient: Ported: registration_swap_cannot_cache_another_inodes_watch and the seeded deletion oracle cover disappearing and replaced watch targets. - dropping_usage_wakes_the_idle_event_worker: Ported: background_notices_a_created_and_replaced_users_tree and background_does_not_count_symlink_targets retain idle worker destruction checks; the new listener also has a continuous-writer shutdown test. - fallback_recounts_when_due_even_with_queued_events: Ported: the old event-batch gather loop is deleted; watcher_failure_only_reports_provisional_bytes_and_polling_health requires a fallback deadline, independent of queued events. - gathering_stops_at_its_deadline_even_with_queued_events: Ported: the old event-batch gather loop is deleted; complete Auth deadlines and the continuous-writer listener shutdown test bound the current interfaces. - failed_measurement_with_missing_native_coverage_keeps_fallback: Ported: watcher_failure_only_reports_provisional_bytes_and_polling_health verifies the new fallback. - successful_recount_with_restored_watches_turns_polling_off: Ported: restored_inode_coverage_turns_fallback_off verifies the new coverage lifecycle. 754g was not merged into a6b1fa14; its behavioral cases are explicitly carried forward in review_containment and lifecycle_tests. Its remaining internal-only cases are accounted for here: - review_754g_coalescing_permutations_keep_all_paths_and_exact_counts: No retry paths or coalescing orders exist; actual failure counts and recovery are tested at 1/2/63/64/65/1000 entries. - review_754g_timer_bound_and_health_at_63_64_65_and_1000_failures: Replaced by review_754g_failure_boundaries_keep_exact_health_without_path_retry_records with actual inaccessible entries. - review_754g_deletion_during_running_repair_cancels_timers: Ported as review_754g_home_deleted_inside_running_scheduled_retry; no path timers remain. - review_754g_occupied_directory_records_compact_after_deletion_walk: The directory index is deleted; deleted_home_releases_watch_index_capacity also checks the new watch maps release their allocations. - review_754g_100k_directory_capacity_bytes: The measured directory-record type no longer exists; native watches still necessarily use memory per observed inode. Other behavioral adaptations, listed rather than claimed unchanged: - Unreadable regular-file tests still require exact metadata bytes. They now also require polling Health and retryable admission when inode watches cannot be installed; mode 000 alone cannot authorize a fresh total. - Existing tests that read confirmed totals after app writes now call ensure_home before asserting. App writes add a conservative provisional total; only fenced recounts authorize uploads. - Concurrent-edit recount tests allow the bounded retryable result under active churn, then require exact recovery once quiescent. - Old private scanner and ready-field hooks are replaced with whole-Home entry/exit checkpoints and freshness observations. - The shipping upload storm keeps its exact deletion totals and waits for people.list to reach them through a bounded condition loop. Upload success/refusal assertions are unchanged. - Auth reliability and contacts correctness fixtures now confirm the owning Home before their same exact byte assertions. The Health fixture places its failing subtree inside the affected person’s Home, and seed uploads retry explicitly retryable errors within a deadline. - Contacts Limit fixtures now derive their artificial limits from confirmed bytes. This preserves the existing exact StorageQuota and undo assertions under provisional display accounting. - The old 754d mount oracles had already changed from inclusion to exclusion before this branch. This branch preserves those integration files from a6b1fa14 and does not claim byte-for-byte preservation from their earliest review branch.
kayg commented 2026-10-29 12:54:31 +00:00
Author
Owner
Copy link

Local recount measurements on 6by32-compute02 — not a perf VM number.

Single dev-build samples, one Home, 17 logical bytes per file, one 10,000-entry folder or two 50,000-entry folders. All runs used one build slot. No cache drop and no cold-cache or server latency claim. Timing comes from untraced runs; syscall counts come from separate strace -f -c runs. Initial means a new counter installs its inode watches; warm means those watches already exist. Both counters reported exactly 170,000 / 1,700,000 bytes.

Files New initial ms, untraced New warm ms, untraced Warm syscall total Syscalls per entry
10000 196.684801 49.359255 20123 2.012098790
100000 2066.224019 522.725821 200194 2.001899962

Entry denominators include the ordinary folder entries (10,001 and 100,002); all fixed probe and fence overhead remains in the syscall totals. Per-file values are 2.0123 and 2.00194. Warm walks have one fstatat and one statx per file, and no openat2 calls.

Comparison on a6b1fa14 with the same public recount probe and datasets: initial/warm untraced times were 172.820904/150.731700 ms at 10k and 1598.787518/1598.861014 ms at 100k. Its successful warm 10k trace had 120,264 calls (12.025197480 per entry). These single samples are not a statistical performance gate.

Tracing the initial 100k count exceeded the 30-second recount deadline on both revisions. Those partial traces are not accepted counts and are not used for syscall-per-entry results. The new complete warm 100k trace succeeded in 8985.617739 ms under instrumentation; its untraced wall sample is the 522.725821 ms value above. Initial watch setup is slower than the old initial count in these samples; startup remains asynchronous and first uploads may return a bounded retryable result.

Warm 10000-file strace output, verbatim:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
 69.20    0.224938       37489         6           ppoll
 15.18    0.049331           4     10001           newfstatat
 13.10    0.042593           4     10003           statx
  1.55    0.005048         280        18           close
  0.66    0.002133         112        19           getdents64
  0.09    0.000278          55         5           futex
  0.08    0.000275          91         3           munmap
  0.05    0.000153           7        21           fcntl
  0.03    0.000083           7        11           openat
  0.02    0.000062           6        10           inotify_add_watch
  0.02    0.000049           9         5         2 read
  0.01    0.000034          11         3           open
  0.01    0.000027           2        11           fstat
  0.01    0.000026          26         1           unlinkat
  0.01    0.000018          18         1           madvise
  0.00    0.000009           9         1           write
  0.00    0.000006           6         1         1 mkdirat
  0.00    0.000005           5         1           rt_sigprocmask
  0.00    0.000005           5         1           sigaltstack
  0.00    0.000004           4         1           getrandom
------ ----------- ----------- --------- --------- ------------------
100.00    0.325077          16     20123         3 total

Warm 100000-file strace output, verbatim:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
 69.20    2.273789      378964         6           ppoll
 15.34    0.504209           5    100002           newfstatat
 13.29    0.436804           4    100005           statx
  1.55    0.051070        2553        20           close
  0.60    0.019707         252        78           getdents64
  0.00    0.000084           3        24           fcntl
  0.00    0.000076           5        13           openat
  0.00    0.000074          74         1           unlinkat
  0.00    0.000054           4        11           inotify_add_watch
  0.00    0.000046           3        12           fstat
  0.00    0.000046           9         5           futex
  0.00    0.000039           7         5         2 read
  0.00    0.000021           7         3           open
  0.00    0.000004           4         1         1 mkdirat
  0.00    0.000003           3         1           getrandom
  0.00    0.000000           0         1           write
  0.00    0.000000           0         3           munmap
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         1           madvise
  0.00    0.000000           0         1           sigaltstack
------ ----------- ----------- --------- --------- ------------------
100.00    3.286026          16    200194         3 total
Local recount measurements on 6by32-compute02 — not a perf VM number. Single dev-build samples, one Home, 17 logical bytes per file, one 10,000-entry folder or two 50,000-entry folders. All runs used one build slot. No cache drop and no cold-cache or server latency claim. Timing comes from untraced runs; syscall counts come from separate strace -f -c runs. Initial means a new counter installs its inode watches; warm means those watches already exist. Both counters reported exactly 170,000 / 1,700,000 bytes. | Files | New initial ms, untraced | New warm ms, untraced | Warm syscall total | Syscalls per entry | |---:|---:|---:|---:|---:| | 10000 | 196.684801 | 49.359255 | 20123 | 2.012098790 | | 100000 | 2066.224019 | 522.725821 | 200194 | 2.001899962 | Entry denominators include the ordinary folder entries (10,001 and 100,002); all fixed probe and fence overhead remains in the syscall totals. Per-file values are 2.0123 and 2.00194. Warm walks have one fstatat and one statx per file, and no openat2 calls. Comparison on a6b1fa14 with the same public recount probe and datasets: initial/warm untraced times were 172.820904/150.731700 ms at 10k and 1598.787518/1598.861014 ms at 100k. Its successful warm 10k trace had 120,264 calls (12.025197480 per entry). These single samples are not a statistical performance gate. Tracing the initial 100k count exceeded the 30-second recount deadline on both revisions. Those partial traces are not accepted counts and are not used for syscall-per-entry results. The new complete warm 100k trace succeeded in 8985.617739 ms under instrumentation; its untraced wall sample is the 522.725821 ms value above. Initial watch setup is slower than the old initial count in these samples; startup remains asynchronous and first uploads may return a bounded retryable result. Warm 10000-file strace output, verbatim: ```text % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ 69.20 0.224938 37489 6 ppoll 15.18 0.049331 4 10001 newfstatat 13.10 0.042593 4 10003 statx 1.55 0.005048 280 18 close 0.66 0.002133 112 19 getdents64 0.09 0.000278 55 5 futex 0.08 0.000275 91 3 munmap 0.05 0.000153 7 21 fcntl 0.03 0.000083 7 11 openat 0.02 0.000062 6 10 inotify_add_watch 0.02 0.000049 9 5 2 read 0.01 0.000034 11 3 open 0.01 0.000027 2 11 fstat 0.01 0.000026 26 1 unlinkat 0.01 0.000018 18 1 madvise 0.00 0.000009 9 1 write 0.00 0.000006 6 1 1 mkdirat 0.00 0.000005 5 1 rt_sigprocmask 0.00 0.000005 5 1 sigaltstack 0.00 0.000004 4 1 getrandom ------ ----------- ----------- --------- --------- ------------------ 100.00 0.325077 16 20123 3 total ``` Warm 100000-file strace output, verbatim: ```text % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ 69.20 2.273789 378964 6 ppoll 15.34 0.504209 5 100002 newfstatat 13.29 0.436804 4 100005 statx 1.55 0.051070 2553 20 close 0.60 0.019707 252 78 getdents64 0.00 0.000084 3 24 fcntl 0.00 0.000076 5 13 openat 0.00 0.000074 74 1 unlinkat 0.00 0.000054 4 11 inotify_add_watch 0.00 0.000046 3 12 fstat 0.00 0.000046 9 5 futex 0.00 0.000039 7 5 2 read 0.00 0.000021 7 3 open 0.00 0.000004 4 1 1 mkdirat 0.00 0.000003 3 1 getrandom 0.00 0.000000 0 1 write 0.00 0.000000 0 3 munmap 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 1 madvise 0.00 0.000000 0 1 sigaltstack ------ ----------- ----------- --------- --------- ------------------ 100.00 3.286026 16 200194 3 total ```
kayg commented 2026-10-29 14:28:48 +00:00
Author
Owner
Copy link

Independent verification of PR #764 at 798f1bbe: fix first.

Tests-only branch: review/764-tests, head 49b67f9f. No product fixes, merges, deployments, or perf-test VM access.

  • P2 — Native initialization failure never recovers. An isolated subprocess temporarily exhausts its own file descriptors, constructs Usage, then restores them. A new Usage can count/admit the same Home; the original instance refuses forever because OnceLock<Option<Native>> permanently contains None. Reproducer: cargo test -p calternal-fsio -j6 review_764_new_native_coverage_recovers_after_transient_fd_exhaustion -- --nocapture.
  • P2 — An unchanged unreadable subtree triggers a full Home walk every second. Exact next deadline is about 1s; two additional full Home walks occur in 2.2s without changes. Native coverage remains true, so metadata failures never enter fallback backoff. Reproducer: cargo test -p calternal-fsio -j6 review_764_unchanged_unreadable_subtree_does_not_walk_every_second -- --nocapture.
  • P2 — The production HTTP placement queue is outside the admission budget. Four concurrent same-person uploads behind a held Auth guard return retryable 500/internal after approximately 1.9, 3.8, 5.7 and 7.6 seconds. The handler awaits uploads.lock(user) before entering Auth's 1900ms timeout. This caller predates this PR but invalidates the claimed single bound over the upload queue. Reproducer: cargo test -p calternal-auth -j6 --test review_764 queued_http_storm -- --nocapture.
  • P3 — The claimed inode-reuse test is vacuous. Its exit hook still holds the old directory fd, pinning the inode throughout all 10,000 create attempts. Recording/asserting that reuse occurred fails. Reproducer: cargo test -p calternal-fsio -j6 review_754d_reused_directory_inode -- --nocapture. This is a coverage gap, not a demonstrated quota bypass.

Positive independent verification: 30 rename cases across descendant/Home/users levels and entry/exit checkpoints, inbound/outbound/cross-Home moves, parent replaced by symlink during walk, real inotify overflow (host queue capacity 1,048,576), queued file-watch retirement, 18 mid-walk bind/tmpfs/file mount cases in private namespaces plus unmount recovery, raw names/unreadable sparse hard-link aliases across Homes, SIGKILL inside recount with offline edits, and two real server processes enforcing exclusive ownership. No over-quota admission was observed. Concurrent HTTP publication admits exactly three files fitting a 500-byte allowance; next upload is HTTP 413/storage_quota/nonretryable. Unmeasurable counts map to HTTP 500/internal/retryable; receiver concurrency refusals map to retryable 429. The held-guard storm separately demonstrates the latency failure.

Both properties passed 50,000 cases with fresh seeds: filesystem 15411026587234305019 (including real mount operations); Auth 4622018493099982131 (23,513 accepted, 26,487 quota-refused). The original code fixes its seeds; reviewer test code adds an environment seed override.

Five-minute idle observation used retained schedule/last-walk/generation counters: 2,983 samples, zero new walks, zero Home deadlines, zero invalidations for a healthy unchanged Home. This does not erase the degraded-Home idle finding, and is not a CPU/RSS/perf-budget claim.

Gates: original head fmt, strict workspace clippy, full workspace pass (1,241 passed, 0 failed, 21 ignored; Tasks #757 test passed, no rerun required). Final reviewer tree fmt and strict workspace clippy pass. The reviewer suite intentionally keeps the four finding reproducers enabled; their failing results are not green gates. The complete workspace run with reviewer tests exited 101 only for the fsio library target (the three enabled fsio finding reproducers). Its Auth binary was built before the final queue reproducer was added; the final Auth reviewer integration suite was separately run and reports two passes and the expected HTTP-queue failure. Tasks again passed. Thus four distinct finding reproducers fail in the final tree; no unrelated gate failure was seen.

Deleted-reviewer-test audit: every missing original reviewer test function has a named disposition in comment 69530; no unexplained deletion found. However, removal of review_754f_same_scope_deduplicates_and_quiet_failures_do_no_work loses the quiet-failure/backoff assertion, and the replacement checks only that some deadline exists. The plan's recovery row covers EACCES/fence recovery but misses native initialization recovery. The inode-reuse row remains unproven. Local complete plan-row audit and raw logs are in /home/agent/codex-jobs/verify-764/; no job artifacts are committed.

(Posted by the coordinator; the verifier's Forgejo credential request timed out.)

🤖 Generated with Claude Code

Independent verification of PR #764 at `798f1bbe`: **fix first**. Tests-only branch: [`review/764-tests`](https://git.kayg.org/kayg/calternal/src/branch/review/764-tests), head `49b67f9f`. No product fixes, merges, deployments, or perf-test VM access. - **P2 — Native initialization failure never recovers.** An isolated subprocess temporarily exhausts its own file descriptors, constructs Usage, then restores them. A new Usage can count/admit the same Home; the original instance refuses forever because `OnceLock<Option<Native>>` permanently contains `None`. Reproducer: `cargo test -p calternal-fsio -j6 review_764_new_native_coverage_recovers_after_transient_fd_exhaustion -- --nocapture`. - **P2 — An unchanged unreadable subtree triggers a full Home walk every second.** Exact next deadline is about 1s; two additional full Home walks occur in 2.2s without changes. Native coverage remains true, so metadata failures never enter fallback backoff. Reproducer: `cargo test -p calternal-fsio -j6 review_764_unchanged_unreadable_subtree_does_not_walk_every_second -- --nocapture`. - **P2 — The production HTTP placement queue is outside the admission budget.** Four concurrent same-person uploads behind a held Auth guard return retryable 500/internal after approximately 1.9, 3.8, 5.7 and 7.6 seconds. The handler awaits `uploads.lock(user)` before entering Auth's 1900ms timeout. This caller predates this PR but invalidates the claimed single bound over the upload queue. Reproducer: `cargo test -p calternal-auth -j6 --test review_764 queued_http_storm -- --nocapture`. - **P3 — The claimed inode-reuse test is vacuous.** Its exit hook still holds the old directory fd, pinning the inode throughout all 10,000 create attempts. Recording/asserting that reuse occurred fails. Reproducer: `cargo test -p calternal-fsio -j6 review_754d_reused_directory_inode -- --nocapture`. This is a coverage gap, not a demonstrated quota bypass. Positive independent verification: 30 rename cases across descendant/Home/users levels and entry/exit checkpoints, inbound/outbound/cross-Home moves, parent replaced by symlink during walk, real inotify overflow (host queue capacity 1,048,576), queued file-watch retirement, 18 mid-walk bind/tmpfs/file mount cases in private namespaces plus unmount recovery, raw names/unreadable sparse hard-link aliases across Homes, SIGKILL inside recount with offline edits, and two real server processes enforcing exclusive ownership. No over-quota admission was observed. Concurrent HTTP publication admits exactly three files fitting a 500-byte allowance; next upload is HTTP 413/storage_quota/nonretryable. Unmeasurable counts map to HTTP 500/internal/retryable; receiver concurrency refusals map to retryable 429. The held-guard storm separately demonstrates the latency failure. Both properties passed 50,000 cases with fresh seeds: filesystem `15411026587234305019` (including real mount operations); Auth `4622018493099982131` (23,513 accepted, 26,487 quota-refused). The original code fixes its seeds; reviewer test code adds an environment seed override. Five-minute idle observation used retained schedule/last-walk/generation counters: 2,983 samples, zero new walks, zero Home deadlines, zero invalidations for a healthy unchanged Home. This does not erase the degraded-Home idle finding, and is not a CPU/RSS/perf-budget claim. Gates: original head fmt, strict workspace clippy, full workspace pass (1,241 passed, 0 failed, 21 ignored; Tasks #757 test passed, no rerun required). Final reviewer tree fmt and strict workspace clippy pass. The reviewer suite intentionally keeps the four finding reproducers enabled; their failing results are not green gates. The complete workspace run with reviewer tests exited 101 only for the fsio library target (the three enabled fsio finding reproducers). Its Auth binary was built before the final queue reproducer was added; the final Auth reviewer integration suite was separately run and reports two passes and the expected HTTP-queue failure. Tasks again passed. Thus four distinct finding reproducers fail in the final tree; no unrelated gate failure was seen. Deleted-reviewer-test audit: every missing original reviewer test function has a named disposition in comment 69530; no unexplained deletion found. However, removal of `review_754f_same_scope_deduplicates_and_quiet_failures_do_no_work` loses the quiet-failure/backoff assertion, and the replacement checks only that some deadline exists. The plan's recovery row covers EACCES/fence recovery but misses native initialization recovery. The inode-reuse row remains unproven. Local complete plan-row audit and raw logs are in `/home/agent/codex-jobs/verify-764/`; no job artifacts are committed. (Posted by the coordinator; the verifier's Forgejo credential request timed out.) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kayg commented 2026-10-29 14:52:31 +00:00
Author
Owner
Copy link

Fixes pushed to perf/quota-simple at 55622509ae. Reviewer branch review/764-tests (49b67f9f) was fast-forwarded first. Five focused commits; no PR merge or deploy.

  • Native initialization caches only success. Background setup retries at bounded 1/2/4/.../60-second backoff. Health reports the 60-second polling fallback; every fallback admission runs or joins a fresh bounded recount, never a cached polling total. An isolated fd-exhaustion regression verifies both fallback admissions after outside edits and automatic watcher recovery.
  • Stable unmeasurable entries stay in Health without dirtying their Home again. Covered failures have no idle timer. Events or explicit admission retry them. Concurrent requests share a recount's success/failure for the generation actually walked.
  • One 1900 ms admission deadline starts before shared measurement and the HTTP placement queue, covering Auth reservation, filesystem, provider and DB waits in backdrop and attachment placement. Publication starts only after admission and retains its guards on cancellation. The app quietly retries retryable backdrop refusals up to three times at 250/500/1000 ms; quota refusals are final.
  • The inode-reuse test now deletes at Home exit, after the child fd closes, and checks real st_ino equality within 10,000 attempts. This build host's btrfs allocator did not repeat the inode: it reports a clear physical-reuse skip and still asserts replacement accounting. If reuse occurs, the test asserts the reused replacement and a later mutation are charged; no claim that physical reuse was exercised here.
  • Updated docs/plans/0754-quota.md failure table. No prior fsio/Auth function names were removed. Reviewer failure-count, refusal, byte-total and recovery assertions remain. The old timer assertions were strengthened to require no idle timer, matching the decided zero-idle-work rule; the 2.2-second walk-count assertion remains.

Gates (all heavy commands via /usr/local/bin/build-slot.sh, Rust at -j6, on 10by32-02-std aarch64):

  • cargo fmt --all -- --check: pass.
  • cargo clippy -j6 --workspace --all-targets -- -D warnings: pass.
  • cargo test -j6 -p calternal-fsio: pass, 117 library tests plus mount integrations; zero failures. Four manual/helper tests retain their ignore annotations; unreadable idle and real overflow were additionally run explicitly.
  • Auth quota tests and cargo test -j6 -p calternal-auth --test review_764 -- --nocapture: pass; all four enabled reviewer/queue tests, zero failures.
  • Filesystem property: 50,000 cases, seed 172964050764, pass (including namespace mount cases).
  • Auth property: 50,000 cases, seed 272964050764, pass (23,560 admitted, 26,440 quota-refused).
  • cargo test -j6 --workspace: pass, zero failures. Tasks kill_9_during_field_writes_loses_nothing_acknowledged passed on the first run; no Tasks rerun or #763 merge was needed.
  • Three client retry tests: pass. Web type/Svelte check: zero errors, 14 warnings.

One earlier standalone fsio run failed only at beneath::tests::paths_that_climb_out_are_refused during fixture remove_dir_all cleanup with NotFound. Its targeted rerun and the subsequent complete fsio gate passed without changing that test. The full workspace run also passed it.

Five-minute unreadable-subtree trace on the committed implementation: 2,895 observations, zero walk deadlines, zero full-Home walks, zero invalidations, one retained Health entry. Both usage-events and usage-recount had zero additional CPU ticks over 300 seconds. Permissions were then restored and admission charged all 118 logical bytes with Health clear. This observes worker CPU at the host's tick resolution; the sampling test itself wakes separately.

HTTP latency — build host, not perf VM. Debug build, warmed metadata and native coverage, sparse 17-byte logical files on /tmp tmpfs; one file externally rewritten before each simultaneous batch. Ten batches per cell, no cold-cache/perf-budget claim. These times include body staging, admission, placement and response. The existing receiver may return retryable 429 before accounting; both all-response and successful-publication distributions are shown so those quick refusals cannot hide upload latency. No perf-test VM was accessed.

Files Concurrent Requests 201 429 Retryable 500 All p50 ms All p95 ms All max ms Successful p50 ms Successful p95 ms Successful max ms
10000 1 10 10 0 0 39.163 40.402 40.402 39.163 40.402 40.402
10000 4 40 40 0 0 79.803 156.883 158.855 79.803 156.883 158.855
10000 16 160 62 98 0 0.016 238.422 351.594 137.388 310.386 351.594
100000 1 10 10 0 0 446.672 520.666 520.666 446.672 520.666 520.666
100000 4 40 40 0 0 963.927 1782.596 1865.818 963.927 1782.596 1865.818
100000 16 160 40 106 14 0.012 1901.525 1903.454 906.448 1762.203 1838.329

Separate held-publication batches hold a real Auth reservation on a healthy, measurable Home, so early measurement failure cannot bypass the queue regression. Every response was retryable 500 or receiver-limit 429, every request stayed below two seconds, and no file was published:

Files Concurrent p50 ms p95 ms Max ms
10000 1 1903.404 1903.404 1903.404
10000 4 1901.418 1901.532 1901.532
10000 16 0.008 1911.229 1911.229
100000 1 1906.171 1906.171 1906.171
100000 4 1902.541 1902.696 1902.696
100000 16 0.010 1904.248 1904.248

Local raw evidence and reproducible test output: /home/agent/codex-jobs/quota-fix/ (not committed). The permanent ignored latency probe is quota_http_latency_build_host; the permanent idle trace is unreadable_subtree_five_minute_idle_trace.

(Posted by the coordinator; the job's Forgejo credential request timed out.)

Fixes pushed to `perf/quota-simple` at **55622509aef821a49006eb85fbc8d7803b23fbc9**. Reviewer branch `review/764-tests` (`49b67f9f`) was fast-forwarded first. Five focused commits; no PR merge or deploy. - Native initialization caches only success. Background setup retries at bounded 1/2/4/.../60-second backoff. Health reports the 60-second polling fallback; every fallback admission runs or joins a fresh bounded recount, never a cached polling total. An isolated fd-exhaustion regression verifies both fallback admissions after outside edits and automatic watcher recovery. - Stable unmeasurable entries stay in Health without dirtying their Home again. Covered failures have no idle timer. Events or explicit admission retry them. Concurrent requests share a recount's success/failure for the generation actually walked. - One 1900 ms admission deadline starts before shared measurement and the HTTP placement queue, covering Auth reservation, filesystem, provider and DB waits in backdrop and attachment placement. Publication starts only after admission and retains its guards on cancellation. The app quietly retries retryable backdrop refusals up to three times at 250/500/1000 ms; quota refusals are final. - The inode-reuse test now deletes at Home exit, after the child fd closes, and checks real `st_ino` equality within 10,000 attempts. This build host's btrfs allocator did not repeat the inode: it reports a clear physical-reuse skip and still asserts replacement accounting. If reuse occurs, the test asserts the reused replacement and a later mutation are charged; no claim that physical reuse was exercised here. - Updated `docs/plans/0754-quota.md` failure table. No prior fsio/Auth function names were removed. Reviewer failure-count, refusal, byte-total and recovery assertions remain. The old timer assertions were strengthened to require no idle timer, matching the decided zero-idle-work rule; the 2.2-second walk-count assertion remains. Gates (all heavy commands via `/usr/local/bin/build-slot.sh`, Rust at `-j6`, on `10by32-02-std` aarch64): - `cargo fmt --all -- --check`: pass. - `cargo clippy -j6 --workspace --all-targets -- -D warnings`: pass. - `cargo test -j6 -p calternal-fsio`: pass, 117 library tests plus mount integrations; zero failures. Four manual/helper tests retain their ignore annotations; unreadable idle and real overflow were additionally run explicitly. - Auth quota tests and `cargo test -j6 -p calternal-auth --test review_764 -- --nocapture`: pass; all four enabled reviewer/queue tests, zero failures. - Filesystem property: 50,000 cases, seed `172964050764`, pass (including namespace mount cases). - Auth property: 50,000 cases, seed `272964050764`, pass (23,560 admitted, 26,440 quota-refused). - `cargo test -j6 --workspace`: pass, zero failures. Tasks `kill_9_during_field_writes_loses_nothing_acknowledged` passed on the first run; no Tasks rerun or #763 merge was needed. - Three client retry tests: pass. Web type/Svelte check: zero errors, 14 warnings. One earlier standalone fsio run failed only at `beneath::tests::paths_that_climb_out_are_refused` during fixture `remove_dir_all` cleanup with `NotFound`. Its targeted rerun and the subsequent complete fsio gate passed without changing that test. The full workspace run also passed it. Five-minute unreadable-subtree trace on the committed implementation: 2,895 observations, zero walk deadlines, zero full-Home walks, zero invalidations, one retained Health entry. Both `usage-events` and `usage-recount` had zero additional CPU ticks over 300 seconds. Permissions were then restored and admission charged all 118 logical bytes with Health clear. This observes worker CPU at the host's tick resolution; the sampling test itself wakes separately. HTTP latency — **build host, not perf VM**. Debug build, warmed metadata and native coverage, sparse 17-byte logical files on `/tmp` tmpfs; one file externally rewritten before each simultaneous batch. Ten batches per cell, no cold-cache/perf-budget claim. These times include body staging, admission, placement and response. The existing receiver may return retryable 429 before accounting; both all-response and successful-publication distributions are shown so those quick refusals cannot hide upload latency. No perf-test VM was accessed. | Files | Concurrent | Requests | 201 | 429 | Retryable 500 | All p50 ms | All p95 ms | All max ms | Successful p50 ms | Successful p95 ms | Successful max ms | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 10000 | 1 | 10 | 10 | 0 | 0 | 39.163 | 40.402 | 40.402 | 39.163 | 40.402 | 40.402 | | 10000 | 4 | 40 | 40 | 0 | 0 | 79.803 | 156.883 | 158.855 | 79.803 | 156.883 | 158.855 | | 10000 | 16 | 160 | 62 | 98 | 0 | 0.016 | 238.422 | 351.594 | 137.388 | 310.386 | 351.594 | | 100000 | 1 | 10 | 10 | 0 | 0 | 446.672 | 520.666 | 520.666 | 446.672 | 520.666 | 520.666 | | 100000 | 4 | 40 | 40 | 0 | 0 | 963.927 | 1782.596 | 1865.818 | 963.927 | 1782.596 | 1865.818 | | 100000 | 16 | 160 | 40 | 106 | 14 | 0.012 | 1901.525 | 1903.454 | 906.448 | 1762.203 | 1838.329 | Separate held-publication batches hold a real Auth reservation on a healthy, measurable Home, so early measurement failure cannot bypass the queue regression. Every response was retryable 500 or receiver-limit 429, every request stayed below two seconds, and no file was published: | Files | Concurrent | p50 ms | p95 ms | Max ms | |---:|---:|---:|---:|---:| | 10000 | 1 | 1903.404 | 1903.404 | 1903.404 | | 10000 | 4 | 1901.418 | 1901.532 | 1901.532 | | 10000 | 16 | 0.008 | 1911.229 | 1911.229 | | 100000 | 1 | 1906.171 | 1906.171 | 1906.171 | | 100000 | 4 | 1902.541 | 1902.696 | 1902.696 | | 100000 | 16 | 0.010 | 1904.248 | 1904.248 | Local raw evidence and reproducible test output: `/home/agent/codex-jobs/quota-fix/` (not committed). The permanent ignored latency probe is `quota_http_latency_build_host`; the permanent idle trace is `unreadable_subtree_five_minute_idle_trace`. (Posted by the coordinator; the job's Forgejo credential request timed out.)
kayg commented 2026-10-29 15:09:40 +00:00
Author
Owner
Copy link

Independent fix-round review of 55622509 against 798f1bbe and docs/plans/0754-quota.md, on 10by32-02-std. Verdict: fix first.

  • P2: missing-native fallback can authorize an inconsistent count. Reproducer: cargo test -j6 -p calternal-fsio fallback_admission_must_not_accept -- --ignored --nocapture through build-slot.sh, on the reviewer test branch below. Force native initialization to fail using an isolated child's RLIMIT_NOFILE, then restore its fd limit. During a real two-subtree scan, transfer bytes from the not-yet-counted subtree to the already-counted subtree. The Home is never empty, but admission returns Ok(()), counted bytes 0, final independent oracle 1009. This cannot be linearized at any instant during the walk. usage.rs accepts complete && ... self.native().is_none() and shares that unfenced success with joiners. A new walk per request does not establish freshness without a watcher/fence; missing native coverage must refuse retryably until recovery, unless another actual freshness mechanism is provided. The final test is opt-in/ignored so default gates do not mask their ordinary results with a known contract failure.
  • P3: actual directory inode reuse remains unverified. The Home-exit pause fixes the pinned-child-fd mistake, but this Btrfs host still reports SKIP physical inode reuse after 10,000 attempts. The replacement assertions do not exercise reuse; relative to review/764-tests, assert!(reused) was weakened to an early return. This is a coverage gap, not a demonstrated bypass. A local ext4 loop filesystem cannot be mounted with this account's current privileges: zero effective capabilities, no passwordless sudo, root:disk 0660 loop devices, and ext4 mounting denied in a private user/mount namespace. No shared host block device was touched. Close the gap on a prepared inode-recycling filesystem and require actual equality there; do not cite this run as physical-reuse evidence.

Verified independently:

  • Real repeated watcher setup failures back off through observed retry states 4/8/16 seconds, then recover automatically after fd-limit restoration. Health clears and no clean-Home timer remains.
  • Sixteen concurrent fallback recount callers share one walk. Sixteen real fallback-mode HTTP uploads exercise the full Auth/backdrop path; six publish in the final isolated run, the rest refuse retryably, final bytes are exactly 899, and every response is under 2 seconds. This quiescent-tree success does not resolve the mutation finding above.
  • Readability restoration clears retained Health via the watcher/background path without an explicit admission. A partial-recovery test keeps the second failure recorded and admission refused. The old open-child-fd deletion adversary, accidentally moved later by the fix round, is independently restored and passes.
  • Five-minute healthy and unreadable-subtree CPU observations both show zero event/recount worker CPU ticks, no full walks, invalidations, or Home deadlines. The unreadable run retains one Health failure until recovery. These observations have kernel clock-tick resolution on a shared build host.
  • Client retries are bounded to four sequential requests per caller with 250/500/1000 ms backoff. Sixteen persistent-failure clients make exactly 64 requests, never overlap their own attempts, and surface the final error after about 1.8 seconds for immediate refusals. There is no jitter, so clients stay synchronized; there is no unbounded retry loop. The helper supplies no transport/abort deadline for a fetch that never resolves.
  • Fresh seed 764029, 50,000 filesystem cases (including real private-namespace mounts) and 50,000 Auth cases pass.

HTTP measurements: build host, not perf VM, ten stale batches per cell. All responses and held-publication queue waits stay below 2 seconds. Receiver 429s are included in all-response quantiles; accepted-only quantiles are retained in the local log.

Files Concurrent Requests Accepted 429 Retryable 500 All p50/p95/max ms Held-publication max ms
10k 1 10 10 0 0 41.747 / 44.133 / 44.133 1904.347
10k 4 40 40 0 0 80.792 / 135.456 / 147.767 1909.686
10k 16 160 61 99 0 0.014 / 237.190 / 335.613 1907.147
100k 1 10 10 0 0 385.045 / 439.839 / 439.839 1902.010
100k 4 40 40 0 0 946.811 / 1695.303 / 1879.373 1903.217
100k 16 160 40 96 24 0.023 / 1910.556 / 1914.204 1917.077

Reviewer preservation: compared test bodies at review/754-tests through 754g and review/764-tests against current code. Literal preservation is false: deleted internal scanner/index/scope tests have dispositions in comment 69530 and adapted Home-boundary cases. The quiet-failure assertion now requires no timer, consistent with zero idle CPU. HTTP/process assertions from review/764-tests are preserved. Physical-reuse anti-vacuity remains the P3 exception; independent tests restore partial-failure recovery and deletion with an open child descriptor rather than silently losing those assertions.

Gates: fmt check, workspace/all-target clippy -D warnings, full workspace tests. Initial workspace invocation hit an Auth migration flake (schema_two_copy_keeps_real_credentials_sessions_and_recovery: rows 1 vs 2); isolated rerun and full subsequent runs passed. Tasks abrupt-stop tests passed; no Tasks flake rerun was necessary. The known fallback-contract reproducer fails as documented and is not included in default passing gates.

Logs, full-precision measurements, test-body inventory/diffs, and incremental findings are under /home/agent/codex-jobs/verify-764b/ on this machine. No merge, deployment, or perf-test VM access.

Reviewer tests pushed only to review/764b-tests, commit 4222b5eb. No production behavior changes; the Auth fd-exhaustion fixture adds a dev-only rustix dependency. Final fmt, clippy, and full workspace commands all exited 0.

Independent fix-round review of **55622509** against 798f1bbe and `docs/plans/0754-quota.md`, on **10by32-02-std**. Verdict: **fix first**. - **P2: missing-native fallback can authorize an inconsistent count.** Reproducer: `cargo test -j6 -p calternal-fsio fallback_admission_must_not_accept -- --ignored --nocapture` through build-slot.sh, on the reviewer test branch below. Force native initialization to fail using an isolated child's RLIMIT_NOFILE, then restore its fd limit. During a real two-subtree scan, transfer bytes from the not-yet-counted subtree to the already-counted subtree. The Home is never empty, but admission returns `Ok(())`, counted bytes **0**, final independent oracle **1009**. This cannot be linearized at any instant during the walk. `usage.rs` accepts `complete && ... self.native().is_none()` and shares that unfenced success with joiners. A new walk per request does not establish freshness without a watcher/fence; missing native coverage must refuse retryably until recovery, unless another actual freshness mechanism is provided. The final test is opt-in/ignored so default gates do not mask their ordinary results with a known contract failure. - **P3: actual directory inode reuse remains unverified.** The Home-exit pause fixes the pinned-child-fd mistake, but this Btrfs host still reports `SKIP physical inode reuse` after 10,000 attempts. The replacement assertions do not exercise reuse; relative to review/764-tests, `assert!(reused)` was weakened to an early return. This is a coverage gap, not a demonstrated bypass. A local ext4 loop filesystem cannot be mounted with this account's current privileges: zero effective capabilities, no passwordless sudo, root:disk 0660 loop devices, and ext4 mounting denied in a private user/mount namespace. No shared host block device was touched. Close the gap on a prepared inode-recycling filesystem and require actual equality there; do not cite this run as physical-reuse evidence. Verified independently: - Real repeated watcher setup failures back off through observed retry states 4/8/16 seconds, then recover automatically after fd-limit restoration. Health clears and no clean-Home timer remains. - Sixteen concurrent fallback recount callers share one walk. Sixteen real fallback-mode HTTP uploads exercise the full Auth/backdrop path; six publish in the final isolated run, the rest refuse retryably, final bytes are exactly 899, and every response is under 2 seconds. This quiescent-tree success does not resolve the mutation finding above. - Readability restoration clears retained Health via the watcher/background path without an explicit admission. A partial-recovery test keeps the second failure recorded and admission refused. The old open-child-fd deletion adversary, accidentally moved later by the fix round, is independently restored and passes. - Five-minute healthy and unreadable-subtree CPU observations both show **zero event/recount worker CPU ticks**, no full walks, invalidations, or Home deadlines. The unreadable run retains one Health failure until recovery. These observations have kernel clock-tick resolution on a shared build host. - Client retries are bounded to four sequential requests per caller with 250/500/1000 ms backoff. Sixteen persistent-failure clients make exactly 64 requests, never overlap their own attempts, and surface the final error after about 1.8 seconds for immediate refusals. There is no jitter, so clients stay synchronized; there is no unbounded retry loop. The helper supplies no transport/abort deadline for a fetch that never resolves. - Fresh seed **764029**, **50,000 filesystem cases** (including real private-namespace mounts) and **50,000 Auth cases** pass. HTTP measurements: **build host, not perf VM**, ten stale batches per cell. All responses and held-publication queue waits stay below 2 seconds. Receiver 429s are included in all-response quantiles; accepted-only quantiles are retained in the local log. | Files | Concurrent | Requests | Accepted | 429 | Retryable 500 | All p50/p95/max ms | Held-publication max ms | |---|---:|---:|---:|---:|---:|---|---:| | 10k | 1 | 10 | 10 | 0 | 0 | 41.747 / 44.133 / 44.133 | 1904.347 | | 10k | 4 | 40 | 40 | 0 | 0 | 80.792 / 135.456 / 147.767 | 1909.686 | | 10k | 16 | 160 | 61 | 99 | 0 | 0.014 / 237.190 / 335.613 | 1907.147 | | 100k | 1 | 10 | 10 | 0 | 0 | 385.045 / 439.839 / 439.839 | 1902.010 | | 100k | 4 | 40 | 40 | 0 | 0 | 946.811 / 1695.303 / 1879.373 | 1903.217 | | 100k | 16 | 160 | 40 | 96 | 24 | 0.023 / 1910.556 / 1914.204 | 1917.077 | Reviewer preservation: compared test bodies at review/754-tests through 754g and review/764-tests against current code. Literal preservation is false: deleted internal scanner/index/scope tests have dispositions in comment 69530 and adapted Home-boundary cases. The quiet-failure assertion now requires no timer, consistent with zero idle CPU. HTTP/process assertions from review/764-tests are preserved. Physical-reuse anti-vacuity remains the P3 exception; independent tests restore partial-failure recovery and deletion with an open child descriptor rather than silently losing those assertions. Gates: fmt check, workspace/all-target clippy `-D warnings`, full workspace tests. Initial workspace invocation hit an Auth migration flake (`schema_two_copy_keeps_real_credentials_sessions_and_recovery`: rows 1 vs 2); isolated rerun and full subsequent runs passed. Tasks abrupt-stop tests passed; no Tasks flake rerun was necessary. The known fallback-contract reproducer fails as documented and is not included in default passing gates. Logs, full-precision measurements, test-body inventory/diffs, and incremental findings are under `/home/agent/codex-jobs/verify-764b/` on this machine. No merge, deployment, or perf-test VM access. Reviewer tests pushed only to [`review/764b-tests`](https://git.kayg.org/kayg/calternal/src/branch/review/764b-tests), commit [`4222b5eb`](https://git.kayg.org/kayg/calternal/commit/4222b5ebaf99663bea16555862c20ba2612c0a61). No production behavior changes; the Auth fd-exhaustion fixture adds a dev-only rustix dependency. Final fmt, clippy, and full workspace commands all exited 0.
kayg commented 2026-10-29 15:23:02 +00:00
Author
Owner
Copy link

Round 2 implemented at d0e6d6f60f688a8e1f71abbc54f28a558d471ab1 on perf/quota-simple.

Fetched first and incorporated origin/review/764b-tests (4222b5eb) by fast-forward. Three small commits: 241d48f0 (verified admission), 8bd896ea (physical inode reuse), d0e6d6f6 (real Auth fallback property).

P2 root cause: a complete but unfenced fallback walk returned success, and joiners were also allowed to accept it despite having no fresh counted generation. The unchanged adversarial reviewer reproducer failed before the fix with a zero count while Home was never empty; it now returns retryable WouldBlock (final oracle 1009, minimum during transfer 202). Fallback recounts remain provisional and never authorize admission until native coverage recovers. An optional VerifiedCount owns its completed total and generation; fresh_bytes is the single Auth admission read and rejects absent/stale generations, including invalidation while providers run. Default display bytes cannot become admission bytes.

P3: the inode test uses CALTERNAL_INODE_REUSE_DIR, skipping clearly only when unset. With it set, failure to reuse is a failure. Inside the mutation checkpoint it verifies deletion of the old object, identical st_ino for the new directory, and absence of the old contents before checking quota behavior. On /srv/inode-reuse, actual directory inode 16 reuse passed; replacement bytes and the later write were both charged. Updated the plan failure table.

Reviewer tests were retained. Explicit behavioral adaptations: the fsio 16-caller fallback test now requires WouldBlock while retaining its exactly-one-walk, exact display bytes, stale generation and polling assertions; the real HTTP test now requires retryable refusal and zero publications while retaining its concurrent requests, timing, accounting-reachability, Health and exact final bytes checks. The physical-reuse test replaces its unsuccessful-reuse early return with mandatory reuse assertions when configured. No reviewer test was deleted or relaxed; the non-linearizable-walk reproducer itself is unchanged and was run explicitly.

Final gates on 10by32-02-std, all exit 0 / zero test failures. Heavy commands used /usr/local/bin/build-slot.sh, Cargo builds/tests -j6; filesystem/Auth and workspace gates used CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse:

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -j6 -- -D warnings
  • cargo test -j6 -p calternal-fsio (reviewer tests included)
  • cargo test -j6 -p calternal-auth, plus final cargo test -j6 -p calternal-auth --lib quota after adding the Auth fallback property
  • cargo test -j6 -p calternal-fsio fallback_admission_must_not_accept -- --ignored --nocapture
  • Seed 7641029, QUOTA_PROPERTY_CASES=50000: original fsio property and original Auth property both passed; new fallback recount property passed with 25,000 confirmed outside writes during scans; new real Auth fallback property passed 50,000 randomized Home/upload decisions, all retryably refused. The original Auth property exercised 23,534 admissions and 26,466 quota refusals.
  • cargo test --workspace -j6

Logs: /home/agent/codex-jobs/fix-764-round2/ (including red/green reproducer, inode proof, shipping fmt/clippy/Auth quota, final fsio/workspace, and all four 50,000-case property runs). Standard pre-existing ignored long/perf tests remain ignored; the contract reproducer was enabled explicitly. No perf VM work, performance claim, PR merge or deployment.

Round 2 implemented at `d0e6d6f60f688a8e1f71abbc54f28a558d471ab1` on `perf/quota-simple`. Fetched first and incorporated `origin/review/764b-tests` (`4222b5eb`) by fast-forward. Three small commits: `241d48f0` (verified admission), `8bd896ea` (physical inode reuse), `d0e6d6f6` (real Auth fallback property). P2 root cause: a complete but unfenced fallback walk returned success, and joiners were also allowed to accept it despite having no fresh counted generation. The unchanged adversarial reviewer reproducer failed before the fix with a zero count while Home was never empty; it now returns retryable WouldBlock (final oracle 1009, minimum during transfer 202). Fallback recounts remain provisional and never authorize admission until native coverage recovers. An optional `VerifiedCount` owns its completed total and generation; `fresh_bytes` is the single Auth admission read and rejects absent/stale generations, including invalidation while providers run. Default display bytes cannot become admission bytes. P3: the inode test uses `CALTERNAL_INODE_REUSE_DIR`, skipping clearly only when unset. With it set, failure to reuse is a failure. Inside the mutation checkpoint it verifies deletion of the old object, identical st_ino for the new directory, and absence of the old contents before checking quota behavior. On `/srv/inode-reuse`, actual directory inode 16 reuse passed; replacement bytes and the later write were both charged. Updated the plan failure table. Reviewer tests were retained. Explicit behavioral adaptations: the fsio 16-caller fallback test now requires WouldBlock while retaining its exactly-one-walk, exact display bytes, stale generation and polling assertions; the real HTTP test now requires retryable refusal and zero publications while retaining its concurrent requests, timing, accounting-reachability, Health and exact final bytes checks. The physical-reuse test replaces its unsuccessful-reuse early return with mandatory reuse assertions when configured. No reviewer test was deleted or relaxed; the non-linearizable-walk reproducer itself is unchanged and was run explicitly. Final gates on 10by32-02-std, all exit 0 / zero test failures. Heavy commands used `/usr/local/bin/build-slot.sh`, Cargo builds/tests `-j6`; filesystem/Auth and workspace gates used `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse`: - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -j6 -- -D warnings` - `cargo test -j6 -p calternal-fsio` (reviewer tests included) - `cargo test -j6 -p calternal-auth`, plus final `cargo test -j6 -p calternal-auth --lib quota` after adding the Auth fallback property - `cargo test -j6 -p calternal-fsio fallback_admission_must_not_accept -- --ignored --nocapture` - Seed `7641029`, `QUOTA_PROPERTY_CASES=50000`: original fsio property and original Auth property both passed; new fallback recount property passed with 25,000 confirmed outside writes during scans; new real Auth fallback property passed 50,000 randomized Home/upload decisions, all retryably refused. The original Auth property exercised 23,534 admissions and 26,466 quota refusals. - `cargo test --workspace -j6` Logs: `/home/agent/codex-jobs/fix-764-round2/` (including red/green reproducer, inode proof, shipping fmt/clippy/Auth quota, final fsio/workspace, and all four 50,000-case property runs). Standard pre-existing ignored long/perf tests remain ignored; the contract reproducer was enabled explicitly. No perf VM work, performance claim, PR merge or deployment.
kayg commented 2026-10-29 15:33:12 +00:00
Author
Owner
Copy link

Independent verification, round 3 — ship for d0e6d6f60f688a8e1f71abbc54f28a558d471ab1. No remaining P1/P2/P3 findings from the requested admission-freshness audit. Reviewed 55622509..d0e6d6f6, called counter/watcher/scanner/Auth/client code, plan 0754, and both prior review reports.

The previous P2 is closed: its explicitly enabled non-linearizable fallback reproducer now returns retryable WouldBlock with provisional display=0, actual Home=1,009, and minimum observed Home=202. Admission reads fresh_bytes; never-counted/provisional totals cannot stand in for a completed fenced total. Concurrent joiners also refuse the unfenced outcome.

Tests added independently on review/764c-tests at 9eddde38 (tests only):

  • 50,000 random mode-switch cases per fresh seed, seeds 8943029071 and 3829174059. Missing-native fallback, failed-watcher/recovered coverage, overflow, deletion/empty count/recreated Home, expired recount, delayed events, poisoned display total, display recount and startup/restart. Respectively 38,857 and 38,835 exact accepted reads against the independent filesystem oracle; every fallback/failed-watcher read refused. Deterministic runtime-failure simulation uses healthy=false plus production dirty-all transition, then restores the still-live reader; separate real missing-native initialization recovery is also asserted.
  • 16 fresh readers share one successful recount, then share one deliberately timed-out recount; none can escape with the older total, and a retry reads current 1,009 bytes. A watcher failure during provider evaluation invalidates the result after the fence.
  • Real Auth: startup charges preexisting bytes; deletion permits empty-Home admission; recreation over quota yields 64 refusals across 16 concurrent callers/retries.
  • Client: transient fallback followed by recovered success stops retries and preserves the upload body; a permanent quota refusal after fallback stops immediately. All six client tests pass, including retained 16-caller bounded-backoff coverage.

Additional evidence:

  • All four existing properties pass 50,000 cases each, fresh seed 9183674521: filesystem oracle, real Auth quota, fallback recount with 25,000 outside writes during scans, and real Auth fallback admission.
  • Real kernel overflow received and asserted at this host's 1,048,576-event capacity; stale admission refused or repaired and subsequent count exact.
  • Prior P3 is closed: CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse actually reuses st_ino=16; mandatory assertions verify deletion of the old object, equality of the reused inode, replacement charge=16 and later mutation charge=117. Also included in full workspace gate with the env set.
  • Both separate 300-second idle traces pass. Healthy event/recount worker CPU ticks remain 0→0. Unreadable-subtree workers remain 0→0 across 2,886 samples, with zero walks/deadlines/invalidations and retained Health=1; readability recovery then succeeds. Clock-tick-resolution evidence on this shared build host, not perf-VM numbers or proof of zero instructions.

Required gates all exit 0: cargo fmt --all -- --check; cargo clippy --workspace --all-targets -j6 -- -D warnings; CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test --workspace -j6. Workspace log: 106 successful test-process results, zero failures (1,279 passing test reports including nested-child repetitions). No Tasks failure/rerun. Heavy work through /usr/local/bin/build-slot.sh at -j6 on 10by32-02-std; no perf-test VMs, merges or deployments.

Findings, exact reproducer commands and logs: /home/agent/codex-jobs/verify-764c/ on this host; full report findings.md. Nothing except tests committed/pushed.

Independent verification, round 3 — **ship** for `d0e6d6f60f688a8e1f71abbc54f28a558d471ab1`. No remaining P1/P2/P3 findings from the requested admission-freshness audit. Reviewed `55622509..d0e6d6f6`, called counter/watcher/scanner/Auth/client code, plan 0754, and both prior review reports. The previous P2 is closed: its explicitly enabled non-linearizable fallback reproducer now returns retryable `WouldBlock` with provisional display=0, actual Home=1,009, and minimum observed Home=202. Admission reads `fresh_bytes`; never-counted/provisional totals cannot stand in for a completed fenced total. Concurrent joiners also refuse the unfenced outcome. Tests added independently on `review/764c-tests` at `9eddde38` (tests only): - 50,000 random mode-switch cases **per fresh seed**, seeds `8943029071` and `3829174059`. Missing-native fallback, failed-watcher/recovered coverage, overflow, deletion/empty count/recreated Home, expired recount, delayed events, poisoned display total, display recount and startup/restart. Respectively 38,857 and 38,835 exact accepted reads against the independent filesystem oracle; every fallback/failed-watcher read refused. Deterministic runtime-failure simulation uses healthy=false plus production dirty-all transition, then restores the still-live reader; separate real missing-native initialization recovery is also asserted. - 16 fresh readers share one successful recount, then share one deliberately timed-out recount; none can escape with the older total, and a retry reads current 1,009 bytes. A watcher failure during provider evaluation invalidates the result after the fence. - Real Auth: startup charges preexisting bytes; deletion permits empty-Home admission; recreation over quota yields 64 refusals across 16 concurrent callers/retries. - Client: transient fallback followed by recovered success stops retries and preserves the upload body; a permanent quota refusal after fallback stops immediately. All six client tests pass, including retained 16-caller bounded-backoff coverage. Additional evidence: - All four existing properties pass **50,000 cases each**, fresh seed `9183674521`: filesystem oracle, real Auth quota, fallback recount with 25,000 outside writes during scans, and real Auth fallback admission. - Real kernel overflow received and asserted at this host's 1,048,576-event capacity; stale admission refused or repaired and subsequent count exact. - Prior P3 is closed: `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse` actually reuses `st_ino=16`; mandatory assertions verify deletion of the old object, equality of the reused inode, replacement charge=16 and later mutation charge=117. Also included in full workspace gate with the env set. - Both separate **300-second idle traces pass**. Healthy event/recount worker CPU ticks remain 0→0. Unreadable-subtree workers remain 0→0 across 2,886 samples, with zero walks/deadlines/invalidations and retained Health=1; readability recovery then succeeds. Clock-tick-resolution evidence on this shared build host, not perf-VM numbers or proof of zero instructions. Required gates all exit 0: `cargo fmt --all -- --check`; `cargo clippy --workspace --all-targets -j6 -- -D warnings`; `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test --workspace -j6`. Workspace log: 106 successful test-process results, zero failures (1,279 passing test reports including nested-child repetitions). No Tasks failure/rerun. Heavy work through `/usr/local/bin/build-slot.sh` at `-j6` on `10by32-02-std`; no perf-test VMs, merges or deployments. Findings, exact reproducer commands and logs: `/home/agent/codex-jobs/verify-764c/` on this host; full report `findings.md`. Nothing except tests committed/pushed.
kayg commented 2026-10-29 18:39:56 +00:00
Author
Owner
Copy link

Perf-VM measurement (perf-test-02, x86_64 4c, uncached 4K disks, release builds; coordinator 29 Oct). dev = d83e54db, #764 = d0e6d6f6.

Idle (6 × 300 s windows per size)

dev #764
10k idle CPU max 0.063% 0.010%
100k idle CPU max 0.630% (budget 0.1%) 0.000%
100k RSS max 107.3 MiB (budget 100) 84.0 MiB
Health unmeasured entries — 0

Upload right after /healthz = 200 (1 KB)

dev #764
10k, 1 upload 201 ~217 ms 500 internal ~300 ms
10k, 4/16 concurrent 201 mostly 201 ~570 ms, 1×500 each
100k, 1/4/16 concurrent 201 ~1.75–2.1 s all 500 at ~2.0 s

Server log: storage measurement is pending; retry the upload surfaced as internal. Blocking: the pending/deadline case must be a proper retryable response, recounts should start at startup, and the cold 100k recount must get faster. Fix job quota-fix-3 is running.

🤖 Generated with Claude Code

Perf-VM measurement (perf-test-02, x86_64 4c, uncached 4K disks, release builds; coordinator 29 Oct). dev = d83e54db, #764 = d0e6d6f6. **Idle (6 × 300 s windows per size)** | | dev | #764 | |---|---|---| | 10k idle CPU max | 0.063% | 0.010% | | 100k idle CPU max | **0.630%** (budget 0.1%) | **0.000%** | | 100k RSS max | **107.3 MiB** (budget 100) | 84.0 MiB | | Health unmeasured entries | — | 0 | **Upload right after /healthz = 200 (1 KB)** | | dev | #764 | |---|---|---| | 10k, 1 upload | 201 ~217 ms | **500 internal** ~300 ms | | 10k, 4/16 concurrent | 201 | mostly 201 ~570 ms, 1×500 each | | 100k, 1/4/16 concurrent | 201 ~1.75–2.1 s | **all 500** at ~2.0 s | Server log: `storage measurement is pending; retry the upload` surfaced as `internal`. Blocking: the pending/deadline case must be a proper retryable response, recounts should start at startup, and the cold 100k recount must get faster. Fix job quota-fix-3 is running. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kayg commented 2026-10-29 19:02:42 +00:00
Author
Owner
Copy link

Implemented the startup-upload follow-up at b6b971151aeb4d590daa578b8c0bc8f78cb96655 on perf/quota-simple.

  • Pending measurement and admission deadlines return 503 storage_busy, retryable: true, and Retry-After: 1, using the existing API error envelope. This includes Auth prepare/admit and the shared HTTP measurement/placement wrapper used by attachments and backdrops. Contacts/import/action paths retain the same typed error. DAV upload and tus adapters are not implemented in this checkout.
  • Initial Home counts are due immediately, without the edit debounce. The existing asynchronous startup worker runs at lower Linux priority, with one background walk at a time; readiness stays independent of counting. Subsequent edits retain debouncing and clean Homes park without timers.
  • The Linux walker combines identity, size, type and mount ID in one statx, skips non-chargeable known d_type values, and reads 64 KiB directory batches through the pinned fd. Regular files still require metadata for size and file bind-mount exclusion. Inode pin/watch installation, event fences, generation revalidation and conservative admission remain intact. Subtree traversal remains serial. A warm-cache release probe of two independent 50k-file Homes sharing one Usage/watch registry took a median 265.6 ms serialized versus 538.4 ms with two simultaneous recounts (three runs each). This supports keeping one background walker on this host; uncached-disk behavior still needs the coordinator’s perf-VM measurement.
  • Added startup incomplete-measurement -> busy -> successful retry, immediate background counting, shared HTTP deadline, and bounded quiet client retry regressions. storage_busy retries stay quiet for several seconds (up to six attempts/eight seconds of retry scheduling, capped two-second backoff). Existing other-error retry behavior is retained. Included the newer review/764c-tests cases; legacy reviewer expectations for 500/internal now assert 503/storage_busy instead. Regression teardown now drains worker-held references before deleting fixtures, preventing a fence-directory cleanup race; the affected coverage regression also passed ten consecutive repeat runs.

Gates: fmt; clippy -D warnings; full workspace tests; the properties — PASS.

Commands use /usr/local/bin/build-slot.sh; Rust builds/tests use -j8:

  • cargo fmt --all -- --check
  • cargo clippy -j8 --workspace --all-targets -- -D warnings
  • CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 --workspace --no-fail-fast
  • QUOTA_PROPERTY_CASES=50000 cargo test -j8 -p calternal-fsio -p calternal-auth property -- --nocapture, plus the separately named seeded_fallback_admission_never_uses_an_underestimate at 50,000 cases. All four properties pass; Auth quota decisions include 23,398 admissions and 26,602 quota refusals.
  • Explicit reviewer native overflow, non-linearizable fallback, and all three five-minute idle checks pass. Healthy and retained-failure worker CPU ticks are unchanged over 300 seconds. Physical inode reuse is exercised with /srv/inode-reuse.
  • Web upload retry tests: 8 passed, including reviewer 764b/764c; web type/check gate passes (0 errors; 14 existing warnings in unchanged Svelte files).
  • Explicit build-host reviewer HTTP sweep passes at 10k/100k files and 1/4/16 concurrent requests. Held placement queues return retryable 503 around 1.9 seconds. At 100k/16, the measured sweep had 43 publications, 98 receiver-limit 429s and 19 storage-busy 503s, with no 500s.

Build host measurements — shared arm64 10by32-02-std, release probe, btrfs, 100 directories of 1 KiB files; median of three initial verified recounts (includes initial watch installation and verification):

Files Previous This head
10k 33.9 ms 28.5 ms
100k 394.8 ms 346.1 ms

These are warm filesystem-cache / initially unmeasured counter results, not cold-cache timings. /usr/local/sbin/perf-drop-caches does not exist here and noninteractive sudo is unavailable, so I cannot provide a valid cold-cache measurement on this host. The coordinator should repeat cold measurements on the perf VM.

The same 10k-file syscall probe (initial verification plus one subsequent recount) reduced newfstatat 30,300 -> 0, getdents64 1,216 -> 610, and total calls 114,646 -> 82,837. A steady regular-file pass now needs one statx; a new file additionally needs its pinned open/fstat/inotify-watch/close. usage-timing is included as a release example for repeating the recount probe.

Implemented the startup-upload follow-up at `b6b971151aeb4d590daa578b8c0bc8f78cb96655` on `perf/quota-simple`. - Pending measurement and admission deadlines return **503 `storage_busy`**, `retryable: true`, and `Retry-After: 1`, using the existing API error envelope. This includes Auth prepare/admit and the shared HTTP measurement/placement wrapper used by attachments and backdrops. Contacts/import/action paths retain the same typed error. DAV upload and tus adapters are not implemented in this checkout. - Initial Home counts are due immediately, without the edit debounce. The existing asynchronous startup worker runs at lower Linux priority, with one background walk at a time; readiness stays independent of counting. Subsequent edits retain debouncing and clean Homes park without timers. - The Linux walker combines identity, size, type and mount ID in one `statx`, skips non-chargeable known `d_type` values, and reads 64 KiB directory batches through the pinned fd. Regular files still require metadata for size and file bind-mount exclusion. Inode pin/watch installation, event fences, generation revalidation and conservative admission remain intact. Subtree traversal remains serial. A warm-cache release probe of two independent 50k-file Homes sharing one Usage/watch registry took a median 265.6 ms serialized versus 538.4 ms with two simultaneous recounts (three runs each). This supports keeping one background walker on this host; uncached-disk behavior still needs the coordinator’s perf-VM measurement. - Added startup incomplete-measurement -> busy -> successful retry, immediate background counting, shared HTTP deadline, and bounded quiet client retry regressions. `storage_busy` retries stay quiet for several seconds (up to six attempts/eight seconds of retry scheduling, capped two-second backoff). Existing other-error retry behavior is retained. Included the newer `review/764c-tests` cases; legacy reviewer expectations for 500/internal now assert 503/storage_busy instead. Regression teardown now drains worker-held references before deleting fixtures, preventing a fence-directory cleanup race; the affected coverage regression also passed ten consecutive repeat runs. Gates: **fmt; clippy -D warnings; full workspace tests; the properties — PASS.** Commands use `/usr/local/bin/build-slot.sh`; Rust builds/tests use `-j8`: - `cargo fmt --all -- --check` - `cargo clippy -j8 --workspace --all-targets -- -D warnings` - `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 --workspace --no-fail-fast` - `QUOTA_PROPERTY_CASES=50000 cargo test -j8 -p calternal-fsio -p calternal-auth property -- --nocapture`, plus the separately named `seeded_fallback_admission_never_uses_an_underestimate` at 50,000 cases. All four properties pass; Auth quota decisions include 23,398 admissions and 26,602 quota refusals. - Explicit reviewer native overflow, non-linearizable fallback, and all three five-minute idle checks pass. Healthy and retained-failure worker CPU ticks are unchanged over 300 seconds. Physical inode reuse is exercised with `/srv/inode-reuse`. - Web upload retry tests: 8 passed, including reviewer 764b/764c; web type/check gate passes (0 errors; 14 existing warnings in unchanged Svelte files). - Explicit build-host reviewer HTTP sweep passes at 10k/100k files and 1/4/16 concurrent requests. Held placement queues return retryable 503 around 1.9 seconds. At 100k/16, the measured sweep had 43 publications, 98 receiver-limit 429s and 19 storage-busy 503s, with no 500s. **Build host measurements** — shared arm64 `10by32-02-std`, release probe, btrfs, 100 directories of 1 KiB files; median of three initial verified recounts (includes initial watch installation and verification): | Files | Previous | This head | | --- | ---: | ---: | | 10k | 33.9 ms | 28.5 ms | | 100k | 394.8 ms | 346.1 ms | These are **warm filesystem-cache / initially unmeasured counter** results, not cold-cache timings. `/usr/local/sbin/perf-drop-caches` does not exist here and noninteractive sudo is unavailable, so I cannot provide a valid cold-cache measurement on this host. The coordinator should repeat cold measurements on the perf VM. The same 10k-file syscall probe (initial verification plus one subsequent recount) reduced `newfstatat` 30,300 -> 0, `getdents64` 1,216 -> 610, and total calls 114,646 -> 82,837. A steady regular-file pass now needs one `statx`; a new file additionally needs its pinned open/fstat/inotify-watch/close. `usage-timing` is included as a release example for repeating the recount probe.
kayg commented 2026-10-29 19:20:08 +00:00
Author
Owner
Copy link

Independent verification of d0e6d6f6..b6b97115 on 10by32-02-std (aarch64). Verdict: fix first.

  • P2 — client retries have no jitter. Sixteen simultaneous 503 storage_busy / Retry-After: 1 clients all schedule attempts at exactly 0/1000/2000/3000/5000/7000 ms: 96 requests in synchronized waves. Attempts/time are bounded and nonretryable errors stop immediately, but the requested anti-herd property is absent. The shorter prior loop already lacked jitter; the extended busy loop retains it. Independent deterministic reproducer: REVIEW_764D_REQUIRE_JITTER=1 node --experimental-strip-types --test apps/web/src/lib/upload-retry-review-764d.test.mjs (expected failure: all 16 schedule the same retry). Add bounded jitter while retaining Retry-After and attempt/time limits.
  • P3 — immediate startup recount exposes a cancellation-test race. The first workspace run failed at reliability_round3.rs:592, recursive file count 1 vs 0. A diagnostic copy preserving the original assertions failed twice in 100 isolated runs; both counted paths were .calternal-tmp-quota-state/fence-*, not leaked upload files. The original blanket recursive count races the quota fence. Reproducer, repeat as needed: /usr/local/bin/build-slot.sh cargo test -j8 -p calternal-auth --test review_764d_cancel -- --ignored --nocapture. Synchronize counting or distinguish only these private fence markers while preserving real upload-cleanup assertions. Original tests were not edited; the added diagnostic is opt-in.

Verified independently:

  • Both implemented raw upload adapters, attachments and backdrops, return 503/storage_busy/retryable=true/Retry-After: 1 for pending measurement and held-admission deadlines. The same idempotency key succeeds after recovery and replays correctly. DAV/WebDAV upload and tus adapters are not implemented here.
  • One low-priority startup worker converges after Home creation/deletion and parks when clean. A real initialized 100k-file installation, populated offline and restarted, answers /healthz before its initial quota watch set completes. Both separate 300-second healthy/unreadable-Home observations show zero additional event/recount worker CPU ticks; unreadable Health retention and exact recovery pass. Build-host warm-cache / kernel-clock-tick evidence, no cold-cache perf claim.
  • Five 50,000-case property runs pass, fresh seeds 764430102907 (filesystem oracle, real Auth quota, real Auth fallback), 764430102911 (mutating fallback), 764430102919 (mode switches). No stale/provisional approval found; 38,749 exact accepted mode-switch reads, all missing/failed-native fallback reads refused.
  • Containment, renames during walks, symlinks, non-UTF-8 names, unreadable files, hard links/outside aliases and real private-namespace mounts pass. /srv/inode-reuse actually reuses directory inode 16, with mandatory replacement/later-write assertions. The explicitly enabled non-linearizable fallback reproducer passes by refusing the unfenced count.
  • Compared reviewer snapshots from review/754* and review/764*-tests. No latest-diff deletion, added skip, reduced cases or weakened accounting assertions. Status expectations now require 503/storage_busy; teardown drains worker references. Historical internal scope/index tests were already removed/adapted before d0e6d6f6, so literal preservation against every old 754 snapshot is not claimed. Snapshot diffs and function inventory are saved.

Required gates: fmt PASS; workspace/all-target clippy -D warnings PASS; full workspace tests PASS on the subsequent complete --no-fail-fast run. The first gate failure remains the P3 above, rather than being hidden by that green rerun. Added normal client checks pass; the opt-in jitter contract fails as documented. Heavy work used /usr/local/bin/build-slot.sh, Cargo -j8.

Tests only: review/764d-tests, commit 7025c29b. Findings, exact commands, reviewer diffs and logs: /home/agent/codex-jobs/verify-764d/findings.md on this machine. No production edits, merges, deployments or perf-test machine access.

Independent verification of `d0e6d6f6..b6b97115` on **10by32-02-std (aarch64)**. Verdict: **fix first**. - **P2 — client retries have no jitter.** Sixteen simultaneous `503 storage_busy` / `Retry-After: 1` clients all schedule attempts at exactly 0/1000/2000/3000/5000/7000 ms: 96 requests in synchronized waves. Attempts/time are bounded and nonretryable errors stop immediately, but the requested anti-herd property is absent. The shorter prior loop already lacked jitter; the extended busy loop retains it. Independent deterministic reproducer: `REVIEW_764D_REQUIRE_JITTER=1 node --experimental-strip-types --test apps/web/src/lib/upload-retry-review-764d.test.mjs` (expected failure: all 16 schedule the same retry). Add bounded jitter while retaining Retry-After and attempt/time limits. - **P3 — immediate startup recount exposes a cancellation-test race.** The first workspace run failed at `reliability_round3.rs:592`, recursive file count 1 vs 0. A diagnostic copy preserving the original assertions failed twice in 100 isolated runs; both counted paths were `.calternal-tmp-quota-state/fence-*`, not leaked upload files. The original blanket recursive count races the quota fence. Reproducer, repeat as needed: `/usr/local/bin/build-slot.sh cargo test -j8 -p calternal-auth --test review_764d_cancel -- --ignored --nocapture`. Synchronize counting or distinguish only these private fence markers while preserving real upload-cleanup assertions. Original tests were not edited; the added diagnostic is opt-in. Verified independently: - Both implemented raw upload adapters, attachments and backdrops, return **503/storage_busy/retryable=true/Retry-After: 1** for pending measurement and held-admission deadlines. The same idempotency key succeeds after recovery and replays correctly. DAV/WebDAV upload and tus adapters are not implemented here. - One low-priority startup worker converges after Home creation/deletion and parks when clean. A real initialized 100k-file installation, populated offline and restarted, answers `/healthz` before its initial quota watch set completes. Both separate 300-second healthy/unreadable-Home observations show **zero additional event/recount worker CPU ticks**; unreadable Health retention and exact recovery pass. Build-host warm-cache / kernel-clock-tick evidence, no cold-cache perf claim. - **Five 50,000-case property runs pass**, fresh seeds `764430102907` (filesystem oracle, real Auth quota, real Auth fallback), `764430102911` (mutating fallback), `764430102919` (mode switches). No stale/provisional approval found; 38,749 exact accepted mode-switch reads, all missing/failed-native fallback reads refused. - Containment, renames during walks, symlinks, non-UTF-8 names, unreadable files, hard links/outside aliases and real private-namespace mounts pass. `/srv/inode-reuse` actually reuses directory inode 16, with mandatory replacement/later-write assertions. The explicitly enabled non-linearizable fallback reproducer passes by refusing the unfenced count. - Compared reviewer snapshots from review/754* and review/764*-tests. No latest-diff deletion, added skip, reduced cases or weakened accounting assertions. Status expectations now require 503/storage_busy; teardown drains worker references. Historical internal scope/index tests were already removed/adapted before d0e6d6f6, so literal preservation against every old 754 snapshot is not claimed. Snapshot diffs and function inventory are saved. Required gates: **fmt PASS; workspace/all-target clippy -D warnings PASS; full workspace tests PASS on the subsequent complete `--no-fail-fast` run**. The first gate failure remains the P3 above, rather than being hidden by that green rerun. Added normal client checks pass; the opt-in jitter contract fails as documented. Heavy work used `/usr/local/bin/build-slot.sh`, Cargo `-j8`. Tests only: [`review/764d-tests`, commit 7025c29b](https://git.kayg.org/kayg/calternal/commit/7025c29be8eae11f9e653d20f541b48ebb06a411). Findings, exact commands, reviewer diffs and logs: `/home/agent/codex-jobs/verify-764d/findings.md` on this machine. No production edits, merges, deployments or perf-test machine access.
kayg commented 2026-10-29 19:29:01 +00:00
Author
Owner
Copy link

Fixed P2/P3 from comment 69856 in a506b739f001c85336ae8a234aadbfcd2d78e3f2 on perf/quota-simple. First incorporated reviewer branch review/764d-tests (7025c29b).

  • Upload retries now add full exponential jitter above the server's Retry-After minimum (including HTTP dates). Busy jitter grows from 200 ms to an 800 ms cap; the existing five-retry limit remains. Retry scheduling is bounded by 30 seconds including elapsed request time. A Retry-After beyond the remaining budget returns the refusal instead of retrying early. Other retryable errors retain their short minimum backoff and gain jitter too.
  • Added deterministic 16-concurrent-client coverage: no more than four first retries in any sliding 50 ms window for stratified random samples. Added Retry-After date/minimum and total-budget tests. The new dispersion and oversized-header tests failed before the fix; the reviewer's opt-in jitter reproducer also failed before and passes now.
  • Cancellation checks now await the actual zero-file condition with a 30-second failure deadline. Every filename is still counted, including quota fences; pending-attachment and capacity assertions remain. No reviewer tests were removed or newly skipped. The fence implementation already unlinks its marker on completion; no product cancellation change was needed.

Validation on 10by32-02-std (arm64), using /usr/local/bin/build-slot.sh and Cargo -j8:

  • cargo fmt --all --check: PASS.
  • cargo clippy --workspace --all-targets -j8 -- -D warnings: PASS.
  • CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 -p calternal-fsio -p calternal-auth --no-fail-fast: PASS, including the actual inode-reuse test.
  • CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test --workspace -j8 --no-fail-fast: PASS.
  • REVIEW_764D_REQUIRE_JITTER=1 node --experimental-strip-types --test apps/web/src/lib/*.test.mjs apps/web/scripts/test/*.test.mjs: 37/37 PASS, no skips.
  • Original cancellation test: 200/200 isolated process runs PASS. Reviewer cancellation diagnostic (explicitly enabled): 200/200 PASS. The stress runner checks that each invocation executes exactly one passing test.

Evidence retained at /home/agent/codex-jobs/quota-fix/ (client-red.log, web-unit.log, clippy.log, quota-gate.log, workspace.log, cancel-stress-checked.log). Worktree is clean and its target/ has been deleted. No PR merge, deployment, or perf-test machine access.

Fixed P2/P3 from comment 69856 in `a506b739f001c85336ae8a234aadbfcd2d78e3f2` on `perf/quota-simple`. First incorporated reviewer branch `review/764d-tests` (`7025c29b`). - Upload retries now add full exponential jitter above the server's Retry-After minimum (including HTTP dates). Busy jitter grows from 200 ms to an 800 ms cap; the existing five-retry limit remains. Retry scheduling is bounded by 30 seconds including elapsed request time. A Retry-After beyond the remaining budget returns the refusal instead of retrying early. Other retryable errors retain their short minimum backoff and gain jitter too. - Added deterministic 16-concurrent-client coverage: no more than four first retries in any sliding 50 ms window for stratified random samples. Added Retry-After date/minimum and total-budget tests. The new dispersion and oversized-header tests failed before the fix; the reviewer's opt-in jitter reproducer also failed before and passes now. - Cancellation checks now await the actual zero-file condition with a 30-second failure deadline. Every filename is still counted, including quota fences; pending-attachment and capacity assertions remain. No reviewer tests were removed or newly skipped. The fence implementation already unlinks its marker on completion; no product cancellation change was needed. Validation on `10by32-02-std` (arm64), using `/usr/local/bin/build-slot.sh` and Cargo `-j8`: - `cargo fmt --all --check`: PASS. - `cargo clippy --workspace --all-targets -j8 -- -D warnings`: PASS. - `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test -j8 -p calternal-fsio -p calternal-auth --no-fail-fast`: PASS, including the actual inode-reuse test. - `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse cargo test --workspace -j8 --no-fail-fast`: PASS. - `REVIEW_764D_REQUIRE_JITTER=1 node --experimental-strip-types --test apps/web/src/lib/*.test.mjs apps/web/scripts/test/*.test.mjs`: 37/37 PASS, no skips. - Original cancellation test: 200/200 isolated process runs PASS. Reviewer cancellation diagnostic (explicitly enabled): 200/200 PASS. The stress runner checks that each invocation executes exactly one passing test. Evidence retained at `/home/agent/codex-jobs/quota-fix/` (`client-red.log`, `web-unit.log`, `clippy.log`, `quota-gate.log`, `workspace.log`, `cancel-stress-checked.log`). Worktree is clean and its `target/` has been deleted. No PR merge, deployment, or perf-test machine access.
kayg commented 2026-10-30 04:14:47 +00:00
Author
Owner
Copy link

Plan-first amendment: #769, targeting perf/quota-simple from a506b739. Only docs/plans/0754-quota.md changes.

  • Covers 22 workloads at 10k/100k/1M, including outside churn, 100 Homes, overflow, #761's two-tree Home, legacy single-root app state, crash/cancellation and policy changes. Predicted passes, unknowns and expected failures are distinguished from measurements.
  • Specifies exact signed deltas for owned publications and checked receipts using pinned identities, watch generations and move cookies. Public content events are never suppressed; an outside write to the same path during publication invalidates freshness. No path/time-window echo exemption.
  • Recommends (c): optimize the complete cold recount and require successful 100k admission within 2 s. This head already has 64 KiB getdents and d_type skipping. Persisted totals alone, unchanged mtimes and “far below quota” do not prove offline continuity or bound external growth. If optimization misses the gate, release remains blocked pending a design revision.
  • Adds race/oracle tests, VM gates for every matrix row, zero own-save recounts during typing, burst/import/restart measurements, idle checks and a failure-table amendment. Legacy single-root SQLite/WAL writes remain an explicit unresolved fast-path requirement rather than a silent exclusion.

Validation: 1,828 amendment words; three A4 pages at 10.5 pt body / 9.5 pt tables, 14 mm margins; Forgejo GFM HTTP 200 with all 22 rows and both amendment tables; git diff --check and docs-only scope checks passed. New perf values are not claimed. No product code, build/test run or deployment.

Head: 6c3f7f8a4f.

Plan-first amendment: https://git.kayg.org/kayg/calternal/pulls/769, targeting `perf/quota-simple` from a506b739. Only `docs/plans/0754-quota.md` changes. - Covers 22 workloads at 10k/100k/1M, including outside churn, 100 Homes, overflow, #761's two-tree Home, legacy single-root app state, crash/cancellation and policy changes. Predicted passes, unknowns and expected failures are distinguished from measurements. - Specifies exact signed deltas for owned publications and checked receipts using pinned identities, watch generations and move cookies. Public content events are never suppressed; an outside write to the same path during publication invalidates freshness. No path/time-window echo exemption. - Recommends (c): optimize the complete cold recount and require successful 100k admission within 2 s. This head already has 64 KiB getdents and d_type skipping. Persisted totals alone, unchanged mtimes and “far below quota” do not prove offline continuity or bound external growth. If optimization misses the gate, release remains blocked pending a design revision. - Adds race/oracle tests, VM gates for every matrix row, zero own-save recounts during typing, burst/import/restart measurements, idle checks and a failure-table amendment. Legacy single-root SQLite/WAL writes remain an explicit unresolved fast-path requirement rather than a silent exclusion. Validation: 1,828 amendment words; three A4 pages at 10.5 pt body / 9.5 pt tables, 14 mm margins; Forgejo GFM HTTP 200 with all 22 rows and both amendment tables; `git diff --check` and docs-only scope checks passed. New perf values are not claimed. No product code, build/test run or deployment. Head: 6c3f7f8a4f68b180c79f377b076d07fa93d7e62b.
kayg commented 2026-10-30 05:17:47 +00:00
Author
Owner
Copy link

Receipt accounting and bounded parallel cold enumeration are implemented, but this is not merge-ready. Default-layout database writes still cause Home recounts and fail the typing gate; state migration or a proven in-place database accounting protocol and final perf-VM qualification remain outstanding. The final-head workspace run also exposed an intermittent startup Home-deletion/fence race; its rerun results are recorded below.

Implementation head: 86f48a39bcace15c13dbcd812cb2e991552d1e17 on perf/quota-simple.

Plan amendment: 85f4c31c38becd61000c2efd4ae6dff416729892 on docs/quota-plan-2, quoted in #769 (comment) and merged into the implementation branch.

Closed-image atomic writes, prepared note saves, streamed uploads and tracked batch file publications use pinned inode identities, watch incarnations, covered size observations, ordered fences and bounded cookie-paired receipts to commit signed byte deltas exactly once. Content notifications are never consumed as echoes. Unmatched events, outside replacement/link changes, overflow, watch loss and failed or canceled operations invalidate authority. Publication/accounting and provider transactions share an owned per-Home permit; the event reader never takes it. Recounts check both external generation and accounting sequence, and serialize their watch-set changes/final commit. Contacts provider errors refuse admission instead of contributing zero. Relative state roots are matched against SQLite's absolute filenames. Complete display-only recounts with an unhealthy listener may publish a provisional snapshot without clearing newer dirtiness or authorizing admission; accounting-sequence changes still reject old snapshots.

Cold enumeration retains fd-relative no-follow/mount checks and descriptor reuse, moves independent substantial directories to at most eight workers, and coalesces new-watch invalidations within each rejected scan attempt. d_type remains a hint; size, identity and mount metadata checks remain.

Added deterministic coverage for pre-watch growth/truncate/replacement, post-watch changes, outside grow/truncate/rename-over/hard links/equal-size writes/restored mtime, two prepared same/different-destination writes, delayed ordered delivery, recount/prune versus a newer delta, receipt overflow, cancel/reject/retirement, checked-provider failures and transitions, relative state roots, and a prepared save racing real Auth upload admission with positive recovery. Kernel order is preserved; userspace batching and concurrent operation scheduling are varied.

Reviewer tests and assertions are retained. One superseded product assertion was adapted: app_publication_adds_known_bytes_without_authorizing_a_provisional_total is now covered_app_publication_adds_exact_bytes_without_a_recount; a separate dirty/unmeasured test retains the refusal obligation. simple_tests.rs adds only a hook-presence helper so deterministic single-thread scanner hooks remain deterministic. No reviewer assertion was removed.

Performance qualification remains blocked for the default layout: public in-place DB/WAL/SHM writes retain conservative Home invalidation. This change does not implement a state migration or a replacement physical-DB accounting protocol. Namespace/directory transitions without a complete receipt proof also retain conservative recounts. The full W01–W22 VM evidence checklist, including long-note/search/history variants, capacity/failure recovery, 1M/100-Home cases, flat folders, upload concurrency and both state layouts, remains required before product merge; unrun cells are missing evidence.

Required gates (verbatim): fmt; clippy -D warnings; full workspace tests; the properties. Formatting, clippy, and the properties passed on the head above; the full workspace suite passed on rerun with the intermittent failure below still unresolved. The first final-head workspace run failed unchanged reviewer test one_low_priority_startup_worker_converges_after_home_creation_and_deletion_then_parks at its recursive Home deletion (DirectoryNotEmpty). The isolated test and full suite then passed on unchanged source. The background Home fence creation versus recursive deletion race is an observed intermittent failure, not a resolved issue; no reviewer assertion was changed or retried inside the test. All cargo build/test commands used /usr/local/bin/build-slot.sh, -j8 where supported, and CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse.

cargo fmt --all -- --check
cargo clippy -j8 --workspace --all-targets -- -D warnings
cargo test -j8 --workspace
QUOTA_PROPERTY_CASES=50000 QUOTA_PROPERTY_SEED=76976420261030 cargo test -j8 -p calternal-fsio -p calternal-auth property -- --nocapture
QUOTA_PROPERTY_CASES=50000 QUOTA_PROPERTY_SEED=76976420261030 cargo test -j8 -p calternal-fsio seeded_fallback_admission_never_uses_an_underestimate -- --nocapture
QUOTA_PROPERTY_CASES=50000 QUOTA_PROPERTY_SEED=76976420261030 cargo test -j8 -p calternal-fsio review_764c_random_mode_switches -- --nocapture
cargo test -j8 -p calternal-fsio review_764_real_kernel_overflow -- --ignored --nocapture
cargo test -j8 -p calternal-fsio fallback_admission_must_not_accept_a_non_linearizable_walk -- --ignored --nocapture

The five 50,000-case properties cover whole-Home accounting, real Auth admission, Auth fallback admission, filesystem fallback admission, and random mode switches. Auth admission had 23,513 admitted and 26,487 quota-refused cases. Fallback properties deliberately verify retryable refusal; the normal Auth property and save/upload race also require positive admission. Actual kernel overflow and the explicit non-linearizable fallback reproducer passed. An earlier combined runner stopped with signal 143 during mode switches. A rerun exposed a failed-listener display recount exhausting its retries while queued events advanced the dirty generation. The follow-up fix retains newer dirtiness and refuses admission while allowing a complete provisional display count only when the listener is unhealthy; healthy coverage gaps retain the existing retry behavior; a deterministic regression was added. Required gates were rerun on the final head. No success was inferred from the interrupted or failed run.

The existing five-minute clean-idle/deadline probe, clean usage-worker CPU probe, and unreadable-subtree idle/recovery probe each passed in a separate process. Existing ignored diagnostic/privileged tests remain marked as such; full workspace tests does not imply every ignored diagnostic ran.

Build host measurements: 10by32-02-std, arm64, shared host; release build on predecessor 88a61f3df8b8e04add12eb9e1be2ed4cc1287a75. The final follow-up changes only unhealthy-listener display recounts; these measurements were not repeated after that fix, so final-head VM measurements remain required. Each save loop runs 60 seconds, 300 edits and 60 explicit saves through real Notes + Engine. Notes file watcher and auto-rename are disabled, so these isolate the save path and do not qualify the full client/search/long-note W01/W02/W07/W09 scenario. Fixtures have 1 KiB files distributed across directories of 1,000 files, plus the note and app state. CPU is process user+system time as a percentage of one core; setup/shutdown are excluded.

Layout Fixture names 60 s CPU Recount starts during loop Receipt commits / fallback Coverage-cold recount
split state 10,000 1.0333% 0 60 / 0 44.823 ms
split state 100,000 0.8833% 0 60 / 0 301.484 ms
default 10,000 4.0499% 119 1 / 59 33.664 ms
default 100,000 22.5494% 125 0 / 60 315.754 ms

Every loop ended with verified bytes equal to an independent quiescent physical-file oracle. Coverage-cold starts from a fresh Usage counter and native watch coverage and includes all passes needed for a verified count. OS page cache was not dropped on this shared host; these are not cold-I/O or perf-VM measurements. The coordinator must remeasure on the perf VM. Default layout fails both the 2% CPU and zero-own-save-recount criteria. Split-state figures are isolated build-host evidence, not final VM acceptance.

Raw measurement log:

build host layout=split names=10000 cold-coverage recount_ms=44.823 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped)
build host layout=split names=10000 save-loop_seconds=60.002 edits=300 saves=60 cpu_seconds=0.620 one_core_cpu_pct=1.0333 recount_starts=0 receipt_commits=60 receipt_fallbacks=0
build host layout=split names=10000 quiescent_charge_oracle=10241256 verified=10241256
build host layout=split names=100000 cold-coverage recount_ms=301.484 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped)
build host layout=split names=100000 save-loop_seconds=60.000 edits=300 saves=60 cpu_seconds=0.530 one_core_cpu_pct=0.8833 recount_starts=0 receipt_commits=60 receipt_fallbacks=0
build host layout=split names=100000 quiescent_charge_oracle=102401256 verified=102401256
build host layout=default names=10000 cold-coverage recount_ms=33.664 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped)
build host layout=default names=10000 save-loop_seconds=60.001 edits=300 saves=60 cpu_seconds=2.430 one_core_cpu_pct=4.0499 recount_starts=119 receipt_commits=1 receipt_fallbacks=59
build host layout=default names=10000 quiescent_charge_oracle=17881687 verified=17881687
build host layout=default names=100000 cold-coverage recount_ms=315.754 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped)
build host layout=default names=100000 save-loop_seconds=60.001 edits=300 saves=60 cpu_seconds=13.530 one_core_cpu_pct=22.5494 recount_starts=125 receipt_commits=0 receipt_fallbacks=60
build host layout=default names=100000 quiescent_charge_oracle=110041687 verified=110041687

Explicit property/overflow evidence:

Auth property initial seed: 76976420261030
Auth fallback property initial seed: 76976420261030
Auth fallback admission property: 50000 random Home/upload cases; all refused retryably
quota Auth admission property: 50000 seeded cases passed (23513 admitted, 26487 quota-refused)
filesystem property initial seed: 76976420261030
quota whole-Home property: 50000 seeded cases passed
fallback property initial seed: 76976420261030
fallback admission property: 50000 random Home/upload/quota cases; 25000 outside writes during recount; all refused retryably
round-3 mode-switch property seed=76976420261030, cases=50000
mode counts=[5518, 5470, 5674, 5597, 5540, 5542, 5549, 5561, 5549]; 39012 exact accepted reads; actual missing-native recovery passed
real kernel queue overflow at capacity 1048576: stale total refused or repaired

Idle logs:

review_764_five_minutes_clean_idle_has_no_recount_deadlines_or_walks: exit 0

running 1 test
test usage::review_764::review_764_five_minutes_clean_idle_has_no_recount_deadlines_or_walks has been running for over 60 seconds
300s idle: 2963 observations; zero walk deadlines, zero new walks, zero invalidations
test usage::review_764::review_764_five_minutes_clean_idle_has_no_recount_deadlines_or_walks ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.01s


clean_five_minute_idle_cpu: exit 0

running 1 test

running 1 test
test usage::review_764b::clean_five_minute_idle_cpu has been running for over 60 seconds
test usage::review_764b::clean_five_minute_idle_cpu has been running for over 60 seconds
healthy idle worker ticks over 300s: [("3281170", 0), ("3281172", 0)] -> [("3281170", 0), ("3281172", 0)]
test usage::review_764b::clean_five_minute_idle_cpu ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.33s

test usage::review_764b::clean_five_minute_idle_cpu ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.33s


unreadable_subtree_five_minute_idle_trace: exit 0

running 1 test
test usage::fix_tests::unreadable_subtree_five_minute_idle_trace has been running for over 60 seconds
quota worker CPU ticks before/after 300s: [("3281169:usage-events", 0), ("3281171:usage-recount", 0)] / [("3281169:usage-events", 0), ("3281171:usage-recount", 0)]
300s unreadable-subtree idle: 2954 samples; zero walk deadlines, zero walks, zero invalidations; retained Health entries: 1
test usage::fix_tests::unreadable_subtree_five_minute_idle_trace ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.28s

W01–W22 qualification checklist on this head (all perf-VM cells remain missing):

Workload Evidence status
W01 Typing, saves every few seconds Build-host isolated loop only; default failed; VM evidence missing.
W02 Autosave 10k-line note VM evidence missing; no acceptance claim.
W03 Upload one / 1,000 photos VM evidence missing; no acceptance claim.
W04 Outside import 50k files VM evidence missing; no acceptance claim.
W05 Phone offline a week VM evidence missing; no acceptance claim.
W06 Trash/restore/purge 10k VM evidence missing; no acceptance claim.
W07 Version history VM evidence missing; no acceptance claim.
W08 Contacts .vcf companions VM evidence missing; no acceptance claim.
W09 Search/index/app state VM evidence missing; no acceptance claim.
W10 Syncthing/rsync/editor swap/temp VM evidence missing; no acceptance claim.
W11 git checkout / node_modules VM evidence missing; no acceptance claim.
W12 Several people typing VM evidence missing; no acceptance claim.
W13 Startup, 100 Homes VM evidence missing; no acceptance claim.
W14 inotify overflow in import VM evidence missing; no acceptance claim.
W15 #761 two-tree Home VM evidence missing; no acceptance claim.
W16 Near quota / far below VM evidence missing; no acceptance claim.
W17 Clock skew VM evidence missing; no acceptance claim.
W18 Home move / backup restore VM evidence missing; no acceptance claim.
W19 Crash/cancel/retry/disk full VM evidence missing; no acceptance claim.
W20 Hardlinks/sparse files/mounts VM evidence missing; no acceptance claim.
W21 Deep/empty dirs, huge siblings VM evidence missing; no acceptance claim.
W22 Quota/provider/root policy change VM evidence missing; no acceptance claim.

Product merge remains blocked by default-layout accounting and missing VM qualification. No dev merge, deployment, or perf-test-* access was performed. Logs and benchmark sources remain at /home/agent/codex-jobs/quota-owned; no logs were committed.

Receipt accounting and bounded parallel cold enumeration are implemented, but this is **not merge-ready**. Default-layout database writes still cause Home recounts and fail the typing gate; state migration or a proven in-place database accounting protocol and final perf-VM qualification remain outstanding. The final-head workspace run also exposed an intermittent startup Home-deletion/fence race; its rerun results are recorded below. Implementation head: `86f48a39bcace15c13dbcd812cb2e991552d1e17` on `perf/quota-simple`. Plan amendment: `85f4c31c38becd61000c2efd4ae6dff416729892` on `docs/quota-plan-2`, quoted in https://git.kayg.org/kayg/calternal/pulls/769#issuecomment-70238 and merged into the implementation branch. Closed-image atomic writes, prepared note saves, streamed uploads and tracked batch file publications use pinned inode identities, watch incarnations, covered size observations, ordered fences and bounded cookie-paired receipts to commit signed byte deltas exactly once. Content notifications are never consumed as echoes. Unmatched events, outside replacement/link changes, overflow, watch loss and failed or canceled operations invalidate authority. Publication/accounting and provider transactions share an owned per-Home permit; the event reader never takes it. Recounts check both external generation and accounting sequence, and serialize their watch-set changes/final commit. Contacts provider errors refuse admission instead of contributing zero. Relative state roots are matched against SQLite's absolute filenames. Complete display-only recounts with an unhealthy listener may publish a provisional snapshot without clearing newer dirtiness or authorizing admission; accounting-sequence changes still reject old snapshots. Cold enumeration retains fd-relative no-follow/mount checks and descriptor reuse, moves independent substantial directories to at most eight workers, and coalesces new-watch invalidations within each rejected scan attempt. d_type remains a hint; size, identity and mount metadata checks remain. Added deterministic coverage for pre-watch growth/truncate/replacement, post-watch changes, outside grow/truncate/rename-over/hard links/equal-size writes/restored mtime, two prepared same/different-destination writes, delayed ordered delivery, recount/prune versus a newer delta, receipt overflow, cancel/reject/retirement, checked-provider failures and transitions, relative state roots, and a prepared save racing real Auth upload admission with positive recovery. Kernel order is preserved; userspace batching and concurrent operation scheduling are varied. Reviewer tests and assertions are retained. One superseded product assertion was adapted: `app_publication_adds_known_bytes_without_authorizing_a_provisional_total` is now `covered_app_publication_adds_exact_bytes_without_a_recount`; a separate dirty/unmeasured test retains the refusal obligation. `simple_tests.rs` adds only a hook-presence helper so deterministic single-thread scanner hooks remain deterministic. No reviewer assertion was removed. Performance qualification remains blocked for the default layout: public in-place DB/WAL/SHM writes retain conservative Home invalidation. This change does not implement a state migration or a replacement physical-DB accounting protocol. Namespace/directory transitions without a complete receipt proof also retain conservative recounts. The full W01–W22 VM evidence checklist, including long-note/search/history variants, capacity/failure recovery, 1M/100-Home cases, flat folders, upload concurrency and both state layouts, remains required before product merge; unrun cells are missing evidence. Required gates (verbatim): **fmt; clippy -D warnings; full workspace tests; the properties.** Formatting, clippy, and the properties passed on the head above; the full workspace suite passed on rerun with the intermittent failure below still unresolved. The first final-head workspace run failed unchanged reviewer test `one_low_priority_startup_worker_converges_after_home_creation_and_deletion_then_parks` at its recursive Home deletion (`DirectoryNotEmpty`). The isolated test and full suite then passed on unchanged source. The background Home fence creation versus recursive deletion race is an observed intermittent failure, not a resolved issue; no reviewer assertion was changed or retried inside the test. All cargo build/test commands used `/usr/local/bin/build-slot.sh`, `-j8` where supported, and `CALTERNAL_INODE_REUSE_DIR=/srv/inode-reuse`. ```sh cargo fmt --all -- --check cargo clippy -j8 --workspace --all-targets -- -D warnings cargo test -j8 --workspace QUOTA_PROPERTY_CASES=50000 QUOTA_PROPERTY_SEED=76976420261030 cargo test -j8 -p calternal-fsio -p calternal-auth property -- --nocapture QUOTA_PROPERTY_CASES=50000 QUOTA_PROPERTY_SEED=76976420261030 cargo test -j8 -p calternal-fsio seeded_fallback_admission_never_uses_an_underestimate -- --nocapture QUOTA_PROPERTY_CASES=50000 QUOTA_PROPERTY_SEED=76976420261030 cargo test -j8 -p calternal-fsio review_764c_random_mode_switches -- --nocapture cargo test -j8 -p calternal-fsio review_764_real_kernel_overflow -- --ignored --nocapture cargo test -j8 -p calternal-fsio fallback_admission_must_not_accept_a_non_linearizable_walk -- --ignored --nocapture ``` The five 50,000-case properties cover whole-Home accounting, real Auth admission, Auth fallback admission, filesystem fallback admission, and random mode switches. Auth admission had 23,513 admitted and 26,487 quota-refused cases. Fallback properties deliberately verify retryable refusal; the normal Auth property and save/upload race also require positive admission. Actual kernel overflow and the explicit non-linearizable fallback reproducer passed. An earlier combined runner stopped with signal 143 during mode switches. A rerun exposed a failed-listener display recount exhausting its retries while queued events advanced the dirty generation. The follow-up fix retains newer dirtiness and refuses admission while allowing a complete provisional display count only when the listener is unhealthy; healthy coverage gaps retain the existing retry behavior; a deterministic regression was added. Required gates were rerun on the final head. No success was inferred from the interrupted or failed run. The existing five-minute clean-idle/deadline probe, clean usage-worker CPU probe, and unreadable-subtree idle/recovery probe each passed in a separate process. Existing ignored diagnostic/privileged tests remain marked as such; full workspace tests does not imply every ignored diagnostic ran. Build host measurements: 10by32-02-std, arm64, shared host; release build on predecessor `88a61f3df8b8e04add12eb9e1be2ed4cc1287a75`. The final follow-up changes only unhealthy-listener display recounts; these measurements were not repeated after that fix, so final-head VM measurements remain required. Each save loop runs 60 seconds, 300 edits and 60 explicit saves through real Notes + Engine. Notes file watcher and auto-rename are disabled, so these isolate the save path and do not qualify the full client/search/long-note W01/W02/W07/W09 scenario. Fixtures have 1 KiB files distributed across directories of 1,000 files, plus the note and app state. CPU is process user+system time as a percentage of one core; setup/shutdown are excluded. | Layout | Fixture names | 60 s CPU | Recount starts during loop | Receipt commits / fallback | Coverage-cold recount | |---|---:|---:|---:|---:|---:| | split state | 10,000 | 1.0333% | 0 | 60 / 0 | 44.823 ms | | split state | 100,000 | 0.8833% | 0 | 60 / 0 | 301.484 ms | | default | 10,000 | 4.0499% | 119 | 1 / 59 | 33.664 ms | | default | 100,000 | 22.5494% | 125 | 0 / 60 | 315.754 ms | Every loop ended with verified bytes equal to an independent quiescent physical-file oracle. Coverage-cold starts from a fresh Usage counter and native watch coverage and includes all passes needed for a verified count. OS page cache was **not dropped** on this shared host; these are not cold-I/O or perf-VM measurements. The coordinator must remeasure on the perf VM. Default layout fails both the 2% CPU and zero-own-save-recount criteria. Split-state figures are isolated build-host evidence, not final VM acceptance. Raw measurement log: ```text build host layout=split names=10000 cold-coverage recount_ms=44.823 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped) build host layout=split names=10000 save-loop_seconds=60.002 edits=300 saves=60 cpu_seconds=0.620 one_core_cpu_pct=1.0333 recount_starts=0 receipt_commits=60 receipt_fallbacks=0 build host layout=split names=10000 quiescent_charge_oracle=10241256 verified=10241256 build host layout=split names=100000 cold-coverage recount_ms=301.484 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped) build host layout=split names=100000 save-loop_seconds=60.000 edits=300 saves=60 cpu_seconds=0.530 one_core_cpu_pct=0.8833 recount_starts=0 receipt_commits=60 receipt_fallbacks=0 build host layout=split names=100000 quiescent_charge_oracle=102401256 verified=102401256 build host layout=default names=10000 cold-coverage recount_ms=33.664 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped) build host layout=default names=10000 save-loop_seconds=60.001 edits=300 saves=60 cpu_seconds=2.430 one_core_cpu_pct=4.0499 recount_starts=119 receipt_commits=1 receipt_fallbacks=59 build host layout=default names=10000 quiescent_charge_oracle=17881687 verified=17881687 build host layout=default names=100000 cold-coverage recount_ms=315.754 activity=Activity { recount_starts: 2, recount_completions: 1, receipt_commits: 0, receipt_fallbacks: 0 } (page cache not dropped) build host layout=default names=100000 save-loop_seconds=60.001 edits=300 saves=60 cpu_seconds=13.530 one_core_cpu_pct=22.5494 recount_starts=125 receipt_commits=0 receipt_fallbacks=60 build host layout=default names=100000 quiescent_charge_oracle=110041687 verified=110041687 ``` Explicit property/overflow evidence: ```text Auth property initial seed: 76976420261030 Auth fallback property initial seed: 76976420261030 Auth fallback admission property: 50000 random Home/upload cases; all refused retryably quota Auth admission property: 50000 seeded cases passed (23513 admitted, 26487 quota-refused) filesystem property initial seed: 76976420261030 quota whole-Home property: 50000 seeded cases passed fallback property initial seed: 76976420261030 fallback admission property: 50000 random Home/upload/quota cases; 25000 outside writes during recount; all refused retryably round-3 mode-switch property seed=76976420261030, cases=50000 mode counts=[5518, 5470, 5674, 5597, 5540, 5542, 5549, 5561, 5549]; 39012 exact accepted reads; actual missing-native recovery passed real kernel queue overflow at capacity 1048576: stale total refused or repaired ``` Idle logs: ```text review_764_five_minutes_clean_idle_has_no_recount_deadlines_or_walks: exit 0 running 1 test test usage::review_764::review_764_five_minutes_clean_idle_has_no_recount_deadlines_or_walks has been running for over 60 seconds 300s idle: 2963 observations; zero walk deadlines, zero new walks, zero invalidations test usage::review_764::review_764_five_minutes_clean_idle_has_no_recount_deadlines_or_walks ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.01s clean_five_minute_idle_cpu: exit 0 running 1 test running 1 test test usage::review_764b::clean_five_minute_idle_cpu has been running for over 60 seconds test usage::review_764b::clean_five_minute_idle_cpu has been running for over 60 seconds healthy idle worker ticks over 300s: [("3281170", 0), ("3281172", 0)] -> [("3281170", 0), ("3281172", 0)] test usage::review_764b::clean_five_minute_idle_cpu ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.33s test usage::review_764b::clean_five_minute_idle_cpu ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.33s unreadable_subtree_five_minute_idle_trace: exit 0 running 1 test test usage::fix_tests::unreadable_subtree_five_minute_idle_trace has been running for over 60 seconds quota worker CPU ticks before/after 300s: [("3281169:usage-events", 0), ("3281171:usage-recount", 0)] / [("3281169:usage-events", 0), ("3281171:usage-recount", 0)] 300s unreadable-subtree idle: 2954 samples; zero walk deadlines, zero walks, zero invalidations; retained Health entries: 1 test usage::fix_tests::unreadable_subtree_five_minute_idle_trace ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 146 filtered out; finished in 300.28s ``` W01–W22 qualification checklist on this head (all perf-VM cells remain missing): | Workload | Evidence status | |---|---| | W01 Typing, saves every few seconds | Build-host isolated loop only; default failed; VM evidence missing. | | W02 Autosave 10k-line note | VM evidence missing; no acceptance claim. | | W03 Upload one / 1,000 photos | VM evidence missing; no acceptance claim. | | W04 Outside import 50k files | VM evidence missing; no acceptance claim. | | W05 Phone offline a week | VM evidence missing; no acceptance claim. | | W06 Trash/restore/purge 10k | VM evidence missing; no acceptance claim. | | W07 Version history | VM evidence missing; no acceptance claim. | | W08 Contacts .vcf companions | VM evidence missing; no acceptance claim. | | W09 Search/index/app state | VM evidence missing; no acceptance claim. | | W10 Syncthing/rsync/editor swap/temp | VM evidence missing; no acceptance claim. | | W11 git checkout / node_modules | VM evidence missing; no acceptance claim. | | W12 Several people typing | VM evidence missing; no acceptance claim. | | W13 Startup, 100 Homes | VM evidence missing; no acceptance claim. | | W14 inotify overflow in import | VM evidence missing; no acceptance claim. | | W15 #761 two-tree Home | VM evidence missing; no acceptance claim. | | W16 Near quota / far below | VM evidence missing; no acceptance claim. | | W17 Clock skew | VM evidence missing; no acceptance claim. | | W18 Home move / backup restore | VM evidence missing; no acceptance claim. | | W19 Crash/cancel/retry/disk full | VM evidence missing; no acceptance claim. | | W20 Hardlinks/sparse files/mounts | VM evidence missing; no acceptance claim. | | W21 Deep/empty dirs, huge siblings | VM evidence missing; no acceptance claim. | | W22 Quota/provider/root policy change | VM evidence missing; no acceptance claim. | Product merge remains blocked by default-layout accounting and missing VM qualification. No dev merge, deployment, or perf-test-* access was performed. Logs and benchmark sources remain at `/home/agent/codex-jobs/quota-owned`; no logs were committed.
kayg commented 2026-10-31 06:27:59 +00:00
Author
Owner
Copy link

Baseline on origin/dev d83e54db (= prod), perf-02, 100,000 notes, 6 repetitions (calternal-perf --sizes 100k --runs 1 --cold-runs 0 --idle-secs 300; each rep is a fresh server on the same indexed Home; method as in docs/perf/budgets.md). Run via perf/vm/typing-idle.sh (PR #787).

metric median min max budget
load.typing_cpu_pct (5 chars/s, 60 s, save check every 5, 1 watcher) 1.125 % 1.117 1.217 <= 2 %
load.typing_wakeups_per_s 60.2 57.7 62.2
idle.cpu_pct (300 s) 0.647 % 0.530 0.670 <= 0.1 % (broken)
idle.wakeups_per_s 0.102 0.093 0.120 <= 0.2
idle.rss 106.1 MB 104.9 109.4 <= 100 MB (broken)

Per rep (idle % / typing %): 0.67/1.15, 0.53/1.22, 0.64/1.12, 0.65/1.12, 0.65/1.12, 0.65/1.13.

Noise: typing is tight (about 9 % spread, one outlier). Idle CPU is read in 10 ms ticks over 300 s (0.01 % per tick; 159-201 ticks), so 0.53 is the one low run and the rest sit at 0.64-0.67. Matches the coordinator's earlier perf-test-02 number (0.630 % / 107.3 MiB). Setup: claude@calternal-perf-0{1,2}-backup01, AMD Ryzen 7 3700X guests (KVM), 4 vCPUs, 7.7 GiB, ext4 on a non-rotational virtual disk (uncached), no cpufreq interface (MHz constant at 3593; frequency cannot be observed). Binaries: origin/dev d83e54db, release, built on dev-01 (sha256 of calternal 2ac46c10...). Steal time 0.07-0.12 % per run. One measurement at a time via flock /tmp/perf.lock.

Raw files on perf-02: ~/perf/out-typing-dev/.

Baseline on `origin/dev` d83e54db (= prod), perf-02, 100,000 notes, 6 repetitions (`calternal-perf --sizes 100k --runs 1 --cold-runs 0 --idle-secs 300`; each rep is a fresh server on the same indexed Home; method as in docs/perf/budgets.md). Run via `perf/vm/typing-idle.sh` (PR #787). | metric | median | min | max | budget | |---|---|---|---|---| | `load.typing_cpu_pct` (5 chars/s, 60 s, save check every 5, 1 watcher) | **1.125 %** | 1.117 | 1.217 | <= 2 % | | `load.typing_wakeups_per_s` | 60.2 | 57.7 | 62.2 | | | `idle.cpu_pct` (300 s) | **0.647 %** | 0.530 | 0.670 | <= 0.1 % (broken) | | `idle.wakeups_per_s` | 0.102 | 0.093 | 0.120 | <= 0.2 | | `idle.rss` | **106.1 MB** | 104.9 | 109.4 | <= 100 MB (broken) | Per rep (idle % / typing %): 0.67/1.15, 0.53/1.22, 0.64/1.12, 0.65/1.12, 0.65/1.12, 0.65/1.13. Noise: typing is tight (about 9 % spread, one outlier). Idle CPU is read in 10 ms ticks over 300 s (0.01 % per tick; 159-201 ticks), so 0.53 is the one low run and the rest sit at 0.64-0.67. Matches the coordinator's earlier perf-test-02 number (0.630 % / 107.3 MiB). Setup: `claude@calternal-perf-0{1,2}-backup01`, AMD Ryzen 7 3700X guests (KVM), 4 vCPUs, 7.7 GiB, ext4 on a non-rotational virtual disk (uncached), no cpufreq interface (MHz constant at 3593; frequency cannot be observed). Binaries: `origin/dev` d83e54db, release, built on dev-01 (sha256 of `calternal` 2ac46c10...). Steal time 0.07-0.12 % per run. One measurement at a time via `flock /tmp/perf.lock`. Raw files on perf-02: `~/perf/out-typing-dev/`.
kayg commented 2026-11-01 11:51:48 +00:00
Author
Owner
Copy link

quota/cold r4 retained results

Budgets: typing CPU <=2%; idle CPU <=0.1%; idle RSS <=100 MB; wakeups <=0.2/s. Six matched pairs per comparison required; matrix stopped at quota-4.

sample typing CPU % idle CPU % RSS MB wakeups/s verdict
cold-1 0.567 0.000 83.953 0.000 PASS
cold-2 0.483 0.000 76.297 0.000 PASS
cold-3 0.583 0.000 84.355 0.000 PASS
dev-cold-1 1.383 0.740 125.262 0.097 FAIL idle CPU and RSS
dev-cold-2 1.333 0.697 129.840 0.073 FAIL idle CPU and RSS
dev-cold-3 1.333 0.720 122.289 0.087 FAIL idle CPU and RSS
dev-quota-1 1.250 0.600 112.441 0.117 EXCLUDED: diagnostic walk overlapped
dev-quota-2 1.283 0.710 114.918 0.117 FAIL idle CPU and RSS
dev-quota-3 1.233 0.687 118.234 0.080 FAIL idle CPU and RSS
dev-quota-4 1.367 0.717 120.625 0.087 FAIL idle CPU and RSS; unpaired
quota-1 0.600 0.000 85.453 0.000 EXCLUDED: paired baseline excluded
quota-2 0.567 0.000 82.914 0.000 PASS
quota-3 0.567 0.000 83.332 0.000 PASS
quota-4 — — — — INVALID: first page empty/incomplete

13 runs completed, one failed run (quota-4). Two completed samples are excluded from matched comparisons. Available accepted pairs: quota rounds 2–3 (2/6); cold rounds 1–3 (3/6). dev-quota-4 is valid but unpaired. No six-pair qualification or cold-growth gate is claimed. Remaining runs: quota-4 retry, dev-cold-4/cold-4, and every arm of rounds 5–6 (11 invocations total).

disposition, diagnosis and reproduction

The r4 script had already exited after quota-4; no quota r4 process or measurement lock remained to stop. typing-idle.sh writes DONE and rc=0 even when the driver records scenario failures (it does not pass --gate). The outer r4 script correctly rejects a nonempty sizes[0].failures; that check caught quota-4. Treating DONE or rc=0 alone as success would be a harness error.

Exact failure: HTTP 200, result={"incomplete":true,"items":[],"next":null,"total":0}. There was no quota refusal or failed HTTP status. The failed fixture's derived notes/meta.db had zero notes, schema 4, stats notes=0, and PRAGMA quick_check returned ok. The retained warm server log contains the development-mode warning, with no recorded scan/disk error. The first-page harness in crates/perf/src/scenarios/home.rs rejects any empty list immediately; server.rs waits only for /healthz. Notes explicitly allow a partial list before Home discovery completes (Notes::page and the_first_page_never_waits_for_the_home_to_open). incomplete:true alone is not evidence of a product defect.

Functional reproduction on dev-01, using a copy of the exact failed quota-4 userdata and appdata and the existing quota binary (SHA256 9c055318e954f79d0309dc6c99637eff98df35fba06c506cb28167730a10078d), under ~/bin/slot, with no build or perf claim:

  1. Start calternal serve --create --listen 127.0.0.1:<free-port> --dev-user 0192f0a0-0000-7000-8000-00000000f1f0 --data <copy>/runs/n100000/indexed --state <copy>/appdata/runs/n100000/indexed, with GLIBC_TUNABLES=glibc.malloc.arena_max=2.
  2. Wait only for healthz, then POST /api/v1/actions/notes.list with {"limit":50}, Content-Type: application/json and X-Requested-With: calternal. At 0.021s: 0 total / 0 items / incomplete=true, reproducing the r4 assertion failure.
  3. Repeat the request once per second. At 10.457s: 32,768 total; at 20.530s: 61,440; at 30.965s: 100,000 but still incomplete; at 32.975s: 100,000 / 50 items / incomplete=false. Stop this exact child with SIGTERM and wait; exit 0.
  4. Restart the same repaired copy. At 0.021s: 100,000 / 50 items / incomplete=true; at 5.031s: incomplete=false. SIGTERM and wait; exit 0.

Verdict: observed failure is a harness readiness/precondition failure on a rebuilding cache, not evidence of a persistent list failure under quota or cold growth. The source of the earlier derived-cache reset is not established. MetaDb::open_with on the measured quota head silently removes/rebuilds the derived DB after any open error, so a storage/open problem is a possible cause; neither corruption, ENOSPC nor a quota-specific regression has been proved. Do not call this root cause closed: a separate clean rerun needs logs of the first open error and cache counts before/after cold-to-warm handoff. Keep a genuine complete-first-page warm-start gate if that is the intended product requirement; otherwise explicitly wait for readiness outside typing/idle measurement, with a deadline and preserved startup timing. Do not silently weaken that gate or count quota-4 as passed.

Measured stack is #764 75978ddb5ed30152a85fbb1d766a62d5c5a73113 and #767 8b5b8cdc74385b3bcbf1540d1c612c075952ef06 (the quota/cold-on-tiering work), using the common driver SHA256 2e3656cd3255e5d6b590705e2c5ea521f8d1dd7642fcb26b7f3aa8eaad94e4fc, and typing script 13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b. Successful retained candidate samples pass all four absolute typing/idle budgets. This is not a completed paired qualification and does not establish the cold-growth <=10% gate.

Small JSON/MD reports, run logs, scripts and reproduction observations are retained on perf-02 at ~/perf-reports-kept/takeover-20261101/quota-r4/ and on control at /home/kayg/perf-reports/takeover-20261101/. Seeded r4 work/Homes are removed after copying the diagnostic fixture; the dev-01 copy is removed after both exact child processes have exited. The previous r2 cleanup was left untouched until it finished; only then were seeded Markdown Homes inadvertently copied into its report directory removed, retaining its real top-level reports. No merge or deployment.

Final cleanup verification

Both 759b units stopped; quota r4 had already exited. All takeover cleanup and reproduction processes have exited. Earlier r2 cleanup completed untouched. Final free: perf-01 root 51G, split Home 58G; perf-02 root 37G, split Home 97G. Shared locks are held by #792 qualification (159697 / 1325428), not the stopped jobs. Cleanup/fixture-copy I/O overlapped the first #792 samples; those samples need overlap exclusion. Retained observations do not establish why quota-4 cache was reset.

Perf-02 cleanup and fixture-copy overlap was approximately 17:13–17:22 IST on 1 November (11:43–11:52 UTC). The earlier r2 file walk also overlapped #792 startup until it completed. #792 first overlapping samples must not be treated as uncontended measurements. No #792 process was stopped.

# quota/cold r4 retained results Budgets: typing CPU <=2%; idle CPU <=0.1%; idle RSS <=100 MB; wakeups <=0.2/s. Six matched pairs per comparison required; matrix stopped at quota-4. | sample | typing CPU % | idle CPU % | RSS MB | wakeups/s | verdict | |---|---:|---:|---:|---:|---| | cold-1 | 0.567 | 0.000 | 83.953 | 0.000 | PASS | | cold-2 | 0.483 | 0.000 | 76.297 | 0.000 | PASS | | cold-3 | 0.583 | 0.000 | 84.355 | 0.000 | PASS | | dev-cold-1 | 1.383 | 0.740 | 125.262 | 0.097 | FAIL idle CPU and RSS | | dev-cold-2 | 1.333 | 0.697 | 129.840 | 0.073 | FAIL idle CPU and RSS | | dev-cold-3 | 1.333 | 0.720 | 122.289 | 0.087 | FAIL idle CPU and RSS | | dev-quota-1 | 1.250 | 0.600 | 112.441 | 0.117 | EXCLUDED: diagnostic walk overlapped | | dev-quota-2 | 1.283 | 0.710 | 114.918 | 0.117 | FAIL idle CPU and RSS | | dev-quota-3 | 1.233 | 0.687 | 118.234 | 0.080 | FAIL idle CPU and RSS | | dev-quota-4 | 1.367 | 0.717 | 120.625 | 0.087 | FAIL idle CPU and RSS; unpaired | | quota-1 | 0.600 | 0.000 | 85.453 | 0.000 | EXCLUDED: paired baseline excluded | | quota-2 | 0.567 | 0.000 | 82.914 | 0.000 | PASS | | quota-3 | 0.567 | 0.000 | 83.332 | 0.000 | PASS | | quota-4 | — | — | — | — | INVALID: first page empty/incomplete | 13 runs completed, one failed run (quota-4). Two completed samples are excluded from matched comparisons. Available accepted pairs: quota rounds 2–3 (2/6); cold rounds 1–3 (3/6). dev-quota-4 is valid but unpaired. No six-pair qualification or cold-growth gate is claimed. Remaining runs: quota-4 retry, dev-cold-4/cold-4, and every arm of rounds 5–6 (11 invocations total). ## disposition, diagnosis and reproduction The r4 script had already exited after quota-4; no quota r4 process or measurement lock remained to stop. `typing-idle.sh` writes DONE and `rc=0` even when the driver records scenario failures (it does not pass `--gate`). The outer r4 script correctly rejects a nonempty `sizes[0].failures`; that check caught quota-4. Treating DONE or rc=0 alone as success would be a harness error. Exact failure: HTTP 200, `result={"incomplete":true,"items":[],"next":null,"total":0}`. There was no quota refusal or failed HTTP status. The failed fixture's derived `notes/meta.db` had zero notes, schema 4, stats `notes=0`, and `PRAGMA quick_check` returned `ok`. The retained warm server log contains the development-mode warning, with no recorded scan/disk error. The first-page harness in `crates/perf/src/scenarios/home.rs` rejects *any* empty list immediately; `server.rs` waits only for `/healthz`. Notes explicitly allow a partial list before Home discovery completes (`Notes::page` and `the_first_page_never_waits_for_the_home_to_open`). `incomplete:true` alone is not evidence of a product defect. Functional reproduction on **dev-01**, using a copy of the exact failed quota-4 userdata and appdata and the existing quota binary (SHA256 `9c055318e954f79d0309dc6c99637eff98df35fba06c506cb28167730a10078d`), under `~/bin/slot`, with no build or perf claim: 1. Start `calternal serve --create --listen 127.0.0.1:<free-port> --dev-user 0192f0a0-0000-7000-8000-00000000f1f0 --data <copy>/runs/n100000/indexed --state <copy>/appdata/runs/n100000/indexed`, with `GLIBC_TUNABLES=glibc.malloc.arena_max=2`. 2. Wait only for healthz, then POST `/api/v1/actions/notes.list` with `{"limit":50}`, `Content-Type: application/json` and `X-Requested-With: calternal`. At 0.021s: **0 total / 0 items / incomplete=true**, reproducing the r4 assertion failure. 3. Repeat the request once per second. At 10.457s: 32,768 total; at 20.530s: 61,440; at 30.965s: 100,000 but still incomplete; at **32.975s: 100,000 / 50 items / incomplete=false**. Stop this exact child with SIGTERM and wait; exit 0. 4. Restart the same repaired copy. At **0.021s: 100,000 / 50 items / incomplete=true**; at **5.031s: incomplete=false**. SIGTERM and wait; exit 0. **Verdict: observed failure is a harness readiness/precondition failure on a rebuilding cache, not evidence of a persistent list failure under quota or cold growth.** The source of the earlier derived-cache reset is not established. `MetaDb::open_with` on the measured quota head silently removes/rebuilds the derived DB after any open error, so a storage/open problem is a possible cause; neither corruption, ENOSPC nor a quota-specific regression has been proved. Do not call this root cause closed: a separate clean rerun needs logs of the first open error and cache counts before/after cold-to-warm handoff. Keep a genuine complete-first-page warm-start gate if that is the intended product requirement; otherwise explicitly wait for readiness outside typing/idle measurement, with a deadline and preserved startup timing. Do not silently weaken that gate or count quota-4 as passed. Measured stack is #764 `75978ddb5ed30152a85fbb1d766a62d5c5a73113` and #767 `8b5b8cdc74385b3bcbf1540d1c612c075952ef06` (the quota/cold-on-tiering work), using the common driver SHA256 `2e3656cd3255e5d6b590705e2c5ea521f8d1dd7642fcb26b7f3aa8eaad94e4fc`, and typing script `13c2e7bf6489663fcf99f21ddc010118d27e5a33e773c9f89be0a1ff113df90b`. Successful retained candidate samples pass all four absolute typing/idle budgets. This is not a completed paired qualification and does not establish the cold-growth <=10% gate. Small JSON/MD reports, run logs, scripts and reproduction observations are retained on perf-02 at `~/perf-reports-kept/takeover-20261101/quota-r4/` and on control at `/home/kayg/perf-reports/takeover-20261101/`. Seeded r4 work/Homes are removed after copying the diagnostic fixture; the dev-01 copy is removed after both exact child processes have exited. The previous r2 cleanup was left untouched until it finished; only then were seeded Markdown Homes inadvertently copied into its report directory removed, retaining its real top-level reports. No merge or deployment. ### Final cleanup verification Both 759b units stopped; quota r4 had already exited. All takeover cleanup and reproduction processes have exited. Earlier r2 cleanup completed untouched. Final free: perf-01 root 51G, split Home 58G; perf-02 root 37G, split Home 97G. Shared locks are held by #792 qualification (159697 / 1325428), not the stopped jobs. Cleanup/fixture-copy I/O overlapped the first #792 samples; those samples need overlap exclusion. Retained observations do not establish why quota-4 cache was reset. Perf-02 cleanup and fixture-copy overlap was approximately **17:13–17:22 IST on 1 November** (11:43–11:52 UTC). The earlier r2 file walk also overlapped #792 startup until it completed. #792 first overlapping samples must not be treated as uncontended measurements. No #792 process was stopped.
kayg commented 2026-11-02 03:18:56 +00:00
Author
Owner
Copy link

paired requalification blocked by rebase conflict

Current PR head: 75978ddb5ed30152a85fbb1d766a62d5c5a73113 (perf/quota-simple). Target checked: origin/dev 55f35fcde3bc91ca3781055577b93e93819f6cd4.

An isolated test branch ran git rebase origin/dev. It stopped at step 2/64, applying aef9715e (WIP fixes), with content conflicts in docs/storage.md:

  • storage root table: current dev's data/state paths and .calternal/settings.json versus the stacked userdata/appdata paths, prefs and additional task/contact state;
  • migration instructions: current dev's coordinated manual move versus the stacked storage split procedure.

These describe storage behavior and recovery procedures, so I treated them as non-trivial and aborted the rebase, as requested. No conflict resolution, source edit, PR branch push or merge was performed. Both isolated worktrees returned clean to their original heads. Further conflicts beyond this first stop have not been assessed.

qualification gate new evidence verdict
rebase onto current dev conflict above; rebase aborted blocked
bounded nonempty-first-page harness readiness known r4 failure reviewed; fix not implemented or tested pending
complete candidate/dev pairs, same VM 0/6 new pairs not qualified
typing CPU ≤2% no new samples unmeasured
idle CPU ≤0.1% no new samples unmeasured
idle RSS ≤100 MB no new samples unmeasured
idle wakeups ≤0.2/s no new samples unmeasured
cold-growth ≤10% (where applicable) no new samples unmeasured

The r4 template on perf-02 was read; its partial retained results remain historical evidence, not a completed qualification. No builds or performance jobs were started, no performance locks were taken, and no seeded Homes or large artifacts were created. Both VMs remain available. Resume after the storage conflict is resolved by the implementation owner: fix the bounded readiness precondition, build on dev-01, relay and SHA256-check binaries, then run six fresh matched pairs per comparison split across perf-01/perf-02 under ~/bin/perf-lock.

Verdict: qualification FAIL (blocked/unmeasured), not a measured product regression.

## paired requalification blocked by rebase conflict Current PR head: `75978ddb5ed30152a85fbb1d766a62d5c5a73113` (`perf/quota-simple`). Target checked: origin/dev `55f35fcde3bc91ca3781055577b93e93819f6cd4`. An isolated test branch ran `git rebase origin/dev`. It stopped at step 2/64, applying `aef9715e` (`WIP fixes`), with content conflicts in `docs/storage.md`: - storage root table: current dev's data/state paths and `.calternal/settings.json` versus the stacked userdata/appdata paths, prefs and additional task/contact state; - migration instructions: current dev's coordinated manual move versus the stacked `storage split` procedure. These describe storage behavior and recovery procedures, so I treated them as non-trivial and aborted the rebase, as requested. No conflict resolution, source edit, PR branch push or merge was performed. Both isolated worktrees returned clean to their original heads. Further conflicts beyond this first stop have not been assessed. | qualification gate | new evidence | verdict | |---|---|---| | rebase onto current dev | conflict above; rebase aborted | blocked | | bounded nonempty-first-page harness readiness | known r4 failure reviewed; fix not implemented or tested | pending | | complete candidate/dev pairs, same VM | 0/6 new pairs | not qualified | | typing CPU ≤2% | no new samples | unmeasured | | idle CPU ≤0.1% | no new samples | unmeasured | | idle RSS ≤100 MB | no new samples | unmeasured | | idle wakeups ≤0.2/s | no new samples | unmeasured | | cold-growth ≤10% (where applicable) | no new samples | unmeasured | The r4 template on perf-02 was read; its partial retained results remain historical evidence, not a completed qualification. No builds or performance jobs were started, no performance locks were taken, and no seeded Homes or large artifacts were created. Both VMs remain available. Resume after the storage conflict is resolved by the implementation owner: fix the bounded readiness precondition, build on dev-01, relay and SHA256-check binaries, then run six fresh matched pairs per comparison split across perf-01/perf-02 under `~/bin/perf-lock`. **Verdict: qualification FAIL (blocked/unmeasured), not a measured product regression.**
kayg commented 2026-11-07 18:21:40 +00:00
Author
Owner
Copy link

Housekeeping #1009 status: audited head 75978ddb5ed30152a85fbb1d766a62d5c5a73113 against dev cd7f84432feb. git cherry origin/dev <head> reports 64 unmatched commits and 0 equivalent patches. Separate stable patch-id indexing and key-file checks do not establish full landing. Absent from dev: apps/web/src/lib/upload-retry-jitter.test.mjs, apps/web/src/lib/upload-retry-review-764b.test.mjs, apps/web/src/lib/upload-retry-review-764c.test.mjs. Left unchanged as WIP, per kayg’s instruction; no rebase, close, tests, or merge. The #1009 audit records the unmatched commit list.

Housekeeping #1009 status: audited head `75978ddb5ed30152a85fbb1d766a62d5c5a73113` against dev `cd7f84432feb`. `git cherry origin/dev <head>` reports 64 unmatched commits and 0 equivalent patches. Separate stable patch-id indexing and key-file checks do not establish full landing. Absent from dev: `apps/web/src/lib/upload-retry-jitter.test.mjs`, `apps/web/src/lib/upload-retry-review-764b.test.mjs`, `apps/web/src/lib/upload-retry-review-764c.test.mjs`. Left unchanged as WIP, per kayg’s instruction; no rebase, close, tests, or merge. The #1009 audit records the unmatched commit list.
Some checks failed
CI / server (pull_request) Successful in 1m52s
Details
CI / rust (pull_request) Failing after 10m28s
Details
CI / web (pull_request) Failing after 42m18s
Details
This pull request has changes conflicting with the target branch.
  • apps/web/src/lib/backdrops.ts
  • crates/attachments/src/actions.rs
  • crates/contacts-actions/src/tests/review3.rs
  • crates/fsio/src/lib.rs
  • crates/fsio/src/swap.rs
  • crates/fsio/src/tests.rs
  • crates/fsio/src/usage.rs
  • crates/notes/src/meta.rs
  • crates/notes/src/peek.rs
  • crates/notes/src/room/actor.rs
  • crates/notes/src/scan.rs
  • crates/notes/src/search.rs
  • crates/notes/src/tests/listing.rs
  • crates/notes/src/watch.rs
  • docs/storage.md
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin perf/quota-simple:perf/quota-simple
git switch perf/quota-simple

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff perf/quota-simple
git switch perf/quota-simple
git rebase dev
git switch dev
git merge --ff-only perf/quota-simple
git switch perf/quota-simple
git rebase dev
git switch dev
git merge --no-ff perf/quota-simple
git switch dev
git merge --squash perf/quota-simple
git switch dev
git merge --ff-only perf/quota-simple
git switch dev
git merge perf/quota-simple
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!764
No description provided.