Keep storage usage correct for hard links and moved folders #754
Closed
kayg
wants to merge 3 commits from
perf/idle-honest-2 into dev
pull from: perf/idle-honest-2
merge into: kayg:dev
kayg:main
kayg:feat/style-sheet-fixes
kayg:storage/wal-checkpoint-off-save-1051
kayg:t3/paged-search-reconcile-final
kayg:feat/pass8
kayg:review/calendar-2
kayg:fix/shared-row-swipes
kayg:feat/contrast
kayg:dev
kayg:feat/collections-db
kayg:fix/instant-keys-1056
kayg:feat/money-round
kayg:perf/stalls-1054
kayg:wip/tiering-v3-cold4
kayg:wip/tiering-v3-cold5
kayg:port/rows-actions
kayg:port/rows-drop
kayg:port/hidden-blocks
kayg:port/rows
kayg:port/port-tables
kayg:port/port-tables-2
kayg:t3/mail-1011-liveness
kayg:port/devices
kayg:port/port-attachments
kayg:port/port-attachments-2
kayg:port/port-attachments-3
kayg:docs/design-consistency
kayg:docs/engineering-standards
kayg:fix/rusqlite-0.41-rustsec-2026-0459
kayg:feat/iphone-fixes
kayg:shared/restore-and-stale-918-919
kayg:fix/phone-top-band-contrast
kayg:feat/notes-approve
kayg:perf/soak-max-gate
kayg:feat/files-s1-identity
kayg:fix/lifecycle-event-id
kayg:feat/quickfix
kayg:wip/tiering-v3-cold
kayg:perf/web-mockup-frame-fixes
kayg:port/contacts-merge
kayg:port/contacts-edit
kayg:test/shared-room-no-deleted-text
kayg:perf/first-paint-credits
kayg:fix/shared-history-after-grant
kayg:fix/phone-swipe-regressions
kayg:feat/contrast-r1
kayg:review/sharing-pr3-review
kayg:feat/tiering-v3-s2-tags
kayg:feat/folder-drop
kayg:fix/deskpills-lazy-share
kayg:feat/sharing-s1-pr3
kayg:tmp/money-round-before
kayg:fix/phone-pane-backdrop-roots
kayg:fix/palette-preload-chips
kayg:fix/invite-defaults
kayg:port/tasks-matrix
kayg:fix/restore-green-gates
kayg:fix/sharing-fast-path-hardening
kayg:t3/photos-s1
kayg:feat/tiering-v3-s1-namespace
kayg:notes-archive-hide
kayg:feat/tasks-actions
kayg:t3/photos-plan-slice-one
kayg:feat/whiteboard-s1
kayg:t3/paged-search-reconcile
kayg:t3/smooth-pane-transitions
kayg:t3/index-position-startup-scan
kayg:fix/private-staging-759b-r2
kayg:feat/dav-transport
kayg:feat/imap-notes-fetch
kayg:feat/imap-notes-projection
kayg:feat/imap-notes-readonly
kayg:feat/imap-notes-auth
kayg:feat/imap-notes-server-foundation
kayg:docs/undo-block-ids-plan
kayg:fix/undo-768
kayg:proto/block-identity-v7d
kayg:issue/carddav-read-perf-fix-r2
kayg:issue/carddav-read-r2
kayg:docs/imap-766-plan-v2
kayg:docs/imap-notes-plan
kayg:docs/quota-plan-2
kayg:fix/parallel-test-flakes
kayg:feat/contacts-companion-pure
kayg:docs/calendar-backend-v2-plan
kayg:feat/calendar-ical-converter
kayg:docs/tasks-writes-v2-plan
kayg:docs/carddav-writes-v2-plan
kayg:docs/contacts-import-v2-plan
kayg:issue/610-pm-midnight
kayg:feat/calendar-v2-s1-containment
kayg:docs/dav-recorder-v2-plan
kayg:plan/previews-865
kayg:plan/save-log-durable-entry
kayg:fix/e2e-data-on-ram
kayg:feat/appwindow3
kayg:feat/appwindow2
kayg:feat/appwindow
kayg:feat/money-p1
kayg:feat/keys-sheet
kayg:review/calendar-2-themes
kayg:feat/smooth-cursor
kayg:fix/float-style
kayg:review/calendar-2-float
kayg:release/2026-11-06-hotfix
kayg:archive/port/rows-drop-code
kayg:archive/port/rows-actions
kayg:archive/port/rows
kayg:scratch/tasks-894-ui-gates
kayg:review/calendar-2-night-wip
kayg:release/2026-11-05
kayg:archive/feat/tiering-v3-s1-namespace
kayg:archive/flaky/rows-drop-fixed
kayg:archive/flaky/rows-actions
kayg:archive/feat/tiering-v3-reader
kayg:archive/feat/tiering-v3-roots
kayg:archive/flaky/rows
kayg:sim/bm15-final
kayg:sim/bm15-fix
kayg:sim/bm15-bis-932
kayg:sim/bm15-bis-909
kayg:sim/bm15
kayg:sim/bm15-bis-887
kayg:perf/tiering-option-b
kayg:port/style-3
kayg:port/style-2
kayg:port/style
kayg:fix/sw-lazy-precache
kayg:fix/vim-w-waits-for-save
kayg:fix/signout-test-races
kayg:fp-scratch/all
kayg:sim/bm15-first-paint
kayg:archive/bm15/rows-drop
kayg:archive/bm15/rows-actions
kayg:fix/palette-snippet-contrast
kayg:archive/bm15/rows
kayg:archive/bm15/hidden
kayg:archive/plan/save-log-durable-entry
kayg:mock/money-fix2
kayg:archive/fix/e2e-data-on-ram
kayg:mock/appearance-all
kayg:mock/pill-press
kayg:archive/docs/design-consistency
kayg:mock/auto-swipes
kayg:mock/mail-actions
kayg:archive/feat/mail-p1
kayg:mock/money-static
kayg:archive/review/standards-917
kayg:archive/docs/engineering-standards
kayg:perf/tiering-761-fix
kayg:mock/type-audit
kayg:mock/third-pane
kayg:feat/contacts-import-v2-s1
kayg:perf/executor-hot-path
kayg:archive/port/hidden-blocks
kayg:archive/port/devices
kayg:scratch/executor-on-894
kayg:scratch/executor-spans2
kayg:scratch/executor-spans
kayg:train/2026-11-03-bm14c
kayg:archive/port/rows-drop
kayg:feat/tiering-761-d-scale
kayg:feat/tiering-761-c-index
kayg:feat/tiering-761-b-reader
kayg:archive/review/sharing-pr3-review-9fec3d1f
kayg:train/2026-11-03-bm14b
kayg:perf/ports-lazy-budgets
kayg:feat/tiering-761-a-migration
kayg:train/2026-11-03-bm14
kayg:train/2026-11-03-bm12
kayg:archive/review/sharing-pr3-review
kayg:archive/feat/sharing-s1-pr3
kayg:train/2026-11-03-bm11
kayg:archive/feat/imap-notes-fetch
kayg:train/2026-11-03-bm10
kayg:archive/review/sharing-pr3-review-code
kayg:archive/plan/previews-865
kayg:train/2026-11-02-bm9
kayg:archive/port/port-attachments-3
kayg:archive/port/contacts-merge
kayg:fix/motion-audit-859
kayg:train/2026-11-02-bm8
kayg:archive/scratch/bm8-prep
kayg:archive/port/port-pickers
kayg:archive/port/port-tables-2
kayg:archive/port/port-tables
kayg:archive/feat/sharing-s1-pr3-code
kayg:archive/scratch/bm7-fixed
kayg:train/2026-11-02-bm7
kayg:archive/fix/parallel-test-flakes-code-minipc01
kayg:train/2026-11-02-bm6
kayg:docs/imap-766-readonly-slice
kayg:feat/dav-recorder-v3
kayg:research/788-diagnosis
kayg:archive/fix/parallel-test-flakes
kayg:archive/perf/pr792-cold-index-edit
kayg:docs/sharing-slice-1
kayg:archive/docs/imap-766-plan-v2
kayg:feat/dav-recorder-v2
kayg:archive/docs/dav-recorder-v2-plan
kayg:spike/imap-766-s0-cases
kayg:archive/fix/undo-768
kayg:feat/dav-recorder
kayg:archive/docs/contacts-import-v2-plan
kayg:archive/feat/calendar-ical-converter
kayg:feat/contacts-import
kayg:perf/cold-growth-on-764
kayg:perf/cold-growth-on-tiering
kayg:archive/docs/tasks-writes-v2-plan
kayg:archive/docs/calendar-backend-v2-plan
kayg:archive/feat/contacts-companion-pure
kayg:feat/tasks-set-status
kayg:feat/tasks-move-reorder
kayg:archive/docs/carddav-writes-v2-plan
kayg:archive/feat/calendar-v2-s1-containment
kayg:feat/tasks-delete-promote-duplicate
kayg:archive/feat/dav-transport
kayg:archive/fix/carddav-rebuild-r1
kayg:archive/fix/carddav-retention-r1
kayg:feat/carddav-writes
kayg:perf/quota-on-tiering
kayg:perf/quota-simple
kayg:perf/cold-growth-on-quota-stack
kayg:feat/calendar-identity
kayg:feat/calendar-event-parser
kayg:feat/calendar-props-maps
kayg:port/glass-when-ring
kayg:feat/tiering-761-s1
kayg:wip/10by32-01-impl-759b-scratch-20261030
kayg:wip/6by32-01-bm3-webkit-pause-20261030
kayg:wip/10by32-01-bm3-chromium-pause-20261030
kayg:wip/6by32-01-tier-761-s1-pause-20261030
kayg:spike/imap-766-s0
kayg:wip/6by32-02-calternal-pause-20261030
kayg:wip/8by64-01-review-688-pause-20261030
kayg:wip/6by32-01-verify-744-pause-20261030
kayg:wip/6by32-01-verify-742-pause-20261030
kayg:fix/private-staging-759b
kayg:wip/6by32-01-review-748-pause-20261030
kayg:wip/6by32-01-review-733c-pause-20261030
kayg:wip/6by32-01-review-733b-pause-20261030
kayg:wip/6by32-01-perf-idle-8-pause-20261030
kayg:wip/6by32-01-perf767-base-pause-20261030
kayg:wip/6by32-01-perf767b-767-pause-20261030
kayg:wip/6by32-01-perf767b-764-pause-20261030
kayg:wip/6by32-01-perf767-767-pause-20261030
kayg:wip/6by32-01-perf767-764-pause-20261030
kayg:wip/6by32-01-lists-b-pause-20261030
kayg:wip/6by32-01-ids-verify5-pause-20261030
kayg:wip/6by32-01-ids-verify4-pause-20261030
kayg:review/calendar-month
kayg:wip/10by32-02-undo-768-20261030
kayg:test/ids-verify3
kayg:wip/10by32-02-bm3-chromium-20261030
kayg:wip/10by32-01-review-753-20261030
kayg:wip/10by32-01-perf-idle-2-20261030
kayg:wip/10by32-01-bm2-webkit2-20261030
kayg:plan/759b-staging
kayg:train/2026-10-30-bm2
kayg:review/758b-tests
kayg:review/758-tests
kayg:review/764d-tests
kayg:review/767b-tests
kayg:review/preview-2026-10-29
kayg:review/767-tests
kayg:docs/tiering-731-plan
kayg:wip/c6b02-when2
kayg:wip/c6b02-tasks-create-fix5
kayg:wip/c6b02-settings-frost-mock
kayg:wip/c6b02-settings-frost-base
kayg:wip/c6b02-settings-frost
kayg:wip/c6b02-ring
kayg:wip/c6b02-review-733d
kayg:wip/c6b02-bm1-full-a
kayg:wip/compute02-when-field
kayg:wip/compute02-webkit-lists-repro
kayg:wip/compute02-train-bl2-dev
kayg:wip/compute02-dev-green-pw164
kayg:docs/deploy-plan
kayg:review/764c-tests
kayg:review/764b-tests
kayg:review/748g-tests
kayg:review/764-tests
kayg:review/759b-tests
kayg:fix/storage-owner-liveness-642
kayg:fix/storage-fix-7
kayg:wip/compute02-storage-fix-7
kayg:review/759-tests
kayg:feat/tiering-731
kayg:wip/tiering-731-first-attempt
kayg:wip/compute02-storage-fix-1
kayg:review/754g-tests
kayg:review/754f-tests
kayg:review/754e-tests
kayg:review/755-tests
kayg:review/754d-tests
kayg:review/754c-tests
kayg:review/754b-tests
kayg:review/754-tests
kayg:fix/storage-private-tmp-642-dev
kayg:wip/compute02-verify-750
kayg:test/ids-verify5
kayg:test/ids-verify4
kayg:fix/storage-exact-file-quarantine-642
kayg:wip/compute02-verify-740
kayg:perf/one-flush-save
kayg:review/calendar-static
kayg:wip/compute02-verify-737
kayg:review/indexing-ring
kayg:review/when-field-2
kayg:wip/compute02-review-733
kayg:proto/block-identity-2-review
kayg:proto/block-identity-review
kayg:wip/c6b02-tc-design
kayg:review/when-field
kayg:issue/tasks-create-ids
kayg:issue/tasks-create
kayg:review/combined
kayg:wip/compute02-review-combined
kayg:review/task-polish
kayg:review/top-buttons
kayg:review/q1-q5
kayg:train/2026-10-26-bl2
kayg:archive/wt-398h
kayg:archive/train-bm-proptest-seed
kayg:archive/integ325-293
kayg:archive/integ325-ci
kayg:archive/client-auth-review-baseline
kayg:archive/wt-sim-4853
kayg:archive/wt-sim-4811
kayg:archive/wt-448m
kayg:archive/wt-398a-allocator
kayg:issue/calendar-adr
kayg:archive/old/train/2026-10-23bk2
kayg:archive/old/train/2026-10-23bk
kayg:issue/469-mockup-9a
kayg:issue/smart-tasks-mockup
kayg:issue/469-menus
kayg:issue/652-signout-flake
kayg:issue/652-history-fix7
kayg:issue/652-history-fix6
kayg:archive/old/issue/spring-check-perf
kayg:issue/spring-check-perf
kayg:issue/659-rename-fast
kayg:archive/old/train/2026-10-21ar
kayg:issue/search-hardening-3
kayg:issue/652-history-fix5
kayg:issue/660-port-sheets-glass
kayg:issue/652-history-fix4
kayg:issue/652-history-fix3
kayg:issue/652-history-fix2
kayg:issue/652-history-flake
kayg:train/2026-10-21an
kayg:issue/settings-admin-port
kayg:issue/search-port
kayg:train/2026-10-20am
kayg:issue/storage-split-fix2
kayg:issue/storage-split
kayg:archive/old/issue/folder-ids-fix
kayg:train/2026-10-20al
kayg:archive/old/issue/folder-ids
kayg:train/2026-10-20ak
kayg:issue/tables-mockup
kayg:train/2026-10-19aj
kayg:train/2026-10-19ai
kayg:issue/469-merge-fixes
kayg:issue/sidebar-band
kayg:archive/old/issue/608-settings-spec
kayg:train/2026-10-19ah
kayg:issue/swipe-offscreen
kayg:train/2026-10-19ag
kayg:archive/old/issue/590-nlp-adr
kayg:archive/old/issue/587-event-after-write-2
kayg:train/2026-10-19af
kayg:archive/old/issue/587-event-after-write
kayg:issue/469-pill-room-c
kayg:issue/469-pill-room
kayg:train/2026-10-19ae
kayg:issue/469-narrow-handle
kayg:issue/397-ios-swipe-2
kayg:archive/old/issue/adrs-photos-files-tasks
kayg:issue/469-desk-fixes
kayg:archive/old/issue/auto-update-adr
kayg:issue/469-filter-gap
kayg:issue/469-port-1b
kayg:issue/442-opacity-flake
kayg:archive/397-ios-swipe-2
kayg:issue/327-web-fast-3
kayg:archive/old/issue/469-port-1
kayg:issue/469-port-1
kayg:issue/327-web-fast-2
kayg:archive/old/issue/327-web-fast
kayg:archive/old/issue/425-stt-bench-vq1
kayg:archive/old/issue/445-no-bleed
kayg:archive/old/issue/400-swipe-mockup-2
kayg:issue/426-client-review
kayg:issue/426-client-2
kayg:issue/426-client
kayg:archive/old/issue/476-safe-area-lab
kayg:archive/445-top-light-2
kayg:archive/old/issue/445-top-light
kayg:issue/471-blocks-mockup
kayg:archive/old/issue/448-links-app-2
kayg:archive/old/issue/448-links-app
kayg:archive/old/issue/398-cold-links
kayg:issue/434-442-small-2
kayg:issue/434-442-small
kayg:archive/old/issue/398-unload
kayg:archive/old/issue/445-bottom-band
kayg:issue/445-bottom-band
kayg:issue/351-rem-3
kayg:archive/312-views-mockup
kayg:issue/351-rem-2
kayg:issue/351-rem
kayg:archive/old/issue/327-first-note
kayg:issue/395-width-phone
kayg:archive/old/issue/327-typing
kayg:issue/396-ios-edges
kayg:issue/397-ios-swipe
kayg:archive/398-allocator
kayg:archive/old/issue/327-notes-index
kayg:issue/345-347-contrast-empty-2
kayg:archive/old/issue/44-arenas
kayg:issue/44-arenas
kayg:archive/integ325
kayg:archive/old/issue/327-perf-harness
kayg:archive/old/issue/44-thp
kayg:issue/44-thp
kayg:archive/old/issue/285-sticky-pills
kayg:archive/old/issue/44-trim-3
kayg:issue/44-trim-3
kayg:archive/old/issue/165-cards
kayg:archive/old/issue/165-picker-2
kayg:archive/old/issue/212-multi-select
No reviewers
Labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/calternal!754
Loading…
Reference in a new issue
No description provided.
Delete branch "perf/idle-honest-2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
For #686, continuing the work reviewed in #753. Writing through a hard link now updates every charged name for that inode, including writes through links outside the watched tree. Moving a newly published folder out releases its charge. Equal clock readings no longer make the deadline test drain a queued event.
Logical bytes retain the full-walk definition: each hard-link name is charged. Linux uses compact inode descriptors and shared paths; file descriptors pin the observed inode during watch installation. Healthy watchers stay event driven. Disappearing files during watch installation are transient, and shutdown remains bounded under continuous writes.
Validated head:
81e6141a2e. Formatting and workspace clippy passed. Workspace tests: 1176 passed, 0 failed, 20 ignored. All 100 counter repetitions passed (28 tests each), executing 1,000,000 operation sequences and 2,000,000 per-Home full-walk comparisons. Both ARM64 and x86_64 release builds passed.All three server gates passed. Six unchanged 300-second idle windows measured zero CPU ticks and zero wakeups:
Reports ran on 10by32-01-std, ARM64, same-root btrfs, under its performance lock with the supplied wrapper copied into the job folder. The VM has no privileged cache-drop helper, so unindexed starts were measured without controlled cache drops. Non-gated storm disk-write targets still exceed 50 MB; the full warnings are retained. Binary, harness and budget hashes stayed unchanged.
Evidence archive, SHA-256
ae082d6ca8ee71b37c69d52e5792ca3d2bb62902c016581ce6df11df9d10331d. Issue #686 contains the findings, original failing reproductions, preliminary failed report, final tables and verbatim gate output.This new branch and PR follow the job's Forgejo rule for updating an already open PR. The reviewed #753 head remains unchanged. No merge or deployment was performed.
Independent verification started. Worktree /home/agent/wt/review-754; review branch review/754-tests; PR head
81e6141a. Fetch complete. Reading the PR base and quota design before independent tests.Independent verification is partially blocked. The requested five-minute idle run must use 10by32-01-std under /usr/local/bin/perf-lock.sh.
ssh -o BatchMode=yes -o ConnectTimeout=15 10by32-01-std ...returnedssh: Could not resolve hostname 10by32-01-std: No address associated with hostname. Tailscale reportsBackendState: NeedsLogin,Health: Tailscale is stopped.The repository's older perf address, root@10.69.69.63, returnsNo route to host. No perf run has been substituted on this shared host. Local sparse, hard-link, move, concurrent replacement and crash tests pass so far. Localcp --reflink=alwaysreturnedOperation not supported; reflink behavior is therefore unverified. Continuing independent edge cases and the required code gates.P2 — Existing startup undercount: unreadable regular files are silently omitted.
Reproducer added in review/754-tests:
cargo test -j3 -p calternal-fsio independent_unreadable_regular_file_on_startup -- --nocapture. Create a 12-byte regular file under users/a, chmod it to 000, then construct a fresh Usage and call ensure(). Independent symlink_metadata reports 12 logical bytes; the fresh counter reports 0. The reconciliation walk opens every file for reading (usage.rs:250; beneath.rs:110), then treats any open failure as a possible directory and silently skips both failures. File content read access is not needed to charge its logical length. Startup returns success, so this is an undercount, not a startup refusal.Verbatim assertion on
81e6141a:Scope: confirmed PREEXISTING on actual PR base dev
d83e54dbwith a separate public-API integration test (unreadable_regular_file_is_charged, 0 versus 12). This is not attributed to the #754 diff, but it disproves the requested universal disk-usage contract. No product fix made during independent review.P2 — Existing startup undercount: non-UTF-8 filenames are silently omitted.
Reproducer added in review/754-tests:
cargo test -j3 -p calternal-fsio independent_non_utf8_filename_on_startup -- --nocapture. Create a 12-byte regular file with basename bytesname-\xff.mdunder users/a, then construct a fresh Usage and call ensure(). An independent walk preserving OsString names finds 12 logical bytes; the counter reports 0. Folder::names converts names using to_string_lossy (beneath.rs:125), so reconciliation opens a different, nonexistent filename. Startup returns success and leaves this usage uncharged.Verbatim assertion on
81e6141a:Scope: confirmed PREEXISTING on actual PR base dev
d83e54dbwith a separate public-API integration test (raw_filename_is_charged, 0 versus 12). This is not attributed to the #754 diff, but it disproves the requested universal disk-usage contract. No product fix made during independent review.Independent verification is partially blocked. The requested five-minute idle run must use 10by32-01-std under /usr/local/bin/perf-lock.sh.
ssh -o BatchMode=yes -o ConnectTimeout=15 10by32-01-std ...returnedssh: Could not resolve hostname 10by32-01-std: No address associated with hostname. Tailscale reportsBackendState: NeedsLogin,Health: Tailscale is stopped.The repository's older perf address, root@10.69.69.63, returnsNo route to host. No perf run has been substituted on this shared host. Local sparse, hard-link, move, concurrent replacement and crash tests pass so far. Localcp --reflink=alwaysreturnedOperation not supported; reflink behavior is therefore unverified. Continuing independent edge cases and the required code gates.Independent verification finished. Verdict: fix first for the requested universal accounting contract and incomplete mandatory verification. No new regression specific to #754 was demonstrated in the tested scenarios.
Reviewed head:
81e6141a2e. Actual PR base: devd83e54dbe5. Independent test branch: review/754-tests, headba30b96f1c. Only test code and its cfg(test) module declaration are committed; no evidence files are in the branch.Findings, both confirmed preexisting on the actual dev base with separate public-API reproductions:
cargo test -j3 -p calternal-fsio independent_unreadable_regular_file_on_startup -- --nocapture.name-\xff.mdyields 0 versus independent 12. Reproduce withcargo test -j3 -p calternal-fsio independent_non_utf8_filename_on_startup -- --nocapture.Six positive independent scenarios pass: inside, cross-Home and outside hard links with growth, truncation, alias detachment and removals; file and 600-file folder moves; outside edits and replacement plus offline changes followed by fresh startup; controlled SIGKILL after observation before publication and restart; sparse/large/dense/zero files and symlinks; and twelve rounds of concurrent app/outside replacement. Native steps compare both counters with an independent regular-file metadata walk, preserving raw filenames and never following symlinks. Logical length charges each hard-link name. The crash test exercises the production counter in an owned child, not the whole HTTP server.
Mandatory coverage remains incomplete: the five-minute idle strace/perf measurement is blocked by perf VM access. Bare and VPN hostnames do not resolve, Tailscale is stopped and NeedsLogin, and the requested host address 10.69.69.73 recorded in issue #686 times out over SSH. No perf workload ran on this shared local host. Reflink verification is unavailable on the local filesystem (
cp --reflink=always: Operation not supported). These are not passed gates. Source inspection alone cannot certify idle CPU or RSS.Formatting and strict workspace clippy pass. Full fsio tests: 63 pass, 2 fail (the independent preexisting undercounts), 1 ignored child helper invoked by the crash parent. All 57 original fsio tests pass. Auth quota tests: 2 pass, 1 perf-only test ignored.
Gate output follows verbatim. Commands ran sequentially under build-slot.sh with the prescribed environment and cargo -j3. Formatting produced no output and exited 0.
cargo clippy --workspace --all-targets -j3 -- -D warnings; exit 0:cargo test -j3 -p calternal-fsio; exit 101:cargo test -j3 -p calternal-auth --test quota_counter -- --nocapture; exit 0:The baseline reproductions, confirming the two findings are preexisting (dev
d83e54db), also returned exit 101:No merge, deployment or issue closure was performed.
Starting startup quota fixes on branch perf/idle-honest-2, base
81e6141a2e. First merging reviewer tests origin/review/754-tests (ba30b96f), then checking metadata-only accounting and every Folder::names caller. Following the task-specific instruction to push only this existing PR branch; no main/dev integration or deploy.Reviewer tests merged by fast-forward to
ba30b96f. Reproduced both findings with cargo test -j3 -p calternal-fsio independent_: independent_non_utf8_filename_on_startup and independent_unreadable_regular_file_on_startup both fail with left: 0, right: 12. Result: FAILED. 6 passed; 2 failed; 1 ignored; 57 filtered out. Caller audit: only quota reconciliation and attachment name allocation call Folder::names in production. Listing, sync, search and trash have no calls to this method. Scoped quota repair also converts path components with to_string_lossy and needs the same raw-name fix. Attachment allocation must avoid treating invalid bytes as replacement characters. Plan: metadata-only statat with SYMLINK_NOFOLLOW for single children of an open directory, raw OsString names throughout walks, explicit failures with a retained count and Health action rather than a partial successful snapshot.The metadata and raw-name fix passes cargo test -j3 -p calternal-fsio. Verbatim result: test result: ok. 67 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 10.59s
Both independent_unreadable_regular_file_on_startup and independent_non_utf8_filename_on_startup report ok, without changing reviewer tests. Added coverage for non-UTF-8 directory and file names (including a backslash) through startup, scoped file repair and scoped directory repair. A separate test checks that two unmeasurable subtrees produce count 2, preserve the prior snapshot, block admission readiness, and retain the failure until a successful full recount. Health is exposed through the existing account plugin as account.storage-health; no paths are returned. Attachment allocation now receives raw OsString names and folds only valid Unicode names: an invalid byte name cannot collide with a UTF-8 candidate. Full requested gates are next.
Static review found a related silent-skip path in reconciliation of concurrent writes: the merge loop removed a changed file from the snapshot and ignored symlink_metadata errors. A new regression changes parent permissions after the walk and records a concurrent refresh. Before the fix, cargo test -j3 -p calternal-fsio a_change_that_becomes_unmeasurable fails with: the merge must not silently drop an unmeasured file. Verbatim result: test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 69 filtered out; finished in 0.00s
The merge now rechecks changed files with the same fd-relative, no-follow metadata lookup and propagates measurement failures to Health. This also prevents intermediate symlinks from escaping the root during the recheck. The first auth test run exposed an incorrect new test input (unit actions take null, not {}); corrected the input, leaving the production action contract consistent with existing account actions. No reviewer tests changed.
Formatting and workspace Clippy with all targets and -D warnings both pass. The first full workspace test run found the expected action-contract mismatch for the newly added account.storage-health action. Verbatim failure: contracts/actions.json is out of date: an action's name or input changed. Verbatim result: test result: FAILED. 30 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 1.96s
Updating the generated action contract and web types using the repository generator, then running the web type check and full workspace tests again. The contract is product source, not an evidence artifact.
Finished startup quota fixes. Head:
883b8ac8ae. Branch: perf/idle-honest-2. Reviewer tests fromba30b96fare in the history and unchanged.Regular files are measured using metadata without content access. Directory entry bytes stay intact during startup and scoped repair. Metadata lookup and traversal never follow symlinks. The concurrent-write merge uses the same lookup and propagates failures.
All Folder::names callers were checked. Quota and attachment name allocation are the only production callers; listing, sync, search and trash do not call it. Attachment allocation keeps raw names and only folds valid Unicode names, since an invalid byte name cannot collide with a UTF-8 candidate.
Unmeasurable entries or subtrees are logged with a count. Failed scans retain a Health failure and cannot mark quota admission ready. The account plugin exposes the count and recovery advice through account.storage-health without disclosing paths. The generated action contract and web types are updated.
All requested gates pass. Full workspace result: 1190 passed, 0 failed, 21 ignored across 87 result groups. Ignored helper and perf tests were left unchanged. Web type checking also passes with 0 errors and 14 warnings in unchanged files. No perf workloads were run.
Gate output excerpts below are verbatim; full output is attached.
No evidence artifacts were committed. No main or dev push, merge, or deploy was performed.
Full gate output
Starting independent verification of PR #754 delta 81e6141a..883b8ac8. Test branch: review/754b-tests. Base SHA:
883b8ac8ae. I will check raw file names, quota metadata accounting, symlink and rename containment, Health reporting, and run fmt, clippy with warnings denied, and full workspace tests.P2: A scoped failure overwrites the retained global failure count
Location: crates/fsio/src/usage.rs:237-249 (new in delta).
Reproducer: cargo test -p calternal-fsio -j3 scoped_failure_must_not_forget_other_retained_unmeasured_subtrees -- --nocapture
Create two independent over-depth subtrees. A full startup scan fails and retains count 2. Retry only the first subtree with Usage::repair; it fails and replaces the Health count with 1 although both subtrees are still unmeasured. Expected retained count: 2. Actual: 1. The new account.storage-health action reads this same value. Keep failure scopes until a full successful recount, or retain the global count when a scoped retry fails.
Evidence, verbatim:
assertion
left == rightfailed: a scoped retry cannot erase the other inaccessible subtreeleft: 1
right: 2
P2: The metadata walker follows a directory descriptor after its parent leaves the Home
Location: crates/fsio/src/usage/scan.rs:38-39 and crates/fsio/src/beneath.rs:89-103.
Reproducer: cargo test -p calternal-fsio -j3 opened_parent_moved_outside_home_must_not_be_walked -- --nocapture
Open Home/parent with the walker's open_child, pause at the boundary before recursive Snapshot::walk, rename parent outside the Home, create a replacement Home/parent, and create foreign only in the moved outside directory. Walking the already-open descriptor measures foreign and stores it under Home/parent/foreign. It returns success with no measurement failure. O_NOFOLLOW prevents symlink traversal but does not enforce current ancestry of an already-open directory. This violates the explicit renamed-parent containment contract. The descriptor-lifetime weakness already existed before this delta; the rewritten scan still does not meet the required contract. This test deterministically exercises the same syscall boundary without adding production hooks.
Evidence, verbatim:
walk measured a new outside file through a renamed parent descriptor
Independent verification finished: fix first.
Reviewed only delta
81e6141a..883b8ac8. Tests-only branch: review/754b-tests. Head SHA:db5b5276259f6401c881bdbd4d9e8370dc6d8b71. No product code changes, merges, deployments, or evidence committed.Two P2 findings with deterministic reproducers are in the earlier comments:
Eight independent tests were added. Six pass: raw directory listings and quota distinguish two invalid byte names from their real replacement-character spelling; an unreadable one-TiB sparse regular file is charged using metadata; symlinks and a socket cause no charge or false Health failure; retained Health clears after a full successful recount; attachment uploads preserve raw files; UTF-8 case and normalization collision behavior is preserved. Two tests fail to reproduce the findings and are intentionally left enabled.
Caller audit: the only production Folder::names consumers are quota and attachment placement. Listing, sync, search, and trash have no changed callers in this delta and use separate walkers; this review does not claim their pre-existing walkers are fixed.
Gates: fmt passed (exit 0), clippy with warnings denied passed (exit 0), full workspace tests completed with --no-fail-fast (exit 101): 1,196 passed, 2 failed, 21 ignored across 87 target summaries. Only the two new reproducers failed. A final targeted rerun reproduces both failures. The final lint-only change spells a zero permission mask as octal; it does not change test behavior. All commands used one local build slot and cargo -j3 where applicable.
Full verbatim gate output, with commands and exit codes. Formatting emitted no output. Final clippy output, verbatim:
Full workspace failure summary, verbatim:
Local reports:
~/codex-jobs/verify-754b/findings.mdand~/codex-jobs/verify-754b/result.md.Starting the two P2 fixes on branch perf/idle-honest-2, base SHA
883b8ac8ae. I will merge the approved verifier tests from origin/review/754b-tests first, preserve the existing tests, fix retained measurement failures and descriptor ancestry checks, and run all requested gates. No deploy or perf workloads.Reproduced both findings after merging
db5b5276. Command: cargo test -j3 -p calternal-fsio review_754b -- --nocaptureVerbatim evidence:
The fix will retain exact failing paths until a full successful recount and validate each open child against its parent name through the root descriptor before accepting a walk. Production open_child callers are confined to crates/fsio/src/usage/scan.rs (recursive walks and scoped path traversal). Listing and search discovery use path-based traversal outside fsio, and trash uses read_dir; none retains open_child descriptors across recursion, so this particular descriptor lifetime fix has no other walker call sites.
Both fixes now pass the original verifier tests without changing them. Retained Health failures are a set of failing paths, so repeated retries deduplicate and independent failures accumulate; only a full successful recount clears them. The walker checks each open child against its name from the parent fd by device and inode, following the recorded chain to the root descriptor. It checks again at the end of enumeration; any mismatch or missing directory fails the snapshot before publication. Scoped file repair and concurrent-write rechecks validate the same ancestry. No periodic walk was added to the native idle path.
Added deterministic tests cover a rename after the last metadata read, a moved ancestor with an unchanged child descriptor, deletion of an open directory, repeated/disjoint failures, and retention across a failed full recount. Other walker audit: notes listing/search discovery use crates/notes/src/watch.rs::walk via crates/notes/src/scan.rs::stat_all, which store paths and reopen read_dir at each level; trash enumeration in crates/notes/src/trash.rs and trash/links.rs uses read_dir. No other production open_child callers exist in the repository, so there is no other recursive open_child walker to patch.
Verbatim fsio result:
Workspace formatting passed. Workspace Clippy and full workspace tests remain to complete.
The first full workspace gate exposed a compatibility regression in the new ancestry guard. directory_move_during_initial_recount_must_track_later_deletes expects recount to recover from an ordinary in-Home directory move. The guard correctly refused the stale snapshot but returned an error immediately. Verbatim evidence:
I am adding at most two fresh retries of the requested scope on a directory-location change. Each retry drops the invalid snapshot completely. Persistent movement still becomes an unmeasured failure, and permission/depth failures are not retried. This adds no periodic idle work. Existing tests remain unchanged.
Finished both P2 fixes on perf/idle-honest-2.
Head SHA:
8675015f64Commit:
8675015f64The remote branch was verified at that SHA after pushing. The required verifier branch was merged by fast-forward at
db5b5276; neither verifier test set was changed.Health now retains the union of failing paths across scoped retries and failed full recounts, deduplicates repeated failures, and clears them only after a full successful recount. Directory walks compare open directories with their parent names by device and inode through the recorded root chain before and after enumeration. A rename discards the snapshot and retries the requested scope at most twice; persistent movement remains unmeasured. Scoped file repairs use the same checks. No periodic work was added to the native idle path.
The walker audit found no other production open_child callers outside quota scanning. Listing and search discovery use path-based traversal in notes/watch.rs, and trash uses read_dir; they do not retain open_child descriptors across recursion.
All gates passed. fsio: 79 passed, 0 failed, 1 existing ignored test. Full workspace: 1202 passed, 0 failed, 21 existing ignored tests across 87 test groups. The existing auth directory-move regression also passes after adding the bounded retry.
Gate runner output verbatim:
Complete stdout and stderr for all final gates, plus the targeted auth regression, are attached verbatim: 754-quota-gates-8675015f.txt.
Independent verification started. Branch: review/754c-tests. Base SHA:
8675015f64. Reviewing only delta 883b8ac8..8675015f and called code; adding independent tests and running fmt, clippy with warnings denied, and full workspace tests.Independent verification of 883b8ac8..8675015f
P1: Replacing users during traversal accepts new outside files
Location: crates/fsio/src/usage/scan.rs:104; crates/fsio/src/beneath.rs:115-143.
The scan opens data/users with Folder::open. That Folder has parent=None, so verify_location stops without checking whether data/users still names the open directory. All descendants compare against that detached descriptor and pass. This is not a trusted data-root move: data stays in place, only its users child is replaced. watch.rs explicitly expects a missing or replaced users tree to be noticed.
Deterministic reproducer: review_754c_replaced_users_root_cannot_charge_new_outside_files. Open users, begin walking its previously enumerated child a, rename users to data/outside, create replacement users/a, and create outside/a/foreign only after the rename. The real recursive scanner returns Ok and includes users/a/foreign. No scheduler timing or sleeps are used.
Observed output:
accepted new outside bytes under the replacement Home: result=Ok(()), files=["/tmp/review-754c-2029040-5/users/a/foreign"]
test result: FAILED. 6 passed; 1 failed; 0 ignored; 0 measured; 80 filtered out; finished in 0.11s
Reproduce: source /etc/calternal/env.sh; /usr/local/bin/build-slot.sh cargo test -p calternal-fsio -j3 review_754c_replaced_users_root_cannot_charge_new_outside_files -- --nocapture
Required behavior: anchor users to the stable data directory and verify that edge too, then retry or retain an unmeasured scope when it moves. Test-only branch leaves product implementation unchanged.
Independent verification finished. Verdict: fix first.
Reviewed delta 883b8ac8..8675015f and its called code. Test-only branch: review/754c-tests. Head SHA:
ee5a225dbb.P1 remaining containment gap: replacing data/users while a scan holds its descriptor lets the recursive walk accept files created afterward in the moved outside tree. The new dev+ino checks stop at the unverified users descriptor. Keep data stable, rename users to outside, create replacement users/a, then create outside/a/foreign. The deterministic review_754c_replaced_users_root_cannot_charge_new_outside_files test returns Ok and observes that outside file under users/a/foreign. Anchor users to the stable data directory, check that edge, and retry or retain an unmeasured scope. This is a remaining gap in the fix, not a claim that the delta introduced it.
The other six independent tests pass: rename at the Home root and each descendant depth, rename back in, swapping two folders, deletion after the final stat, repeated mixed scoped repairs retaining the exact failure union, full recount clearing all failures, twenty concurrent ordinary-edit rounds without false Health failures, and unreadable sparse raw-byte names with symlinks excluded.
Other walkers: production Folder::open_child recursion exists only in quota scanning. Listing, trash and search import enumerate names and reopen paths; they do not recurse through retained child directory descriptors. This documents why this specific descriptor-containment check does not apply to them. Healthy native idle watching blocks without a recount timer. The preexisting degraded-mode fallback still does full recounts every 60 seconds; this delta adds no timer.
Gate status output verbatim (the initial fixture lint warning was fixed; final Clippy was rerun):
Full stdout/stderr from the gates is attached verbatim. Formatting emitted zero bytes. Findings and local raw logs are in /home/agent/codex-jobs/verify-754c. Only tests and test registrations were pushed. No merges or deployments.
Gate output files: clippy-initial.log, clippy.log, workspace-tests.log, targeted.log.
Started quota containment work on perf/idle-honest-2, base SHA
8675015f. First merging origin/review/754c-tests (ee5a225d). I will anchor scans at the data root, validate every recorded directory chain after traversal, add deterministic randomized coverage, and run all requested gates. The final task instruction explicitly authorizes pushing this existing branch only; I will follow that instruction.Finding: scan::read_scope opened data/users with Folder::open, so its parent chain stopped before the users edge. Also, walk_names checked each directory only when its recursion finished; a later sibling could move an already measured directory before acceptance. Evidence: scan.rs read_scope and walk_names, beneath.rs verify_location, and the newly merged review_754c_replaced_users_root_cannot_charge_new_outside_files test. The fix anchors at the configured data-root fd and validates all retained directory chains after the complete scope walk.
Containment evidence: the new directory_moved_after_its_recursion_returns_rejects_the_whole_scope test rejects a directory moved after recursion has already returned; repeated_users_replacement_exhausts_three_attempts_without_charging verifies exactly three attempts, no published files or bytes, and one retained Health scope. The seeded property test passed all 2,000 cases, covering random trees and renames, replacements, swaps, deletions, and moves back at users, Home, and nested levels. Every accepted snapshot is checked against an independent final-tree oracle. cargo test -j3 -p calternal-fsio: test result: ok. 89 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 12.87s. All reviewer tests remain unchanged. Formatting and workspace clippy with -D warnings also passed; workspace tests are running.
Gate finding: the first full workspace run failed in the unchanged Tasks crash test tests::kill_9_during_field_writes_loses_nothing_acknowledged (crates/tasks/src/tests.rs:809): ACK 1: due Text("2026-11-04"), wanted one of ["2026-11-02", "2026-11-03"]. Its result was 27 passed; 1 failed. Formatting, clippy, the fsio gate, and the workspace fsio tests passed. I am inspecting and rerunning the crash test before repeating the full workspace gate; no reviewer or existing test will be weakened.
Crash-test follow-up: the unchanged targeted test passed: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 27 filtered out; finished in 0.63s. Source evidence: kill_at reads a stdout marker then calls child.kill(), while crash_child continues updates without a handshake (crates/tasks/src/tests.rs:723-783). Advancing past the allowed next write before the parent is scheduled is consistent with the observed failure. This is a scheduling-race inference, not a proven product regression. The complete workspace retry is running with cargo test --workspace -j3 -- --test-threads=3 to reduce contention. No existing test was changed, skipped, or weakened.
Verbatim gate output for head
967dd3e892. Command: cargo fmt --all --check. Part 1 of 1.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo clippy --workspace --all-targets -j3 -- -D warnings. Part 1 of 1.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test -j3 -p calternal-fsio. Part 1 of 1.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test --workspace -j3 (first run, exit 101). Part 1 of 3.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test --workspace -j3 (first run, exit 101). Part 2 of 3.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test --workspace -j3 (first run, exit 101). Part 3 of 3.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test -j3 -p calternal-tasks tests::kill_9_during_field_writes_loses_nothing_acknowledged -- --exact. Part 1 of 1.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test --workspace -j3 -- --test-threads=3. Part 1 of 3.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test --workspace -j3 -- --test-threads=3. Part 2 of 3.Gate output
Verbatim gate output for head
967dd3e892. Command: cargo test --workspace -j3 -- --test-threads=3. Part 3 of 3.Gate output
Finished quota containment at head
967dd3e892, pushed to perf/idle-honest-2. The PR head was checked through the repo API and matches this SHA. Reviewer branch origin/review/754c-tests (ee5a225d) was fast-forwarded first. All reviewer tests remain unchanged.The scan opens the configured data root as its trusted descriptor and opens users and every Home below it. Each measured directory saves its name and (dev, ino) chain. Before accepting the complete scope, the central location verifier resolves every saved chain name-by-name from the data-root fd and requires the same identities. This also catches directories moved after their recursion has returned. Rejected snapshots publish no bytes. The existing three-attempt retry and retained Health union remain in place. Saved chains share the root descriptor without holding one descriptor per measured directory. The central code comment documents the rule. The idle path and watch fallback are unchanged.
The new seeded property test passed all 2,000 random trees and traversal pauses, covering renames, replacements, swaps, deletions, and moves back at users, Home, and nested levels. Accepted file snapshots match an independent final-tree oracle. Separate tests cover a previously measured directory moved later and exactly three users replacements leaving Health unmeasured with no published bytes.
Final formatting, clippy, fsio, and complete workspace gates passed. The workspace retry ran all tests with three test threads: 1,212 passed, zero failed, 21 existing ignored tests. The fsio gate had 89 passed, zero failed, and one existing ignored crash helper. The initial workspace run failed in the unchanged Tasks crash test; its targeted retry and the complete workspace retry passed. No test was changed or skipped to resolve that failure.
Gate driver output verbatim:
Full command output is preserved verbatim in these comments:
Only product code and tests were committed. No evidence or artifacts were committed. No PR merge, deployment, or perf workload was run.
Starting independent verification of PR #754. Branch: review/754d-tests. Base and reviewed head:
967dd3e892. Reviewing containment, bounded retries, property coverage, retained reviewer tests and the Tasks crash, followed by fmt, clippy and full workspace tests.P3: the whole-Folder ancestry contract still has two bypasses outside the quota recursion path. In crates/fsio/src/beneath.rs, open_sub returns parent=None both for existing subfolders and after creation (lines 82 and 105 at the reviewed head). location() consequently saves an empty chain, and verify_location() accepts it after users or Home has moved outside the anchor. The quota scanner uses open_child, so this is not a demonstrated bypass in its recursive path. It does violate the explicit rule that only the trusted root may have parent=None. Reproducer added: review_754d_open_sub_must_keep_its_ancestry, which opens users/a/nested from a trusted data Folder, replaces users, and asks the nested Folder to verify its location. Static result: the current verifier returns Ok for an empty chain. Test execution is pending behind the workspace build.
P2: the three-attempt quota-scan limit does not bound reconciliation. crates/fsio/src/usage.rs:272 restarts the entire walk whenever directory_generation or the observed generation changes, with no attempt or time limit. A directory-move notification increments directory_generation before blocking on the reconciliation mutex, so another thread can force a fresh full walk on every iteration.
Deterministic reproducer: review_754d_outer_reconciliation_must_bound_restarts performs eight empty-directory moves out and back at traversal entry, reports each through the real crate::usage::moved notification on a separate thread, and waits until its generation increment is visible. Every individual snapshot passes containment. The outer recount performed nine full walks and returned Ok; continued notifications can keep startup/admission from finishing. This gap predates the reviewed delta, but remains in the whole design and violates the requested bounded-retry behavior. No timers or fixed sleeps are used.
Verbatim test evidence:
The open_sub reproducer also failed as predicted:
P3: mounts are currently in scope. Private user/mount namespace checks confirmed that both a same-device bind mount of a directory outside users and a new tmpfs mounted below Home are traversed and charged. Each contributed 12 bytes and Health remained clear. There is no NO_XDEV resolution flag or mount-identity check in open_child or Location::verify. Checking only st_dev would still permit same-device bind mounts.
This is a containment-policy limit, not a violation of the literal saved-chain rule: the mounted directory is reachable by the expected name below the trusted root and its saved identity still matches. Ordinary directory writers cannot create mounts without mount privileges in the server namespace. I am recording the present behavior in passing observation tests rather than imposing an unstated no-mount policy. If backing-storage confinement is required, reject mount transitions centrally, including bind mounts; otherwise document that mounted content is quota content.
The initial adversarial tests requested mount rejection and failed with:
The final tests are named review_754d_bind_mount_is_currently_in_scope and review_754d_tmpfs_mount_is_currently_in_scope and assert those observed results. All mounts are created in disposable private namespaces and unmounted before cleanup.
The 20,000 additional seeded topology cases passed, covering seeds 2001 through 12000 and 10000 seeds beginning at 0xd754000000000001. The initial inode-reuse probe in /tmp did not reuse the directory inode within 10000 allocations; a workspace-filesystem probe is queued because inode allocation differs between filesystems.
P3: the idle guarantee is conditional, and fallback is sticky. watch.rs:74-76 enables periodic recounts even after an initial measurement failure with healthy native watchers. run also sets polling=true on any later apply/sync failure, and never sets it back to false. After the underlying problem disappears and successful repairs resume, the same instance continues doing a full walk every 60 seconds until restarted. Reproducer lifecycle: force the initial three scans to fail under churn, allow the tree to settle, then keep the process alive for at least one POLL deadline. Static evidence is the one-way polling assignments and apply(timeout) calling Usage::recount. No elapsed idle/perf run is claimed. Healthy instances without any failure use blocking recv and have no periodic walk. This behavior predates the reviewed delta.
P2: incremental charging still bypasses the central no-follow containment rule. Usage::refresh in crates/fsio/src/usage.rs uses std::fs::symlink_metadata on the complete path; this does not follow the final symlink but does follow symlinks in parent components. accounted_path returns a cached path immediately if its parent appears in state.directories, without revalidating that directory's identity. Replacing a previously scanned parent with a symlink to an outside directory therefore lets refresh publish the outside file's size with no scan or Health failure.
Reproducer added: review_754d_refresh_must_not_follow_a_replaced_parent_symlink. Recount users/a/child/file containing 3 bytes, create an outside file of 4096 bytes, move the measured child out, replace its original name with a symlink to the outside directory, and call the real incremental refresh on the old charged path. The test requires that the 4096 outside bytes are not published. Execution is pending in the final workspace gate. This path predates the reviewed delta, but remains part of the requested whole-design audit; fixing only scan::read does not enforce one charging invariant in one place.
P3: the unchanged Tasks crash test is demonstrably flaky on the pre-PR dev baseline, not evidence of quota-scan data loss.
head: 50 runs; 48 passed; 2 failed.
crash-head-22:
thread 'tests::kill_9_during_field_writes_loses_nothing_acknowledged' (2183577) panicked at crates/tasks/src/tests.rs:809:9:
SENT 41: due Text("2026-12-13"), wanted one of ["2026-12-11", "2026-12-12"]
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 27 filtered out; finished in 0.72s
crash-head-37:
thread 'tests::kill_9_during_field_writes_loses_nothing_acknowledged' (2185874) panicked at crates/tasks/src/tests.rs:809:9:
ACK 7: due Text("2026-11-10"), wanted one of ["2026-11-08", "2026-11-09"]
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 27 filtered out; finished in 0.27s
dev: 50 runs; 49 passed; 1 failed.
crash-dev-31:
thread 'tests::kill_9_during_field_writes_loses_nothing_acknowledged' (2193828) panicked at crates/tasks/src/tests.rs:809:9:
SENT 3: due Text("2026-11-05"), wanted one of ["2026-11-03", "2026-11-04"]
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 27 filtered out; finished in 0.16s
Each run used cargo test -j3 -p calternal-tasks --lib tests::kill_9_during_field_writes_loses_nothing_acknowledged -- --exact --nocapture, sequentially in one local build slot. Dev was a detached worktree at
d83e54db. git diffd83e54db967dd3e8-- crates/tasks crates/notes/src is empty. No test, assertion or scheduler was modified for these 100 runs.The child writes a SENT or ACK line and immediately continues updating; the parent reads from a pipe and then sends SIGKILL. Scheduling can allow multiple later commits before that signal. Every observed failure was a later due date, not an older acknowledged value or corrupted data. The assertion incorrectly limits survival to at most one additional update. A robust crash test must accept any later valid committed value (or use an explicit handshake when testing an exact boundary) while still rejecting loss of acknowledged state.
P2: saved directory identities are vulnerable to inode reuse after recursion returns. A Location retains the root descriptor and only numeric dev/ino values for the rest of the chain. Once a child Folder is dropped, deleting that child releases its inode. Recreating the same name can reuse the same inode, so final verification treats a different directory as the one that was walked and accepts stale files from its predecessor.
The final workspace run reproduced review_754d_reused_directory_inode_must_not_certify_a_replacement as FAILED. It measures users/a/child/old, waits until recursion has returned to the users exit checkpoint, removes child, allocates replacement directories until the original inode is reused, and creates child/replacement. read_scope accepts the old snapshot instead of rejecting it. This uses the workspace filesystem, where a separate allocation probe reused the deleted directory inode on the very first allocation. The earlier /tmp probe did not reuse an inode within 10000 allocations; filesystem-dependent behavior is explicit in the bounded test.
The chain must identify the original directory lifetime, not merely a reused inode number. Retaining a reference that prevents reuse, or an available generation-safe identity, is needed for that stronger guarantee. Rechecking dev/ino alone cannot distinguish the replacement. This is a residual gap in the requested post-walk containment design; it does not imply that the root descriptor itself was replaced.
The same workspace run also reproduced review_754d_refresh_must_not_follow_a_replaced_parent_symlink as FAILED, confirming the independently reported incremental-charging bypass. Complete failure output will be posted with the gate logs.
Independent verification: verbatim fmt-final gate output (1/1). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo fmt --all --checkExit code: 0.
The command emitted no output.
Independent verification: verbatim clippy-final gate output (1/1). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo clippy --workspace --all-targets -j3 -- -D warningsExit code: 0.
Independent verification: verbatim workspace gate output (1/4). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo test --workspace -j3 --no-fail-fast -- --test-threads=3Exit code: 101.
Independent verification: verbatim workspace gate output (2/4). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo test --workspace -j3 --no-fail-fast -- --test-threads=3Exit code: 101.
Independent verification: verbatim workspace gate output (3/4). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo test --workspace -j3 --no-fail-fast -- --test-threads=3Exit code: 101.
Independent verification: verbatim workspace gate output (4/4). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo test --workspace -j3 --no-fail-fast -- --test-threads=3Exit code: 101.
Independent verification: verbatim targeted-final gate output (1/1). Tests branch: review/754d-tests, final head
1494bd7a. Reviewed product head967dd3e8.Command:
cargo test -j3 -p calternal-fsio review_754d --no-fail-fast -- --skip review_754d_twenty_thousand_additional_topologies --test-threads=1 --nocaptureExit code: 101.
Independent verification finished: fix first.
Reviewed product head:
967dd3e892, focusing on 8675015f..967dd3e8. The PR API still reports that head. Tests-only branch: review/754d-tests, pushed head1494bd7ae9. The working tree is clean.Three P2 findings have deterministic enabled reproducers:
The fourth failing reproducer is a P3 whole-Folder contract gap: open_sub still returns parent=None in both paths, so its empty chain verifies successfully after users is replaced. The quota recursion uses open_child, so this is outside that specific recursive path. Reproducer: review_754d_open_sub_must_keep_its_ancestry.
The original 2000 seeded cases and 20000 additional cases all passed. The original property test really mutates the filesystem at seeded directory-entry/exit pauses and compares accepted snapshots to an independent final-tree oracle. Its two checkpoints per directory do not cover file-stat, final-verifier or publication interleavings, repeated churn, or inode reuse. All earlier reviewer tests from review/754-tests, 754b and 754c are retained unchanged; the original bytes are preserved where later tests were appended.
Move out and back is acceptable when bytes return inside Home before acceptance; the independent returned-directory test passes. Sequential identity verification cannot prove continuous confinement or one atomic instant across every chain. Bind mounts, tmpfs mounts, and a bind mount introduced during traversal are charged as currently in scope; no mount-transition guard exists. Mounted content satisfies the literal name-chain rule, so these tests record that policy rather than invent a no-mount requirement. A saved mount identity becomes invalid after unmount. Overlay-specific mount creation was not separately exercised. Healthy native idle handling blocks on events; any measurement or backend failure permanently enables the existing 60-second full-walk fallback for that instance, even after recovery. This is a P3 limit on the absolute idle claim; no timed idle/perf run is claimed.
The Tasks test is tests::kill_9_during_field_writes_loses_nothing_acknowledged. It passed 48/50 runs on the PR head and 49/50 on dev
d83e54db, with no test or scheduling changes. The dev failure was SENT 3 observing 2026-11-05 while permitting only 2026-11-03 or 2026-11-04. The child keeps committing after its marker while the parent is scheduled to read and send SIGKILL; the test incorrectly permits only one extra update. All failures showed later committed values, not lost acknowledged values. Tasks and notes sources are identical between dev and this head. This proves the failure mode predates the PR; these small samples do not establish equal failure rates. The Tasks test passed in the full workspace run. Detailed evidence: #754 (comment)Final gates: fmt exit 0; clippy with warnings denied exit 0; complete workspace tests exit 101, with 1217 passed, 4 failed and 21 existing ignored across 88 top-level target summaries. Only the four new reproducers failed. The initial clippy run found two fixture writes in my new integration tests; those now use write_atomic, and the final gate passes. A final targeted rerun reproduced all four findings, while mount and returned-directory checks passed. No existing test was changed or disabled to resolve a failure.
Final clippy completion, verbatim:
Workspace fsio result and final failure output, verbatim:
Full gate output is posted verbatim, with command and exit code:
Local findings and result: ~/codex-jobs/verify-754d/findings.md and result.md. Only test code was pushed. No evidence artifacts were committed; no merge, deployment or issue closure was performed.
Starting the kernel-enforced quota containment rebuild on perf/idle-honest-2. Original base
967dd3e892; merged the approved reviewer tests by fast-forward to1494bd7a. I will retain all reviewer tests, use kernel-relative resolution, run the required gates, and push only this branch. No deploys or perf workloads.Implementation findings: (1) refresh and watcher classification used full-path symlink_metadata and cached canonical parents; the new path removes canonicalization and uses the same openat2 resolver as scans and inode-watch setup. (2) Snapshot directory identity must remain live through publication, not just through enumeration; the rebuild retains generation-bearing handles or pinned directory fds and revalidates immediately before commit. (3) The 754d mount tests explicitly asserted that mounted foreign bytes were charged (12 bytes). I retained these tests and strengthened their policy assertions to require zero mounted bytes and Health count/detail, including a mount introduced during enumeration. The returned-directory test now requires rejection while the directory is outside Home and an exact fresh snapshot after return. Initial targeted execution exposed acceptance/error-handling gaps; those are being fixed before gates. No reviewer test was removed.
Finished idle measurement of PR #754 head
883b8ac8ae, origin/perf/idle-honest-2. The source worktree on perf-test-02 is unchanged. The release build ran on 10by32-01-std (aarch64) through build-slot.sh with cargo -j3; measurements ran there in perf.slice through the previous perf-run.sh and current perf-lock.sh, holding all three VM slots. Homes and state used the same-root btrfs filesystem, as in the previous job.All twelve idle windows pass CPU <= 0.1% and RSS <= 100 MB. Every window recorded zero CPU ticks and zero context-switch wakeups. Maximum sampled idle RSS was 27.89453125 MiB at 10k notes and 90.3203125 MiB at 100k notes (94707712 bytes, also below decimal 100 MB).
Six 300-second windows per size, twelve total. Each window used a new indexed server process, the previous Home and search quiet criteria, and the previous 65-second room-close wait. CPU is process user plus system ticks divided by CLK_TCK=100 and elapsed seconds, as a percentage of one CPU. Wakeups use the previous per-thread voluntary plus involuntary context-switch deltas; they are a proxy for wakeups. RSS max is sampled once per second throughout idle, including both boundaries. RSS MB follows the previous harness convention (KiB / 1024, MiB). Process lifetime high-water RSS is retained separately in raw data and is not substituted for idle RSS. Allocator setting: glibc.malloc.arena_max=2. This dedicated idle run omits the earlier full workload and typing scenarios, so its memory values are for settled read-only Homes.
All raw idle numbers, without rounding:
Startup comparisons use three fresh unindexed starts per size and three indexed starts per size. For the head, the indexed samples are the first three of the six idle-process starts; all six startup samples remain in the raw data. The preserved
81e6141arelease binary matches the previous evidence hash bc36c66b5a7703ff611031ebac472c3c36f1dd2c3386619c16e903dd6b289d1c. Both versions were measured with this run's same HTTP client and fixtures; baseline ran before the indexed head samples. Startup measures process spawn to the first successful /healthz response, polling at 1 ms as in the previous method. The Python client differs from the previous Rust client, so use the same-client table for this comparison.The samples are too few and their cache state too uncontrolled to establish a startup regression or improvement.
Cache drops are UNAVAILABLE: /usr/local/sbin/perf-drop-caches is absent and sudo -n reports that a password is required. These are fresh-process, unindexed starts with uncontrolled page cache, not controlled cold-cache results. The requested controlled cold-cache measurement could not be completed on this VM. No failed repetition was replaced.
883b8ac881e6141a883b8ac881e6141a883b8ac881e6141a883b8ac881e6141aHistorical
81e6141anumbers from issue #686 comment 68678's original reports, quoted verbatim for context. The previous harness called unindexed healthz time cold.startup and indexed healthz time startup.healthz:Gate output verbatim:
Measurement failures retained: 0. Fixture Homes and state were deleted. No code changes, commits, pushes, merges, deployments, or issue closure.
Raw evidence archive, SHA-256 e1fa13216b69d40a1a468436f6207cdf4b538c2567d92f6e857c755971d57c1d. Upload download verified. The archive includes every raw sample, server and build logs, harness and wrapper, hashes, cleanup event, and gate output. No artifacts were committed.
Additional acceptance finding: hard-link alias metadata must be observed before publishing a reconciliation. Late alias reads could fail after the candidate map had begun replacing state, and a successful full recount would then clear that failure. Reconciliation now gathers affected aliases and overlapping writes into the candidate snapshot, retains their directory identities, and validates the combined snapshot before replacing any state. The scope uses one three-attempt/30-second budget, including timed acquisition of its reconciliation gate and directory enumeration. Targeted tests passed 98 tests with zero failures before the final gate run; private namespace tests actually ran 32 seeded bind/tmpfs cases on this host. Missing-openat2, fd-budget exhaustion, expired deadlines and native-watch recovery have explicit regression tests.
The 50,000-case containment gate passed once; full gate elapsed time was 186.11 seconds. The complete fsio gate passed 102 unit tests and both reviewer mount integration tests, with zero failures. During final lifetime review of the recovery refactor I found a wake-sender ownership problem: retaining the sender in the event worker could keep its receiver connected after Usage was dropped. The sender now lives in Usage between batches; the worker clones it only while processing a batch. A new test waits for an initialized idle worker, drops Usage, and requires the worker to exit. Affected gates will be rerun before commit. Full workspace tests are running.
The first workspace gate caught a Health message compatibility regression in crates/auth/tests/quota_counter.rs:164: health_reports_retained_storage_measurement_failures requires the retained-error detail to contain "permissions". I restored the existing permission/recount guidance for measurement failures and append the mount count only when mounts are actually excluded. No test was weakened. The final gate run includes that auth integration target, the idle-worker shutdown fix, fmt, workspace clippy, the full fsio suite and the full workspace suite. The 50,000-case property test already passed once in 184.880933382 seconds (186.11 seconds including command overhead).
The workspace reliability test counter_concurrent_mutations_drift_and_restart exposed normal file callbacks exhausting the new three-attempt bound (crates/auth/tests/reliability_review.rs:278). Ordinary file changes now merge into the candidate snapshot under the counter lock; only directory topology changes restart full walks. Incremental refresh observes metadata and publishes under the same lock, so unrelated concurrent callbacks do not consume its retries. The full walk still runs outside the counter lock, and the merge retains kernel resolution, directory identities and the shared deadline. Reviewer churn assertions remain unchanged. I am testing the concurrency target and will rerun the final gates against this change, including the 50,000-case property gate.
Completed on perf/idle-honest-2. Head:
e8a99a6fee.Every quota lookup now goes through kernel resolution under its Home. Symlinks, magic links and mount crossings are refused. Metadata comes from the opened descriptor, with mount IDs checked as well. Missing kernel resolution or mount-ID support leaves the scope unmeasured. Skipped mounts appear in Health as a count.
Snapshots compare file handles that include inode generations. Filesystems without usable handles retain directory descriptors through publication, with a bounded descriptor budget. Reconciliation shares three attempts and 30 seconds, including lock waits and enumeration. File updates merge before publication; only topology changes restart a walk. Successful recounts with restored watches turn fallback polling off. Folders keep their trusted root and relative name; ancestry chains and cached canonical parents were removed.
All reviewer tests remain. The mount assertions now require exclusion and Health reporting. Added coverage includes replaced symlink parents, directory recreation, descriptor-budget exhaustion, missing kernel support, continuous churn, deadlines, watch recovery and worker shutdown. Private namespace tests ran 32 seeded bind/tmpfs topologies. The bounded directory allocation probe reached inode reuse and rejected the stale snapshot. The Tasks crash test passed without a flake retry. The final 50,000-case property test passed in 145.486164816s; its command took 145.63 seconds.
All required gates passed. Full output is attached verbatim. No perf workloads or deployments ran. Only this branch was pushed.
fmt
clippy
auth-concurrency
auth-health
fsio
property-50000
workspace
inode-probe
fmt full output
clippy full output
auth-concurrency full output
auth-health full output
fsio full output
property-50000 full output
workspace full output
inode-probe full output
Starting cache-dropped idle and startup measurement on perf-test-02. Worktree /home/agent/wt/idle-754b is clean at origin/perf/idle-honest-2, head/base
e8a99a6fee. Comparison revision: devd83e54db. No code changes or pushes. Using the previous method from comment 68858, perf-run.sh and its perf-lock.sh, and successful cache drops before every cold repetition.Independent verification started for PR #754 at head
e8a99a6fee. Test branch: review/754e-tests; base:e8a99a6fee. Reviewing diff 967dd3e8..e8a99a6f, earlier reviewer tests unchanged, independent cases, 50,000-case seeded runs, and required gates. No merge or deployment.Findings before measurement: 10by32-01-std still has no /usr/local/sbin/perf-drop-caches or installed perf-run.sh. This task specifically requests perf-test-02 and requires verified cache drops, so I am using perf-test-02 for both releases rather than repeat the old uncontrolled-cache run. Its installed perf-run.sh invokes perf-lock.sh and perf.slice. Each local heavy command will also use build-slot.sh. Six idle windows per size means 12 total. Source inspection at
e8a99a6f: crates/fsio/src/usage/reconcile.rs logs failures but no successful reconciliation timestamp; crates/fsio/src/usage/watch.rs also has no success log. I will retain logs and report reconciliation timing as unavailable unless an existing log provides it, without modifying the binary. Health will be checked through account.storage-health.Independent verification of
e8a99a6feeP3 — Review-test preservation statement needs qualification (validation evidence, not a product defect).
Reproducer: git diff
1494bd7ae8a99a6f-- crates/fsio/src/usage/review_754d_scan.rs crates/fsio/tests/review_754d_mount.rs.Earlier review/754-tests, review/754b-tests and review/754c-tests files compare byte-for-byte unchanged. Four 754d cases changed: returned-directory traversal now expects rejection while detached, and three mount cases now assert exclusion rather than inclusion. Mount namespace availability checks were added. Those assertions fit the newly approved contract; it would be inaccurate to claim every earlier test was unchanged. This verification does not modify those earlier tests.
P2 — Incremental refresh failures can leave Health unmeasured indefinitely with no watcher retry.
A brief parent-permission denial during Usage::refresh records a retained failure and clears ready. After access returns, the queued directory/file events reconcile that scope successfully. watch::apply returns Ok, so reconcile_batch leaves polling false; successful scoped repairs intentionally do not clear retained Health. No full recount is queued. The healthy worker returns to event-only waiting with stale unmeasured Health. A later explicit ensure/recount can clear it, but idle watcher recovery never happens. A transient openat2 rename race uses the same refresh_failed path.
Reproducer added unchanged as a failing regression: usage:⌚:review_754e::review_754e_refresh_failure_must_schedule_a_full_retry. It uses a real chmod failure as non-root, restores access, then feeds the production event-batch handler the file and parent events. All backends are native; the inode watch is installed before the failure. Expected: Health clears through a full recount or a full retry is scheduled. Actual: bytes are correct, retained Health remains, ready=false, polling=false.
Command: cargo test -j3 -p calternal-fsio review_754e_refresh_failure_must_schedule_a_full_retry -- --exact usage:⌚:review_754e::review_754e_refresh_failure_must_schedule_a_full_retry --nocapture
Cause: usage.rs refresh_failed records failure without waking a retry; watch_recovery.rs only starts polling on returned errors and only clears it after full recount. refresh returns (), so the event worker cannot observe a refresh failure through apply's Result. Scope repair success cannot certify/clear earlier retained failures.
P3 — The 50,000-case knob does not scale every contracted property-test dimension.
Reproducer: inspect usage/containment_property.rs two_thousand_seeded_topology_changes_match_final_tree_or_retry and run_case, and usage/containment_mounts.rs seeded_bind_and_tmpfs_topologies_exclude_mounts_and_recover. QUOTA_PROPERTY_CASES scales the topology loop only. That generator performs one of seven operations at one traversal checkpoint; remove/recreate is included but actual inode reuse is not asserted. Namespace mount topologies remain a separate fixed 32 seeds (1..=32), and repeated three-attempt exhaustion is a separate deterministic test. The earlier 754d inode-reuse probe also returns successfully if it cannot obtain reuse.
The suite does exercise symlink-parent replacement, deletion/recreation, namespace mounts, and bounded churn collectively. However, a claim that all four dimensions receive 50,000 randomized cases would overstate the coverage. The strict single-generator reading of contract item 7 is not met. This run reports the separate case counts and inode-reuse evidence explicitly rather than claiming broader coverage.
Finding: all six 10k cold-start cache drops exited 0. Raw startup.healthz samples (ms), same run/client/fixture: [{"kind": "dev-cold", "rep": 1, "startup.healthz_ms": 87.04084700002568, "cache_drop": true}, {"kind": "dev-cold", "rep": 2, "startup.healthz_ms": 83.3353529960732, "cache_drop": true}, {"kind": "dev-cold", "rep": 3, "startup.healthz_ms": 78.9145499948063, "cache_drop": true}, {"kind": "head-cold", "rep": 1, "startup.healthz_ms": 85.57564200600609, "cache_drop": true}, {"kind": "head-cold", "rep": 2, "startup.healthz_ms": 84.42939299857244, "cache_drop": true}, {"kind": "head-cold", "rep": 3, "startup.healthz_ms": 93.90227599942591, "cache_drop": true}]. All four Health checks so far returned HTTP 200, error null, unmeasured_entries 0. First 300-second idle window is preparing after the previous 65-second room-close wait. Quota reconciliation log timing remains blocked: this release emits no successful completion event, so the requested time cannot be derived from its logs without changing the binary. Continuing every other measurement unchanged.
Independent verification checkpoint: all three 50,000-case topology runs passed. That is 150,000 cases total, including 100,000 fresh seeds. The main generator is unchanged; the new test-only wrapper supplies fresh seed ranges. Mount coverage is reported separately, as above. Required workspace gates are now running sequentially through the build-slot wrapper.
property-original-50000 (verbatim):
property-fresh-50000-a (verbatim):
property-fresh-50000-b (verbatim):
Measurement progress for head
e8a99a6fon perf-test-02. 1/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 6 attempted, 6 exited 0. Health: 6 checks, 6 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
Measurement progress for head
e8a99a6fon perf-test-02. 2/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 6 attempted, 6 exited 0. Health: 8 checks, 8 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
P2 — The reconciliation scope deadline does not bound state-mutex waits.
Reproducer added: usage::reconcile::review_754e::review_754e_state_mutex_wait_must_obey_the_scope_deadline. It calls the unchanged production repair_snapshot method with a 100 ms scope deadline while another thread owns Usage.state. More than one second past that deadline, no result is returned. Releasing that mutex allows the worker to return TimedOut. The test releases its own guard and joins its own worker before asserting, and uses a channel deadline rather than a fixed sleep.
Verbatim failure:
Source evidence: usage/reconcile.rs acquires state with ordinary Mutex::lock at the start of repair_snapshot and again before merging/publication; those waits do not consult the shared deadline. Only the separate reconciliation Gate has a timed wait. Production refresh now owns that same state mutex while resolving paths and observing aliases through filesystem I/O, so waiting behind a slow or queued writer can exceed the advertised 30-second scope limit. The test scales the passed deadline for speed; it does not claim to have measured a public 30-second run. This is a lock-wait gap, separate from the inherent inability of userspace deadline checks to preempt a blocked kernel filesystem call.
The first complete workspace run passed every existing test, including the known Tasks abrupt-stop test, and failed only the earlier Health-retry reproducer (fsio: 109 passed, 1 failed, 1 ignored). Final gates are being rerun with this additional regression enabled. Product code is unchanged.
Measurement progress for head
e8a99a6fon perf-test-02. 3/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 6 attempted, 6 exited 0. Health: 10 checks, 10 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
P2 — Discovering a new mount incrementally leaves cached siblings charged.
Enabled public-API reproducer: review_754e_incremental_mount_discovery_matches_a_fresh_recount, in crates/fsio/tests/review_754e_mount.rs. This actually ran in a private user/mount namespace on this host.
Reproduction: recount a directory containing files of 11 and 13 bytes (24 total); bind-mount an outside directory over it on the same filesystem; write one mounted name using calternal_fsio::write_atomic, exercising its production quota callback; call Usage::ensure; compare maintained usage with a fresh counter's recount.
Verbatim observation:
The callback removes the changed file's old 11-byte entry but leaves its hidden 13-byte sibling charged. ensure returns successfully without fixing the cache. A fresh full recount excludes the entire mount and charges zero. Health correctly records one excluded mount, but the maintained total does not match that exclusion policy. Cached siblings can therefore keep a Home over quota until a broader repair/recount occurs.
Source evidence: scan::read_until records the encountered mount's ancestor path, but changed_file only extends Usage.mounts and returns an observation for the requested file. Usage::refreshed replaces only that event name. When a mount is newly discovered, all cached names beneath that mount need invalidation or repair through the bounded scope mechanism. Existing mount tests begin with the mount already present, so they do not exercise this transition.
Command: cargo test -j3 -p calternal-fsio --test review_754e_mount -- --nocapture. As in the targeted gate, the parent and namespace child both report the failure; this is one regression, not two distinct findings. The child unmounts and removes its fixture before asserting.
Measurement progress for head
e8a99a6fon perf-test-02. 4/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 6 attempted, 6 exited 0. Health: 12 checks, 12 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
Independent verification finished: fix first.
Reviewed product head:
e8a99a6fee; delta 967dd3e8..e8a99a6f. Tests-only branch: review/754e-tests, head1f7e08279e. Product code is unchanged.Three P2 findings have enabled reproducers:
Earlier containment regressions pass. Independent positive tests pass for a 10,000-file Home, 60 nested directories, Unicode and non-UTF-8 names, sparse logical bytes, per-name hard-link accounting, Home replacement before publication, eight native Home replacements and stale-parent event handling. Original fsio: 103 passed, 0 failed, 1 existing ignored, plus both mount integration tests passed. Namespace mount cases actually ran, and the earlier inode-reuse probe reached reuse rather than taking its early return.
All three dedicated 50,000-case runs passed: the original range plus 100,000 fresh seeds. Seed starts: 15739536603800761161 and 8468869847220311323. Verbatim results.
Two P3 qualifications: seven earlier reviewer files are byte-for-byte unchanged; two 754d files changed four cases to match the new policy. The 50,000-case knob does not scale the separate fixed 32-seed namespace mount suite or force inode reuse; repeated churn is tested separately. The attached audit explains these limits and the remaining Path-based native watch registration interfaces. Home-relative quota measurements use the shared resolver; no production std::fs metadata/canonicalization bypass was found.
Final gates: fmt exit 0; clippy with warnings denied exit 0; complete workspace tests exit 101, with 1234 passed, 3 failed and 21 existing ignored across 89 top-level targets. Only the three added regressions fail. The Tasks abrupt-stop test passed in all three full runs, so no retry was needed. All existing tests remain enabled and unchanged by this verification.
Final gate excerpts, verbatim; complete stdout/stderr for all gates is attached without truncation.
The namespace child and its parent both print the mount-test failure; that is one failing integration test. Reports, raw logs, seeds and provenance are attached to this comment, and none were committed. Only review/754e-tests was pushed. No merges, deployments, perf workloads or issue closure.
Complete verbatim gate output, SHA-256
9df1b24f8ac762f13a8d0563016aa69ba3c8f423ffbc29c790fa02c1c764e009.Evidence archive, SHA-256
6d5496318e40f376764e9ab22d597373912d79662a6913f607ecc2601ff39fd2.Finding: head
e8a99a6fat 10k notes, indexed warm startup.healthz raw ms (all five requested samples, unchanged): 16.186341003049165, 5.410058001871221, 5.4032159969210625, 5.35294899600558, 7.553593000920955. Nearest-rank p95: 16.186341003049165 ms <= 300 ms PASS. These are process-spawn to first successful /healthz with the same client as the cache-dropped cold comparisons. The sixth start will be retained as an extra, as in the previous method. Idle collection continues.Starting the approved quota robustness fixes on perf/idle-honest-2. Original base
e8a99a6f; current base1f7e08279e. I will merge reviewer branch review/754e-tests, retain all reviewer tests, address the three P2 findings and property scaling, and run the requested gates. No deployment or merge to main/dev.Baseline after merging reviewer tests at
1f7e0827: cargo test -j3 -p calternal-fsio review_754e -- --nocapture reproduced two failures, 6 passed / 2 failed. review_754e_state_mutex_wait_must_obey_the_scope_deadline exceeded its 100 ms budget by over one second; review_754e_refresh_failure_must_schedule_a_full_retry left Health unmeasured with healthy native watches and no retry. Mount code inspection confirms changed_file only removes the event path and keeps cached siblings below newly discovered mounts. Fixes will add scoped backoff retries, bound state locking and admission, and evict mount descendants. Ordinary explicit scoped repairs will preserve historical failure union, as required by existing reviewer tests; successful scheduled retries will clear validated failure scopes.Measurement progress for head
e8a99a6fon perf-test-02. 5/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 6 attempted, 6 exited 0. Health: 14 checks, 14 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
The first complete fsio run after the P2 fixes passed: 115 unit tests, zero failures, plus all integration tests. The reviewer incremental mount test passed in a private mount namespace (incremental and recount both zero), as did the previously failing state-mutex deadline and watcher-retry tests. New tests exercise event-free recovery of one failed path, 5/30/120/600-second backoff, cached mount sibling eviction, and bounded admission. Final verification now also includes cleanup after a timeout and preserving failures newer than a successful attempt. The property setting now scales generated tree size, operations and independently selected traversal pauses; at 50,000 it runs two mutations with two pause points per case on trees with twice the generated directory budget. No reviewer tests were changed or removed.
Measurement progress for head
e8a99a6fon perf-test-02. 6/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 6 attempted, 6 exited 0. Health: 15 checks, 15 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
Property scaling review found the first version ran extra mutations on fresh fixtures. I stopped only my own recorded property-test PID and strengthened the generator before counting a 50,000-case gate: mutations now run successively on the same evolving tree, with a fresh traversal pause for each operation. At 50,000 the directory-generation budget doubles and each case performs two mutations and two pause points. The final 50,000 run is now in progress; the stopped preliminary run is not counted as a passing gate.
Finding: all twelve requested cold-start cache drops exited 0. 100k raw startup.healthz samples (ms), same run/client/fixture: [{"kind": "dev-cold", "rep": 1, "startup.healthz_ms": 87.45171099872096, "cache_drop": true}, {"kind": "dev-cold", "rep": 2, "startup.healthz_ms": 129.11793799867155, "cache_drop": true}, {"kind": "dev-cold", "rep": 3, "startup.healthz_ms": 94.55018399603432, "cache_drop": true}, {"kind": "head-cold", "rep": 1, "startup.healthz_ms": 313.30598299973644, "cache_drop": true}, {"kind": "head-cold", "rep": 2, "startup.healthz_ms": 82.38098199944943, "cache_drop": true}, {"kind": "head-cold", "rep": 3, "startup.healthz_ms": 116.80540799716255, "cache_drop": true}]. dev-cold nearest-rank p95=129.11793799867155 ms <= 300 ms PASS. head-cold nearest-rank p95=313.30598299973644 ms <= 300 ms FAIL. No repetitions were replaced. The final cold process is settling; six 100k idle windows and five warm samples follow.
The final strengthened property gate passed. The stopped preliminary run was excluded. Command: QUOTA_PROPERTY_CASES=50000 cargo test -j3 -p calternal-fsio usage::scan::property -- --nocapture, run through the build slot. The same evolving tree now receives successive mutations at separate traversal pauses. The 50,000 setting doubles the tree-generation budget and runs two operations and pause points per case. Both 50,000 seed ranges, the existing 20,000-case reviewer range, and mount boundary tests passed.
Verbatim gate output:
The final fmt, workspace Clippy with warnings denied, fsio and full workspace gates are running next, one build-slot job at a time.
The first full workspace run found a contract conflict in the older auth integration test signin_finishes_before_a_large_recount (not a reviewer test). It unwraps every admission while 50,000 files are being reconciled. With the requested bounded admission behavior, those uploads now correctly return retryable Internal errors instead of waiting for stale totals. The sign-in ordering assertions completed successfully; the failure occurred when joining an admission task. The test predates the reviewer branches (history:
046e7661,6f7e5525,883b8ac8); review/754-tests only adds fsio independent-review tests. I am updating the auth test to assert the new bounded, retryable admission contract while keeping its sign-in/recount checks, and checking the upload serialization wait as part of the same admission budget.Measurement progress for head
e8a99a6fon perf-test-02. 7/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 12 attempted, 12 exited 0. Health: 21 checks, 21 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
The strengthened auth test reproduced a real queue-bound problem: multiple admissions failed its two-second bound while serialized behind reconciliation. A separate fsio regression also reproduced that merely waiting for an active recount invalidated Health. Fix: admission now has one 500 ms budget covering the per-person queue and freshness check; waiting for an existing measurement does not create a new failed measurement. Actual scope walks still have bounded lock acquisition, mark Health on timeout and schedule retry. The older auth test retains its sign-in-before-recount assertions, now checks bounded retryable admission results, and additionally requires upload admission to recover once measurement catches up. All reviewer tests remain unchanged.
The next workspace run passed the updated quota_counter tests, then found the same immediate-success assumption in independent_login_completes_before_recount from the older auth reliability suite (history predates PR #754: 8703faef/162935a8/932d22ec/6f7e5525). Its helper now retries only retryable busy errors with a two-second assertion on each admission and a 60-second recovery bound. All existing sign-in ordering checks and eventual successful-admission requirements remain in place. The PR #754 reviewer files are unchanged, verified against origin/review/754e-tests.
Measurement progress for head
e8a99a6fon perf-test-02. 8/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 12 attempted, 12 exited 0. Health: 22 checks, 22 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
The recovery assertion caught admission-driven scan starvation: repeated unready admissions could keep starting 250 ms full walks instead of letting the worker perform its longer repair. Admission now waits briefly for an active measurement and, when a background worker exists, returns a retryable pending error without starting another full walk or changing retry generations. Initialization and failed-scope walks belong to that worker. The independent login/recovery test now passes in 1.91 seconds. A new fsio test holds the state lock while a retry is pending and verifies admission performs no walk, leaves the timer and generation intact, and succeeds after the retry. The Health integration test now waits for automatic recovery before requiring successful admission.
The reviewer refresh fixture caught a worker-detection mistake in the admission change: it sets started before creating a worker. The check now uses the installed wake channel as evidence that a worker exists, so standalone initialization still works. The unchanged review_754e_refresh_failure_must_schedule_a_full_retry test passes again. Targeted admission and missing-backend fallback tests also pass. Missing native watches still enable fallback polling even when a measurement fails; healthy native watches use only failed-scope retry timers.
Measurement progress for head
e8a99a6fon perf-test-02. 9/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 12 attempted, 12 exited 0. Health: 25 checks, 25 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
Final gate output for head
ec4f3e1769. Formatting exited 0 with no output. All local heavy commands ran through the build slot. The output below is verbatim.cargo clippy -j3 --workspace --all-targets -- -D warnings
cargo test -j3 -p calternal-fsio
QUOTA_PROPERTY_CASES=50000 cargo test -j3 -p calternal-fsio usage::scan::property -- --nocapture
Full workspace gate output for head
ec4f3e1769, part 1 of 3. Command: cargo test -j3 --workspace. Verbatim output, continued across these comments.Full workspace gate output for head
ec4f3e1769, part 2 of 3. Command: cargo test -j3 --workspace. Verbatim output, continued across these comments.Full workspace gate output for head
ec4f3e1769, part 3 of 3. Command: cargo test -j3 --workspace. Verbatim output, continued across these comments.Finished and pushed perf/idle-honest-2 at
ec4f3e1769. PR #754 reports this same head. Reviewer branch review/754e-tests was merged at1f7e0827before the fixes; all PR #754 reviewer test files remain unchanged.The three P2 fixes and P3 scaling are complete. Failed measurement scopes retry after 5 seconds, 30 seconds, 2 minutes, then every 10 minutes, and successful retries clear Health. Healthy native Homes have no periodic walk. Reconciliation state-lock waits and cleanup obey the attempt budget. Admission has a 500 ms queue and freshness budget and returns retryable errors while unmeasured; it does not start competing walks when the worker owns measurement. Mount discovery removes cached descendants, including siblings without events. Property cases now grow trees and perform successive mutations at separate pause points on the same evolving tree.
Older auth integration tests now check bounded retryable busy errors, successful eventual admission, and unchanged sign-in ordering. The final fsio gate passed 120 unit tests and all integration tests with zero failures. Full workspace tests passed with zero failures. The Tasks flake from #757 did not occur. The stopped preliminary property run was excluded; the completed strengthened 50,000 run took 400.840 seconds. No evidence or artifacts were committed. No PR was merged and no deployment was performed.
Verbatim gate status:
Verbatim property timing output:
Complete gate output is in the preceding comments.
Measurement progress for head
e8a99a6fon perf-test-02. 10/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 12 attempted, 12 exited 0. Health: 27 checks, 27 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
Finding: head
e8a99a6fat 100k notes, indexed warm startup.healthz raw ms (all five requested samples, unchanged): 16.89317700220272, 8.83206900471123, 8.710842004802544, 4.311869000957813, 8.948848997533787. Nearest-rank p95: 16.89317700220272 ms <= 300 ms PASS. The 100k cache-dropped cold p95 remains 313.30598299973644 ms FAIL; it has not been replaced. Ten idle windows are complete, with two more to collect.Independent verification 754f started. Branch: review/754f-tests. Base and reviewed head:
ec4f3e1769. Reviewing delta e8a99a6f..ec4f3e17, retry lifecycle, admission bounds and HTTP error mapping, mount churn, prior reviewer test integrity, and the requested gates. Tests only will be pushed; no merges or deployments.754f finding 1 — P1: the first failed scope drops later repairs from the same event batch.
At crates/fsio/src/usage/watch.rs,
for scope in scopes { usage.repair(&scope, || {})?; }returns early. The event batch has already been consumed. Only the first scope owns a retry; with the new event-only failure recovery, later scopes have neither a retry nor a pending event. Once the first scope recovers, ready becomes true with stale totals for the later scope.Deterministic reproducer: initialize an empty Home; create a-fails with 66 nested directories and z-healthy/file with 17 bytes; apply one Pending batch with both repair scopes; delete a-fails and run the sole scheduled retry. Health clears, the retry queue empties, ensure succeeds, but usage remains 0 instead of 17. This permits stale quota admission.
Test: review_754f_failed_batch_preserves_later_scopes. Run
cargo test -j3 -p calternal-fsio review_754f_failed_batch_preserves_later_scopes -- --ignored --nocapturethrough build-slot.sh after sourcing env.sh. Actual assertion: left 0, right 17. Reproducer is explicitly ignored in normal gates because it documents an unfixed product defect. No product fix in this verification branch.754f finding 2 — P2: distinct failed paths retain an unbounded number of retry records.
Retries.scopes in crates/fsio/src/usage/retry.rs has neither a count bound nor ancestor coalescing. schedule_retry deduplicates the same path only. The event queue is capped at 4096, but successive batches or application refresh failures can accumulate arbitrarily many distinct paths in both Health and retries. Each retry_at scans every retained record, and each due batch allocates all due paths and performs a separate repair per path. A large permanently inaccessible subtree therefore has no bounded bookkeeping or repair cost.
Reproducer: refresh_failed for 8192 distinct paths in one Home leaves 8192 timers; no ancestor scope replaces them. review_754f_failure_timers_are_bounded_like_event_queue fails with “8192 retained timers exceed the bounded event queue; no ancestor coalescing”. The test uses 4096 as a concrete existing queue budget, not a claimed documented timer limit. Source inspection establishes that any chosen finite cap is absent. Repeated failures of one path do deduplicate correctly.
Run the named test with
-- --ignored --nocapturethrough build-slot.sh. The ignored reproducer documents an unfixed defect.754f finding 3 — P3: deleting a Home does not immediately cancel historical failed-repair timers.
resolve_retries only clears nonincremental failed-repair records when scheduled is true. A successful explicit repair of a deleted Home calls it with scheduled=false, so descendant historical failures retain their timers and Health state until their next retry, potentially ten minutes later. Incremental failure timers do clear correctly. This is delayed cleanup, not an indefinitely retained timer: the next scheduled retry of the absent path succeeds and removes it.
Reproducer: initialize; fail repair of a 66-level subtree; advance retries through 30 s, 120 s, 600 s backoff; delete Home; successfully repair Home. retry_at is still Some. review_754f_deleted_home_cancels_historical_timers_immediately fails with “deleted Home still owns a ten-minute retry”. Run with
-- --ignored --nocapturethrough build-slot.sh. The ignored reproducer documents the cancellation contract gap.754f prior reviewer test integrity check:
All dedicated reviewer tests from 754, 754b, 754c and 754e match their review branch bytes. The 754b attachment HTTP tests also match exactly. No existing dedicated reviewer test file was changed by e8a99a6f..ec4f3e17; this delta only adds the five 754e reviewer files.
The historical 754d branch does NOT match the reviewed tree in two files: crates/fsio/src/usage/review_754d_scan.rs and crates/fsio/tests/review_754d_mount.rs. Git history attributes these changes to
e8a99a6f, before the present delta. Mount expectations were changed from counting foreign mounted bytes to excluding them and retaining a Health mount count; one detached-Home snapshot test was changed to expect rejection followed by a fresh read, and unavailable user namespaces now skip mount tests. These are real test changes and must not be described as unchanged across every historical review branch. They are unchanged within this delta.The shared containment property generator also changed in this delta as requested: QUOTA_PROPERTY_CASES scales tree construction, operations and pause selections from 1 through 8, at 25,000-case boundaries. Existing reviewer seed wrappers remain byte-identical.
754f measurement limitation: the designated perf VM is unavailable by its supplied name.
Read-only connection attempt:
ssh -o BatchMode=yes -o ConnectTimeout=5 10by32-01-std ...returnedssh: Could not resolve hostname 10by32-01-std: No address associated with hostname(exit 255). No perf command ran and no other VM was touched. I will not claim a measured CPU budget for permanently failing scopes. Functional tests cover quiet retries, deduplication, backoff and teardown, and source inspection verifies event/deadline waiting. Verification continues through the required gates and functional reproducers; this limitation does not prevent a fix-first verdict for the reproduced stale-admission defect.Measurement progress for head
e8a99a6fon perf-test-02. 11/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 12 attempted, 12 exited 0. Health: 29 checks, 29 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
754f finding 4 — P2: a failed full recount periodically rewalks healthy Homes.
repair_until receives exact failed paths in scan::Unmeasured and adds those paths to Health, but unconditionally calls schedule_retry(scope, false). When scope is users, retries retain users rather than the failing subtree. retry_due then repairs users repeatedly, traversing every healthy Home at each 5 s / 30 s / 2 min / 10 min retry. A stable failure in one Home therefore makes background CPU and I/O proportional to the entire vault, contrary to the failed-scope-only contract.
Reproducer: initialize a healthy Home b and complete its recount; add a 66-level failing subtree under Home a; fail one full recount; install the existing traversal checkpoint hook and execute the scheduled retry. The hook observes Home b during the retry even though b has no failure. Test review_754f_retry_does_not_walk_healthy_homes is explicitly ignored for normal gates and run with
-- --ignored --nocapturethrough build-slot.sh.Observed output: “scheduled failed-scope retry walked a healthy Home”, left 2, right 0 (entry and exit checkpoints in Home b). The explicit ignored-test run reproduced all four findings, exit 101.
Measurement progress for head
e8a99a6fon perf-test-02. 12/12 idle windows complete. New raw results, without rounding:Each RSS gate uses 100000000 bytes and CPU uses 0.1%. Cache drops: 12 attempted, 12 exited 0. Health: 30 checks, 30 returned HTTP 200 with zero unmeasured entries. Measurement failures retained: 0.
754f independent verification finished. Verdict: fix first.
Reviewed delta: e8a99a6f..ec4f3e1769c26748d93c6979a2ee3914b4e2ac1d. Test-only branch review/754f-tests is pushed at
d0545088b2. The working tree is clean.Findings and reproducers:
The four contract reproducers are explicitly ignored in ordinary test runs and were all executed separately; all four fail on the reviewed product code. The green workspace gate does not erase these findings.
Passing independent checks: paused long reconciliation refuses stale admission within its bound and recovers to fresh totals; the production backdrop HTTP API returns 500/internal with retryable=true for both a held admission queue (501.35425 ms) and unmeasured accounting (521.998 microseconds). Twelve private-namespace bind mount/unmount cycles correctly evict hidden cached siblings and recover after scoped repair. Healthy counters have no retry records, same-scope failures deduplicate, quiet retries do no scan work, and teardown checks pass. Thirty-two fresh property seeds passed at maximum tree, operation and pause scale 8 with QUOTA_PROPERTY_CASES=200000; this was a targeted 32-case run, not 200000 full cases.
Prior reviewer test comparison: no existing dedicated reviewer tests changed within this delta. The two historical 754d file differences already exist at
e8a99a6f; all dedicated 754/754b/754c/754e tests and the 754b attachment HTTP tests match their review branches exactly. Comparison detail.Required gates passed on the first attempt: cargo fmt --all -- --check; cargo clippy --workspace --all-targets -j3 -- -D warnings; cargo test --workspace -j3. The #757 Tasks flake did not recur, so no rerun was needed.
The gate runner output, verbatim:
Complete evidence and verbatim gate output contains unabridged fmt, clippy, full workspace, independent HTTP, maximum-scale property and failing-reproducer logs, SHA-256 checksums, findings, result.md and prior test comparisons. The empty fmt.log reflects a successful fmt check with no output. Reproduce the four findings with cargo test -j3 -p calternal-fsio review_754f -- --ignored --nocapture, through build-slot.sh after sourcing env.sh.
Measurement limitation: SSH could not resolve the designated 10by32-01-std hostname, so permanent-failure CPU cost was not measured. No perf command ran on another VM. Connection evidence.
Only tests and test module registrations were committed. Evidence stays outside the repo. No merges, deployments or issue closure.
Finding from final coverage verification: the zero wakeup proxy is incomplete in all twelve idle windows. Each start/end per-thread sample loses at least one thread that exited during the window. The proxy records zero for surviving threads but cannot recover exited threads final context-switch counters, so zero total wakeups is not established. All twelve wakeups-thread-coverage gates fail. Process-wide CPU ticks are valid and remain zero. No repetition was replaced. Evidence, unchanged raw values:
The final 100k RSS gate also failed: 106921984 bytes > 100000000. The 100k cold-start p95 failed at 313.30598299973644 ms > 300. The finish report will retain all failures and distinguish coverage limits from the zero harness-exception count.
Observed budget or integrity gate failures: 100000 notes repetition 6: idle.rss_max_bytes FAIL 106921984 <= 100000000; 100000 head-cold startup.healthz_p95_ms FAIL 313.30598299973644 <= 300.
Wakeup coverage is incomplete in every idle window. The raw proxy records zero for surviving threads, but each window lost at least one thread between boundary samples; those exited threads have no final context-switch counter. All twelve wakeups-thread-coverage gates fail, so zero total wakeups is NOT established. The previous proxy is retained verbatim, with its coverage limits explicit. Process-wide CPU ticks remain valid at zero. These coverage failures are separate from harness exceptions, whose count is zero.
Quota reconciliation completion time is unavailable because the measured head emits no success log event. Full raw results and verbatim gates follow.
Finished idle and startup measurement of PR #754 head
e8a99a6fee, origin/perf/idle-honest-2. Comparison: devd83e54dbe5. Both release binaries were built with --release --locked -j3 and measured on perf-test-02 (x86_64) in this run. No code changes, commits, pushes, merges, deployments or issue closure.The task-specific instruction to use perf-test-02 was selected over the generic 10by32-01-std instruction because the required working cache-drop helper is on perf-test-02. The old VM still lacks that helper. This changes the machine and architecture from comment 68858; compare these two revisions within this run, not against the old absolute numbers.
Measurements ran under /usr/local/bin/perf-run.sh, which holds perf-lock.sh and enters perf.slice. The harness ran through build-slot.sh inside that held lock. Every cold repetition used sudo -n /usr/local/sbin/perf-drop-caches immediately before launch; all twelve helper return codes are retained. Cold starts use separate fresh unindexed copies of the same seed-1 fixture. Warm starts use the head's indexed copy. The first five of six indexed process starts supply the requested warm samples, and all six supply separate idle windows. p95 uses nearest rank: the maximum of three or five samples. Failed repetitions are retained and never replaced.
Same idle method as comment 68858: notes.list complete and search.query no longer indexing, then a 65-second room-close wait, then 300 seconds of idle. CPU is process user plus system ticks / CLK_TCK / actual elapsed seconds, percent of one CPU. Wakeups are the previous proxy: summed per-thread voluntary plus involuntary context-switch deltas, not hardware wakeups. RSS is sampled every second including boundaries; MiB is KiB / 1024, and the gate also enforces decimal 100 MB (100000000 bytes). Process lifetime high-water RSS remains separate in raw records. GLIBC_TUNABLES=glibc.malloc.arena_max=2. Homes and state share one root, as before. This is settled read-only idle, not a full workload run.
Raw idle numbers, without rounding:
Raw startup numbers, without rounding:
d83e54dbe8a99a6fe8a99a6fd83e54dbe8a99a6fe8a99a6fAdditional sixth warm-start samples, excluded only from the requested n=5 startup statistic and retained verbatim: [{"event": "startup", "notes": 10000, "kind": "head-warm", "rep": 6, "startup.healthz_ms": 5.2609189951908775, "cache_drop": false, "pid": 310006, "startup_sample": false, "observed_utc": "2026-10-29T03:48:49.548636+00:00"}, {"event": "startup", "notes": 100000, "kind": "head-warm", "rep": 6, "startup.healthz_ms": 8.069229996181093, "cache_drop": false, "pid": 311677, "startup_sample": false, "observed_utc": "2026-10-29T04:34:03.237064+00:00"}]
Quota reconciliation completion from log timestamps is UNAVAILABLE. At this exact head, Usage::background starts the usage-events worker, which calls recount and installs inode watches, but neither successful recount nor completed initialization emits a log event (crates/fsio/src/usage/watch.rs and reconcile.rs). Logs are retained with calternal and calternal_fsio at info. Health zero is not proof that background initialization has finished: storage_failure reports retained failures and does not expose readiness. No timing is inferred from Health or note indexing, and no binary or source was modified to add instrumentation. This requested metric remains incomplete.
Health responses for every head start after notes/search settle, and again after every idle window:
Gate output verbatim:
All measurement failures retained verbatim:
Fixture Homes and state were deleted. Raw samples, server logs, build logs and measurement scripts are retained outside the repository at /home/agent/wt/idle-754b-evidence. No artifacts were committed.
Raw evidence for head
e8a99a6feeand comparisond83e54dbe5: download archive. SHA-256 ec13f259020a4dbbbc0abfd144ef398d911147cb5eb37a058af67853ab7dba6c. Upload download verified. The archive contains every raw sample, cache-drop return code, server and build logs, gates, scripts, binary hashes and a file manifest. It contains no fixture Homes or state; those were deleted. No evidence was committed. The complete raw tables and verbatim gate output are in the finish comments immediately above. Quota reconciliation completion timing remains unavailable because the exact measured release has no success log event.Starting the approved retry/repair and memory fixes on perf/idle-honest-2, base
ec4f3e1769. First merging reviewer tests origin/review/754f-tests (d0545088). I will preserve all reviewer tests, run the required gates, and report evidence here. No perf VM workloads, main/dev pushes, merges into main/dev, or deployments. The final task instruction to push only perf/idle-honest-2 is taken as the specific branch instruction for this job.Baseline
d0545088reproduces all four findings with cargo test -j3 -p calternal-fsio review_754f -- --include-ignored --nocapture. The bounded-timer test reports 8192 retained timers; deletion reports a surviving ten-minute retry; the healthy-Home checkpoint records 2 visits instead of 0; the batch test publishes 0 instead of 17 bytes after recovery. Full output is retained locally, not committed.There is one contradictory reviewer assertion: review_754f_healthy_and_deleted_incremental_scopes_have_no_timers requires scopes.len() == 1000 before deletion. The approved bounded-retry requirement makes that impossible. I will strengthen that assertion to require at most 64 actual timers and exactly 1000 unmeasured entries in Health, retaining all its deletion/readiness assertions. All previously ignored defect reproductions will be enabled.
Memory investigation: State retains a HashSet for every walked directory (24 bytes per record, plus hash-table control/capacity and the allocated full path). Snapshot also holds Location identities only until publication. The retained directory map is redundant: an absent observed file already triggers repair, and repair can remove descendants by prefix. I am removing retained directory records while preserving temporary identities through snapshot acceptance.
Additional correctness finding: merely replacing a failed full recount's root retry with failing-Home retries would discard newly measured bytes in healthy Homes. Added regressions first; both failed on
d0545088because healthy Home totals stayed zero after the failed full scan. Reconciliation now validates and publishes measured entries, preserves cached entries under unmeasured scopes, and leaves admission closed until scoped retries recover. Initial partial traversal records initialization without changing readiness semantics.The first scan optimization attempt removed per-entry parent verification. Reviewer test review_754d_returned_directory_charges_bytes_that_returned_with_it rejected that change: a detached Home could be read through its pinned descriptor before being returned. I restored per-entry verification, reusing the saved generation identity instead of creating a fresh handle each time. The final optimization preserves the reviewer containment fence.
Memory and hot-path review found that directory ancestry hints cannot be removed completely without losing directory-to-file replacement detection or making ordinary file repair scan the vault. A new regression first reproduced 38 cached bytes instead of 7 after an application callback replaced a directory with a file. The final design stores only a 64-bit randomized fingerprint and a reference count for directories containing cached files. Collisions request extra repair; they cannot skip repair. Empty directories keep no records. Updates cost bounded path depth, and existing-file replacements need no directory update. A unit test asserts 16 bytes per directory record (previously 24 bytes plus a full allocated PathBuf string), and another asserts zero retained bytes for 1000 empty directories. Temporary generation identities are dropped after snapshot acceptance.
Retries coalesce to one nearest-common-ancestor timer per Home, which is stricter than the proposed cap of 64. A Home lookup index makes failure insertion independent of the number of prior failures; Health retains exact distinct paths and counts. New tests check separation between Homes and account for two failed scopes surrounding a healthy scope in one batch.
The explicitly requested local syscall check used the same standalone debug scan probe, one Home with 10,000 empty regular files, and strace -f -c before and after. Each probe ran inside a build slot. Total calls: 130278 to 120277, or 13.0278 to 12.0277 per file (7.68% fewer). name_to_handle_at: 20028 to 10027. statx/openat2 counts are unchanged, preserving the mount and containment checks. This is syscall evidence, not a cold-start latency or RSS result. No perf VM workloads were run.
Before (verbatim):
After (verbatim):
Further memory finding from the rebuild path: publication calls replace for every scanned file while recording is still true. That unnecessarily builds a second path-to-generation map for the entire vault, then clears it. If Recording::drop cannot acquire the state lock immediately, the empty map can retain its large allocation. This is separate from directory records, and matters for the mostly flat 100k-note fixture too.
Once snapshot validation and merging are complete, publication already holds the state lock. It now stops overlap recording and drops its map before publishing replacements, so synthetic recount entries allocate no write-history records. Actual callbacks during the walk still record and merge normally; the existing concurrent-write and reviewer topology tests cover that boundary. The cleanup guard also requests trimming after failed attempts, after snapshot identities have dropped. RSS budget confirmation remains with the coordinator; I am not running perf workloads.
The 50,000-case property run passed in 357.03 seconds, but final retry-specific regressions found two additional holes before delivery.
Both regressions were run and failed before fixes. I am correcting these paths and will refresh the final gates before pushing. No artifacts are being committed.
Gate output verbatim for
a6b1fa14d2; fmt-final; part 1/1; exit 0.Command:
cargo fmt --all -- --checkGate output verbatim for
a6b1fa14d2; clippy-final; part 1/1; exit 0.Command:
cargo clippy -j3 --workspace --all-targets --locked -- -D warningsGate output verbatim for
a6b1fa14d2; fsio-final; part 1/1; exit 0.Command:
cargo test -j3 -p calternal-fsioGate output verbatim for
a6b1fa14d2; property-final; part 1/1; exit 0.Command:
QUOTA_PROPERTY_CASES=50000 cargo test -j3 -p calternal-fsio two_thousand_seeded_topology_changes_match_final_tree_or_retry -- --nocaptureGate output verbatim for
a6b1fa14d2; workspace; part 1/3; exit 0.Command:
cargo test -j3 --workspace --lockedGate output verbatim for
a6b1fa14d2; workspace; part 2/3; exit 0.Command:
cargo test -j3 --workspace --lockedGate output verbatim for
a6b1fa14d2; workspace; part 3/3; exit 0.Command:
cargo test -j3 --workspace --lockedFinished on perf/idle-honest-2 at a6b1fa14d21944b25fa1538615501f414ae9ddbf. The push succeeded, the remote branch matches, and PR #754 reports this head. Reviewer tests
d0545088are included.Every consumed repair scope is processed. Retries coalesce to their nearest common ancestor within the same Home, with one timer per Home and exact distinct failure counts in Health. Validated partial walks publish healthy measurements and retry only failing Homes. An older root timer is retired when the failing Homes become known. Unmeasurable overlapping callbacks keep dirty scopes instead of discarding healthy measurements. Home deletion cancels historical timers immediately. New failures retain their generation protection.
All four ignored 754f reproductions are enabled. The contradictory expectation of 1000 actual timers was strengthened to at most 64 timers plus exactly 1000 unmeasured Health entries; its deletion and readiness assertions remain. Additional tests cover multiple failed batch scopes, partial initialization, failure during overlap merging, narrowing an initial root timeout, Home separation, and directory-to-file replacement.
Memory: retained directory records changed from a 24-byte PathBuf plus an allocated full path to a 16-byte fingerprint and reference count, without path allocation. These are record sizes before hash-table capacity overhead. A unit test measures the actual map key and value sizes. Empty directories retain zero records; another unit test checks zero retained bytes for 1000 empty directories. Publication also stops building a second vault-sized write-history map, and cleanup requests trimming after snapshot handles and paths are dropped, including failed walks.
The same debug scan probe over a 10000-file Home used 130278 syscalls before and 120273 after: 13.0278 to 12.0273 per file, 7.68% fewer. name_to_handle_at fell from 20028 to 10027; openat2 and statx counts are unchanged. This measures syscall counts, not cold-start latency. RSS and cold-start budget confirmation remain with the coordinator. No perf VM workloads were run.
Final gates passed. fsio reported 139 unit tests passed, zero failures, and one ignored crash helper exercised by its parent test; integration tests passed too. The final extended property run completed 50000 seeds at tree scale 2 with 2 operations and pause points per case in 367.699467866 seconds. The workspace passed without a retry; no #757 flake occurred. The extended run was repeated after the two final retry edge cases were reproduced and fixed, so these results cover the final code. Earlier attempts and findings are described above.
Complete gate stdout and stderr are posted verbatim in the preceding comments. Final gate status, verbatim:
Final strace output, verbatim:
The commit contains only product code and tests. Evidence and gate output were not committed.
Independent verification 754g started. Branch review/754g-tests, base
a6b1fa14d2. Reviewing delta ec4f3e17..a6b1fa14, adding independent contract tests, and running fmt, clippy with warnings denied, and full workspace tests. No merges, deploys, or perf workloads.Starting the requested idle and startup measurement on perf-test-02. Worktree /home/agent/wt/idle-754c is clean and detached at origin/perf/idle-honest-2, head/base SHA
a6b1fa14d2. Comparison devd83e54dbe5. No branch will be created or pushed: this is measurement only. Following comment 69074 fixture and idle method, with 3 cold and 5 warm starts for each revision at 10k and 100k, and six 300-second head idle windows per size. Raw failed repetitions will remain. Evidence stays outside the repo; fixtures will be deleted. perf-run.sh takes the lock. Each cold launch requires a successful cache drop. Using the specifically requested perf-test-02 rather than the generic machine instruction.754g finding: P1: admission returns success while a newer dirty scope still owns a retry
Reproducer:
cargo test -p calternal-fsio -j3 review_754g_admission_rejects_a_newer_failure_during_its_walk -- --nocapture.A deterministic traversal hook reports an incremental failure after the root repair captures its retry generation. The walk succeeds, and generation protection correctly retains Health, the retry and ready=false. Usage::ensure nevertheless returns Ok because it checks ready only before repair_until. The quota caller treats Ok as permission to read totals and admit. This violates the admission freshness contract. This code predates the reviewed delta, but remains a contract blocker at
a6b1fa14.Observed:
admission returned success with a newer dirty scope and live retry.754g finding: P2: empty directory index is not compacted after a deletion walk
Reproducer:
cargo test -p calternal-fsio -j3 review_754g_occupied_directory_records_compact_after_deletion_walk -- --nocapture.Create 1000 occupied directories, recount, delete the Home, and repair its deletion. The directory HashMap has no live entries but still retains payload capacity. Observed:
directory map capacity bytes before=28672, after deletion walk=28160; assertion expected 0. Directories::update removes entries but no successful publication shrinks or replaces the map. This violates post-walk compaction. HashMap capacity after removals is a conservative lower bound on the allocation because tombstones reduce the reported capacity.At 100000 occupied directories the independent in-memory index test observed 100002 records, capacity 114688, payload capacity 1835008 bytes = 18.35008 bytes/directory, excluding controls and allocator overhead. After removing every file it retained reported payload capacity 1125600 bytes; the original backing allocation is not freed.
754g finding: P2: reviewer tests cannot be confirmed unchanged
Reproducer:
git diff origin/review/754f-tests a6b1fa14 -- crates/fsio/src/usage/review_754f.rsandgit diff origin/review/754d-tests a6b1fa14 -- crates/fsio/tests/review_754d_mount.rs.754f's first timer assertion changed from exactly 1000 to <=64 and an exact Health-count assertion was added. Four ignored tests were enabled across the 754f files. 754d's mount test now skips when namespaces are unavailable and changed its charge/Health oracle from 12/no Health to 0/one excluded mount. The 754d changes predate this delta. This is a verification provenance finding: these edits may reflect intended new semantics, but the requested unchanged-tests condition is false. Full per-file comparison is retained outside the repository in reviewer-tests.txt.
Measurement setup finding at
a6b1fa14: account.storage-health returns only unmeasured_entries and error (crates/auth/src/quota.rs). Usage::storage_failure reads retained failure scopes, while watch.rs sets an internal initialized flag after recount and watch setup; that flag is not exposed. There is no existing environment-controlled success log. I will report the requested Health polling fallback at 250 ms until one second of unchanged scope count, both immediately after healthz and after note/search catch-up. These are observed stabilization timings; an initial zero cannot prove reconciliation completion. No code instrumentation will be added. perf is absent and perf_event_paranoid=3, so complete per-thread /proc/TID/sched nr_switches coverage will use kernel clone and exit stops; raw counters include observer overhead.754g additional P2 finding: Reproducer:
cargo test -p calternal-fsio -j3 review_754g_timer_bound_and_health_at_63_64_65_and_1000_failures -- --nocaptureestablishes that one Home retains one timer but 1000 unique Health paths. Inspect usage/retry.rs resolve_retries: after collecting resolved scopes it calls health.retain over every retained failure, and for each failure searches resolved scopes and remaining retry scopes. Even a healthy unrelated repair with no resolved timers scans that whole Health set while holding both Health and retry locks. With N failed paths and a one-timer cap, completion bookkeeping is at least O(N), and can be O(N * number of failed Homes), rather than O(cap). schedule_retry itself has bounded ancestor work, but that does not establish the requested bound for completion bookkeeping. This retain loop predates the reviewed delta and remains a contract gap at the head. No perf workload was run to establish the static bound.Finding for
a6b1fa14on perf-test-02: 1/12 idle windows recorded. Raw result, without rounding:CPU gate PASS: 0.0 <= 0.1 percent. RSS gate PASS: 29921280 <= 100000000 bytes. Scheduler counter coverage PASS. Context-switch counts include the observer boundary stops and exit-stop overhead; these are the complete scheduler-counter proxy, not literal hardware wakeups.
6 cold cache drops recorded, 6 successful. 6 normal-fixture Health checks recorded, 6 with zero unmeasured entries. Failed repetitions are retained and never replaced.
754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim fmt.log, part 1/1. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim clippy.log, part 1/1. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim workspace-tests.log, part 1/4. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim workspace-tests.log, part 2/4. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim workspace-tests.log, part 3/4. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim workspace-tests.log, part 4/4. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim targeted-final.log, part 1/1. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g finished verification output. Head
bf715d8446, branch review/754g-tests. Verbatim repro.log, part 1/1. The repro log intentionally runs the two known failing tests; they are ignored in the normal workspace gate.754g independent verification finished. Verdict: fix first.
Reviewed ec4f3e17..a6b1fa14 and called accounting/resolver code. Tests-only branch review/754g-tests is pushed at
bf715d8446. Product code was not changed.Findings:
Seven new ordinary checks pass: all six coalescing orders with duplicate reports and exact counts; mixed failing batches in every sorted position; 1/2/63/64/65/1000 failure boundaries; healthy-Home exclusion; deletion during a paused repair and an actual scheduled retry; and the 100k-directory index measurement. The implementation keeps one timer per Home; there is no separate numeric cap, so a distinct numeric overflow-to-root policy could not be certified.
Gates: cargo fmt --all -- --check exited 0; cargo clippy --workspace --all-targets -j3 -- -D warnings exited 0; cargo test --workspace -j3 exited 0. #757 did not reproduce, so no rerun was needed. Both known failing reproducers are explicitly ignored in the normal suite and were run separately with --ignored --nocapture: 0 passed, 2 failed, exit 101. Green normal gates do not resolve these findings.
Verbatim outputs are in the preceding comments:
Findings and result are saved outside the repository at ~/codex-jobs/verify-754g/findings.md and result.md. No merges, deploys, perf workloads, or issue closure.
Finding for
a6b1fa14on perf-test-02: 2/12 idle windows recorded. Raw result, without rounding:CPU gate PASS: 0.0 <= 0.1 percent. RSS gate PASS: 29732864 <= 100000000 bytes. Scheduler counter coverage PASS. Context-switch counts include the observer boundary stops and exit-stop overhead; these are the complete scheduler-counter proxy, not literal hardware wakeups.
6 cold cache drops recorded, 6 successful. 8 normal-fixture Health checks recorded, 8 with zero unmeasured entries. Failed repetitions are retained and never replaced.
Starting the XFS project quota feasibility probe. Tracking branch: job/xfs-quota-probe-20261029. Base SHA:
a779a46f42. No product code changes planned. Scripts and evidence stay in ~/codex-jobs/xfs-quota-probe; tests touch only a new scratch tree on /srv/quota-probe. I will test setup permissions, accounting, reflinks, query cost, enforcement and restore behavior.First finding: project setup and usage reads have different permissions on the supplied XFS mount (kernel 6.12.113+deb13-amd64).
Verbatim initial probe output:
The server can label directories it owns without capabilities on this kernel. Reading project usage requires a privileged path; a one-time directory setup helper alone is insufficient. I am checking the matching kernel source and accounting semantics next. No product code changed.
Accounting and reflink finding from /srv/quota-probe:
The 92 semantic checks passed against an independent stat walk for settled allocated blocks. The kernel counts allocated inode space, not logical bytes per name. A 16 MiB sparse file with one written byte charged 4096 bytes. Two hard-link names to a 65536-byte file charged 65536, not 131072. Rename and hard link across inherited project boundaries returned EXDEV. Ordinary mv copied and unlinked and inherited the destination ID. Moving a file out into an unprojected directory succeeded but retained its original project ID and charge.
Cross-project cp --reflink=always and FIDEDUPERANGE both succeeded and charged both projects fully. Peer edits/deletion did not immediately change the other project's charge. However, the following controlled result prevents claiming strict dedupe invisibility with raw allocated usage (verbatim):
The writer's own charge can reveal whether shared extents still have a peer. The final report will distinguish this from the successful full-charging test. GNU tar --xattrs --acls also lost source project IDs when restoring into an unprepared tree; restoring into a pre-labelled inherited root used the destination project correctly. No product changes.
Query and enforcement findings:
Warm quota reads do not grow with the number of test projects. With 1, 100 and 10,000 test projects, seven-sample median times for 10,000 requests to the same Home were 43.086, 41.510 and 41.665 ms (about 4.2 microseconds/request including CLI formatting and amortized startup). Random-Home reads were 41.695, 43.140 and 43.509 ms per 10,000. Spawning sudo and xfs_quota for one request cost about 8 ms. This mount-specific measurement ran in one local build slot because the supplied disk exists here; no application perf gates ran. No cold-cache claim: cache dropping is not authorized on this host. The dataset had 1/100/10,000 NEW test projects plus the earlier semantic probe projects, not exactly those total filesystem cardinalities.
Enforcement does work, but the app must handle ENOSPC as well as EDQUOT. Verbatim outside writer result at a 1 MiB project hard limit:
A cross-Home clone over the destination limit also failed with "No space left on device". Matching Linux v6.12 source explicitly returns ENOSPC for project dquots and EDQUOT for other types. The product HTTP server has not been exercised; the raw kernel error alone does not prove clean user-facing error handling.
Final detail checks matched GNU du exactly: 1000 one-byte files used 4,132,864 allocated bytes, of which 36,864 were directory blocks; apparent size was 1000 bytes. A buffered 10,001-byte write was visible to quota inspection before fsync/close as 12,288 bytes. Matching kernel source shows returned usage includes current allocation plus reservations, so concurrent reads are admission accounting, not an exact logical walk snapshot. A first unlink observation also retained an inode charge briefly during reclaim.
Soft limit result after polling the actual allocation error (no fixed sleep):
Startup Q_XGETQSTATV succeeded without capabilities and reported project flags 0x30 (accounting and enforcement on). Missing accounting was tested only through the decoder's simulated state; I did not disable accounting or remount the supplied disk. ext4 project quotas and rsync restores remain explicitly untested (no configured ext4 project-quota mount and no rsync executable). The report will preserve these limits.
Finding for
a6b1fa14on perf-test-02: 3/12 idle windows recorded. Raw result, without rounding:CPU gate PASS: 0.0 <= 0.1 percent. RSS gate PASS: 31854592 <= 100000000 bytes. Scheduler counter coverage PASS. Context-switch counts include the observer boundary stops and exit-stop overhead; these are the complete scheduler-counter proxy, not literal hardware wakeups.
6 cold cache drops recorded, 6 successful. 10 normal-fixture Health checks recorded, 10 with zero unmeasured entries. Failed repetitions are retained and never replaced.
XFS project quotas feasibility — #754 / #731
29 October 2026. Host: 6by32-compute02. Kernel: 6.12.113+deb13-amd64. xfs_quota 6.13.0. Supplied test filesystem: /srv/quota-probe, /dev/loop0, 20 GiB XFS, 4 KiB blocks, reflink enabled, prjquota. All file mutations ran as uid 1002 with CapEff zero; only inspection and limit commands used the permitted sudo xfs_quota. Scratch tree: /srv/quota-probe/xfs-probe-20261029. Scripts: ~/codex-jobs/xfs-quota-probe/probe/.
Decision
NO-GO as a direct replacement for #754's logical, per-filename Home counter, and NO-GO for the strict claim that raw allocated usage cannot reveal cross-user dedupe. The kernel facility itself works and is fast. It could be an allocated-space accounting/enforcement backend if the product explicitly accepts the differences and resolves the dedupe visibility result.
Reasons established by tests:
No product code, commits, pushes, merges, deployments, or new PRs. Tracking branch job/xfs-quota-probe-20261029 points to the base/head
a779a46f42. The shared checkout stayed on dev. Its pre-existing untracked result.md was left untouched.1. Setup and privileges
Test: FS_IOC_FSGETXATTR / FS_IOC_FSSETXATTR on an owned directory, first setting fsx_projid, then FS_XFLAG_PROJINHERIT. Both succeeded with no capabilities. The separate unprivileged xfs_quota project -s -p PATH ID test recursively labelled a populated tree, including its existing nested file. No edits to /etc/projects or /etc/projid were needed.
On this host, the server can set up and repair directories it owns without root, in the initial user namespace. The Linux v6.12 VFS path requires ownership or CAP_FOWNER; it restricts changes to project IDs/inheritance in a noninitial user namespace. CAP_FOWNER is relevant to operating on someone else's inode, not required for the owner. CAP_SYS_ADMIN is required for project quota reads and limit changes. Those distinctions come from VFS file attributes and quotactl permission checks, and the owner/read tests below. Nonowner capability grants and user namespaces were not experimentally varied.
The minimal deployment setup is an administrator enabling prjquota on each backing XFS mount and provisioning a narrow runtime helper with CAP_SYS_ADMIN to read project quotas and administer limits. Do not give the entire server that capability just to query usage. The helper should accept fixed operations on configured mounts and registered project IDs. Empty new Homes can be labelled by the owning server before publication. Alternatively, the helper can perform project -s once per new Home, and after restores or migration of existing trees. Existing nonempty trees need a recursive labelling pass while writers are quiesced. Record a stable unique Home-to-project mapping per filesystem; do not reuse an ID while any charged inode remains, including outside paths and open unlinked files. /etc/projects is a userspace convenience, not a kernel requirement.
Usage read test: direct unprivileged quotactl(QCMD(Q_XGETQUOTA, PRJQUOTA), /dev/loop0, ID, ...) returned EPERM. Unprivileged xfs_quota quota silently returned no rows with exit zero, whereas sudo returned the row. Therefore do not interpret an empty CLI response or exit zero as a zero-byte Home. An unprivileged limit command explicitly failed. Q_XGETQSTATV state inspection did succeed without capabilities.
Verbatim initial output:
Verbatim setup, reflink control, dedupe and restore output (the later sections discuss the same tests):
2. Accuracy, outside programs, renames and hard links
Test: after each operation, compare sudo quota records to an independent bytes-path os.walk/lstat count. The walker reports regular-file logical bytes per name, logical bytes per distinct inode, allocated st_blocks * 512 per distinct inode including directories, and unique inode count. Filenames are bytes, so invalid UTF-8 is supported. File writes were fsynced unless the test explicitly says otherwise. CLI block output is in KiB and converted to bytes; the direct XFS quota ABI uses 512-byte basic blocks. See fs_disk_quota ABI.
The 92 assertions passed in the final semantic run. Settled allocated block counts matched the walk. The initial run observed a recently unlinked inode still in quota after the walk stopped seeing it; inode accounting can await reclaim. The final checks assert it covers the live inode count rather than claiming instant path equality. Source confirms the returned usage includes current usage plus reservations, so it is an admission accounting snapshot rather than an atomic logical tree snapshot. See quota values returned by XFS.
Observed behavior:
Cross-project moves in the server therefore need explicit copy/clone-then-unlink with failure cleanup; they cannot rely on rename atomicity. Staging files must already carry the destination project (prefer staging inside that Home), otherwise rename into its inherited directory returns EXDEV. Distinct projects prevent hard links between Homes, but same-UID tools can relabel files or clear inheritance. This is accounting for correctly labelled, trusted OS writers, not a boundary against an inode owner deliberately escaping a quota. Restrict access to unprojected directories and audit/repair labels after administrative operations.
Verbatim complete final semantic run:
3. Reflink and dedupe privacy
Tests: GNU cp --reflink=always across projects, plus direct FIDEDUPERANGE with a 4 MiB identical range. Both succeeded; FIDEDUPERANGE returned status 0 and 4194304 bytes deduplicated. Both projects continued to charge the full 4194304 bytes. Clone edits preserved source content. Deleting either peer left the surviving peer fully charged. The full output appears in sections 1 and 2.
The control test used three separate pairs: no clone; live clone in another project; clone created and then deleted. Each original was 4 MiB, then received the same 4096-byte overwrite followed by fsync/close. Verbatim:
This distinguishes the existence of a peer through the writer's resulting usage. No claim is made that an idle surviving file's quota changes directly on peer deletion; that did not happen in the tested sequence. The failed strict-privacy condition is the subsequent local edit. Reducing CoW hints, trimming reservations, or hiding/coarsening displayed usage were not tested as fixes; they would need separate privacy and enforcement tests. Raw XFS usage must not be represented as proven invisible dedupe for T-A2.
4. Read cost at 1, 100 and 10000 projects
Test: create 1, then 100, then 10000 newly labelled projects, retaining earlier semantic-probe projects. These are dataset sizes, not exact totals on the filesystem. For each stage run one quota read, 10000 repeats of one Home, and 10000 seeded random Home reads through xfs_quota quota -p. One warmup then seven measured processes; check every requested row was returned. All output was captured. The batch includes formatting and amortized sudo/CLI startup. No privileged direct-syscall benchmark was possible under the sudo allowlist; the unprivileged direct syscall only tests EPERM.
Warm lookup cost is effectively constant per Home over this range, around 4.2 microseconds/request for the batch, easily small enough for upload admission. Single-process-per-upload CLI use costs about 8 ms; a persistent helper avoids most startup overhead. The kernel uses an indexed per-ID dquot lookup, with radix-tree cached records, rather than walking the Home or all projects. See XFS dquot lookup. This supports the requested O(1)-per-Home operational conclusion; it is not a proof of constant cold disk latency under every filesystem size or contention level. Mutex contention, cold cache, many-mount summation and production load were not measured.
This specific local mount measurement ran through timeout 360 /usr/local/bin/build-slot.sh python3 .../query-cost.py, holding one build slot. The task specifically supplied the local mount for this measurement; no application perf harness or VM performance gate ran. No cache drop was authorized on this host, so these are warm results only, not cold or production budget claims.
Verbatim output:
5. Enforcement and application errors
Tests: set a 1 MiB hard limit; a separate Python process with O_SYNC repeatedly writes 4096-byte blocks. It stopped after exactly 1048576 bytes with ENOSPC (28). Creating another file under the exhausted project also returned ENOSPC. Cloning a 2 MiB file into a project with a 1 MiB hard limit failed with the same error. Thus outside tools are constrained, including reflinks.
XFS project quota failure is not EDQUOT on this kernel. The project-specific ENOSPC return is explicit in XFS quota reservation; other quota types return EDQUOT. A tiny 64 KiB limit even rejected file creation at open before any data write, consistent with filesystem reservation needs. A limit need not translate to exactly usable payload bytes, especially with directories, metadata and staging/replacement space.
With bsoft=1m and bhard=4m, a 2 MiB write was allowed during the default seven-day grace. Only the test project's timer was changed to one second; poll allocations until the actual refusal, not a fixed sleep or rounded CLI timer text. The expired soft limit returned ENOSPC. Global grace was not changed, and the probe limits were cleared afterwards.
The OS caller gets an ordinary catchable I/O error and may have a partial file. The product server's HTTP error response is untested. A future adapter must map ENOSPC/EDQUOT, clean temporary/partial files, and handle errors at create/write/fsync/clone/rename. Distinguish a quota refusal from a genuinely full mount via the helper where useful. Current core error catalogue maps StorageQuota to HTTP 413, and promises small files/notes still save; a blanket filesystem hard limit blocks allocations from those writers too. Preserve that policy deliberately if adding enforcement; this feasibility probe does not change it.
Verbatim hard-limit and initial soft-grace output:
Verbatim successful soft-grace enforcement test:
6. Inheritance, restore and absent accounting
Tests: a new subdirectory inherited both the project ID and PROJINHERIT; its new file inherited the ID. A project-zero file could not rename into the inherited Home: EXDEV. GNU mv then copied it into a newly created inode with the Home's ID and removed the source. The project ID is not silently lost on a normal rename into a correctly inherited root; that rename is rejected. Moves outward and explicit owner relabelling are the converse risks described in section 2.
GNU tar 1.35 --xattrs --acls -cpf / -xpf did not preserve the source project ID or inheritance in a newly created, unprepared destination. Both nested dir and file had project zero. The exact same archive restored into a prepared inherited root used its destination project, including inherited subdirectories. Running unprivileged project -s recursively after restoring repaired the unprepared tree. These are ioctl-backed inode attributes, not ordinary named xattrs that --xattrs alone promises to preserve. Do not admit writes based on restored quotas until recursive setup/check finishes. rsync is not installed and was not tested; do not assume rsync -a or -X preserves these attributes. Reapply project setup after any restore unless that restore path is specifically verified. The actual tar and inheritance output is in sections 1 and 2; project -c verification is below.
Startup test: direct unprivileged Q_XGETQSTATV reported flags 0x30: project accounting ON, enforcement ON. Decoder tests for zero flags and accounting-only flags are explicitly simulated. A live no-prjquota XFS mount was NOT tested: no mount privilege is available, and turning accounting off would require remounting to restore it. I did not alter the supplied mount's accounting or enforcement state.
At startup, inspect each configured data mount, query Q_XGETQSTATV and require FS_QUOTA_PDQ_ACCT for this backend; require FS_QUOTA_PDQ_ENFD too if relying on kernel hard limits. Confirm the privileged usage read actually works and validate the Home's project ID/inheritance. Handle unsupported/version/error states explicitly; EPERM means the reader lacks permission, not that usage is zero. If accounting is absent or unavailable, use the requested full-Home recount backend on change and before upload admission while a change is pending. Keep pending state through any concurrent changes and serialize upload reservations. An unavailable watcher needs a recount before every admission or an explicit readiness failure, not a falsely clean cached count. If the fallback remains logical while XFS is allocated, backend changes change user-visible usage: this is another reason not to ship a silent direct substitution.
For #731's two-disk Home, each filesystem has independent project accounting. Label both trees, sum both mount records and coordinate admission/reservations across them. Giving each mount the full Home limit permits twice the global quota. Tiering moves across mounts require copy/unlink and can temporarily need both charges. Those multi-mount crash and concurrency cases were not exercised here.
Verbatim startup test:
Verbatim GNU du, buffered-write and recursive project-check tests:
7. ext4
Untested. A read-only mount inventory showed / is ext4 with rw,relatime,discard,errors=remount-ro, without configured project quotas. There is no supplied ext4 project-quota test mount. No ext4 image was made or mounted, and no test files were written to another disk for this question. XFS results must not be generalized to ext4. rsync was also explicitly unavailable.
Verbatim inventory:
Evidence and limits
Scripts are one-shot probes with fixed, reserved scratch names/project IDs; use a fresh namespace before rerunning. The query dataset uses IDs 1760000 through 1769999; semantic/setup tests use IDs near 1754001 through 1754405. Scratch files were retained for inspection and limits cleared. Nothing was committed to the repo.
Harness corrections are preserved in the raw logs: the first parser read the wrong CLI column; an initial strict inode-equality assertion exposed deferred reclaim; enforcement initially assumed EDQUOT and then assumed rounded zero timer text meant the kernel had expired it. The final tests use the correct column, state the reclaim limitation, check ENOSPC, and poll the actual allocation error. These are not hidden application test successes. The early runs are in semantics-first.log / extra.log / enforcement.log; successful final evidence and explicitly labelled excerpts are reproduced above. The extra and enforcement scripts were corrected afterwards, syntax checked, and a fresh independent verification gate below rechecks the actual filesystem evidence. No full product test suite was needed or run because product code was unchanged.
The remaining untested items are ext4 project quotas, rsync-specific preservation, a live XFS mount without accounting, privilege-granted/nonowner/user-namespace variations, product HTTP behavior, cold cache, restart/crash persistence and multi-mount admission. The no-go conclusion does not depend on those gaps: logical/allocated incompatibility and the controlled sharing-dependent charge were directly observed.
Completion and final gate
Finished. Head SHA:
a779a46f42(unchanged from base). Fresh verification reran the sharing control, hard-limit error, live usage comparisons and permission/state checks. This is a probe evidence gate, not a claim that product builds or the proposed replacement pass. Verbatim gate output:Started the approved whole-Home recount redesign on perf/quota-simple, based on
a6b1fa14d2(fetched origin/perf/idle-honest-2 and verified it matches). Existing crates/fsio/src/usage.rs plus usage/*.rs: 5,987 lines. First checking generation acceptance against delayed native notifications and mount operations, then implementation and gates if that acceptance rule holds.Finding for
a6b1fa14on perf-test-02: 4/12 idle windows recorded. Raw result, without rounding:CPU gate PASS: 0.0 <= 0.1 percent. RSS gate PASS: 30539776 <= 100000000 bytes. Scheduler counter coverage PASS. Context-switch counts include the observer boundary stops and exit-stop overhead; these are the complete scheduler-counter proxy, not literal hardware wakeups.
6 cold cache drops recorded, 6 successful. 11 normal-fixture Health checks recorded, 11 with zero unmeasured entries. Failed repetitions are retained and never replaced.
Finding: unchanged callback generations do not prove containment. A real native inotify queue can retain events until after the counter accepts its walk. This is ordinary callback scheduling delay; there is no overflow and no lost event. I opened a child directory fd, moved that child outside Home, and created a new 1,009-byte file only outside Home. Before releasing the event consumer, the required before/after rule accepted 1,013 bytes although the quiescent Home oracle was zero. The counter would be marked counted at generation 1. Three native events were delivered afterward. The current descriptor location validation addresses this case; deleting it requires a synchronous native event fence, not merely an atomic in the callback. A timeout, debounce, or three attempts does not help: the first attempt sees unchanged generations and succeeds.
Command: python3 generation-race.py
Output:
This reproduces the specified replacement acceptance rule with real kernel events; it does not claim that the existing chain-verifying scanner accepted these bytes. Reproduction source follows.
Finding: same-device bind mounts do not produce the native notification that generation acceptance needs. I verified this in an isolated user and mount namespace on this host; no host mount changed. Every existing Home file and directory was watched for every inotify event. After opening a child directory fd, a bind mount covered that child. The mount returned with zero bytes pending in the native event queue. The opened child fd still had the Home root mount ID and yielded seven now-hidden bytes, while the current path had a different mount ID and the required mount-excluding oracle was zero. Comparing st_dev and mount IDs on each object being walked does not detect a mount placed over a directory already opened. This needs mount-namespace change detection or renewed path validation; inotify overflow handling cannot detect an operation that produces no event.
Command: unshare --user --map-root-user --mount python3 mount-race.py
Output:
Reproduction source follows.
Finding: admission can use a stale total even with counted_gen equal to dirty_gen. With a cached four-byte Home and a ten-byte quota, an external write grew its file to 100 bytes. While its native notification was queued, both generations remained 1. The specified freshness condition did not request a recount and admitted another byte. The native queue contained a modification event, so this does not depend on an event being lost. Admission needs the same synchronous event fence before checking freshness, including when no recount would otherwise run.
Output:
Reproduction source follows.
Startup finding at 10000 notes on perf-test-02,
a6b1fa14versus devd83e54db. Requested three cold and five warm starts per revision are complete. Every cold launch had a successful cache drop. Raw numbers without rounding; nearest-rank p95 is the maximum for these sample counts. Failed repetitions remain.Blocked finish for this unattended job. The three executable counterexamples above show that the specified acceptance and admission rules do not satisfy the required outside-byte, mount, and no-stale-admission assertions. I have not replaced safe containment validation with a rule demonstrated to accept invalid counts. The needed refinement is a synchronous native event fence at admission and recount acceptance, plus mount-namespace change detection (or retaining path validation for this case). Sleeping to allow callbacks to catch up is not a correctness bound.
Branch: perf/quota-simple. Base and current head:
a6b1fa14d2. Fetched origin/perf/idle-honest-2 before inspection. Product diff is empty and the worktree is clean. No tests were removed. usage* line count before: 5,987; after: 5,987. Reproduction source and exact output are retained in issue comments, and the temporary local scripts were removed.Gate commands were not run, so there is no gate output to quote. fmt --check: NOT RUN. clippy -D warnings: NOT RUN. cargo test -j3 -p calternal-fsio: NOT RUN. 50,000-case property test: NOT RUN. Full workspace tests and #757 rerun: NOT RUN. Recount timing and strace measurements: NOT RUN because no replacement was implemented. These are uncompleted gates, not passes.
No commit, push, draft PR, merge, or deploy was made. A draft replacement PR would have no changes to review. Evidence and this blocked status have been posted on both #754 and #753. The requested redesign remains unfinished.
Starting the approved whole-Home recount revision on branch perf/quota-simple, base
a6b1fa14(origin/perf/idle-honest-2 fetched). I will add event/admission barriers and mount-table invalidation, retain applicable reviewer regressions, and run the requested gates. No merge or deployment.Finding: the requested Home-root sentinel barrier does not cover writes through hard links outside Home. A native inotify reproducer observed the sentinel with zero other events, accepted 1,014 bytes while the quiescent tree contained 2,022, and admitted a 1-byte upload against a 1,500-byte quota. An unreadable but writable regular file can still be measured by stat, but inotify_add_watch on that inode returns EACCES. I will preserve per-inode event coverage on the same inotify fd; a Home whose inode watches cannot be installed must stay stale for admission, rather than trusting an unfenced count. Its metadata bytes can still be reported and Health can report unavailable coverage. This is a conservative choice for the otherwise ambiguous degraded-watcher case. Full reproducer and output will be included with the final evidence.
Finding for
a6b1fa14on perf-test-02: 5/12 idle windows recorded. Raw result, without rounding:CPU gate PASS: 0.0 <= 0.1 percent. RSS gate PASS: 30511104 <= 100000000 bytes. Scheduler counter coverage PASS. Context-switch counts include the observer boundary stops and exit-stop overhead; these are the complete scheduler-counter proxy, not literal hardware wakeups.
6 cold cache drops recorded, 6 successful. 14 normal-fixture Health checks recorded, 14 with zero unmeasured entries. Failed repetitions are retained and never replaced.
Finding from the seeded mount property: an unmount can make mountinfo poll ready without generating an inotify event. The reader first polled mountinfo in its blocking poll set, then polled it again in a helper; the first poll had already consumed the namespace event indication. The test consequently admitted using 0 bytes after unmount exposed 26 bytes. Exact assertion:
assertion left == right failed: unmount case 0; left: 0; right: 26. Fix: retain the original poll revents and invalidate Homes when either that original result or the sentinel-time nonblocking poll reports the change. This is now a permanent property regression and will be rerun at 50,000 cases.Finding from independent static review, reproduced as a permanent test: statting inode A then installing a watch through its mutable name can cache inode B's watch under A. After A returns to Home, an outside hard-link write is missed. The deterministic registration-swap test failed with
the watched inode must be the charged inode; left: 1; right: 1009. Fix: pin each NEW regular-file watch with O_PATH|O_NOFOLLOW, verify its identity, and add the watch through that trusted proc fd. Warm recounts reuse the observed identity and need no extra file open. Review also found unchecked cache lookups racing overflow; registration now returns its watch descriptor directly, and private watch lifecycle events are handled before filtering sentinel traffic.Starting the continuation on perf/quota-simple at
bb36381dd1(base from the previous job). I will commit docs/plans/0754-quota.md before more code, cover the review findings through 754g and comment 69219, then run the required gates. No merge or deployment.Plan committed first at
3f7eb017. The unpublished WIP is preserved in a local backup ref; this branch now starts with the plan and excludes the prior job's accidentally committed artifacts.Finding: a deleted directory's inode can be reused before its queued IN_IGNORED event is read. The watch cache then mistakes the replacement private state directory for the retired watch. The seeded property made almost no progress because recount fences repeatedly waited for their full 30-second budget. A debugger showed recount_home waiting in barrier while the single event reader was idle in poll. A 20-case strace reproducer shows openat of the replacement state folder and sentinel without an inotify_add_watch for that state folder; only its parent CREATE arrives. Counts remain stale, so this is a recovery and cost defect, not evidence of an unsafe admission.
Small fix: refresh directory watches from their already pinned fd rather than reusing numeric dev/ino alone. A live watch returns the same descriptor and does not dirty the Home; a retired watch is reinstalled and dirties it. Warm regular-file walks keep their two metadata calls per entry. Permanent watch-retirement and property checks will verify this.
The watch-retirement regression failed before the fix (old and current watch both 1) and passed afterward. The same 2,000-case property, including private-namespace mounts and delayed delivery, now passes in 3.76 seconds rather than spending 30 seconds on repeated missing fences.
Finding: the unfinished admission timeout covered only the upload queue and fsio freshness check. It excluded both plugin byte providers and the database queue. Permanent tests reproduced admission exceeding two seconds behind the database queue and returning success after a provider blocked for ten seconds. The fix applies one 1,900 ms timeout to the complete admission decision and runs provider work on a blocking thread. Both regressions now pass; result: ok. 2 passed; 0 failed; finished in 1.92s. The existing Internal error remains retryable through the HTTP catalogue.
Finding: after a stable recount failure, attempted_gen equaled dirty_gen and the worker suppressed its next retry. Native coverage could be healthy while a depth or measurement failure stayed stale forever without another event. A new regression failed with “unmeasured Home lost its retry” after two recounts of one over-depth Home; a separate healthy Home had already been counted.
Fix: stale Homes retain one debounce deadline until an accepted complete count. Clean Homes have no deadline. Native coverage failures still use the 60-second fallback, and recovery disables it. Background walks now have a per-Home 30-second budget so a large initial count can complete independently of a short admission; requests still return retryably within two seconds. The retry regression passes. A companion Auth property passed 2,000 seeded outside-write, rename and hard-link cases against the real quota decision, rather than merely checking arithmetic in fsio.
Two more permanent regressions exposed gaps in the inherited replacement.
Both failing tests ran before these fixes. The complete fsio suite is being rerun. Watch installation, sentinel fences, event consumption and native tests have also been split into small files, without changing the interfaces.
The first workspace run found three quota-display failures. Root cause was a missing part of the required app-write contract: ordinary atomic writes, streamed placements and compare-and-swap saves still called changed(path) with zero bytes. A permanent regression failed with cached 0 versus a five-byte app publication. These paths now pass their known written lengths to the existing wrote hook, provisionally adding bytes and invalidating the generation without scanning the Home.
After that fix, the duplicate-upload and backdrop tests pass unchanged. The remaining storm test sees 2,097,319 cached bytes immediately after deleting a 165-byte note, versus the eventual exact 2,097,154. Deletions are now confirmed by debounce, so its two display-after-deletion assertions wait on the exact same expected values via people.list, with a bounded condition loop and no fixed sleep. Upload admission assertions remain synchronous and unchanged.
A further watch-lifecycle regression also failed: a permanently unmeasured Home retained 89 watches after starting with 69 when 20 files were moved outside and replaced. Watch pruning had only run after complete accepted counts. Failed walks now explicitly invalidate freshness and prune their last observed watch set too. They still cannot authorize admission. The complete fsio suite then passed: 103 passed, zero failed, one existing ignored child helper, plus all mount integration tests.
Reviewer test inventory for the whole-Home replacement. Names below are no longer present at their old internal locations; each line states its replacement or the removed internal mechanism. Existing public mount integration tests remain unchanged from
a6b1fa14.754g was not merged into
a6b1fa14; its behavioral cases are explicitly carried forward in review_containment and lifecycle_tests. Its remaining internal-only cases are accounted for here:Other behavioral adaptations, listed rather than claimed unchanged:
a6b1fa14and does not claim byte-for-byte preservation from their earliest review branch.The 50,000-case gate passed for both properties. Auth: 23,398 admitted and 26,602 quota-refused; 34.10 seconds. Filesystem: all 50,000 cases passed, including delayed delivery, users/Home renames, true exchanges and private-namespace mounts; 81.53 seconds.
The next full workspace run exposed one old global-counter assumption in health_reports_retained_storage_measurement_failures: its failing tree belonged to users/unmeasurable, but it expected admission for the unrelated signed-in person's Home to fail. Independent per-Home admission is required by this design. The fixture now creates that same over-depth subtree inside the signed-in person's Home, preserving every Health, refusal and recovery assertion. Other-Home independence is covered separately. No product behavior changed for this adjustment. Full workspace tests are being rerun; the Tasks flake has not occurred so far.
The workspace rerun reached an intermittent seed-upload assumption in the shipping isolation tests. Concurrent seeding and rename jobs legitimately produced HTTP 500 with error.code=internal and retryable=true; the fixture asserted immediate 201. The test upload helper now retries only explicitly retryable HTTP 500 responses within ten seconds, without fixed sleeps. Permanent quota refusals and all isolation assertions remain unchanged. The dedicated 754f HTTP tests still observe and assert retryable admission errors directly. The next full workspace run uses --no-fail-fast so all remaining compatibility checks finish together.
The --no-fail-fast workspace run identified five additional old-counter assumptions in reliability fixtures. Four read cached totals immediately after deletions or cross-Home moves (30 vs 17, 10,000 vs 9,999, 4,096 vs 0, and rename replacement). Those fixtures now call ensure_home before their existing exact byte assertions, matching the public confirmed-total contract. One concurrent-writer fixture unwrapped every recount under sustained churn; it now accepts only the specified bounded WouldBlock/TimedOut result during churn, then requires successful freshness and the same exact total after all writers stop. None of these changes relaxes the byte or quota assertions. The full run is continuing to collect every remaining result before rerunning these adjusted fixtures.
The full run found the same provisional-display assumption in three contacts undo/recovery tests: the helper compared an unfenced cached total (517/537/4686) with a complete recount (318/332/4464). Its existing history-database oracle and exact assertions remain; the helper now confirms the Home with ensure_home before comparing confirmed totals. The auth security deletion fixture similarly performs its existing successful fresh admission before reading the exact 31-byte display. No contacts, auth or quota product code changed for these fixture adjustments.
The complete workspace run passed: exit 0 in 296.66 seconds. The Tasks kill-9 test passed, so #757 did not need a rerun.
A final permanent failure-case test then found a Health gap: replacing the private sentinel directory with a regular file correctly refused admission, but the initial admission fence returned before retaining the unavailable coverage. The regression failed with “failed fence must report unavailable coverage”. Admission now falls through to its existing bounded recount on that fence failure; the same recount failure path records polling Health and recovery. A dead native reader still fails immediately. This does not authorize a failed fence or trust a stale total. I am refreshing all required gates after this small fix.
Local recount measurements on 6by32-compute02 — not a perf VM number.
Single dev-build samples, one Home, 17 logical bytes per file, one 10,000-entry folder or two 50,000-entry folders. All runs used one build slot. No cache drop and no cold-cache or server latency claim. Timing comes from untraced runs; syscall counts come from separate strace -f -c runs. Initial means a new counter installs its inode watches; warm means those watches already exist. Both counters reported exactly 170,000 / 1,700,000 bytes.
Entry denominators include the ordinary folder entries (10,001 and 100,002); all fixed probe and fence overhead remains in the syscall totals. Per-file values are 2.0123 and 2.00194. Warm walks have one fstatat and one statx per file, and no openat2 calls.
Comparison on
a6b1fa14with the same public recount probe and datasets: initial/warm untraced times were 172.820904/150.731700 ms at 10k and 1598.787518/1598.861014 ms at 100k. Its successful warm 10k trace had 120,264 calls (12.025197480 per entry). These single samples are not a statistical performance gate.Tracing the initial 100k count exceeded the 30-second recount deadline on both revisions. Those partial traces are not accepted counts and are not used for syscall-per-entry results. The new complete warm 100k trace succeeded in 8985.617739 ms under instrumentation; its untraced wall sample is the 522.725821 ms value above. Initial watch setup is slower than the old initial count in these samples; startup remains asynchronous and first uploads may return a bounded retryable result.
Warm 10000-file strace output, verbatim:
Warm 100000-file strace output, verbatim:
The latest full run found one more test-only quota assumption in the contacts Limit fixture. undo_of_a_contact_change_is_held_to_the_limit set its artificial quota from an unfenced conservative display total. The later real recount correctly reduced those bytes, leaving room under that artificially enlarged test limit and allowing undo. The fixture now confirms bytes before deriving limits or comparing post-write totals, and its test UploadPolicy uses ensure_home like Auth. The expected StorageQuota assertion and undo behavior remain unchanged. The real Auth 50,000-case property still passed every case; this finding does not show product admission accepting an over-quota upload.
Publication check: origin/perf/quota-simple is at
bb36381dd1, and there is no open PR for that branch. I earlier described the saved WIP as unpublished; that description was wrong. The old WIP already contains the job artifacts. The final branch history starts ata6b1fa14with the plan commit, followed by product code and tests only. Publishing it will use --force-with-lease=refs/heads/perf/quota-simple:bb36381dd14878c3bdafba1794f4ef89eb0f07a8, so any concurrent branch change rejects the push. The original WIP is preserved in a local backup ref. Only perf/quota-simple will be pushed.Two permanent users-root boundary regressions failed after the otherwise green workspace run.
Small fixes: missing roots return the already-known Home ids for recount, so their totals are cleared through the normal fence. A single discovery-failure flag retains failed enumeration or anchor coverage in Health and keeps the existing 60-second fallback active even with no registered Homes. Successful discovery clears it. Both regressions now pass, including recovery. The walk, byte rules and admission design are unchanged. I am running the full required gates again after these boundary fixes.
Gate output verbatim for
798f1bbe91; fmt; part 1/1; exit 0.Command:
cargo fmt --all -- --checkGate output verbatim for
798f1bbe91; clippy; part 1/1; exit 0.Command:
cargo clippy -j3 --workspace --all-targets --locked -- -D warningsGate output verbatim for
798f1bbe91; fsio; part 1/1; exit 0.Command:
cargo test -j3 -p calternal-fsioGate output verbatim for
798f1bbe91; property; part 1/1; exit 0.Command:
QUOTA_PROPERTY_CASES=50000 cargo test -j3 -p calternal-fsio -p calternal-auth --lib seeded_whole_home_property -- --nocaptureGate output verbatim for
798f1bbe91; workspace; part 1/3; exit 0.Command:
cargo test -j3 --workspace --locked --no-fail-fastGate output verbatim for
798f1bbe91; workspace; part 2/3; exit 0.Command:
cargo test -j3 --workspace --locked --no-fail-fastGate output verbatim for
798f1bbe91; workspace; part 3/3; exit 0.Command:
cargo test -j3 --workspace --locked --no-fail-fastFinished on perf/quota-simple at
798f1bbe91. Draft PR: #764. The plan was committed first in3f7eb017. All required gates passed; their complete output is in the preceding comments. Both 50,000-case properties passed on the final code. The Tasks kill-9 test passed; #757 did not reproduce, so no flake rerun was needed. Local measurements are explicitly not perf VM results; initial 100k traces timed out and were excluded. Test inventory and measurement evidence are linked in the PR. No merge, deployment, issue closure, or push to main/dev. The own-branch update used an exact lease on the previous WIP SHA, and the published history contains no job artifacts.kayg referenced this pull request2026-11-05 23:21:11 +00:00
housekeeping #1009: needs-decision.
Incremental quota counter is explicitly replaced by whole-Home recount WIP #764 and amendment #769. Retain hard-link/move regressions; do not replay the discarded counter.
original head:
a6b1fa14d21944b25fa1538615501f414ae9ddbf. against devcd7f84432febfc63b788fbb9b9f9950452982d74, git cherry reports 17 unmatched and 0 equivalent patches. the landing audit lists each candidate missing commit and stable patch match inpr-754.md; unmatched patches can include superseded behavior.left open, source branch unchanged. no rebase or new test claim for this decision lane.
Superseded by whole-Home recount #764/#769. The branch and original tips are archived under
archive/(see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.Pull request closed