kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

storage: portable pinned readers and resilient Notes scans (#761 B) #791

Closed
kayg wants to merge 12 commits from feat/tiering-761-b-reader into feat/tiering-761-a-migration
pull from: feat/tiering-761-b-reader
merge into: kayg:feat/tiering-761-a-migration
Conversation 9 Commits 12 Files changed 46 +558 -64
kayg commented 2026-10-31 16:15:27 +00:00
Owner
Copy link

Plan — pinned readers and resilient note discovery (#761), B of A → B → C → D

Stacked on A, ultimately into dev. Separate runtime reader behavior from the offline migration in #784. Keep the existing identity index here; C changes its representation.

  • Pin each Home/Notes root by descriptor, device, inode and owner (plus Linux mount id). Revalidate the named root and caller authority before returning bytes.
  • Linux opens below the descriptor with openat2 BENEATH/NO_SYMLINKS/NO_MAGICLINKS/NO_XDEV. macOS opens each component with openat/O_NOFOLLOW and enforces the same device.
  • Do not follow symlinks in Notes: a folder link could escape the Home or reach another person. The migration preserves links as links; discovery reports and skips them. Keep raw on-disk UTF-8 spelling; use Unicode folding only for lookup, including old NFC ids.jsonl entries with NFD filenames.
  • Skip an unreadable/vanishing entry or non-UTF-8 name without failing the Home scan; report skipped reads/names. Failure to list the Notes root or root replacement remains an incomplete/retryable scan, never mass deletion.
  • Resolve content, metadata, scan enumeration and polling through the retained Root. Polling opens each regular note for descriptor metadata without reading its contents. No async request waits for a whole-tree scan.

Cost: O(depth) descriptor operations per read (one openat2 on Linux); O(entries × depth) descriptor operations for metadata discovery and polling in background, including absolute-root validation; content reads add O(bytes). Raw names do not gain normalization copies. Inventory access remains an explicit privileged method; product paths reject hidden names.

Failure → effect → detection → recovery → test

Failure Effect / detection Recovery Test
Missing root / hidden or escaping path Refuse before reading; never create root Restore mount/path reader::tests::missing_root_is_never_created_and_protected_names_are_never_served
Symlink / mount escape / sparse or aliased file No symlink traversal; regular descriptors and full logical lengths Operator resolves symlink; ordinary hard links readable reader::tests::nofollow_raw_identity_aliases_and_sparse_lengths
Replaced root or authorization revoked Validation fails; no descriptor returned Restart after coordinated root/mount maintenance reader::tests::pinned_descriptor_survives_replacement_and_rechecks_auth_and_roots
One unreadable entry / symlink / non-UTF-8 name Report/skip; other notes index and Home completes Fix entry; later scan rediscovers it one_unreadable_or_odd_entry_never_stops_the_scan
Note disappears after inventory Skip its head, keep other discovered ids Reconcile next changes vanished_note_does_not_drop_other_discovered_heads
Mac-origin decomposed filename and legacy normalized log Raw path retained; folded lookup keeps id No rewrite required migrated_decomposed_name_keeps_its_id_and_remains_readable
Directory rename/replacement while walking Root validation refuses incomplete walk Retry on a stable root Reader replacement test and inherited scan tests

Validation

PASS at 58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8 on dev-01: fmt, clippy with -D warnings, and the full workspace suite; 1187 passed, 0 failed, 20 ignored, job exit 0. All heavy commands ran in Nix through ~/bin/slot. Evidence: ~/tier-r2-final-b.log and ~/tier-r2-final-b.exit.

cargo fmt --all --check
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings
cargo test -j4 --workspace --locked --no-fail-fast

Round-2 findings

Finding Fix Failing-before / passing-after evidence
R2-B1 P1, ancestor symlink metadata escape Remove path-parent re-rooting adapter. All note reads and metadata opens use the retained Home/Notes Root, with descriptor metadata. Root pin/validation walks absolute ancestors with openat/O_NOFOLLOW; relative resolution shares Linux openat2 BENEATH/NO_SYMLINKS/NO_XDEV or the macOS no-follow component walk. Enumeration uses retained directory descriptors and no-follow statat. r2_metadata_refresh_never_reads_through_an_ancestor_symlink reproduces Sub/Deep/a.md, swaps Sub for a foreign-tree symlink, rejects refresh and preserves title/snippet/link index. r2_a_root_ancestor_symlink_cannot_repin_authority covers an alias pointing back to the same moved root. Both failed on original B, pass after.
R2-B2 P2, replaced root reports complete Validate retained authority before/after enumeration and before scan reconciliation, metadata commit and state save. Root replacement propagates an incomplete/retryable scan error. r2_replaced_root_scan_stays_incomplete_and_keeps_indexes_and_state swaps Notes after Home construction, requires incomplete status, identical saved state/list and no foreign index entry. Original B failed, fixed B passes.
Inherited R2-A1–A4 Rebased on fixed #790. See #790 status table; inherited migration regressions run in this workspace gate.

Earlier broader Mac suites were not green: Linux /proc/self/task assumptions, delayed WAL/SHM cleanup, and the case-only collision expectation remain separately recorded portability failures. No final-head Mac pass is claimed.

Decisions and remaining risks

  • Root/mount/ancestor changes require offline coordination. Linux detects a remount and asks for restart; no privileged remount campaign was run. macOS lacks Linux mount-id identity and enforces device + inode + owner.
  • Product note discovery skips non-UTF-8 names with a diagnostic; offline migration copies their byte names intact. Links remain on disk but are not traversed by discovery or reads.
  • Unicode folding still permits historical portable-name collisions; the index's shadowing rules remain. No arbitrary ancestor-move sandbox or full NFC/NFD conflict migration is claimed.
  • Full privileged mount races, 100k-note poll measurements, successful-open/FO1 and final perf-VM qualification remain owed. No cutover authorization.

Stack: A #790 → B #791 → C #792 → D #793. Independent review of this replacement stack remains pending.

Final-head macOS checks were not run: the shared lock is held by dav-recorder PR 789. Earlier targeted Mac results apply only to older heads. This change rejects symlinked ancestors, including aliases to the same physical root; configure physical paths. Real media errors, power cuts, 100k-note polling costs and latency qualification remain unrun.

Stack: A #790 → B #791 → C #792 → D #793. Ready for final round-3 review after these fixes; nothing merged.

## Plan — pinned readers and resilient note discovery (#761), B of A → B → C → D Stacked on A, ultimately into `dev`. Separate runtime reader behavior from the offline migration in #784. Keep the existing identity index here; C changes its representation. - Pin each Home/Notes root by descriptor, device, inode and owner (plus Linux mount id). Revalidate the named root and caller authority before returning bytes. - Linux opens below the descriptor with openat2 BENEATH/NO_SYMLINKS/NO_MAGICLINKS/NO_XDEV. macOS opens each component with openat/O_NOFOLLOW and enforces the same device. - Do not follow symlinks in Notes: a folder link could escape the Home or reach another person. The migration preserves links as links; discovery reports and skips them. Keep raw on-disk UTF-8 spelling; use Unicode folding only for lookup, including old NFC ids.jsonl entries with NFD filenames. - Skip an unreadable/vanishing entry or non-UTF-8 name without failing the Home scan; report skipped reads/names. Failure to list the Notes root or root replacement remains an incomplete/retryable scan, never mass deletion. - Resolve content, metadata, scan enumeration and polling through the retained Root. Polling opens each regular note for descriptor metadata without reading its contents. No async request waits for a whole-tree scan. Cost: O(depth) descriptor operations per read (one openat2 on Linux); O(entries × depth) descriptor operations for metadata discovery and polling in background, including absolute-root validation; content reads add O(bytes). Raw names do not gain normalization copies. Inventory access remains an explicit privileged method; product paths reject hidden names. ## Failure → effect → detection → recovery → test | Failure | Effect / detection | Recovery | Test | |---|---|---|---| | Missing root / hidden or escaping path | Refuse before reading; never create root | Restore mount/path | `reader::tests::missing_root_is_never_created_and_protected_names_are_never_served` | | Symlink / mount escape / sparse or aliased file | No symlink traversal; regular descriptors and full logical lengths | Operator resolves symlink; ordinary hard links readable | `reader::tests::nofollow_raw_identity_aliases_and_sparse_lengths` | | Replaced root or authorization revoked | Validation fails; no descriptor returned | Restart after coordinated root/mount maintenance | `reader::tests::pinned_descriptor_survives_replacement_and_rechecks_auth_and_roots` | | One unreadable entry / symlink / non-UTF-8 name | Report/skip; other notes index and Home completes | Fix entry; later scan rediscovers it | `one_unreadable_or_odd_entry_never_stops_the_scan` | | Note disappears after inventory | Skip its head, keep other discovered ids | Reconcile next changes | `vanished_note_does_not_drop_other_discovered_heads` | | Mac-origin decomposed filename and legacy normalized log | Raw path retained; folded lookup keeps id | No rewrite required | `migrated_decomposed_name_keeps_its_id_and_remains_readable` | | Directory rename/replacement while walking | Root validation refuses incomplete walk | Retry on a stable root | Reader replacement test and inherited scan tests | ## Validation **PASS at `58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8`** on dev-01: fmt, clippy with `-D warnings`, and the full workspace suite; **1187 passed, 0 failed, 20 ignored**, job exit 0. All heavy commands ran in Nix through `~/bin/slot`. Evidence: `~/tier-r2-final-b.log` and `~/tier-r2-final-b.exit`. ```sh cargo fmt --all --check cargo clippy -j4 --workspace --all-targets --locked -- -D warnings cargo test -j4 --workspace --locked --no-fail-fast ``` ## Round-2 findings | Finding | Fix | Failing-before / passing-after evidence | |---|---|---| | R2-B1 P1, ancestor symlink metadata escape | Remove path-parent re-rooting adapter. All note reads and metadata opens use the retained Home/Notes Root, with descriptor metadata. Root pin/validation walks absolute ancestors with openat/O_NOFOLLOW; relative resolution shares Linux openat2 BENEATH/NO_SYMLINKS/NO_XDEV or the macOS no-follow component walk. Enumeration uses retained directory descriptors and no-follow statat. | `r2_metadata_refresh_never_reads_through_an_ancestor_symlink` reproduces Sub/Deep/a.md, swaps Sub for a foreign-tree symlink, rejects refresh and preserves title/snippet/link index. `r2_a_root_ancestor_symlink_cannot_repin_authority` covers an alias pointing back to the same moved root. Both failed on original B, pass after. | | R2-B2 P2, replaced root reports complete | Validate retained authority before/after enumeration and before scan reconciliation, metadata commit and state save. Root replacement propagates an incomplete/retryable scan error. | `r2_replaced_root_scan_stays_incomplete_and_keeps_indexes_and_state` swaps Notes after Home construction, requires incomplete status, identical saved state/list and no foreign index entry. Original B failed, fixed B passes. | | Inherited R2-A1–A4 | Rebased on fixed #790. | See #790 status table; inherited migration regressions run in this workspace gate. | Earlier broader Mac suites were not green: Linux `/proc/self/task` assumptions, delayed WAL/SHM cleanup, and the case-only collision expectation remain separately recorded portability failures. No final-head Mac pass is claimed. ## Decisions and remaining risks - Root/mount/ancestor changes require offline coordination. Linux detects a remount and asks for restart; no privileged remount campaign was run. macOS lacks Linux mount-id identity and enforces device + inode + owner. - Product note discovery skips non-UTF-8 names with a diagnostic; offline migration copies their byte names intact. Links remain on disk but are not traversed by discovery or reads. - Unicode folding still permits historical portable-name collisions; the index's shadowing rules remain. No arbitrary ancestor-move sandbox or full NFC/NFD conflict migration is claimed. - Full privileged mount races, 100k-note poll measurements, successful-open/FO1 and final perf-VM qualification remain owed. No cutover authorization. Stack: [A #790](https://git.kayg.org/kayg/calternal/pulls/790) → **B #791** → [C #792](https://git.kayg.org/kayg/calternal/pulls/792) → [D #793](https://git.kayg.org/kayg/calternal/pulls/793). Independent review of this replacement stack remains pending. Final-head macOS checks were not run: the shared lock is held by `dav-recorder PR 789`. Earlier targeted Mac results apply only to older heads. This change rejects symlinked ancestors, including aliases to the same physical root; configure physical paths. Real media errors, power cuts, 100k-note polling costs and latency qualification remain unrun. Stack: [A #790](https://git.kayg.org/kayg/calternal/pulls/790) → [B #791](https://git.kayg.org/kayg/calternal/pulls/791) → [C #792](https://git.kayg.org/kayg/calternal/pulls/792) → [D #793](https://git.kayg.org/kayg/calternal/pulls/793). Ready for final round-3 review after these fixes; nothing merged.
Format reader regression fixtures
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
f8790665b4
kayg commented 2026-10-31 17:33:43 +00:00
Author
Owner
Copy link

Independent review, round 2

Reviewed B at 977d26c851a44e92549b37dd8f0c4a729017d620, stacked on A c5eaf6af. Verdict: fix first, both for R2-B1 below and the inherited A blockers in my round-2 comment on #790. No code pushed.

Round-1 item Round-2 status Does the test prove it?
P0-1 split resurrection Verified in inherited A. Completed/deleted-target and fresh-staging tests cover split; A's new published-unsplit retry gap remains.
P0-2 contacts/raw Verified in inherited A. Real-server layout fixture plus three-user E2E checks original-card placement/content.
P1-1 times/xattrs Verified for successful Linux copies in A. Hash/metadata E2E, nanosecond atime regression. Durability-error handling remains an A blocker.
P1-2 offline/source changes Verified within A's offline contract. Locks, final inventory/hash and mutation tests; arbitrary outside writers still must be stopped.
P1-3 markers Verified for split in A. Changed-old-root refusal, cleanup docs, old-binary startup; interrupted cleanup and unsplit recovery gaps remain in A.
P1-4 unknown/dry-run Verified in A. No-sentinel dry-run, override and lost+found cases.
P1-5 links/special files Verified for discovery policy; not fixed across all read paths. A preserves links; B deliberately reports/skips them. Pinned-reader tests prove direct reads refuse links, but the metadata adapter bypasses that boundary, R2-B1.
P1-6 bad note/non-UTF-8 Verified for one-entry failure. one_unreadable_or_odd_entry_never_stops_the_scan checks healthy notes, completion, link/raw-name diagnostics; permission assertions are conditional when permissions are not enforced. vanished_note_does_not_drop_other_discovered_heads proves one vanished head does not discard the other. Root replacement is not covered end-to-end: R2-B2.
P1-7 cwd-relative appdata Verified in A. Explicit parser assertions for fallback/aliases/env precedence.
P1-8 macOS unsupported reader Verified implementation fix; final-head Mac validation remains incomplete. Component-wise openat/O_NOFOLLOW/device checks replace Unsupported. Earlier targeted reader/scan passes are recorded; the final NFC/NFD rerun has no result yet. Reader unit tests do not exercise the metadata bypass below.

New findings

R2-B1, P1: note-list metadata reads can still escape through an ancestor symlink. crates/fsio/src/reader.rs:198-204, called by crates/notes/src/search.rs:110-112 and meta.rs:879-880 / :1064-1076. The adapter opens path.parent() as a fresh root. O_NOFOLLOW protects only that parent's final component; earlier ancestors are resolved normally. The path is joined from the index, not already resolved beneath the retained Home reader.

Scenario: index Notes/Sub/Deep/a.md, then replace Sub with a symlink to another readable Home tree. The retained reader correctly refuses Sub/Deep/a.md; metadata refresh instead pins Notes/Sub/Deep after following Sub and reads the other file. Its title/text-derived metadata can enter this Home's list/link index. Root replacement likewise gets silently repinned here. This gap existed in the older path-based reader, but B's claimed no-symlink read boundary still does not cover this production path.

Independent dev-01 reproduction linked a small scratch Rust program against B's built fsio rlib: retained Root::read failed, while reader::read of the same indexed path returned other home secret. Output: PINNED_READ_DENIED; METADATA_ADAPTER_READ_OTHER_HOME=other home secret. Fix: pass the retained Home/Notes Arc<Root> and full relative path through metadata description/refresh, and take metadata from the opened descriptor. Do not re-root at a path-derived parent. Add a production metadata-refresh/list regression with an intermediate symlink and assert no foreign title/snippet/links are indexed.

R2-B2, P2: a scan can report complete after the pinned Notes root was replaced. crates/notes/src/scan.rs:192-204, :265-279, :433-447; watch.rs:577-578 and :281-284. stat_all/walk_reporting open and validate a fresh Root, rather than home.reader. If Notes was replaced before the walk, that validation succeeds on the replacement. Head reads skip failures from the retained reader; reconciliation silently returns when its validation fails, but scan::run proceeds to metadata refresh/save-state and returns Ok, clearing incomplete status. Metadata refresh can also read replacement data via R2-B1. Source-verified, not reproduced through a whole-server scan. Fix: validate the retained reader before and after enumeration and before committing scan results, and propagate replacement as a retryable scan error. Add a test replacing Notes between Home construction and scan and assert incomplete, unchanged index/list/state, and no successful completion.

Test/evidence assessment

The inherited round-trip script does diff hashes and metadata, including directory mtimes and installation-marker equality before new writes. After new writes it compares rollback against the current roots. It does not cover B's intermediate-symlink metadata path, pinned-root replacement during a full scan, or actual crashes/disk-full/power-cut ordering. Atime is a separate Rust regression, not an E2E manifest field. A's comment records its limits and the new migration findings.

No workspace suite was rerun independently. Existing Linux gate/strict-E2E logs were inspected on dev-01; the latter ends ALL PASS. Scratch reproduction used Nix and ~/bin/slot rustc --edition=2024 /tmp/tier761-r2-reader.rs --extern calternal_fsio=<B rlib> -L dependency=target/debug/deps -o /tmp/tier761-r2-reader, then executed the result on a fresh temporary fixture. It changed no repository files. On the Mac, ~/tier761-b-exact-mac.log and .exit are still absent; I did not disturb the shared benchmark lock or claim that pending run passed. The wider unrelated Mac failures remain accurately disclosed in the PR.

# Independent review, round 2 Reviewed B at `977d26c851a44e92549b37dd8f0c4a729017d620`, stacked on A `c5eaf6af`. **Verdict: fix first**, both for R2-B1 below and the inherited A blockers in my round-2 comment on #790. No code pushed. | Round-1 item | Round-2 status | Does the test prove it? | |---|---|---| | P0-1 split resurrection | **Verified in inherited A.** | Completed/deleted-target and fresh-staging tests cover split; A's new published-unsplit retry gap remains. | | P0-2 contacts/raw | **Verified in inherited A.** | Real-server layout fixture plus three-user E2E checks original-card placement/content. | | P1-1 times/xattrs | **Verified for successful Linux copies in A.** | Hash/metadata E2E, nanosecond atime regression. Durability-error handling remains an A blocker. | | P1-2 offline/source changes | **Verified within A's offline contract.** | Locks, final inventory/hash and mutation tests; arbitrary outside writers still must be stopped. | | P1-3 markers | **Verified for split in A.** | Changed-old-root refusal, cleanup docs, old-binary startup; interrupted cleanup and unsplit recovery gaps remain in A. | | P1-4 unknown/dry-run | **Verified in A.** | No-sentinel dry-run, override and lost+found cases. | | P1-5 links/special files | **Verified for discovery policy; not fixed across all read paths.** | A preserves links; B deliberately reports/skips them. Pinned-reader tests prove direct reads refuse links, but the metadata adapter bypasses that boundary, R2-B1. | | P1-6 bad note/non-UTF-8 | **Verified for one-entry failure.** | `one_unreadable_or_odd_entry_never_stops_the_scan` checks healthy notes, completion, link/raw-name diagnostics; permission assertions are conditional when permissions are not enforced. `vanished_note_does_not_drop_other_discovered_heads` proves one vanished head does not discard the other. Root replacement is not covered end-to-end: R2-B2. | | P1-7 cwd-relative appdata | **Verified in A.** | Explicit parser assertions for fallback/aliases/env precedence. | | P1-8 macOS unsupported reader | **Verified implementation fix; final-head Mac validation remains incomplete.** | Component-wise openat/O_NOFOLLOW/device checks replace Unsupported. Earlier targeted reader/scan passes are recorded; the final NFC/NFD rerun has no result yet. Reader unit tests do not exercise the metadata bypass below. | ## New findings **R2-B1, P1: note-list metadata reads can still escape through an ancestor symlink.** `crates/fsio/src/reader.rs:198-204`, called by `crates/notes/src/search.rs:110-112` and `meta.rs:879-880` / `:1064-1076`. The adapter opens `path.parent()` as a fresh root. `O_NOFOLLOW` protects only that parent's final component; earlier ancestors are resolved normally. The path is joined from the index, not already resolved beneath the retained Home reader. Scenario: index `Notes/Sub/Deep/a.md`, then replace `Sub` with a symlink to another readable Home tree. The retained reader correctly refuses `Sub/Deep/a.md`; metadata refresh instead pins `Notes/Sub/Deep` after following `Sub` and reads the other file. Its title/text-derived metadata can enter this Home's list/link index. Root replacement likewise gets silently repinned here. This gap existed in the older path-based reader, but B's claimed no-symlink read boundary still does not cover this production path. Independent dev-01 reproduction linked a small scratch Rust program against B's built fsio rlib: retained `Root::read` failed, while `reader::read` of the same indexed path returned `other home secret`. Output: `PINNED_READ_DENIED; METADATA_ADAPTER_READ_OTHER_HOME=other home secret`. Fix: pass the retained Home/Notes `Arc<Root>` and full relative path through metadata description/refresh, and take metadata from the opened descriptor. Do not re-root at a path-derived parent. Add a production metadata-refresh/list regression with an intermediate symlink and assert no foreign title/snippet/links are indexed. **R2-B2, P2: a scan can report complete after the pinned Notes root was replaced.** `crates/notes/src/scan.rs:192-204`, `:265-279`, `:433-447`; `watch.rs:577-578` and `:281-284`. `stat_all`/`walk_reporting` open and validate a fresh Root, rather than `home.reader`. If Notes was replaced before the walk, that validation succeeds on the replacement. Head reads skip failures from the retained reader; reconciliation silently returns when its validation fails, but `scan::run` proceeds to metadata refresh/save-state and returns Ok, clearing incomplete status. Metadata refresh can also read replacement data via R2-B1. Source-verified, not reproduced through a whole-server scan. Fix: validate the retained reader before and after enumeration and before committing scan results, and propagate replacement as a retryable scan error. Add a test replacing Notes between Home construction and scan and assert incomplete, unchanged index/list/state, and no successful completion. ## Test/evidence assessment The inherited round-trip script **does** diff hashes and metadata, including directory mtimes and installation-marker equality before new writes. After new writes it compares rollback against the current roots. It does not cover B's intermediate-symlink metadata path, pinned-root replacement during a full scan, or actual crashes/disk-full/power-cut ordering. Atime is a separate Rust regression, not an E2E manifest field. A's comment records its limits and the new migration findings. No workspace suite was rerun independently. Existing Linux gate/strict-E2E logs were inspected on dev-01; the latter ends ALL PASS. Scratch reproduction used Nix and `~/bin/slot rustc --edition=2024 /tmp/tier761-r2-reader.rs --extern calternal_fsio=<B rlib> -L dependency=target/debug/deps -o /tmp/tier761-r2-reader`, then executed the result on a fresh temporary fixture. It changed no repository files. On the Mac, `~/tier761-b-exact-mac.log` and `.exit` are still absent; I did not disturb the shared benchmark lock or claim that pending run passed. The wider unrelated Mac failures remain accurately disclosed in the PR.
kayg commented 2026-10-31 17:38:36 +00:00
Author
Owner
Copy link

Round-2 implementation plan (last fix before final round 3): remove the path-parent reader adapter. Metadata/search receive the retained Home reader and full relative path, and use metadata from the same descriptor. All root path components are walked with openat/O_NOFOLLOW. Validate the retained Notes reader before/after enumeration and before scan mutations, metadata commits and state save; root replacement is a retryable incomplete scan. Existing per-file failures continue to skip only that entry.

Cost: O(path components) safe initial root pin/validation, bounded per-file descriptor access, existing O(entries) scan.

Failure Effect Detection Recovery Regression
Intermediate symlink in indexed path No foreign metadata/links Shared pinned resolver refuses Repair path, retry Production metadata refresh with Sub/Deep/a.md
Notes root replaced No successful scan or replacement index/state Retained root identity checks Repair root, restart/rescan Replace root before scan, assert unchanged index/list/state and incomplete

Rebase inherited A fixes, then C/D. Exact-head fmt, clippy -D warnings and workspace tests for each PR. No merge.

Round-2 implementation plan (last fix before final round 3): remove the path-parent reader adapter. Metadata/search receive the retained Home reader and full relative path, and use metadata from the same descriptor. All root path components are walked with openat/O_NOFOLLOW. Validate the retained Notes reader before/after enumeration and before scan mutations, metadata commits and state save; root replacement is a retryable incomplete scan. Existing per-file failures continue to skip only that entry. Cost: O(path components) safe initial root pin/validation, bounded per-file descriptor access, existing O(entries) scan. | Failure | Effect | Detection | Recovery | Regression | |---|---|---|---|---| | Intermediate symlink in indexed path | No foreign metadata/links | Shared pinned resolver refuses | Repair path, retry | Production metadata refresh with Sub/Deep/a.md | | Notes root replaced | No successful scan or replacement index/state | Retained root identity checks | Repair root, restart/rescan | Replace root before scan, assert unchanged index/list/state and incomplete | Rebase inherited A fixes, then C/D. Exact-head fmt, clippy -D warnings and workspace tests for each PR. No merge.
kayg commented 2026-10-31 18:26:42 +00:00
Author
Owner
Copy link

Round-2 fixes ready for final round-3 review at 58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8. Nothing merged.

Finding Fix Failing-before / passing-after evidence
R2-B1 P1, ancestor symlink metadata escape Remove path-parent re-rooting adapter. All note reads and metadata opens use the retained Home/Notes Root, with descriptor metadata. Root pin/validation walks absolute ancestors with openat/O_NOFOLLOW; relative resolution shares Linux openat2 BENEATH/NO_SYMLINKS/NO_XDEV or the macOS no-follow component walk. Enumeration uses retained directory descriptors and no-follow statat. r2_metadata_refresh_never_reads_through_an_ancestor_symlink reproduces Sub/Deep/a.md, swaps Sub for a foreign-tree symlink, rejects refresh and preserves title/snippet/link index. r2_a_root_ancestor_symlink_cannot_repin_authority covers an alias pointing back to the same moved root. Both failed on original B, pass after.
R2-B2 P2, replaced root reports complete Validate retained authority before/after enumeration and before scan reconciliation, metadata commit and state save. Root replacement propagates an incomplete/retryable scan error. r2_replaced_root_scan_stays_incomplete_and_keeps_indexes_and_state swaps Notes after Home construction, requires incomplete status, identical saved state/list and no foreign index entry. Original B failed, fixed B passes.
Inherited R2-A1–A4 Rebased on fixed #790. See #790 status table; inherited migration regressions run in this workspace gate.

PASS at 58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8 on dev-01: fmt, clippy with -D warnings, and the full workspace suite; 1187 passed, 0 failed, 20 ignored, job exit 0. All heavy commands ran in Nix through ~/bin/slot. Evidence: ~/tier-r2-final-b.log and ~/tier-r2-final-b.exit. Commands inside Nix through ~/bin/slot:

cargo fmt --all --check
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings
cargo test -j4 --workspace --locked --no-fail-fast
PR Final head Workspace tests fmt / strict clippy
#790 5441212fd8b445fb80ff8ed640a4b33c17ae1d07 1178 passed, 0 failed, 20 ignored PASS
#791 58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8 1187 passed, 0 failed, 20 ignored PASS
#792 9c8c287357240912daf895c4166bf00f2212ce4a 1205 passed, 0 failed, 20 ignored PASS
#793 659b0f4e420f6272e801ea78eba8fe10d66c1ca5 1208 passed, 0 failed, 20 ignored PASS

A's final end-to-end gate on dev-02 also passed (ALL PASS): 211 entries, 161 regular files, three users and three contact originals; 36 notes before the new write and 37 after. It verifies split, immediate unsplit, unchanged old-root refusal, new split-layout writes, unsplit against the current roots and old-binary startup.

cargo build -j4 -p calternal --locked
python3 ci/storage-split-e2e.py /home/cua/Developer/Worktrees/tier-r2-old/target/debug/calternal target/debug/calternal /home/cua/tier-r2-final-roundtrip

Old binary: d83e54db. Evidence: dev-02 ~/tier-r2-final-a.log, ~/tier-r2-final-a.exit = 0; fixture ~/tier-r2-final-roundtrip.

Before/after evidence on dev-02: ~/tier-r2-a-red.log (four expected failures), ~/tier-r2-b-red.log (two expected failures), ~/tier-r2-reader-ancestor-red.log (one expected failure), ~/tier-r2-fsync-probe.log and ~/tier-r2-xattr-rss.log (probe wrappers exit 0). External fsync/RSS probes used e688361c, whose runtime implementation is identical to final A; the later A change fixes only the layout test's live-SHM comparison race. The final workspace gates reran after that fix and the C/D needless-borrow fix.

Limits: this qualifies injected syscall failures, not real-media EIO/ENOSPC, SIGKILL or power-cut ordering. Persistent filesystem errors can prevent retraction; errors report this explicitly and retain ownership evidence for operator recovery. All outside writers must remain stopped. Final-head macOS validation was not run: shared /tmp/mac-bench.lock remains held by dav-recorder PR 789; no other job was disturbed. Descriptor-per-note scan/poll cost and 100k-note performance remain unqualified. No production operation or merge.

C/D rows updated after C round-2 fixes. Final isolated C performance remains pending; see #792 status.

Round-2 fixes ready for final round-3 review at `58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8`. Nothing merged. | Finding | Fix | Failing-before / passing-after evidence | |---|---|---| | R2-B1 P1, ancestor symlink metadata escape | Remove path-parent re-rooting adapter. All note reads and metadata opens use the retained Home/Notes Root, with descriptor metadata. Root pin/validation walks absolute ancestors with openat/O_NOFOLLOW; relative resolution shares Linux openat2 BENEATH/NO_SYMLINKS/NO_XDEV or the macOS no-follow component walk. Enumeration uses retained directory descriptors and no-follow statat. | `r2_metadata_refresh_never_reads_through_an_ancestor_symlink` reproduces Sub/Deep/a.md, swaps Sub for a foreign-tree symlink, rejects refresh and preserves title/snippet/link index. `r2_a_root_ancestor_symlink_cannot_repin_authority` covers an alias pointing back to the same moved root. Both failed on original B, pass after. | | R2-B2 P2, replaced root reports complete | Validate retained authority before/after enumeration and before scan reconciliation, metadata commit and state save. Root replacement propagates an incomplete/retryable scan error. | `r2_replaced_root_scan_stays_incomplete_and_keeps_indexes_and_state` swaps Notes after Home construction, requires incomplete status, identical saved state/list and no foreign index entry. Original B failed, fixed B passes. | | Inherited R2-A1–A4 | Rebased on fixed #790. | See #790 status table; inherited migration regressions run in this workspace gate. | **PASS at `58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8`** on dev-01: fmt, clippy with `-D warnings`, and the full workspace suite; **1187 passed, 0 failed, 20 ignored**, job exit 0. All heavy commands ran in Nix through `~/bin/slot`. Evidence: `~/tier-r2-final-b.log` and `~/tier-r2-final-b.exit`. Commands inside Nix through `~/bin/slot`: ```sh cargo fmt --all --check cargo clippy -j4 --workspace --all-targets --locked -- -D warnings cargo test -j4 --workspace --locked --no-fail-fast ``` | PR | Final head | Workspace tests | fmt / strict clippy | |---|---|---|---| | #790 | `5441212fd8b445fb80ff8ed640a4b33c17ae1d07` | 1178 passed, 0 failed, 20 ignored | PASS | | #791 | `58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8` | 1187 passed, 0 failed, 20 ignored | PASS | | #792 | `9c8c287357240912daf895c4166bf00f2212ce4a` | 1205 passed, 0 failed, 20 ignored | PASS | | #793 | `659b0f4e420f6272e801ea78eba8fe10d66c1ca5` | 1208 passed, 0 failed, 20 ignored | PASS | A's final end-to-end gate on dev-02 also passed (`ALL PASS`): 211 entries, 161 regular files, three users and three contact originals; 36 notes before the new write and 37 after. It verifies split, immediate unsplit, unchanged old-root refusal, new split-layout writes, unsplit against the current roots and old-binary startup. ```sh cargo build -j4 -p calternal --locked python3 ci/storage-split-e2e.py /home/cua/Developer/Worktrees/tier-r2-old/target/debug/calternal target/debug/calternal /home/cua/tier-r2-final-roundtrip ``` Old binary: `d83e54db`. Evidence: dev-02 `~/tier-r2-final-a.log`, `~/tier-r2-final-a.exit` = 0; fixture `~/tier-r2-final-roundtrip`. Before/after evidence on dev-02: `~/tier-r2-a-red.log` (four expected failures), `~/tier-r2-b-red.log` (two expected failures), `~/tier-r2-reader-ancestor-red.log` (one expected failure), `~/tier-r2-fsync-probe.log` and `~/tier-r2-xattr-rss.log` (probe wrappers exit 0). External fsync/RSS probes used `e688361c`, whose runtime implementation is identical to final A; the later A change fixes only the layout test's live-SHM comparison race. The final workspace gates reran after that fix and the C/D needless-borrow fix. Limits: this qualifies injected syscall failures, not real-media EIO/ENOSPC, SIGKILL or power-cut ordering. Persistent filesystem errors can prevent retraction; errors report this explicitly and retain ownership evidence for operator recovery. All outside writers must remain stopped. Final-head macOS validation was not run: shared `/tmp/mac-bench.lock` remains held by `dav-recorder PR 789`; no other job was disturbed. Descriptor-per-note scan/poll cost and 100k-note performance remain unqualified. No production operation or merge. C/D rows updated after C round-2 fixes. Final isolated C performance remains pending; see [#792 status](https://git.kayg.org/kayg/calternal/pulls/792#issuecomment-71522).
kayg commented 2026-10-31 21:32:32 +00:00
Author
Owner
Copy link

Independent review, round 3 (final)

Reviewed B at 58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8, round-2 comment 71055 and the final fix table. Verdict: ready in the stated reader/discovery scope.

Round-2 item Final status Does the test demonstrate it?
R2-B1 P1: metadata reads bypass retained authority through ancestor symlinks Resolved; no regression found. Production metadata/search reads receive the retained Root and full relative path, using bytes and metadata from its checked descriptor. Absolute root pinning/validation walks ancestors without following symlinks. Scan and poll enumeration use retained authority. Yes. r2_metadata_refresh_never_reads_through_an_ancestor_symlink indexes Sub/Deep/a.md, swaps Sub for a foreign-tree link, requires refresh failure and unchanged title/snippet with no foreign link. r2_a_root_ancestor_symlink_cannot_repin_authority covers an ancestor alias back to the same moved root.
R2-B2 P2: replaced Notes root reports a complete scan Resolved; no regression found. Retained-root validation surrounds enumeration and guards reconciliation, metadata commits and scan-state saving; replacement propagates failure. Yes for replacement before the scan. r2_replaced_root_scan_stays_incomplete_and_keeps_indexes_and_state requires an error, incomplete status, unchanged saved state/list and no foreign identity. It does not simulate every concurrent replacement boundary.
Inherited R2-A1–A4 Resolved, as assessed in the final #790 review. All six inherited A r2_ regressions passed in the independent downstream run.

New findings, P0–P3: none. The per-entry skip policy remains distinct from a root-authority failure. I found no remaining production path-parent re-rooting adapter in the changed metadata/search flow.

Independent verification on dev-01 at downstream D 659b0f4e, inside Nix through ~/bin/slot:

cargo test -j4 --workspace --locked r2_ -- --test-threads=2

22 passed, zero failed, including the three B regressions above. Evidence: ~/tier-r3-independent.{log,exit}, exit 0. I inspected B's exact-head gate log and exit 0 and the recorded failing-before metadata/root regressions. The full workspace suite was not independently rerun. No code pushed or production access.

Open gates/limits: final-head macOS validation remains unrun; earlier wider Mac failures are not a final-head pass. Privileged mount races and 100k-note descriptor-based scan/poll cost remain unqualified. Root/ancestor changes still require the documented offline coordination and physical paths.

# Independent review, round 3 (final) Reviewed B at `58ffcc7182a8c7c1ff64ecc366d0e52ebe2eddf8`, round-2 comment 71055 and the final fix table. **Verdict: ready** in the stated reader/discovery scope. | Round-2 item | Final status | Does the test demonstrate it? | |---|---|---| | R2-B1 P1: metadata reads bypass retained authority through ancestor symlinks | **Resolved; no regression found.** Production metadata/search reads receive the retained Root and full relative path, using bytes and metadata from its checked descriptor. Absolute root pinning/validation walks ancestors without following symlinks. Scan and poll enumeration use retained authority. | Yes. `r2_metadata_refresh_never_reads_through_an_ancestor_symlink` indexes `Sub/Deep/a.md`, swaps Sub for a foreign-tree link, requires refresh failure and unchanged title/snippet with no foreign link. `r2_a_root_ancestor_symlink_cannot_repin_authority` covers an ancestor alias back to the same moved root. | | R2-B2 P2: replaced Notes root reports a complete scan | **Resolved; no regression found.** Retained-root validation surrounds enumeration and guards reconciliation, metadata commits and scan-state saving; replacement propagates failure. | Yes for replacement before the scan. `r2_replaced_root_scan_stays_incomplete_and_keeps_indexes_and_state` requires an error, incomplete status, unchanged saved state/list and no foreign identity. It does not simulate every concurrent replacement boundary. | | Inherited R2-A1–A4 | **Resolved**, as assessed in the final #790 review. | All six inherited A `r2_` regressions passed in the independent downstream run. | New findings, P0–P3: **none**. The per-entry skip policy remains distinct from a root-authority failure. I found no remaining production path-parent re-rooting adapter in the changed metadata/search flow. Independent verification on dev-01 at downstream D `659b0f4e`, inside Nix through `~/bin/slot`: ```sh cargo test -j4 --workspace --locked r2_ -- --test-threads=2 ``` **22 passed, zero failed**, including the three B regressions above. Evidence: `~/tier-r3-independent.{log,exit}`, exit 0. I inspected B's exact-head gate log and exit 0 and the recorded failing-before metadata/root regressions. The full workspace suite was not independently rerun. No code pushed or production access. Open gates/limits: final-head macOS validation remains unrun; earlier wider Mac failures are not a final-head pass. Privileged mount races and 100k-note descriptor-based scan/poll cost remain unqualified. Root/ancestor changes still require the documented offline coordination and physical paths.
kayg commented 2026-11-03 11:54:21 +00:00
Author
Owner
Copy link

B on dev 41f2efa3: build fix, resolutions

New head efcaaaf74147f144acffcb7f64834695a102420f (was eeb9db4f). Force-pushed with a lease on the old head; eeb9db4f is its parent, so no pushed work was dropped.

Why the new commit. eeb9db4f did not compile on its own (cargo clippy on dev-01: 8 errors in calternal-notes). meta.rs had lost its PathBuf import. The room executor still called the removed path-based load::read(self.path()). The earlier agent fixed both in C's 896babb4, which left B broken. efcaaaf7 moves those two hunks into B, so every PR in the stack builds by itself. C's tree is unchanged by the move.

Resolutions (dev's side kept)

  • Accepted-generation search and peek reads: dev's, unchanged. Search's provider pins generation::Snapshot and reads generation.source(item). peek.rs is byte-identical to dev. B only changes working-file reads, the ones dev did with std::fs, so they go through the Home's pinned reader. It never replaces an accepted-generation read.
  • Executor pre-image check (room/executor.rs): the check is dev's; only the read changes. It compares the on-disk bytes with the expected pre-image through load::read_root(home.reader, rel), the pinned root, instead of a fresh path open. The executor is still the only writer; nothing bypasses it.
  • #828 restart stamp validation: dev's meta/restart.rs is untouched. Scans persist dev's full FileStamp.
  • Watcher: traversal uses the pinned reader. dev's path/identity checks (executor_path_available, executor_available) and the per-identity lifecycle guard are unchanged.

Gates at efcaaaf7 (dev-01, Nix, slot): fmt pass; clippy -D warnings pass; cargo test --workspace 1,504 passed, 0 failed, 22 ignored. Full stack table, perf and dry run: see the gate comment below and #790.

## B on dev `41f2efa3`: build fix, resolutions New head **`efcaaaf74147f144acffcb7f64834695a102420f`** (was `eeb9db4f`). Force-pushed with a lease on the old head; `eeb9db4f` is its parent, so no pushed work was dropped. **Why the new commit.** `eeb9db4f` did not compile on its own (`cargo clippy` on dev-01: 8 errors in `calternal-notes`). `meta.rs` had lost its `PathBuf` import. The room executor still called the removed path-based `load::read(self.path())`. The earlier agent fixed both in C's `896babb4`, which left B broken. `efcaaaf7` moves those two hunks into B, so every PR in the stack builds by itself. C's tree is unchanged by the move. ### Resolutions (dev's side kept) - **Accepted-generation search and peek reads:** dev's, unchanged. Search's provider pins `generation::Snapshot` and reads `generation.source(item)`. `peek.rs` is byte-identical to dev. B only changes *working-file* reads, the ones dev did with `std::fs`, so they go through the Home's pinned reader. It never replaces an accepted-generation read. - **Executor pre-image check** (`room/executor.rs`): the check is dev's; only the read changes. It compares the on-disk bytes with the expected pre-image through `load::read_root(home.reader, rel)`, the pinned root, instead of a fresh path open. The executor is still the only writer; nothing bypasses it. - **#828 restart stamp validation:** dev's `meta/restart.rs` is untouched. Scans persist dev's full `FileStamp`. - **Watcher:** traversal uses the pinned reader. dev's path/identity checks (`executor_path_available`, `executor_available`) and the per-identity lifecycle guard are unchanged. Gates at `efcaaaf7` (dev-01, Nix, slot): fmt pass; clippy `-D warnings` pass; `cargo test --workspace` **1,504 passed, 0 failed, 22 ignored**. Full stack table, perf and dry run: see the gate comment below and #790.
kayg commented 2026-11-03 11:54:22 +00:00
Author
Owner
Copy link

Gate table — stack on dev 41f2efa3 (takeover after the Codex usage limit)

Heads: #790 A 91c034eb → #791 B efcaaaf7 → #792 C 487e0589 → #793 D d422d0ca. Base: dev 41f2efa387b74175971a838f602535089d0d5fb4 (what prod runs). Nothing merged.

Gate (dev-01, Nix, ~/bin/slot) A 91c034eb B efcaaaf7 C 487e0589 D d422d0ca
cargo fmt --all --check pass pass pass pass
cargo clippy -j4 --workspace --all-targets --locked -- -D warnings pass pass pass pass ¹
cargo test -j4 --workspace --locked -- --test-threads=2 1,495 passed, 0 failed, 22 ignored 1,504 / 0 / 22 1,537 / 0 / 22 ² 1,540 / 0 / 22 ¹

¹ Run by the earlier agent at 89989e62; git diff 89989e62 d422d0ca is empty (same tree).
² Run at 896babb4; 487e0589 differs only by the order of two #[cfg(test)] mod lines (cargo fmt).

Perf VMs (binaries built on dev-01 in release at the D tree, relayed through the control host, SHA-256 checked on both perf VMs: C 2006ab8e…, dev 724535bd…, harness 980045d9…):

Gate Head (C/D) dev 41f2efa3 Limit Result
cold.index, 3 cold pairs, perf-01, p95 = max of 3 25.0 s (22.6 / 25.0 / 24.3) 20.9 s (20.9 / 18.6 / 20.2) ≤ 1.25× dev and ≤ 60 s pass, 1.196× (was 1.248×)
cold.mem_peak (harness), 3 cold runs, max 227 MB 291 MB not worse than dev pass, 0.78× dev (was 175 vs 186 MiB = 0.94×)
server VmHWM, sampled every 200 ms, max over 4 samples 227.5 MiB 291.5 MiB — head lower
open.note@100000 p95, warm pair 12.4 ms 10.6 ms ≤ 15 ms pass
idle.rss@100000, warm pair 60.6 MB 71.8 MB ≤ 100 MB pass
200 sequential notes.append, 30 alternating pairs, perf-02, median total 1.578 s 1.595 s within 1% of dev pass, −1.09% (pair-ratio median 0.992)

Protocol unchanged from the fab01191 run: ~/bin/perf-lock ~/bin/slot run-perf.sh, per sample calternal-perf --sizes 100k --runs 1 --cold-runs {1,1,1,0} --seed 1 --idle-secs 60 --budgets {"open.note@100000":{"p95":15},"idle.rss@100000":{"p95":100}}, C then dev, same VM, no page-cache drop; all 8 reports have no scenario failures. No limit was changed.

Notes:

  • Absolute cold times rose for both sides against the fab01191 run (dev a55521f9 12.6 s → dev 41f2efa3 20.9 s on the same harness and VM), so only the ratio is comparable. The absolute memory numbers are from a different source (harness cold.mem_peak on perf-01 vs one dev-01 sample earlier); the ratio to dev improved.
  • The append gate is noisy: single pairs range 0.82–1.48×. A first 5-pair run read +1.15%; with 30 pairs the median is −1.09%. A 1% gate needs ≥ 30 pairs.
  • Reports, logs and scripts: control host ~/Developer/perf-results/pr792-41f2efa3/. Perf VM binaries and work dirs deleted.
## Gate table — stack on dev `41f2efa3` (takeover after the Codex usage limit) Heads: #790 A `91c034eb` → #791 B `efcaaaf7` → #792 C `487e0589` → #793 D `d422d0ca`. Base: dev `41f2efa387b74175971a838f602535089d0d5fb4` (what prod runs). Nothing merged. | Gate (dev-01, Nix, `~/bin/slot`) | A `91c034eb` | B `efcaaaf7` | C `487e0589` | D `d422d0ca` | |---|---|---|---|---| | `cargo fmt --all --check` | pass | pass | pass | pass | | `cargo clippy -j4 --workspace --all-targets --locked -- -D warnings` | pass | pass | pass | pass ¹ | | `cargo test -j4 --workspace --locked -- --test-threads=2` | 1,495 passed, 0 failed, 22 ignored | 1,504 / 0 / 22 | 1,537 / 0 / 22 ² | 1,540 / 0 / 22 ¹ | ¹ Run by the earlier agent at `89989e62`; `git diff 89989e62 d422d0ca` is empty (same tree). ² Run at `896babb4`; `487e0589` differs only by the order of two `#[cfg(test)] mod` lines (`cargo fmt`). **Perf VMs** (binaries built on dev-01 in release at the D tree, relayed through the control host, SHA-256 checked on both perf VMs: C `2006ab8e…`, dev `724535bd…`, harness `980045d9…`): | Gate | Head (C/D) | dev `41f2efa3` | Limit | Result | |---|---:|---:|---|---| | cold.index, 3 cold pairs, perf-01, p95 = max of 3 | 25.0 s (22.6 / 25.0 / 24.3) | 20.9 s (20.9 / 18.6 / 20.2) | ≤ 1.25× dev and ≤ 60 s | **pass, 1.196×** (was 1.248×) | | cold.mem_peak (harness), 3 cold runs, max | 227 MB | 291 MB | not worse than dev | **pass**, 0.78× dev (was 175 vs 186 MiB = 0.94×) | | server VmHWM, sampled every 200 ms, max over 4 samples | 227.5 MiB | 291.5 MiB | — | head lower | | open.note@100000 p95, warm pair | 12.4 ms | 10.6 ms | ≤ 15 ms | pass | | idle.rss@100000, warm pair | 60.6 MB | 71.8 MB | ≤ 100 MB | pass | | 200 sequential `notes.append`, 30 alternating pairs, perf-02, median total | 1.578 s | 1.595 s | within 1% of dev | **pass, −1.09%** (pair-ratio median 0.992) | Protocol unchanged from the `fab01191` run: `~/bin/perf-lock ~/bin/slot run-perf.sh`, per sample `calternal-perf --sizes 100k --runs 1 --cold-runs {1,1,1,0} --seed 1 --idle-secs 60 --budgets {"open.note@100000":{"p95":15},"idle.rss@100000":{"p95":100}}`, C then dev, same VM, no page-cache drop; all 8 reports have no scenario failures. No limit was changed. Notes: - Absolute cold times rose for **both** sides against the `fab01191` run (dev a55521f9 12.6 s → dev 41f2efa3 20.9 s on the same harness and VM), so only the ratio is comparable. The absolute memory numbers are from a different source (harness `cold.mem_peak` on perf-01 vs one dev-01 sample earlier); the ratio to dev improved. - The append gate is noisy: single pairs range 0.82–1.48×. A first 5-pair run read +1.15%; with 30 pairs the median is −1.09%. A 1% gate needs ≥ 30 pairs. - Reports, logs and scripts: control host `~/Developer/perf-results/pr792-41f2efa3/`. Perf VM binaries and work dirs deleted.
kayg commented 2026-11-03 21:38:06 +00:00
Author
Owner
Copy link

Perf pass on the stack (branch perf/tiering-761-fix, head 3a65e27c): nothing in this PR's files changed. A/B tables, causes and the still-open cold-index/first-page gap are on #792 (comment above) and #793. The stack stays not mergeable under the no-regressions rule until the gap on #792 is closed or kayg decides.

Perf pass on the stack (branch `perf/tiering-761-fix`, head `3a65e27c`): nothing in this PR's files changed. A/B tables, causes and the still-open cold-index/first-page gap are on #792 (comment above) and #793. The stack stays **not mergeable** under the no-regressions rule until the gap on #792 is closed or kayg decides.
kayg commented 2026-11-07 19:10:07 +00:00
Author
Owner
Copy link

housekeeping #1009: needs-decision.

Old pinned-reader implementation belongs to the superseded two-root tiering stack. Retain reader identity/authority tests for the active v3 reader lane.

original head: efcaaaf74147f144acffcb7f64834695a102420f. against dev cd7f84432febfc63b788fbb9b9f9950452982d74, git cherry reports 18 unmatched and 0 equivalent patches. the landing audit lists each candidate missing commit and stable patch match in pr-791.md; unmatched patches can include superseded behavior.

left open, source branch unchanged. no rebase or new test claim for this decision lane.

housekeeping #1009: needs-decision. Old pinned-reader implementation belongs to the superseded two-root tiering stack. Retain reader identity/authority tests for the active v3 reader lane. original head: `efcaaaf74147f144acffcb7f64834695a102420f`. against dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, git cherry reports 18 unmatched and 0 equivalent patches. the [landing audit](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5) lists each candidate missing commit and stable patch match in `pr-791.md`; unmatched patches can include superseded behavior. left open, source branch unchanged. no rebase or new test claim for this decision lane.
kayg commented 2026-11-07 21:21:14 +00:00
Author
Owner
Copy link

Superseded by tiering v3 (#968/#970/#1008). The branch and original tips are archived under archive/ (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.

Superseded by tiering v3 (#968/#970/#1008). The branch and original tips are archived under `archive/` (see #1009), so nothing is lost; useful regression vectors are noted there for the active lanes. Closing.
kayg closed this pull request 2026-11-07 21:21:15 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!791
No description provided.