Tiering v3 slice 2: indexed tag pages and counts #1008
Open
kayg
wants to merge 7 commits from
feat/tiering-v3-s2-tags into feat/tiering-v3-s1-namespace
pull from: feat/tiering-v3-s2-tags
merge into: kayg:feat/tiering-v3-s1-namespace
kayg:main
kayg:feat/style-sheet-fixes
kayg:storage/wal-checkpoint-off-save-1051
kayg:t3/paged-search-reconcile-final
kayg:feat/pass8
kayg:review/calendar-2
kayg:fix/shared-row-swipes
kayg:feat/contrast
kayg:dev
kayg:feat/collections-db
kayg:fix/instant-keys-1056
kayg:feat/money-round
kayg:perf/stalls-1054
kayg:wip/tiering-v3-cold4
kayg:wip/tiering-v3-cold5
kayg:port/rows-actions
kayg:port/rows-drop
kayg:port/hidden-blocks
kayg:port/rows
kayg:port/port-tables
kayg:port/port-tables-2
kayg:t3/mail-1011-liveness
kayg:port/devices
kayg:port/port-attachments
kayg:port/port-attachments-2
kayg:port/port-attachments-3
kayg:docs/design-consistency
kayg:docs/engineering-standards
kayg:fix/rusqlite-0.41-rustsec-2026-0459
kayg:feat/iphone-fixes
kayg:shared/restore-and-stale-918-919
kayg:fix/phone-top-band-contrast
kayg:feat/notes-approve
kayg:perf/soak-max-gate
kayg:feat/files-s1-identity
kayg:fix/lifecycle-event-id
kayg:feat/quickfix
kayg:wip/tiering-v3-cold
kayg:perf/web-mockup-frame-fixes
kayg:port/contacts-merge
kayg:port/contacts-edit
kayg:test/shared-room-no-deleted-text
kayg:perf/first-paint-credits
kayg:fix/shared-history-after-grant
kayg:fix/phone-swipe-regressions
kayg:feat/contrast-r1
kayg:review/sharing-pr3-review
kayg:feat/folder-drop
kayg:fix/deskpills-lazy-share
kayg:feat/sharing-s1-pr3
kayg:tmp/money-round-before
kayg:fix/phone-pane-backdrop-roots
kayg:fix/palette-preload-chips
kayg:fix/invite-defaults
kayg:port/tasks-matrix
kayg:fix/restore-green-gates
kayg:fix/sharing-fast-path-hardening
kayg:t3/photos-s1
kayg:feat/tiering-v3-s1-namespace
kayg:notes-archive-hide
kayg:feat/tasks-actions
kayg:t3/photos-plan-slice-one
kayg:feat/whiteboard-s1
kayg:t3/paged-search-reconcile
kayg:t3/smooth-pane-transitions
kayg:t3/index-position-startup-scan
kayg:fix/private-staging-759b-r2
kayg:feat/dav-transport
kayg:feat/imap-notes-fetch
kayg:feat/imap-notes-projection
kayg:feat/imap-notes-readonly
kayg:feat/imap-notes-auth
kayg:feat/imap-notes-server-foundation
kayg:docs/undo-block-ids-plan
kayg:fix/undo-768
kayg:proto/block-identity-v7d
kayg:issue/carddav-read-perf-fix-r2
kayg:issue/carddav-read-r2
kayg:docs/imap-766-plan-v2
kayg:docs/imap-notes-plan
kayg:docs/quota-plan-2
kayg:fix/parallel-test-flakes
kayg:feat/contacts-companion-pure
kayg:docs/calendar-backend-v2-plan
kayg:feat/calendar-ical-converter
kayg:docs/tasks-writes-v2-plan
kayg:docs/carddav-writes-v2-plan
kayg:docs/contacts-import-v2-plan
kayg:issue/610-pm-midnight
kayg:feat/calendar-v2-s1-containment
kayg:docs/dav-recorder-v2-plan
kayg:plan/previews-865
kayg:plan/save-log-durable-entry
kayg:fix/e2e-data-on-ram
kayg:feat/appwindow3
kayg:feat/appwindow2
kayg:feat/appwindow
kayg:feat/money-p1
kayg:feat/keys-sheet
kayg:review/calendar-2-themes
kayg:feat/smooth-cursor
kayg:fix/float-style
kayg:review/calendar-2-float
kayg:release/2026-11-06-hotfix
kayg:archive/port/rows-drop-code
kayg:archive/port/rows-actions
kayg:archive/port/rows
kayg:scratch/tasks-894-ui-gates
kayg:review/calendar-2-night-wip
kayg:release/2026-11-05
kayg:archive/feat/tiering-v3-s1-namespace
kayg:archive/flaky/rows-drop-fixed
kayg:archive/flaky/rows-actions
kayg:archive/feat/tiering-v3-reader
kayg:archive/feat/tiering-v3-roots
kayg:archive/flaky/rows
kayg:sim/bm15-final
kayg:sim/bm15-fix
kayg:sim/bm15-bis-932
kayg:sim/bm15-bis-909
kayg:sim/bm15
kayg:sim/bm15-bis-887
kayg:perf/tiering-option-b
kayg:port/style-3
kayg:port/style-2
kayg:port/style
kayg:fix/sw-lazy-precache
kayg:fix/vim-w-waits-for-save
kayg:fix/signout-test-races
kayg:fp-scratch/all
kayg:sim/bm15-first-paint
kayg:archive/bm15/rows-drop
kayg:archive/bm15/rows-actions
kayg:fix/palette-snippet-contrast
kayg:archive/bm15/rows
kayg:archive/bm15/hidden
kayg:archive/plan/save-log-durable-entry
kayg:mock/money-fix2
kayg:archive/fix/e2e-data-on-ram
kayg:mock/appearance-all
kayg:mock/pill-press
kayg:archive/docs/design-consistency
kayg:mock/auto-swipes
kayg:mock/mail-actions
kayg:archive/feat/mail-p1
kayg:mock/money-static
kayg:archive/review/standards-917
kayg:archive/docs/engineering-standards
kayg:perf/tiering-761-fix
kayg:mock/type-audit
kayg:mock/third-pane
kayg:feat/contacts-import-v2-s1
kayg:perf/executor-hot-path
kayg:archive/port/hidden-blocks
kayg:archive/port/devices
kayg:scratch/executor-on-894
kayg:scratch/executor-spans2
kayg:scratch/executor-spans
kayg:train/2026-11-03-bm14c
kayg:archive/port/rows-drop
kayg:feat/tiering-761-d-scale
kayg:feat/tiering-761-c-index
kayg:feat/tiering-761-b-reader
kayg:archive/review/sharing-pr3-review-9fec3d1f
kayg:train/2026-11-03-bm14b
kayg:perf/ports-lazy-budgets
kayg:feat/tiering-761-a-migration
kayg:train/2026-11-03-bm14
kayg:train/2026-11-03-bm12
kayg:archive/review/sharing-pr3-review
kayg:archive/feat/sharing-s1-pr3
kayg:train/2026-11-03-bm11
kayg:archive/feat/imap-notes-fetch
kayg:train/2026-11-03-bm10
kayg:archive/review/sharing-pr3-review-code
kayg:archive/plan/previews-865
kayg:train/2026-11-02-bm9
kayg:archive/port/port-attachments-3
kayg:archive/port/contacts-merge
kayg:fix/motion-audit-859
kayg:train/2026-11-02-bm8
kayg:archive/scratch/bm8-prep
kayg:archive/port/port-pickers
kayg:archive/port/port-tables-2
kayg:archive/port/port-tables
kayg:archive/feat/sharing-s1-pr3-code
kayg:archive/scratch/bm7-fixed
kayg:train/2026-11-02-bm7
kayg:archive/fix/parallel-test-flakes-code-minipc01
kayg:train/2026-11-02-bm6
kayg:docs/imap-766-readonly-slice
kayg:feat/dav-recorder-v3
kayg:research/788-diagnosis
kayg:archive/fix/parallel-test-flakes
kayg:archive/perf/pr792-cold-index-edit
kayg:docs/sharing-slice-1
kayg:archive/docs/imap-766-plan-v2
kayg:feat/dav-recorder-v2
kayg:archive/docs/dav-recorder-v2-plan
kayg:spike/imap-766-s0-cases
kayg:archive/fix/undo-768
kayg:feat/dav-recorder
kayg:archive/docs/contacts-import-v2-plan
kayg:archive/feat/calendar-ical-converter
kayg:feat/contacts-import
kayg:perf/cold-growth-on-764
kayg:perf/cold-growth-on-tiering
kayg:archive/docs/tasks-writes-v2-plan
kayg:archive/docs/calendar-backend-v2-plan
kayg:archive/feat/contacts-companion-pure
kayg:feat/tasks-set-status
kayg:feat/tasks-move-reorder
kayg:archive/docs/carddav-writes-v2-plan
kayg:archive/feat/calendar-v2-s1-containment
kayg:feat/tasks-delete-promote-duplicate
kayg:archive/feat/dav-transport
kayg:archive/fix/carddav-rebuild-r1
kayg:archive/fix/carddav-retention-r1
kayg:feat/carddav-writes
kayg:perf/quota-on-tiering
kayg:perf/quota-simple
kayg:perf/cold-growth-on-quota-stack
kayg:feat/calendar-identity
kayg:feat/calendar-event-parser
kayg:feat/calendar-props-maps
kayg:port/glass-when-ring
kayg:feat/tiering-761-s1
kayg:wip/10by32-01-impl-759b-scratch-20261030
kayg:wip/6by32-01-bm3-webkit-pause-20261030
kayg:wip/10by32-01-bm3-chromium-pause-20261030
kayg:wip/6by32-01-tier-761-s1-pause-20261030
kayg:spike/imap-766-s0
kayg:wip/6by32-02-calternal-pause-20261030
kayg:wip/8by64-01-review-688-pause-20261030
kayg:wip/6by32-01-verify-744-pause-20261030
kayg:wip/6by32-01-verify-742-pause-20261030
kayg:fix/private-staging-759b
kayg:wip/6by32-01-review-748-pause-20261030
kayg:wip/6by32-01-review-733c-pause-20261030
kayg:wip/6by32-01-review-733b-pause-20261030
kayg:wip/6by32-01-perf-idle-8-pause-20261030
kayg:wip/6by32-01-perf767-base-pause-20261030
kayg:wip/6by32-01-perf767b-767-pause-20261030
kayg:wip/6by32-01-perf767b-764-pause-20261030
kayg:wip/6by32-01-perf767-767-pause-20261030
kayg:wip/6by32-01-perf767-764-pause-20261030
kayg:wip/6by32-01-lists-b-pause-20261030
kayg:wip/6by32-01-ids-verify5-pause-20261030
kayg:wip/6by32-01-ids-verify4-pause-20261030
kayg:review/calendar-month
kayg:wip/10by32-02-undo-768-20261030
kayg:test/ids-verify3
kayg:wip/10by32-02-bm3-chromium-20261030
kayg:wip/10by32-01-review-753-20261030
kayg:wip/10by32-01-perf-idle-2-20261030
kayg:wip/10by32-01-bm2-webkit2-20261030
kayg:plan/759b-staging
kayg:train/2026-10-30-bm2
kayg:review/758b-tests
kayg:review/758-tests
kayg:review/764d-tests
kayg:review/767b-tests
kayg:review/preview-2026-10-29
kayg:review/767-tests
kayg:docs/tiering-731-plan
kayg:wip/c6b02-when2
kayg:wip/c6b02-tasks-create-fix5
kayg:wip/c6b02-settings-frost-mock
kayg:wip/c6b02-settings-frost-base
kayg:wip/c6b02-settings-frost
kayg:wip/c6b02-ring
kayg:wip/c6b02-review-733d
kayg:wip/c6b02-bm1-full-a
kayg:wip/compute02-when-field
kayg:wip/compute02-webkit-lists-repro
kayg:wip/compute02-train-bl2-dev
kayg:wip/compute02-dev-green-pw164
kayg:docs/deploy-plan
kayg:review/764c-tests
kayg:review/764b-tests
kayg:review/748g-tests
kayg:review/764-tests
kayg:review/759b-tests
kayg:fix/storage-owner-liveness-642
kayg:fix/storage-fix-7
kayg:wip/compute02-storage-fix-7
kayg:review/759-tests
kayg:feat/tiering-731
kayg:wip/tiering-731-first-attempt
kayg:wip/compute02-storage-fix-1
kayg:review/754g-tests
kayg:perf/idle-honest-2
kayg:review/754f-tests
kayg:review/754e-tests
kayg:review/755-tests
kayg:review/754d-tests
kayg:review/754c-tests
kayg:review/754b-tests
kayg:review/754-tests
kayg:fix/storage-private-tmp-642-dev
kayg:wip/compute02-verify-750
kayg:test/ids-verify5
kayg:test/ids-verify4
kayg:fix/storage-exact-file-quarantine-642
kayg:wip/compute02-verify-740
kayg:perf/one-flush-save
kayg:review/calendar-static
kayg:wip/compute02-verify-737
kayg:review/indexing-ring
kayg:review/when-field-2
kayg:wip/compute02-review-733
kayg:proto/block-identity-2-review
kayg:proto/block-identity-review
kayg:wip/c6b02-tc-design
kayg:review/when-field
kayg:issue/tasks-create-ids
kayg:issue/tasks-create
kayg:review/combined
kayg:wip/compute02-review-combined
kayg:review/task-polish
kayg:review/top-buttons
kayg:review/q1-q5
kayg:train/2026-10-26-bl2
kayg:archive/wt-398h
kayg:archive/train-bm-proptest-seed
kayg:archive/integ325-293
kayg:archive/integ325-ci
kayg:archive/client-auth-review-baseline
kayg:archive/wt-sim-4853
kayg:archive/wt-sim-4811
kayg:archive/wt-448m
kayg:archive/wt-398a-allocator
kayg:issue/calendar-adr
kayg:archive/old/train/2026-10-23bk2
kayg:archive/old/train/2026-10-23bk
kayg:issue/469-mockup-9a
kayg:issue/smart-tasks-mockup
kayg:issue/469-menus
kayg:issue/652-signout-flake
kayg:issue/652-history-fix7
kayg:issue/652-history-fix6
kayg:archive/old/issue/spring-check-perf
kayg:issue/spring-check-perf
kayg:issue/659-rename-fast
kayg:archive/old/train/2026-10-21ar
kayg:issue/search-hardening-3
kayg:issue/652-history-fix5
kayg:issue/660-port-sheets-glass
kayg:issue/652-history-fix4
kayg:issue/652-history-fix3
kayg:issue/652-history-fix2
kayg:issue/652-history-flake
kayg:train/2026-10-21an
kayg:issue/settings-admin-port
kayg:issue/search-port
kayg:train/2026-10-20am
kayg:issue/storage-split-fix2
kayg:issue/storage-split
kayg:archive/old/issue/folder-ids-fix
kayg:train/2026-10-20al
kayg:archive/old/issue/folder-ids
kayg:train/2026-10-20ak
kayg:issue/tables-mockup
kayg:train/2026-10-19aj
kayg:train/2026-10-19ai
kayg:issue/469-merge-fixes
kayg:issue/sidebar-band
kayg:archive/old/issue/608-settings-spec
kayg:train/2026-10-19ah
kayg:issue/swipe-offscreen
kayg:train/2026-10-19ag
kayg:archive/old/issue/590-nlp-adr
kayg:archive/old/issue/587-event-after-write-2
kayg:train/2026-10-19af
kayg:archive/old/issue/587-event-after-write
kayg:issue/469-pill-room-c
kayg:issue/469-pill-room
kayg:train/2026-10-19ae
kayg:issue/469-narrow-handle
kayg:issue/397-ios-swipe-2
kayg:archive/old/issue/adrs-photos-files-tasks
kayg:issue/469-desk-fixes
kayg:archive/old/issue/auto-update-adr
kayg:issue/469-filter-gap
kayg:issue/469-port-1b
kayg:issue/442-opacity-flake
kayg:archive/397-ios-swipe-2
kayg:issue/327-web-fast-3
kayg:archive/old/issue/469-port-1
kayg:issue/469-port-1
kayg:issue/327-web-fast-2
kayg:archive/old/issue/327-web-fast
kayg:archive/old/issue/425-stt-bench-vq1
kayg:archive/old/issue/445-no-bleed
kayg:archive/old/issue/400-swipe-mockup-2
kayg:issue/426-client-review
kayg:issue/426-client-2
kayg:issue/426-client
kayg:archive/old/issue/476-safe-area-lab
kayg:archive/445-top-light-2
kayg:archive/old/issue/445-top-light
kayg:issue/471-blocks-mockup
kayg:archive/old/issue/448-links-app-2
kayg:archive/old/issue/448-links-app
kayg:archive/old/issue/398-cold-links
kayg:issue/434-442-small-2
kayg:issue/434-442-small
kayg:archive/old/issue/398-unload
kayg:archive/old/issue/445-bottom-band
kayg:issue/445-bottom-band
kayg:issue/351-rem-3
kayg:archive/312-views-mockup
kayg:issue/351-rem-2
kayg:issue/351-rem
kayg:archive/old/issue/327-first-note
kayg:issue/395-width-phone
kayg:archive/old/issue/327-typing
kayg:issue/396-ios-edges
kayg:issue/397-ios-swipe
kayg:archive/398-allocator
kayg:archive/old/issue/327-notes-index
kayg:issue/345-347-contrast-empty-2
kayg:archive/old/issue/44-arenas
kayg:issue/44-arenas
kayg:archive/integ325
kayg:archive/old/issue/327-perf-harness
kayg:archive/old/issue/44-thp
kayg:issue/44-thp
kayg:archive/old/issue/285-sticky-pills
kayg:archive/old/issue/44-trim-3
kayg:issue/44-trim-3
kayg:archive/old/issue/165-cards
kayg:archive/old/issue/165-picker-2
kayg:archive/old/issue/212-multi-select
No reviewers
Labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/calternal!1008
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/tiering-v3-s2-tags"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Depends on #970 and is stacked on feat/tiering-v3-s1-namespace.
Implements Slice 2 of docs/plans/tiering-v3.md:
RED/GREEN evidence is in the branch commits. Full slice gates and perf qualification will be posted here before the PR is marked ready. The required perf session must start after the 14:16 IST epoch with both locks held. Nothing is merged by this PR.
Synced, Rust gates: head
eaeb417e72544a7c0aadcc906d8526f6ebb2de94Merges slice 1
381ce6750(now on dev2fa9e18fb) cleanly.cargo fmt --all --check: PASScargo clippy -j4 --workspace --all-targets --locked -- -D warnings: PASScargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2: PASS, 1,660 passed, 0 failed, 22 ignoredThe first run failed once in
counter_concurrent_mutations_drift_and_restart(one extra 137-byte file). It passed 10/10 here and 10/10 on dev, so it's not blocking: #1021.Release
calternalsha256f7e5ad0f…e810d0,calternal-perfb3682687…e4a2. Perf: see #970. Nothing measured at this head yet.Fresh session H1 (1k/10k/100k): s2 vs dev, a severe regression, now fixed
Session H1: perf-01, 11-09 11:07–11:37 CET (15:37–16:07 IST). Both locks were held: perf-01 through
perf-turn, perf-02 throughperf-lock. Binaries (sha256): devd4b32af94b99ace57…6859, s1381ce67508b5292d6…bf0e, s2eaeb417e7f7e5ad0f…10d0, perf harnessb3682687…e4a2. Each Home had its own setup. Every pair ran in the order control-a, s1, control-b, s2 (--hot-only --reuse-indexed, offset = pair). The gate reading is the one in the earlier comment: changed paths need a 95% upper bound below 1.00; unchanged paths need the point ratio no higher than the control's upper bound (or 1.00 if higher). Before H1, I deleted the predecessor's stale fixtures (96 GBwork-head841, 18 GB ofabab-20261107Homes; reports kept). That took 7 min at the start of the turn.At
eaeb417e7, s2 was much slower than dev, and it got worse with Home size:Root cause. The
notes_tag_order_refreshtrigger runs on every update of a note'supdated,createdortitle_key, which every append or metadata refresh does. It ranUPDATE tag_order … WHERE id = NEW.id.tag_order's key is(tag, id)and nothing indexesidalone, so each note write scanned every order row: 446k at 100k notes.Fix, two commits:
6addfc4ea(RED):tag_order_refresh_cost_does_not_grow_with_other_notescounts the SQLite VM steps of one note's update with 1 and with 3,000 other tagged notes. On dev-01: refresh steps grew from 206 to 42192.ee00d638c(GREEN): the trigger now matchesWHERE tag IN (SELECT tag FROM tags WHERE id = NEW.id) AND id = NEW.id. That goes throughtags_by_idand the(tag, id)key, with no new index and so no extra write cost. The trigger is dropped and recreated on open, so v7 stores made before this fix pick it up. The test passes.Gates at
ee00d638c(dev-01,nix develop,~/bin/slot):cargo fmt --all --checkPASS;cargo clippy -j4 --workspace --all-targets --locked -- -D warningsPASS;cargo test -j4 --workspace --locked --no-fail-fast -- --test-threads=2PASS, 1,661 passed, 0 failed, 22 ignored. Releasecalternalsha25684e5efcb…17d6, staged on perf-01. A re-run turn (control-a, s2fix, control-b; 1k/10k/100k) is queued inperf-turn.Full gate table at the old head
eaeb417e7(s2)OVERALL FAIL
H1b: the append scan is fixed; open-time schema rewrite found and fixed
Session H1b: perf-01, 11-09 16:48–16:54 CET (21:18–21:24 IST). Both locks were held: perf-01 through
perf-turn, perf-02 throughperf-lock. The Homes from H1 were reused (no new setups). Pairs ran in the order control-a, s1, control-b, s2fix (ee00d638c, sha25684e5efcb…17d6), at offsets 0–4.At
ee00d638cthenotes.appendscan is gone (100k p50 1.048, was 52.7×). Two problems remained:Cause of the first-request and open slowdowns: the
ee00d638cfix ranDROP TRIGGER+CREATE TRIGGERon every store open. Each open rewrote the schema, which bumpsschema_versionand makes every connection re-prepare right after start.5dd3419bb(RED):reopening_a_current_store_leaves_its_schema_untouched. On dev-01:left: 24, right: 22.29cb41838(GREEN): the fixed trigger has a new name,notes_tag_order_refresh_by_tag, created only if missing. Dropping the old name is a no-op once it's gone, so later opens never write the schema.Gates at
29cb41838(dev-01): fmt PASS, clippy-D warningsPASS,cargo test --workspace --no-fail-fast1,662 passed, 0 failed, 22 ignored. Release sha256e651f991….Untagged
page_big5.36× at 100k: the perf-01 s2 Home had been built and then put through 20 append runs by the old binary. On fresh 100k Homes on dev-01 (5 pairs, s2 vs s1, both v3 layouts, loaded host, direction only), at29cb41838:The
page_bigquery plan is identical on both Homes (SCAN n USING INDEX notes_updated), and the raw query takes 22 ms on the s2 store vs 106 ms on the s1 store. The s2meta.dbis 14 % larger (236,746 vs 206,921 pages, fromtag_orderand its three indexes), so cache pressure is the likely cause; that's unconfirmed. Next turn (H1c, queued) builds a fresh s2 Home on perf-01 with29cb41838and measures control-a, s2, control-b at 1k/10k/100k.Gate table (s2fix `
ee00d638c`, H1b)OVERALL FAIL
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.