kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

WIP: storage: checkpoint SQLite WALs between saves (#1051) #1064

Draft
kayg wants to merge 14 commits from storage/wal-checkpoint-off-save-1051 into dev
pull from: storage/wal-checkpoint-off-save-1051
merge into: kayg:dev
Conversation 1 Commits 14 Files changed 11 +796 -19
kayg commented 2026-11-10 16:29:52 +00:00
Owner
Copy link

Closes #1051. Includes the test race fix tracked in #1060.

SQLite's default WAL auto-checkpoints ran inside roughly 1.8% of saves. This change schedules checkpoints between writes, with a 4096-page (16 MiB) auto-checkpoint backstop for sustained activity.

Design and cost

  • History uses the shared quiet/idle policy: PASSIVE after 100 ms quiet with at least 100 commits, PASSIVE after 1 s idle, and TRUNCATE after 10 s idle.
  • The same history writer checkpoints meta.db on a separate connection. Its commit hook records count and time together. A meta checkpoint waits for both the actual meta commit deadline and a quiet history writer, so an overdue checkpoint cannot follow a typing ACK straight into notes.save.
  • The journal keeps its original blocking receiver. After 100 commands, one async task waits for a stable commit count and queues an internal checkpoint message. The writer rechecks the count and empty queue before checkpointing; a stale message reschedules without I/O. Its weak sender does not keep a closing store alive.
  • Auto-checkpoint remains a backstop; journal_size_limit is 8 MiB. An active reader can prevent a checkpoint from completing, so neither threshold is a hard WAL-size bound while a snapshot remains held.
  • Each ordinary journal command adds a counter increment; meta adds a short count/time mutex update. Checkpoint policy work is constant per job. No extra OS thread is added.

Failure cases

Failure or activity Effect Recovery / detection Test
Sustained writes without a quiet interval WAL grows to the auto-checkpoint threshold SQLite's backstop runs during a write continuous history stream bound
A reader holds an old snapshot PASSIVE may make partial progress; truncation reports busy zero checkpoint busy timeout; ordinary writer timeout restored busy truncate / timeout restoration
A meta commit is observed late Its original timestamp is retained checkpoint deadline is not shifted into the next save delayed observation policy test
Typing ACK wakes an overdue meta checkpoint It waits for the history writer's quiet interval too no immediate meta I/O after that ACK overdue meta / recent ACK test
A journal timer becomes stale Checkpoint command sees a changed count or waiting work reschedule without I/O active commit-stream witness test
Store closes with a timer pending Weak sender cannot keep the writer alive channel closes and accepted writes drain store shutdown / restore suite
Crash before checkpoint Committed WAL frames remain recoverable SQLite replays the WAL on open crash-image recovery test
Background upload cleanup removes a listed temporary file The crash-test scan ignores only NotFound for that temporary name all other metadata errors still fail 20 fixed repetitions; #1060

Verification

On dev-01 through ~/bin/slot and nix develop, production-equivalent head 7ff88807e. Current head 93ae82ba0 only additionally isolates two test fixture paths:

  • cargo fmt --check: pass.
  • cargo clippy -j4 --workspace --all-targets -- -D warnings: pass.
  • cargo test -j4 --workspace --no-fail-fast -- --test-threads=4: pass, 1745 tests, 0 failures, 22 ignored. Dedicated tmpfs TMPDIR.
  • Seven history checkpoint tests and 75 store tests pass.
  • Crash test: branch failed 1/10, dev failed 1/30; both had the same listing/metadata race. After the test-only fix, 20/20 passed. Logs.

Performance

Qualification is still in progress. The journal-only gate currently reports commit p50 +10.78% versus a +0.38% dev control. This is a failing gate, not an accepted regression. See #1051 for the diagnostic record. Final ABAB, six soak arms per side, and traced-pair results will replace this paragraph before readiness is claimed.

Current journal-only session, milliseconds (median of per-run quantiles):

Metric Quantile Dev Branch Delta Dev control delta
Commit p50 0.026404 0.029250 +10.78% +0.38%
Commit p95 0.041514 0.043823 +5.56% +2.73%
Commit p99 0.062813 0.059367 -5.49% +4.24%
Catch-up page p50 0.164784 0.170650 +3.56% -1.80%
Catch-up page p95 0.200677 0.205396 +2.35% +1.75%
Catch-up page p99 0.234631 0.237536 +1.24% +1.14%

The commit p50 gate fails: paired delta +11.59%, control pair range -2.81% to +7.80%. The isolated backstop experiment and server rebuild are queued behind dev-01's train. Final save-soak, open/list/typing/cold-start, and current-head trace results are pending.

Evidence: rejected diagnostics with raw traces, controls, runners and hashes; passing Rust gates; crash-test reproduction and fixed repetitions.

Closes #1051. Includes the test race fix tracked in #1060. SQLite's default WAL auto-checkpoints ran inside roughly 1.8% of saves. This change schedules checkpoints between writes, with a 4096-page (16 MiB) auto-checkpoint backstop for sustained activity. ## Design and cost - History uses the shared quiet/idle policy: PASSIVE after 100 ms quiet with at least 100 commits, PASSIVE after 1 s idle, and TRUNCATE after 10 s idle. - The same history writer checkpoints meta.db on a separate connection. Its commit hook records count and time together. A meta checkpoint waits for both the actual meta commit deadline and a quiet history writer, so an overdue checkpoint cannot follow a typing ACK straight into notes.save. - The journal keeps its original blocking receiver. After 100 commands, one async task waits for a stable commit count and queues an internal checkpoint message. The writer rechecks the count and empty queue before checkpointing; a stale message reschedules without I/O. Its weak sender does not keep a closing store alive. - Auto-checkpoint remains a backstop; journal_size_limit is 8 MiB. An active reader can prevent a checkpoint from completing, so neither threshold is a hard WAL-size bound while a snapshot remains held. - Each ordinary journal command adds a counter increment; meta adds a short count/time mutex update. Checkpoint policy work is constant per job. No extra OS thread is added. ## Failure cases | Failure or activity | Effect | Recovery / detection | Test | |---|---|---|---| | Sustained writes without a quiet interval | WAL grows to the auto-checkpoint threshold | SQLite's backstop runs during a write | continuous history stream bound | | A reader holds an old snapshot | PASSIVE may make partial progress; truncation reports busy | zero checkpoint busy timeout; ordinary writer timeout restored | busy truncate / timeout restoration | | A meta commit is observed late | Its original timestamp is retained | checkpoint deadline is not shifted into the next save | delayed observation policy test | | Typing ACK wakes an overdue meta checkpoint | It waits for the history writer's quiet interval too | no immediate meta I/O after that ACK | overdue meta / recent ACK test | | A journal timer becomes stale | Checkpoint command sees a changed count or waiting work | reschedule without I/O | active commit-stream witness test | | Store closes with a timer pending | Weak sender cannot keep the writer alive | channel closes and accepted writes drain | store shutdown / restore suite | | Crash before checkpoint | Committed WAL frames remain recoverable | SQLite replays the WAL on open | crash-image recovery test | | Background upload cleanup removes a listed temporary file | The crash-test scan ignores only NotFound for that temporary name | all other metadata errors still fail | 20 fixed repetitions; #1060 | ## Verification On dev-01 through ~/bin/slot and nix develop, production-equivalent head 7ff88807e. Current head 93ae82ba0 only additionally isolates two test fixture paths: - `cargo fmt --check`: pass. - `cargo clippy -j4 --workspace --all-targets -- -D warnings`: pass. - `cargo test -j4 --workspace --no-fail-fast -- --test-threads=4`: pass, 1745 tests, 0 failures, 22 ignored. Dedicated tmpfs TMPDIR. - Seven history checkpoint tests and 75 store tests pass. - Crash test: branch failed 1/10, dev failed 1/30; both had the same listing/metadata race. After the test-only fix, 20/20 passed. [Logs](https://git.kayg.org/kayg/calternal/issues/1060#issuecomment-89774). ## Performance Qualification is still in progress. The journal-only gate currently reports commit p50 +10.78% versus a +0.38% dev control. This is a failing gate, not an accepted regression. See #1051 for the diagnostic record. Final ABAB, six soak arms per side, and traced-pair results will replace this paragraph before readiness is claimed. Current journal-only session, milliseconds (median of per-run quantiles): | Metric | Quantile | Dev | Branch | Delta | Dev control delta | |---|---|---:|---:|---:|---:| | Commit | p50 | 0.026404 | 0.029250 | +10.78% | +0.38% | | Commit | p95 | 0.041514 | 0.043823 | +5.56% | +2.73% | | Commit | p99 | 0.062813 | 0.059367 | -5.49% | +4.24% | | Catch-up page | p50 | 0.164784 | 0.170650 | +3.56% | -1.80% | | Catch-up page | p95 | 0.200677 | 0.205396 | +2.35% | +1.75% | | Catch-up page | p99 | 0.234631 | 0.237536 | +1.24% | +1.14% | The commit p50 gate fails: paired delta +11.59%, control pair range -2.81% to +7.80%. The isolated backstop experiment and server rebuild are queued behind dev-01's train. Final save-soak, open/list/typing/cold-start, and current-head trace results are pending. Evidence: [rejected diagnostics with raw traces, controls, runners and hashes](https://git.kayg.org/kayg/calternal/issues/1051#issuecomment-89816); [passing Rust gates](https://git.kayg.org/kayg/calternal/issues/1051#issuecomment-89786); [crash-test reproduction and fixed repetitions](https://git.kayg.org/kayg/calternal/issues/1060#issuecomment-89774).
kayg added 14 commits 2026-11-10 16:29:53 +00:00 
calternal_core::wal decides when a writer loop checkpoints its SQLite WAL
off the write path (#1051): after a quiet 100 ms once 100 commits piled
up, after an idle second for the rest, TRUNCATE after ten idle seconds,
then sleep. SQLite's own auto-checkpoint, which runs inside the commit
that crosses 1,000 pages, stays only as a 16 MiB backstop.

meta.db gets that backstop and counts its writer's commits with a commit
hook, so the Home's history writer can checkpoint it in quiet moments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The history writer runs calternal_core::wal's policy between jobs: after a
quiet 100 ms once 100 commits piled up, after an idle second for the rest,
TRUNCATE after ten idle seconds, then it sleeps until the next job. It
does the same for the Home's meta.db on its own connection (a PASSIVE
checkpoint never blocks the list's writer), reading the list's commit
count from its commit hook. SQLite's auto-checkpoint stays only as a
16 MiB backstop. Before this, either store checkpointed inside the save
that crossed 1,000 pages (#1051).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The journal writer checkpoints after every 100 commits, once the reply is
out and nothing else is queued, instead of SQLite doing it inside the
commit that crossed 1,000 pages. Auto-checkpoint stays as a 16 MiB
backstop; journal_size_limit cuts the log file back past 8 MiB (#1051).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A commit only appends to the WAL; the main file changes only when a
checkpoint copies pages back. The tests stream writes with gaps shorter
than the quiet window and check the main file is untouched until the
stream stops: history, the note list (checkpointed by the history
writer) and the journal. Also: the WAL stays within the backstop under a
stream with no quiet moment, and a crash image taken before any
checkpoint keeps every committed write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test(storage): isolate checkpoint timing fixtures in temporary directories
Some checks are pending
CI / server (pull_request) Waiting to run
Details
CI / rust (pull_request) Waiting to run
Details
CI / web (pull_request) Waiting to run
Details
Markdown extraction contract / markdown-native-contract (pull_request) Waiting to run
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Waiting to run
Details
93ae82ba04
kayg commented 2026-11-10 16:46:27 +00:00
Author
Owner
Copy link

Live qualification state (work is continuing):

  • Head: 93ae82ba04. The last commit only isolates test fixtures; production code matches 7ff88807e.
  • Rust gates at 7ff88807e: fmt, clippy, full workspace all pass (1745 passed, 0 failed, 22 ignored). Seven history checkpoint tests and 75 store tests pass. The test-only fixture change still needs its targeted rerun.
  • Failing performance gate: journal commit p50 +10.78% vs control +0.38%; source data and rejected traces are attached to #1051 comment 89816. No final readiness claim.
  • dev-01 train lock is train-1110c, owner PID 2642193. Our queued units are wal1051-cap-build-long2 and wal1051-r5-server-long. Logs: /home/claude/wal1051-run.crMA/cap-build.log and r5-server.log. Both use ~/bin/slot; neither has started compiling yet. No perf lock is held by this lane.
  • The isolated cap experiment is /home/claude/Developer/Worktrees/wal1051-cap.57Vg/tree (detached 7ff88807e, only core BACKSTOP_PAGES changed from 4096 to 1000); probe manifest is /home/claude/Developer/Worktrees/wal1051-cap.57Vg/probe/Cargo.toml, output /home/claude/Developer/Worktrees/wal1051-cap.57Vg/probe-cap1000. It does not change the PR branch. Compare that binary against the branch's ordinary probe-cand, not against a different implementation, to isolate the WAL-size effect.
  • Source binaries are under /home/claude/wal1051-build.sJl1/bin. probe-cand is the ordinary 4096-page 7ff88807e implementation; the queued server build will replace server-cand with the recent-history-ACK guard. Do not reuse the older staged server as the final guarded server.
  • On perf-01, /home/claude/wal1051-perf/cap-check has the runner/measurement scripts ready; stage/hash both experimental binaries before starting plain ~/bin/perf-turn wal1051. The measurement requires the normal perf-02 lock handshake too.
  • Control scratch is /tmp/wal1051-final.ZMCO; runner-final-template.py also corrects nearest-rank aggregation for single-observation first-token and first-page metrics. Final qualification still needs one fresh session with journal/startup/room ABAB and dev controls, boot plus open/list smoke, twelve interleaved 180 s soak arms (six per side), and a current-head traced pair. The fixed dev baseline is 4274dafb4. Current dev's subsequent commits do not change the Rust/Cargo paths involved here; a merge-tree check is clean.
Live qualification state (work is continuing): - Head: 93ae82ba043eada4b86dcc35faf939d59e3340d5. The last commit only isolates test fixtures; production code matches 7ff88807e. - Rust gates at 7ff88807e: fmt, clippy, full workspace all pass (1745 passed, 0 failed, 22 ignored). Seven history checkpoint tests and 75 store tests pass. The test-only fixture change still needs its targeted rerun. - Failing performance gate: journal commit p50 +10.78% vs control +0.38%; source data and rejected traces are attached to #1051 comment 89816. No final readiness claim. - dev-01 train lock is train-1110c, owner PID 2642193. Our queued units are wal1051-cap-build-long2 and wal1051-r5-server-long. Logs: /home/claude/wal1051-run.crMA/cap-build.log and r5-server.log. Both use ~/bin/slot; neither has started compiling yet. No perf lock is held by this lane. - The isolated cap experiment is /home/claude/Developer/Worktrees/wal1051-cap.57Vg/tree (detached 7ff88807e, only core BACKSTOP_PAGES changed from 4096 to 1000); probe manifest is /home/claude/Developer/Worktrees/wal1051-cap.57Vg/probe/Cargo.toml, output /home/claude/Developer/Worktrees/wal1051-cap.57Vg/probe-cap1000. It does not change the PR branch. Compare that binary against the branch's ordinary probe-cand, not against a different implementation, to isolate the WAL-size effect. - Source binaries are under /home/claude/wal1051-build.sJl1/bin. probe-cand is the ordinary 4096-page 7ff88807e implementation; the queued server build will replace server-cand with the recent-history-ACK guard. Do not reuse the older staged server as the final guarded server. - On perf-01, /home/claude/wal1051-perf/cap-check has the runner/measurement scripts ready; stage/hash both experimental binaries before starting plain ~/bin/perf-turn wal1051. The measurement requires the normal perf-02 lock handshake too. - Control scratch is /tmp/wal1051-final.ZMCO; runner-final-template.py also corrects nearest-rank aggregation for single-observation first-token and first-page metrics. Final qualification still needs one fresh session with journal/startup/room ABAB and dev controls, boot plus open/list smoke, twelve interleaved 180 s soak arms (six per side), and a current-head traced pair. The fixed dev baseline is 4274dafb4. Current dev's subsequent commits do not change the Rust/Cargo paths involved here; a merge-tree check is clean.
Some checks are pending
CI / server (pull_request) Waiting to run
Details
CI / rust (pull_request) Waiting to run
Details
CI / web (pull_request) Waiting to run
Details
Markdown extraction contract / markdown-native-contract (pull_request) Waiting to run
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Waiting to run
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin storage/wal-checkpoint-off-save-1051:storage/wal-checkpoint-off-save-1051
git switch storage/wal-checkpoint-off-save-1051

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff storage/wal-checkpoint-off-save-1051
git switch storage/wal-checkpoint-off-save-1051
git rebase dev
git switch dev
git merge --ff-only storage/wal-checkpoint-off-save-1051
git switch storage/wal-checkpoint-off-save-1051
git rebase dev
git switch dev
git merge --no-ff storage/wal-checkpoint-off-save-1051
git switch dev
git merge --squash storage/wal-checkpoint-off-save-1051
git switch dev
git merge --ff-only storage/wal-checkpoint-off-save-1051
git switch dev
git merge storage/wal-checkpoint-off-save-1051
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!1064
No description provided.