kayg/calternal

Watch

Fork 0

Daily Log projection rebuild did not run after the #998 upgrade on production #1033

New issue

Closed

opened 2026-10-04 06:15:11 +00:00 by kayg · 4 comments

kayg commented

2026-10-04 06:15:11 +00:00

Owner

Production finding (2026-10-04, small round 3 = commit `1fd5a630c` deployed ~08:20 IST)

The headingless-Daily-note parser (#998) shipped with notes migration 0028 meant to restart the Daily Log projection rebuild. 15+ minutes after start, production's note_calendar_logs count is unchanged (7,212 rows / 505 days; expected about +590 rows from 37 headingless Daily notes, which a read-only check confirms: frontmatter, no headings, timed bullets). index.sqlite reports migrations up to 28. Backup copies of the index from earlier deploys show notes_daily_log_projection_rebuilds with last_path = Notes/20251226-dailynote.md, completed_ms = NULL (the #724 rebuild may never have marked completion), while the live index.sqlite shows no such table in a quick listing. #1011 (same round) also moved upgrade backfills after the HTTP bind onto a bounded low-priority runtime.

Task (Sol; production is READ-ONLY for you)

Diagnose with read-only queries on production (netbird ssh --no-browser --user calternal 10.69.69.83; open SQLite with mode=ro; aggregates only; never print note text) and the code on origin/dev (commit 1fd5a630c): does migration 0028 reset the rebuild state; is the rebuild job scheduled after #1011's change; did the #724 rebuild ever complete; where the rebuild cursor/state lives now.
Fix on a branch from origin/dev with regression tests: an upgrade that bumps the parser version must re-run the projection rebuild exactly once to completion after bind, resumable, and mark completion; add a test that simulates upgrading a Home with headingless Daily notes and asserts the new Log rows appear without manual action.
Report on #998 with the root cause, head SHA, gates, and the exact post-deploy check. Do not touch production.

## Production finding (2026-10-04, small round 3 = commit 1fd5a630c deployed ~08:20 IST) The headingless-Daily-note parser (#998) shipped with notes migration 0028 meant to restart the Daily Log projection rebuild. 15+ minutes after start, production's `note_calendar_logs` count is unchanged (7,212 rows / 505 days; expected about +590 rows from 37 headingless Daily notes, which a read-only check confirms: frontmatter, no headings, timed bullets). `index.sqlite` reports migrations up to 28. Backup copies of the index from earlier deploys show `notes_daily_log_projection_rebuilds` with `last_path = Notes/20251226-dailynote.md`, `completed_ms = NULL` (the #724 rebuild may never have marked completion), while the live index.sqlite shows no such table in a quick listing. #1011 (same round) also moved upgrade backfills after the HTTP bind onto a bounded low-priority runtime. ## Task (Sol; production is READ-ONLY for you) 1. Diagnose with read-only queries on production (`netbird ssh --no-browser --user calternal 10.69.69.83`; open SQLite with mode=ro; aggregates only; never print note text) and the code on origin/dev (commit 1fd5a630c): does migration 0028 reset the rebuild state; is the rebuild job scheduled after #1011's change; did the #724 rebuild ever complete; where the rebuild cursor/state lives now. 2. Fix on a branch from origin/dev with regression tests: an upgrade that bumps the parser version must re-run the projection rebuild exactly once to completion after bind, resumable, and mark completion; add a test that simulates upgrading a Home with headingless Daily notes and asserts the new Log rows appear without manual action. 3. Report on #998 with the root cause, head SHA, gates, and the exact post-deploy check. Do not touch production.

kayg commented

2026-10-04 06:16:14 +00:00

Author

Owner

Started on job/rebuild-1033 at base 1fd5a630c1. I will inspect production with read-only aggregate queries, trace migration 0028 and post-bind scheduling, then add the upgrade regression and fix locally.

Started on job/rebuild-1033 at base 1fd5a630c14b58dafb2649afdf4a8569417d740a. I will inspect production with read-only aggregate queries, trace migration 0028 and post-bind scheduling, then add the upgrade regression and fix locally.

kayg commented

2026-10-04 06:20:23 +00:00

Author

Owner

Read-only evidence: the live Index has notes migration 28 and the rebuild-state table. Log aggregates remain 7,212 rows / 505 days. Six rebuild jobs are dead after three attempts each, all with Notes operation failed; five earlier jobs stopped at 0/204 remaining notes and the new job at 0/392. The backup before this deploy also has an incomplete cursor and five dead jobs, so #724 did not complete there. The next source has repeated Area Tags on two lines. journal_index inserts duplicate area membership into the unique (User, path, block, Tag) key. I will confirm this with a synthetic upgrade regression and deduplicate projection membership without changing Markdown.

kayg commented

2026-10-04 06:31:17 +00:00

Author

Owner

Fixed in c3db4039a54ad18d74ba40dfcec3be2594e10256 on job/rebuild-1033 (base 1fd5a630c14b58dafb2649afdf4a8569417d740a). Fetched origin and merged origin/dev once before final gates: already up to date. No push or deploy. Production access was read-only; output contained aggregates and schema only.

Root cause and evidence

Migration 0028 correctly clears both completed and partial rebuild cursors. The #1011 post-bind path starts the job worker and calls the Notes backfill. The live Index is /srv/calternal/data/.system/index.sqlite; the rebuild state remains in notes_daily_log_projection_rebuilds there. It has one incomplete cursor. The live Notes migration version is 28.

The rebuild ran but stopped at the next Daily note. Six jobs are dead after three attempts each, with Notes operation failed: five have progress 0/204, and the upgrade job has progress 0/392. The pre-small3 backup already has five dead jobs and an incomplete cursor, so the #724 pass had not completed in that backup. Live Calendar Log aggregates were 7,212 rows / 505 days.

The next source has repeated Area Tags on two lines. journal_index inserted each occurrence into journal_resource_areas, whose primary key permits one membership per User/path/block/Tag. The unique-key failure rolls back the projection and leaves the cursor in place. This also explains why the headingless notes later in the pass did not appear. A synthetic regression failed on the old code with post-bind worker must complete the parser upgrade: Elapsed(()), then passed with the fix.

Built and files

crates/plugins/notes/src/store.rs: collect DAV Area memberships as a set before inserting them. Keep source Markdown and its Tags intact.
crates/plugins/notes/src/lib.rs: add an upgrade regression using the real migration runner, post-bind backfill and durable worker. Test both completed and interrupted pre-0028 cursors, repeated Area Tags, headingless Daily notes, completion, one pass across repeated startup, and byte-identical source. Update module documentation in both files.
Existing resumability and 39-headingless-note projection regressions also passed.

Gates (verbatim output)

cargo fmt --check: exit 0, no output.

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 56s

cargo test -p calternal-plugin-notes -- --test-threads=4:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 33.53s
test result: ok. 192 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 120.35s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.46s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Decisions

No new product design. Area membership follows set semantics from DESIGN §46. Retain migration 0028 and existing post-bind scheduling: both work. Add no new migration; startup queues the existing incomplete cursor again and resumes it after the fixed projection. No dependencies changed.

UX gaps closed

Existing imported Daily notes with repeated Area Tags can no longer strand the parser projection rebuild before later headingless Daily notes. The test proves automatic indexing without source changes or manual rebuild calls.

Known gaps / UX gaps left

Production completion must be checked after the orchestrator deploys this commit. The job did not deploy or change production. No web code changed, so no UI screenshots or web gates apply. No performance measurement: this is a correctness issue, under the verify-once policy.

Exact post-deploy check (read-only; aggregates only)

Run on production after startup, using Python because sqlite3 CLI is absent:

import sqlite3
c = sqlite3.connect('file:/srv/calternal/data/.system/index.sqlite?mode=ro', uri=True)
print(c.execute("SELECT MAX(version) FROM _migrations WHERE namespace='calternal-plugin-notes'").fetchall())
print(c.execute("SELECT COUNT(*), SUM(completed_ms IS NULL) FROM notes_daily_log_projection_rebuilds").fetchall())
print(c.execute("SELECT COUNT(*), COUNT(DISTINCT day) FROM note_calendar_logs").fetchall())
print(c.execute("SELECT state, COUNT(*) FROM jobs WHERE kind='notes.daily-log-projection-rebuild' GROUP BY state").fetchall())

Expected: migration remains 28; incomplete count becomes 0; a new rebuild job completes (old dead jobs remain history); Log rows increase by approximately 590 over the observed 7,212, subject to concurrent User edits. A later restart must not create another completed parser pass. Do not reset the cursor or change production manually.

For the merge round

Run cargo test -p calternal-server wire::tests::startup_serves_http_while_upgrade_backfills_wait -- --ignored --exact --nocapture --test-threads=1 to retain the real HTTP-bind guarantee from #1011. Full combined suites and adversarial matrices remain merge-round work under the verification policy.

Fixed in `c3db4039a54ad18d74ba40dfcec3be2594e10256` on `job/rebuild-1033` (base `1fd5a630c14b58dafb2649afdf4a8569417d740a`). Fetched origin and merged origin/dev once before final gates: already up to date. No push or deploy. Production access was read-only; output contained aggregates and schema only. ## Root cause and evidence Migration 0028 correctly clears both completed and partial rebuild cursors. The #1011 post-bind path starts the job worker and calls the Notes backfill. The live Index is `/srv/calternal/data/.system/index.sqlite`; the rebuild state remains in `notes_daily_log_projection_rebuilds` there. It has one incomplete cursor. The live Notes migration version is 28. The rebuild ran but stopped at the next Daily note. Six jobs are dead after three attempts each, with `Notes operation failed`: five have progress 0/204, and the upgrade job has progress 0/392. The pre-small3 backup already has five dead jobs and an incomplete cursor, so the #724 pass had not completed in that backup. Live Calendar Log aggregates were 7,212 rows / 505 days. The next source has repeated Area Tags on two lines. `journal_index` inserted each occurrence into `journal_resource_areas`, whose primary key permits one membership per User/path/block/Tag. The unique-key failure rolls back the projection and leaves the cursor in place. This also explains why the headingless notes later in the pass did not appear. A synthetic regression failed on the old code with `post-bind worker must complete the parser upgrade: Elapsed(())`, then passed with the fix. ## Built and files - `crates/plugins/notes/src/store.rs`: collect DAV Area memberships as a set before inserting them. Keep source Markdown and its Tags intact. - `crates/plugins/notes/src/lib.rs`: add an upgrade regression using the real migration runner, post-bind backfill and durable worker. Test both completed and interrupted pre-0028 cursors, repeated Area Tags, headingless Daily notes, completion, one pass across repeated startup, and byte-identical source. Update module documentation in both files. - Existing resumability and 39-headingless-note projection regressions also passed. ## Gates (verbatim output) `cargo fmt --check`: exit 0, no output. `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 56s ``` `cargo test -p calternal-plugin-notes -- --test-threads=4`: ``` Finished `test` profile [unoptimized + debuginfo] target(s) in 33.53s test result: ok. 192 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 120.35s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.46s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ## Decisions No new product design. Area membership follows set semantics from DESIGN §46. Retain migration 0028 and existing post-bind scheduling: both work. Add no new migration; startup queues the existing incomplete cursor again and resumes it after the fixed projection. No dependencies changed. ## UX gaps closed Existing imported Daily notes with repeated Area Tags can no longer strand the parser projection rebuild before later headingless Daily notes. The test proves automatic indexing without source changes or manual rebuild calls. ## Known gaps / UX gaps left Production completion must be checked after the orchestrator deploys this commit. The job did not deploy or change production. No web code changed, so no UI screenshots or web gates apply. No performance measurement: this is a correctness issue, under the verify-once policy. ## Exact post-deploy check (read-only; aggregates only) Run on production after startup, using Python because sqlite3 CLI is absent: ```python import sqlite3 c = sqlite3.connect('file:/srv/calternal/data/.system/index.sqlite?mode=ro', uri=True) print(c.execute("SELECT MAX(version) FROM _migrations WHERE namespace='calternal-plugin-notes'").fetchall()) print(c.execute("SELECT COUNT(*), SUM(completed_ms IS NULL) FROM notes_daily_log_projection_rebuilds").fetchall()) print(c.execute("SELECT COUNT(*), COUNT(DISTINCT day) FROM note_calendar_logs").fetchall()) print(c.execute("SELECT state, COUNT(*) FROM jobs WHERE kind='notes.daily-log-projection-rebuild' GROUP BY state").fetchall()) ``` Expected: migration remains 28; incomplete count becomes 0; a new rebuild job completes (old dead jobs remain history); Log rows increase by approximately 590 over the observed 7,212, subject to concurrent User edits. A later restart must not create another completed parser pass. Do not reset the cursor or change production manually. ## For the merge round Run `cargo test -p calternal-server wire::tests::startup_serves_http_while_upgrade_backfills_wait -- --ignored --exact --nocapture --test-threads=1` to retain the real HTTP-bind guarantee from #1011. Full combined suites and adversarial matrices remain merge-round work under the verification policy.

kayg commented

2026-10-04 06:58:38 +00:00

Author

Owner

Deployed to production (2026-10-04 ~08:56 IST, small round 4 = `6074f71d1`, with #1029)

Gates on the round branch: web check COMPLETED 2002 FILES 0 ERRORS 0 WARNINGS, web tests Tests 1114 passed (1114), cargo fmt --check clean, clippy clean for calternal-plugin-notes and calternal-server, tests 193 passed / 163 passed, 0 failed. Staging healthy first.

Production check (read-only aggregate): note_calendar_logs went from 7,212 rows over 505 days to 7,804 rows over 542 days (+592 rows, +37 days), matching the 37 headingless Daily notes. Root cause was repeated Area Tags in legacy Daily notes, which stopped the resumable rebuild at a unique membership key; memberships are now a set.

## Deployed to production (2026-10-04 ~08:56 IST, small round 4 = 6074f71d1, with #1029) Gates on the round branch: web check `COMPLETED 2002 FILES 0 ERRORS 0 WARNINGS`, web tests `Tests 1114 passed (1114)`, `cargo fmt --check` clean, clippy clean for calternal-plugin-notes and calternal-server, tests 193 passed / 163 passed, 0 failed. Staging healthy first. Production check (read-only aggregate): `note_calendar_logs` went from 7,212 rows over 505 days to **7,804 rows over 542 days** (+592 rows, +37 days), matching the 37 headingless Daily notes. Root cause was repeated Area Tags in legacy Daily notes, which stopped the resumable rebuild at a unique membership key; memberships are now a set.

kayg closed this issue

2026-10-04 06:58:39 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#1033

No description provided.

Rows
Columns