kayg/calternal

Watch

Fork 0

Expose Daily note Log parse state in the Index #998

New issue

Open

opened 2026-10-03 08:42:28 +00:00 by kayg · 8 comments

kayg commented

2026-10-03 08:42:28 +00:00

Owner

Parent: #719.

Evidence

This read-only audit classified all 428 Daily notes in the audit range that have no indexed Calendar Log rows. The server-side classifier emitted aggregate counts only. It read all 428 files successfully and did not print or copy source text, titles, names, or paths.

5 files are empty or contain whitespace only; 0 are frontmatter-only; 423 have body text.
279 files have a Log heading accepted by the core parser; 4 of those Log sections are empty.
4,655 compact HHMM time-stamped bullets occur inside accepted Log sections in 269 files. Current read_time_prefix requires HH:MM, so parse_log_bullet rejects these. Parser support and the Index rebuild belong to #724.
Colon-form HH:MM bullets total 600 lines: 594 in 39 files have no accepted Log heading, 1 is before a Log heading, 4 in 2 files are after the next H2 heading, and 1 inside a Log section has no title. The first three placements are outside the parser's selected section; the last fails its non-empty-title rule.
5,146 other non-task bullet lines in 272 files have no HH:MM prefix. Task checkboxes are counted separately.

The Index's Calendar Log rows do not retain a parse state for a Daily note with zero rows. The per-day Notes API already returns missing/duplicate heading diagnostics and unparsed top-level list items inside a Log section (#53 and #70). This issue covers an Index-level summary only. It does not duplicate #724's time grammar or rebuild, #856's links and embeds, or #857's stable identity.

Scope

Add a derived Log parse state to the Daily-note Index projection and expose aggregate counts for that state. Refresh it when the Daily note is indexed. At minimum, distinguish an empty note, a missing accepted Log heading, an empty accepted Log section, unparsed list items inside the section, and accepted Log entries. Keep source bytes unchanged and do not store source text in the derived state. Keep repair behavior in #53/#70, time syntax and rebuild work in #724, and do not add a warning to the normal User flow.

Parser and Index locations

crates/calternal-notes-core/src/dayfile.rs: is_log_heading, source_log, parse_log_bullet, read_time_prefix, unparsed_log_lines, and log_heading_diagnostics.
crates/plugins/notes/src/store.rs: index_calendar_logs and the Daily-note Index projection.

Synthetic examples

These examples are invented and contain no production text:

## Log

This has an accepted but empty Log section.

## Log
- 09:00

This has an unparsed top-level bullet because the title is empty.

Regression test idea

Index fixtures for an empty note, a note without a Log heading, an empty Log section, a malformed bullet in the section, a parseable entry, and a note whose Index projection is stale. Assert the derived state and aggregate counts after indexing or rebuilding. Assert the source bytes do not change.

Parent: #719. ## Evidence This read-only audit classified all 428 Daily notes in the audit range that have no indexed Calendar Log rows. The server-side classifier emitted aggregate counts only. It read all 428 files successfully and did not print or copy source text, titles, names, or paths. - 5 files are empty or contain whitespace only; 0 are frontmatter-only; 423 have body text. - 279 files have a Log heading accepted by the core parser; 4 of those Log sections are empty. - 4,655 compact `HHMM` time-stamped bullets occur inside accepted Log sections in 269 files. Current `read_time_prefix` requires `HH:MM`, so `parse_log_bullet` rejects these. Parser support and the Index rebuild belong to #724. - Colon-form `HH:MM` bullets total 600 lines: 594 in 39 files have no accepted Log heading, 1 is before a Log heading, 4 in 2 files are after the next H2 heading, and 1 inside a Log section has no title. The first three placements are outside the parser's selected section; the last fails its non-empty-title rule. - 5,146 other non-task bullet lines in 272 files have no `HH:MM` prefix. Task checkboxes are counted separately. The Index's Calendar Log rows do not retain a parse state for a Daily note with zero rows. The per-day Notes API already returns missing/duplicate heading diagnostics and unparsed top-level list items inside a Log section (#53 and #70). This issue covers an Index-level summary only. It does not duplicate #724's time grammar or rebuild, #856's links and embeds, or #857's stable identity. ## Scope Add a derived Log parse state to the Daily-note Index projection and expose aggregate counts for that state. Refresh it when the Daily note is indexed. At minimum, distinguish an empty note, a missing accepted Log heading, an empty accepted Log section, unparsed list items inside the section, and accepted Log entries. Keep source bytes unchanged and do not store source text in the derived state. Keep repair behavior in #53/#70, time syntax and rebuild work in #724, and do not add a warning to the normal User flow. ## Parser and Index locations - `crates/calternal-notes-core/src/dayfile.rs`: `is_log_heading`, `source_log`, `parse_log_bullet`, `read_time_prefix`, `unparsed_log_lines`, and `log_heading_diagnostics`. - `crates/plugins/notes/src/store.rs`: `index_calendar_logs` and the Daily-note Index projection. ## Synthetic examples These examples are invented and contain no production text: ```markdown ## Log ``` This has an accepted but empty Log section. ```markdown ## Log - 09:00 ``` This has an unparsed top-level bullet because the title is empty. ## Regression test idea Index fixtures for an empty note, a note without a Log heading, an empty Log section, a malformed bullet in the section, a parseable entry, and a note whose Index projection is stale. Assert the derived state and aggregate counts after indexing or rebuilding. Assert the source bytes do not change.

kayg changed title from ~~Distinguish empty Daily note Logs from unparsed Log-shaped text~~ to Expose Daily note Log parse state in the Index

2026-10-03 08:53:29 +00:00

kayg referenced this issue

2026-10-03 08:57:33 +00:00

Moving in by dropping Markdown folders: find and fix what breaks (replaces the dedicated importer) #719

kayg commented

2026-10-03 09:03:54 +00:00

Author

Owner

Starting #998 on job/logheading, based on 7a merge-round head 516faaa698570bdb468626cf6cd75d9c81b33ac2. I am reviewing the production parser and will run the requested count-only production classification. Work will stay in the shared Notes day-file parser and anonymised regression fixtures; the time grammar remains unchanged.

Starting #998 on `job/logheading`, based on 7a merge-round head `516faaa698570bdb468626cf6cd75d9c81b33ac2`. I am reviewing the production parser and will run the requested count-only production classification. Work will stay in the shared Notes day-file parser and anonymised regression fixtures; the time grammar remains unchanged.

kayg commented

2026-10-03 09:17:37 +00:00

Author

Owner

Read-only production classification completed over 662 Daily notes, with no read errors and no note text, names or paths emitted. Counts:

Heading level	Emoji	Heading word	Files	HH:MM bullets
none	no	none	39	590
##	no		2	4

The total is 594 outside-section bullets across 41 files. The 39-file no-preceding-heading group contains 590 bullets; the other 4 bullets are under H2 headings in two distinct files. Of the 590, 589 are in 38 files with no accepted Log heading, and 1 is before a later accepted Log heading. This differs from the brief's 39-file/594-bullet grouping, so I will preserve the exact source-parser boundary rather than infer Log sections from headingless content.

Read-only production classification completed over 662 Daily notes, with no read errors and no note text, names or paths emitted. Counts: | Heading level | Emoji | Heading word | Files | HH:MM bullets | | --- | --- | --- | ---: | ---: | | none | no | none | 39 | 590 | | ## | no | <other> | 2 | 4 | The total is 594 outside-section bullets across 41 files. The 39-file no-preceding-heading group contains 590 bullets; the other 4 bullets are under H2 headings in two distinct files. Of the 590, 589 are in 38 files with no accepted Log heading, and 1 is before a later accepted Log heading. This differs from the brief's 39-file/594-bullet grouping, so I will preserve the exact source-parser boundary rather than infer Log sections from headingless content.

kayg referenced this issue from a commit

2026-10-03 10:09:33 +00:00

Align Daily note Log headings with calternal.js

kayg commented

2026-10-03 11:09:05 +00:00

Author

Owner

Notes plugin gate finding: the full cargo test -p calternal-plugin-notes run reported test result: FAILED. 180 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 524.25s. Two IMAP tests timed out at their 10-second read timeout. The other failure was tests::daily_and_composer_preserve_unrelated_bytes (GET /journal/2026-09-24 returned 404).

The 404 test passed when run alone (1 passed). The two IMAP cases also passed in the focused serial run (cargo test -p calternal-plugin-notes captured_ -- --test-threads=1, 3 passed). The plugin test setup uses one fixed User ID across tests, while journal_snapshot uses a non-blocking per-User lock for the missing-ID repair path; parallel tests can therefore make that test return 404. The timeout cases did not reproduce alone. I did not change any test expectation or plugin code because these failures were parallel-test/load effects, not heading parser regressions.

Notes plugin gate finding: the full `cargo test -p calternal-plugin-notes` run reported `test result: FAILED. 180 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 524.25s`. Two IMAP tests timed out at their 10-second read timeout. The other failure was `tests::daily_and_composer_preserve_unrelated_bytes` (`GET /journal/2026-09-24` returned 404). The 404 test passed when run alone (1 passed). The two IMAP cases also passed in the focused serial run (`cargo test -p calternal-plugin-notes captured_ -- --test-threads=1`, 3 passed). The plugin test setup uses one fixed User ID across tests, while `journal_snapshot` uses a non-blocking per-User lock for the missing-ID repair path; parallel tests can therefore make that test return 404. The timeout cases did not reproduce alone. I did not change any test expectation or plugin code because these failures were parallel-test/load effects, not heading parser regressions.

kayg commented

2026-10-03 11:14:19 +00:00

Author

Owner

#998 implementation report

Built: is_log_heading now accepts calternal.js-compatible decorated H2 Log headings through the existing shared day-file parser and byte-span walk. The first matching Log section can follow earlier sections. The time grammar is unchanged. The fixture uses anonymised text, and its test checks that bytes outside the selected Log section stay unchanged.

Files:

crates/calternal-notes-core/src/dayfile.rs
crates/calternal-notes-core/tests/fixtures/log-heading-variants.md

Head: 4e8f22ad6c08574dc94b8841f293c1de1b726675 (Align Daily note Log headings with calternal.js)

Production shape counts

Read-only scan: 662 Daily notes, 0 read errors; no note text, paths, names or titles were emitted.

Heading level	Emoji	Word	Files	HH:MM bullets
none	no	none	39	590
##	no		2	4

No # or ### heading shapes occurred above these outside-section bullets. The 594 bullets span 41 files: 589 are in 38 files with no accepted Log heading; 1 is before a later accepted Log heading; 4 are after the next H2 in 2 files. This differs from the brief's 39-file/594-bullet grouping. The 39 files in the none row have 590 bullets with no preceding Markdown heading.

Gates

cargo fmt --check — no output; exit 0.

cargo clippy -p calternal-notes-core --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 39s

cargo test -p calternal-notes-core output summaries:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 43.72s
test result: ok. 520 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.27s
test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.00s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.32s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The test output has no failures. The capture wrapper then printed zsh:2: read-only variable: status after Cargo completed; I confirmed the summaries from its log.

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 26m 41s

cargo test -p calternal-plugin-notes:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 44m 57s
test result: FAILED. 180 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 524.25s
error: test failed, to rerun pass `-p calternal-plugin-notes --lib`

Two IMAP tests hit their 10-second read timeout. The third failure returned 404 in daily_and_composer_preserve_unrelated_bytes. Follow-up isolation passed: that test alone reported 1 passed; cargo test -p calternal-plugin-notes captured_ -- --test-threads=1 reported 3 passed. The full suite uses parallel tests with a shared fixed test User ID and a process-wide per-User lock; I did not change expectations or plugin code. Please rerun the full Notes plugin gate in the merge round with cargo test -p calternal-plugin-notes -- --test-threads=4 and confirm it has no timeout or 404 failures.

Cleanup:

Removed 7337 files, 2.9GiB total

Decisions and known gaps

The reference parser and calternal.js docs recognize the exact, case-sensitive word Log; I did not infer Log from Journal, Timeline, Today, Time, Tracking, Day, or translations. Those words are only for the requested audit classification.
I retained the core parser's existing whitespace-tolerant forms while adding the reference parser's leading non-ASCII decoration support.
The 590 bullets with no preceding heading and 4 bullets under another H2 remain outside the Log section. Parsing them would need a separate owner decision; this change does not invent sections or rewrite files.
The fetched #724 branches contain the resumable Daily Log projection rebuild; this base did not. The hotfix rebuild should run with this shared parser so it refreshes the newly accepted headings.
I fetched origin; origin/dev had four later design-only commits. I left the branch unmerged as the job says not to merge. The orchestrator can fold this commit into the #724 hotfix or 7b.

#998 implementation report **Built:** `is_log_heading` now accepts calternal.js-compatible decorated H2 Log headings through the existing shared day-file parser and byte-span walk. The first matching Log section can follow earlier sections. The time grammar is unchanged. The fixture uses anonymised text, and its test checks that bytes outside the selected Log section stay unchanged. **Files:** - `crates/calternal-notes-core/src/dayfile.rs` - `crates/calternal-notes-core/tests/fixtures/log-heading-variants.md` **Head:** `4e8f22ad6c08574dc94b8841f293c1de1b726675` (`Align Daily note Log headings with calternal.js`) ## Production shape counts Read-only scan: 662 Daily notes, 0 read errors; no note text, paths, names or titles were emitted. | Heading level | Emoji | Word | Files | HH:MM bullets | | --- | --- | --- | ---: | ---: | | none | no | none | 39 | 590 | | ## | no | <other> | 2 | 4 | No `#` or `###` heading shapes occurred above these outside-section bullets. The 594 bullets span 41 files: 589 are in 38 files with no accepted Log heading; 1 is before a later accepted Log heading; 4 are after the next H2 in 2 files. This differs from the brief's 39-file/594-bullet grouping. The 39 files in the `none` row have 590 bullets with no preceding Markdown heading. ## Gates `cargo fmt --check` — no output; exit 0. `cargo clippy -p calternal-notes-core --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 39s ``` `cargo test -p calternal-notes-core` output summaries: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 43.72s test result: ok. 520 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.27s test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.00s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.32s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` The test output has no failures. The capture wrapper then printed `zsh:2: read-only variable: status` after Cargo completed; I confirmed the summaries from its log. `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 26m 41s ``` `cargo test -p calternal-plugin-notes`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 44m 57s test result: FAILED. 180 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 524.25s error: test failed, to rerun pass `-p calternal-plugin-notes --lib` ``` Two IMAP tests hit their 10-second read timeout. The third failure returned 404 in `daily_and_composer_preserve_unrelated_bytes`. Follow-up isolation passed: that test alone reported `1 passed`; `cargo test -p calternal-plugin-notes captured_ -- --test-threads=1` reported `3 passed`. The full suite uses parallel tests with a shared fixed test User ID and a process-wide per-User lock; I did not change expectations or plugin code. Please rerun the full Notes plugin gate in the merge round with `cargo test -p calternal-plugin-notes -- --test-threads=4` and confirm it has no timeout or 404 failures. Cleanup: ```text Removed 7337 files, 2.9GiB total ``` ## Decisions and known gaps - The reference parser and calternal.js docs recognize the exact, case-sensitive word `Log`; I did not infer Log from `Journal`, `Timeline`, `Today`, `Time`, `Tracking`, `Day`, or translations. Those words are only for the requested audit classification. - I retained the core parser's existing whitespace-tolerant forms while adding the reference parser's leading non-ASCII decoration support. - The 590 bullets with no preceding heading and 4 bullets under another H2 remain outside the Log section. Parsing them would need a separate owner decision; this change does not invent sections or rewrite files. - The fetched #724 branches contain the resumable Daily Log projection rebuild; this base did not. The hotfix rebuild should run with this shared parser so it refreshes the newly accepted headings. - I fetched `origin`; `origin/dev` had four later design-only commits. I left the branch unmerged as the job says not to merge. The orchestrator can fold this commit into the #724 hotfix or 7b.

kayg commented

2026-10-04 04:07:41 +00:00

Author

Owner

Owner decision (2026-10-04): a Daily note that contains NO headings at all is treated as if its whole body were the Log section; its timed bullets (HH:MM and HHMM forms) become Log entries. Notes with a Log heading keep today's rule (bullets under other headings stay ignored); non-Daily notes are never affected; nothing is rewritten on open; the composer adds '## Log' only when the User adds an entry to such a day. Implement on top of job/logheading (decorated headings) and the #724 parser on dev.

kayg commented

2026-10-04 04:09:33 +00:00

Author

Owner

Starting #998 on job/headless-998, based on merge commit 1163d0b7e6f086e05973c24c8774eff48b784518 (decorated Log headings), whose origin/dev parent is f2f8491ff5c76ab28f140c964542c97362e6b119. I am checking the current #724 parser/rebuild work and will implement the owner decision in the shared day-file parser with anonymised fixtures and composer coverage.

Starting #998 on `job/headless-998`, based on merge commit `1163d0b7e6f086e05973c24c8774eff48b784518` (decorated Log headings), whose `origin/dev` parent is `f2f8491ff5c76ab28f140c964542c97362e6b119`. I am checking the current #724 parser/rebuild work and will implement the owner decision in the shared day-file parser with anonymised fixtures and composer coverage.

kayg referenced this issue from a commit

2026-10-04 04:16:27 +00:00

Merge decorated Log headings (#998) as base

kayg referenced this issue from a commit

2026-10-04 05:13:59 +00:00

notes-core: recognize headingless Daily note Logs

kayg commented

2026-10-04 05:35:30 +00:00

Author

Owner

READY FOR MERGE: yes

Built

The shared calternal-notes-core day-file parser treats the complete body of a Daily note as its implicit Log when it has no Markdown headings. Reads preserve source bytes. The first Composer write adds one bare ## Log heading and keeps existing timed bullets as entries.
Migration 0028 clears completed and partial #724 rebuild cursors, so startup queues a new projection rebuild using the shared parser.
The anonymised 39-file fixture indexed 590 headingless Log entries (both HH:MM and HHMM). The rebuild also accepted one entry within an explicit Log and ignored bullets under another H2 and in a non-Daily Note. The rebuild did not change Markdown bytes.
Extended the existing Notes rebuild profile to include 39 headingless Daily notes. No performance measurement was run because #998 is not a performance issue under the 2026-10-02 verification rule.

Files

crates/calternal-notes-core/src/dayfile.rs
crates/calternal-notes-core/tests/fixtures/headingless-daily-note.md
crates/calternal-notes-core/tests/fixtures/daily-note-log-boundaries.md
crates/calternal-notes-core/tests/fixtures/daily-note-other-heading.md
crates/plugins/notes/src/lib.rs
crates/plugins/notes/src/store.rs
crates/plugins/notes/src/reminders_tests.rs
crates/plugins/notes/migrations/0028_headingless_daily_log_rebuild.sql
bench/notes-daily-log-rebuild-724.sh

Commits

6128e7ff961d6ae783488858b4e79b66b20be61e — recognize headingless Daily note Logs in the shared parser.
f9673fa75346a525f2581fd6df6cc6c958acff25 — restart Notes Daily Log projections after the parser change.
Head: f9673fa75346a525f2581fd6df6cc6c958acff25.

Gates

cargo fmt --check
(no output; exit 0)

cargo clippy -p calternal-notes-core --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.60s

cargo test -p calternal-notes-core
test result: ok. 527 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.20s
test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.58s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.38s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

OPENSSL_NO_VENDOR=1 cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.20s

OPENSSL_NO_VENDOR=1 cargo test -p calternal-plugin-notes -- --test-threads=4
test result: ok. 189 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 88.07s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

bash -n bench/notes-daily-log-rebuild-724.sh
(no output; exit 0)

bun run build
✓ built in 19.59s
✔ done

UX gaps closed

Reading a headingless Daily note reports its missing heading without changing the file.
Composer adds the bare heading once; the existing HH:MM and HHMM entries remain visible and are not duplicated.

UX gaps left

None. This change has no UI surface.

Decisions not stated in DESIGN.md

“Headingless” means no ATX or setext Markdown heading outside a fenced code block. Any such heading, including another H2, disables the implicit Log fallback.
On first Composer write, insert ## Log after frontmatter and before the existing body. Keep that body under the new heading.
Use migration 0028 to reset both partial and completed rebuild cursors; 0027 is the latest migration on the fetched origin/dev.

Known gaps / merge round

The real-server adversarial matrix was not run in this job under the 2026-10-02 verification policy. Run tests/adversarial/run.sh against the merged local server and fix or file every non-SLOW finding.
The benchmark profile was extended but not measured; no performance regression claim is made.

READY FOR MERGE: yes ## Built - The shared `calternal-notes-core` day-file parser treats the complete body of a Daily note as its implicit Log when it has no Markdown headings. Reads preserve source bytes. The first Composer write adds one bare `## Log` heading and keeps existing timed bullets as entries. - Migration 0028 clears completed and partial #724 rebuild cursors, so startup queues a new projection rebuild using the shared parser. - The anonymised 39-file fixture indexed **590** headingless Log entries (both HH:MM and HHMM). The rebuild also accepted one entry within an explicit Log and ignored bullets under another H2 and in a non-Daily Note. The rebuild did not change Markdown bytes. - Extended the existing Notes rebuild profile to include 39 headingless Daily notes. No performance measurement was run because #998 is not a performance issue under the 2026-10-02 verification rule. ## Files - `crates/calternal-notes-core/src/dayfile.rs` - `crates/calternal-notes-core/tests/fixtures/headingless-daily-note.md` - `crates/calternal-notes-core/tests/fixtures/daily-note-log-boundaries.md` - `crates/calternal-notes-core/tests/fixtures/daily-note-other-heading.md` - `crates/plugins/notes/src/lib.rs` - `crates/plugins/notes/src/store.rs` - `crates/plugins/notes/src/reminders_tests.rs` - `crates/plugins/notes/migrations/0028_headingless_daily_log_rebuild.sql` - `bench/notes-daily-log-rebuild-724.sh` ## Commits - `6128e7ff961d6ae783488858b4e79b66b20be61e` — recognize headingless Daily note Logs in the shared parser. - `f9673fa75346a525f2581fd6df6cc6c958acff25` — restart Notes Daily Log projections after the parser change. - Head: `f9673fa75346a525f2581fd6df6cc6c958acff25`. ## Gates ```text cargo fmt --check (no output; exit 0) cargo clippy -p calternal-notes-core --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.60s cargo test -p calternal-notes-core test result: ok. 527 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.20s test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.58s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.38s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s OPENSSL_NO_VENDOR=1 cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.20s OPENSSL_NO_VENDOR=1 cargo test -p calternal-plugin-notes -- --test-threads=4 test result: ok. 189 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 88.07s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s bash -n bench/notes-daily-log-rebuild-724.sh (no output; exit 0) bun run build ✓ built in 19.59s ✔ done ``` ## UX gaps closed - Reading a headingless Daily note reports its missing heading without changing the file. - Composer adds the bare heading once; the existing HH:MM and HHMM entries remain visible and are not duplicated. ## UX gaps left - None. This change has no UI surface. ## Decisions not stated in DESIGN.md - “Headingless” means no ATX or setext Markdown heading outside a fenced code block. Any such heading, including another H2, disables the implicit Log fallback. - On first Composer write, insert `## Log` after frontmatter and before the existing body. Keep that body under the new heading. - Use migration 0028 to reset both partial and completed rebuild cursors; 0027 is the latest migration on the fetched `origin/dev`. ## Known gaps / merge round - The real-server adversarial matrix was not run in this job under the 2026-10-02 verification policy. Run `tests/adversarial/run.sh` against the merged local server and fix or file every non-SLOW finding. - The benchmark profile was extended but not measured; no performance regression claim is made.

kayg referenced this issue

2026-10-04 06:15:11 +00:00

Daily Log projection rebuild did not run after the #998 upgrade on production #1033

kayg commented

2026-10-04 06:31:25 +00:00

Author

Owner

Fixed in c3db4039a54ad18d74ba40dfcec3be2594e10256 on job/rebuild-1033 (base 1fd5a630c14b58dafb2649afdf4a8569417d740a). Fetched origin and merged origin/dev once before final gates: already up to date. No push or deploy. Production access was read-only; output contained aggregates and schema only.

Root cause and evidence

Migration 0028 correctly clears both completed and partial rebuild cursors. The #1011 post-bind path starts the job worker and calls the Notes backfill. The live Index is /srv/calternal/data/.system/index.sqlite; the rebuild state remains in notes_daily_log_projection_rebuilds there. It has one incomplete cursor. The live Notes migration version is 28.

The rebuild ran but stopped at the next Daily note. Six jobs are dead after three attempts each, with Notes operation failed: five have progress 0/204, and the upgrade job has progress 0/392. The pre-small3 backup already has five dead jobs and an incomplete cursor, so the #724 pass had not completed in that backup. Live Calendar Log aggregates were 7,212 rows / 505 days.

The next source has repeated Area Tags on two lines. journal_index inserted each occurrence into journal_resource_areas, whose primary key permits one membership per User/path/block/Tag. The unique-key failure rolls back the projection and leaves the cursor in place. This also explains why the headingless notes later in the pass did not appear. A synthetic regression failed on the old code with post-bind worker must complete the parser upgrade: Elapsed(()), then passed with the fix.

Built and files

crates/plugins/notes/src/store.rs: collect DAV Area memberships as a set before inserting them. Keep source Markdown and its Tags intact.
crates/plugins/notes/src/lib.rs: add an upgrade regression using the real migration runner, post-bind backfill and durable worker. Test both completed and interrupted pre-0028 cursors, repeated Area Tags, headingless Daily notes, completion, one pass across repeated startup, and byte-identical source. Update module documentation in both files.
Existing resumability and 39-headingless-note projection regressions also passed.

Gates (verbatim output)

cargo fmt --check: exit 0, no output.

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 56s

cargo test -p calternal-plugin-notes -- --test-threads=4:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 33.53s
test result: ok. 192 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 120.35s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.46s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Decisions

No new product design. Area membership follows set semantics from DESIGN §46. Retain migration 0028 and existing post-bind scheduling: both work. Add no new migration; startup queues the existing incomplete cursor again and resumes it after the fixed projection. No dependencies changed.

UX gaps closed

Existing imported Daily notes with repeated Area Tags can no longer strand the parser projection rebuild before later headingless Daily notes. The test proves automatic indexing without source changes or manual rebuild calls.

Known gaps / UX gaps left

Production completion must be checked after the orchestrator deploys this commit. The job did not deploy or change production. No web code changed, so no UI screenshots or web gates apply. No performance measurement: this is a correctness issue, under the verify-once policy.

Exact post-deploy check (read-only; aggregates only)

Run on production after startup, using Python because sqlite3 CLI is absent:

import sqlite3
c = sqlite3.connect('file:/srv/calternal/data/.system/index.sqlite?mode=ro', uri=True)
print(c.execute("SELECT MAX(version) FROM _migrations WHERE namespace='calternal-plugin-notes'").fetchall())
print(c.execute("SELECT COUNT(*), SUM(completed_ms IS NULL) FROM notes_daily_log_projection_rebuilds").fetchall())
print(c.execute("SELECT COUNT(*), COUNT(DISTINCT day) FROM note_calendar_logs").fetchall())
print(c.execute("SELECT state, COUNT(*) FROM jobs WHERE kind='notes.daily-log-projection-rebuild' GROUP BY state").fetchall())

Expected: migration remains 28; incomplete count becomes 0; a new rebuild job completes (old dead jobs remain history); Log rows increase by approximately 590 over the observed 7,212, subject to concurrent User edits. A later restart must not create another completed parser pass. Do not reset the cursor or change production manually.

For the merge round

Run cargo test -p calternal-server wire::tests::startup_serves_http_while_upgrade_backfills_wait -- --ignored --exact --nocapture --test-threads=1 to retain the real HTTP-bind guarantee from #1011. Full combined suites and adversarial matrices remain merge-round work under the verification policy.

Fixed in `c3db4039a54ad18d74ba40dfcec3be2594e10256` on `job/rebuild-1033` (base `1fd5a630c14b58dafb2649afdf4a8569417d740a`). Fetched origin and merged origin/dev once before final gates: already up to date. No push or deploy. Production access was read-only; output contained aggregates and schema only. ## Root cause and evidence Migration 0028 correctly clears both completed and partial rebuild cursors. The #1011 post-bind path starts the job worker and calls the Notes backfill. The live Index is `/srv/calternal/data/.system/index.sqlite`; the rebuild state remains in `notes_daily_log_projection_rebuilds` there. It has one incomplete cursor. The live Notes migration version is 28. The rebuild ran but stopped at the next Daily note. Six jobs are dead after three attempts each, with `Notes operation failed`: five have progress 0/204, and the upgrade job has progress 0/392. The pre-small3 backup already has five dead jobs and an incomplete cursor, so the #724 pass had not completed in that backup. Live Calendar Log aggregates were 7,212 rows / 505 days. The next source has repeated Area Tags on two lines. `journal_index` inserted each occurrence into `journal_resource_areas`, whose primary key permits one membership per User/path/block/Tag. The unique-key failure rolls back the projection and leaves the cursor in place. This also explains why the headingless notes later in the pass did not appear. A synthetic regression failed on the old code with `post-bind worker must complete the parser upgrade: Elapsed(())`, then passed with the fix. ## Built and files - `crates/plugins/notes/src/store.rs`: collect DAV Area memberships as a set before inserting them. Keep source Markdown and its Tags intact. - `crates/plugins/notes/src/lib.rs`: add an upgrade regression using the real migration runner, post-bind backfill and durable worker. Test both completed and interrupted pre-0028 cursors, repeated Area Tags, headingless Daily notes, completion, one pass across repeated startup, and byte-identical source. Update module documentation in both files. - Existing resumability and 39-headingless-note projection regressions also passed. ## Gates (verbatim output) `cargo fmt --check`: exit 0, no output. `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`: ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 56s ``` `cargo test -p calternal-plugin-notes -- --test-threads=4`: ``` Finished `test` profile [unoptimized + debuginfo] target(s) in 33.53s test result: ok. 192 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 120.35s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.46s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` ## Decisions No new product design. Area membership follows set semantics from DESIGN §46. Retain migration 0028 and existing post-bind scheduling: both work. Add no new migration; startup queues the existing incomplete cursor again and resumes it after the fixed projection. No dependencies changed. ## UX gaps closed Existing imported Daily notes with repeated Area Tags can no longer strand the parser projection rebuild before later headingless Daily notes. The test proves automatic indexing without source changes or manual rebuild calls. ## Known gaps / UX gaps left Production completion must be checked after the orchestrator deploys this commit. The job did not deploy or change production. No web code changed, so no UI screenshots or web gates apply. No performance measurement: this is a correctness issue, under the verify-once policy. ## Exact post-deploy check (read-only; aggregates only) Run on production after startup, using Python because sqlite3 CLI is absent: ```python import sqlite3 c = sqlite3.connect('file:/srv/calternal/data/.system/index.sqlite?mode=ro', uri=True) print(c.execute("SELECT MAX(version) FROM _migrations WHERE namespace='calternal-plugin-notes'").fetchall()) print(c.execute("SELECT COUNT(*), SUM(completed_ms IS NULL) FROM notes_daily_log_projection_rebuilds").fetchall()) print(c.execute("SELECT COUNT(*), COUNT(DISTINCT day) FROM note_calendar_logs").fetchall()) print(c.execute("SELECT state, COUNT(*) FROM jobs WHERE kind='notes.daily-log-projection-rebuild' GROUP BY state").fetchall()) ``` Expected: migration remains 28; incomplete count becomes 0; a new rebuild job completes (old dead jobs remain history); Log rows increase by approximately 590 over the observed 7,212, subject to concurrent User edits. A later restart must not create another completed parser pass. Do not reset the cursor or change production manually. ## For the merge round Run `cargo test -p calternal-server wire::tests::startup_serves_http_while_upgrade_backfills_wait -- --ignored --exact --nocapture --test-threads=1` to retain the real HTTP-bind guarantee from #1011. Full combined suites and adversarial matrices remain merge-round work under the verification policy.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#998

No description provided.

Rows
Columns