Journal: moving a log entry merges it with the next Log line (data corruption, calendar e2e fails on dev) #94

New issue

Closed

opened 2026-09-25 14:30:49 +00:00 by kayg · 1 comment

kayg commented

2026-09-25 14:30:49 +00:00

Owner

Problem

Moving a log entry by drag (PATCH /api/v1/notes/journal/entries/{block_id}) corrupts the Daily note: after the move, the moved line and the next line read back as one entry, and the next line is lost as a separate log entry. This is data corruption. It blocks a merge by the owner rule.

It happens on dev (9ae55cf) without other changes: bun run test:e2e:calendar fails at step 6b with "drag did not move the entry to 10:00". It also happens when the Log lines are in time order (the first report said out of order; that was wrong).

Reproduce

Found by the calendar e2e (apps/web/e2e/calendar.mjs) on branch job/composer-drafts (#93), with the dev server binary at 9ae55cf.

The Log section of the Daily note for one day is:

- 09:00 - 10:30 [tz=Europe/Berlin] Morning review #area/work ^t…a
- 13:00 - 13:45 [tz=Europe/Berlin] Lunch with Ana #area/life ^t…b
- 14:00 - 15:30 [tz=Europe/Berlin] Wrote the calendar test #area/work ^t…c
- 12:00 [tz=Europe/Berlin] Draft sent once ^t…d
- 12:00 [tz=Europe/Berlin] Frozen snapshot ^t…e

PATCH …/journal/entries/t…a with {"date":"<day>","start":"10:00","end":"11:30","tz":"Europe/Berlin"} and the correct If-Match. The response is 200 and looks correct (title: "Morning review").
GET /api/v1/notes/journal/<day> then returns 4 entries, not 5. The first one is:

["10:00","11:30","Morning review #area/work ^t…a- 13:00 - 13:45 [tz=Europe/Berlin] Lunch with Ana"]

The Lunch line and the block ID of the moved line are now in the title of the moved entry. The rewrite probably loses the line break between the rewritten line and the next line, or it splices at an index computed for a sorted order.

The same failure happens with only the three in-order lines (Morning review, Lunch with Ana, Wrote the calendar test).

Acceptance criteria

A move, a resize and an edit of a log entry change only that line, for any order of Log lines.
A regression test in crates/plugins/notes for a move with a following line (in order and out of order).
An adversarial probe in tests/adversarial/ for PATCH on a Log section with several lines.
bun run test:e2e:calendar passes again.

Relevant files

crates/plugins/notes/src/lib.rs (patch_journal_entry → NotesJournalProvider::patch_event)

## Problem Moving a log entry by drag (PATCH `/api/v1/notes/journal/entries/{block_id}`) corrupts the Daily note: after the move, the moved line and the next line read back as one entry, and the next line is lost as a separate log entry. This is data corruption. It blocks a merge by the owner rule. It happens on `dev` (9ae55cf) without other changes: `bun run test:e2e:calendar` fails at step 6b with "drag did not move the entry to 10:00". It also happens when the Log lines are in time order (the first report said out of order; that was wrong). ## Reproduce Found by the calendar e2e (`apps/web/e2e/calendar.mjs`) on branch `job/composer-drafts` (#93), with the dev server binary at 9ae55cf. 1. The Log section of the Daily note for one day is: ``` - 09:00 - 10:30 [tz=Europe/Berlin] Morning review #area/work ^t…a - 13:00 - 13:45 [tz=Europe/Berlin] Lunch with Ana #area/life ^t…b - 14:00 - 15:30 [tz=Europe/Berlin] Wrote the calendar test #area/work ^t…c - 12:00 [tz=Europe/Berlin] Draft sent once ^t…d - 12:00 [tz=Europe/Berlin] Frozen snapshot ^t…e ``` 2. PATCH `…/journal/entries/t…a` with `{"date":"<day>","start":"10:00","end":"11:30","tz":"Europe/Berlin"}` and the correct If-Match. The response is 200 and looks correct (`title: "Morning review"`). 3. GET `/api/v1/notes/journal/<day>` then returns 4 entries, not 5. The first one is: ``` ["10:00","11:30","Morning review #area/work ^t…a- 13:00 - 13:45 [tz=Europe/Berlin] Lunch with Ana"] ``` The Lunch line and the block ID of the moved line are now in the title of the moved entry. The rewrite probably loses the line break between the rewritten line and the next line, or it splices at an index computed for a sorted order. The same failure happens with only the three in-order lines (Morning review, Lunch with Ana, Wrote the calendar test). ## Acceptance criteria - A move, a resize and an edit of a log entry change only that line, for any order of Log lines. - A regression test in `crates/plugins/notes` for a move with a following line (in order and out of order). - An adversarial probe in `tests/adversarial/` for PATCH on a Log section with several lines. - `bun run test:e2e:calendar` passes again. ## Relevant files - `crates/plugins/notes/src/lib.rs` (`patch_journal_entry` → `NotesJournalProvider::patch_event`)

kayg changed title from ~~Journal: moving a log entry merges it with the next line when the Log section is out of time order (data corruption)~~ to Journal: moving a log entry merges it with the next Log line (data corruption, calendar e2e fails on dev)

2026-09-25 14:33:37 +00:00

kayg referenced this issue

2026-09-25 14:38:51 +00:00

Composer: keep drafts on dismiss, commit-freeze snapshot, and the rest of calternal.js composer parity #93

kayg commented

2026-09-25 16:27:39 +00:00

Author

Owner

Fixed on branch job/log-drag. It is not merged or deployed.

Root cause. replace_log_entry_block (crates/calternal-notes-core/src/dayfile.rs) replaces the whole source record: the bullet, its line ending and its child lines. It wrote the line ending back only when the entry had child lines. A move, resize or edit of an entry without children therefore joined the rewritten line with the next line. PATCH /journal/entries/{id} and CalDAV PUT both use this function. The error was not in the sort order or the byte ranges. It happened with lines in order and out of order. Existing tests did not find it because every fixture had a child line.

Fix. The replacement ends with a line ending exactly when the old record did. All other bytes stay identical, and a file without a final newline keeps that shape.

Sibling writers checked and fixed:

serialize_day_file_preserving (task link and unlink, tag rename): a child added to a last line without a final newline was joined to it. The writer also rewrote the bullet in canonical form when only the children changed. same_entry ignored the zone, so a zone-only change was dropped.
append_log_entry (create, move to another day): it found the heading with a text search for ## 📝 Log. A ## Log heading or that phrase in the preamble gave a wrong offset. A CRLF file without a Log heading got LF lines.
CalDAV PUT, same-day branch of NotesJournalProvider::put: this dropped the child lines (task and Note links) and the hide marker of the moved entry, because an ICS body cannot carry them. They are now kept from the file.
Found by the new probe: a Log title with U+2028 was written, and after that every read of that Daily note panicked in links.rs (scan_block_ids). The composer's connector scan had the same slice error.
These were correct already: replace_log_entry, remove_log_entry, append_log_block_id, replace_unparsed_log_line, set_day_file_attachment_text.

Tests: crates/calternal-notes-core/tests/log_rewrite.rs covers the first, middle and last position, a final newline or none, CRLF, out-of-order lines, multi-byte titles and non-Log content after the entry. It also has two seeded property tests with 3,000 random Daily notes each. They check that every other line stays byte-identical after replace, preserve, remove, append and a child link. Plugin tests do the same through the REST route and a CalDAV ICS round trip. Round 2 of the adversarial probe has a new logrewrite section. The calendar e2e passes, including step 6b.

Damage check (read-only): calternal-server find-joined-log-lines [<user-id>]. With no user ID it scans all users. It reads the Daily notes through calternal-fs and writes nothing. It prints one tab-separated row per suspect line: <user-id> <path> <line> <reason> <text>. The reasons are block-id-followed-by-text and second-time-in-title. The file versions keep the text from before the damage.

Not fixed here (follow-ups):

The parser reads a reversed range (23:00 - 01:00) as having no end time. Any rewrite of such a line removes the end text.
POST /journal/log writes with a plain write under the user lock and not with replace_if. A live editor that writes between the read and the write would lose its change.

Fixed on branch `job/log-drag`. It is not merged or deployed. **Root cause.** `replace_log_entry_block` (crates/calternal-notes-core/src/dayfile.rs) replaces the whole source record: the bullet, its line ending and its child lines. It wrote the line ending back only when the entry had child lines. A move, resize or edit of an entry without children therefore joined the rewritten line with the next line. PATCH `/journal/entries/{id}` and CalDAV PUT both use this function. The error was not in the sort order or the byte ranges. It happened with lines in order and out of order. Existing tests did not find it because every fixture had a child line. **Fix.** The replacement ends with a line ending exactly when the old record did. All other bytes stay identical, and a file without a final newline keeps that shape. **Sibling writers checked and fixed:** - `serialize_day_file_preserving` (task link and unlink, tag rename): a child added to a last line without a final newline was joined to it. The writer also rewrote the bullet in canonical form when only the children changed. `same_entry` ignored the zone, so a zone-only change was dropped. - `append_log_entry` (create, move to another day): it found the heading with a text search for `## 📝 Log`. A `## Log` heading or that phrase in the preamble gave a wrong offset. A CRLF file without a Log heading got LF lines. - CalDAV PUT, same-day branch of `NotesJournalProvider::put`: this dropped the child lines (task and Note links) and the hide marker of the moved entry, because an ICS body cannot carry them. They are now kept from the file. - Found by the new probe: a Log title with U+2028 was written, and after that every read of that Daily note panicked in `links.rs` (`scan_block_ids`). The composer's connector scan had the same slice error. - These were correct already: `replace_log_entry`, `remove_log_entry`, `append_log_block_id`, `replace_unparsed_log_line`, `set_day_file_attachment_text`. **Tests:** `crates/calternal-notes-core/tests/log_rewrite.rs` covers the first, middle and last position, a final newline or none, CRLF, out-of-order lines, multi-byte titles and non-Log content after the entry. It also has two seeded property tests with 3,000 random Daily notes each. They check that every other line stays byte-identical after replace, preserve, remove, append and a child link. Plugin tests do the same through the REST route and a CalDAV ICS round trip. Round 2 of the adversarial probe has a new `logrewrite` section. The calendar e2e passes, including step 6b. **Damage check (read-only):** `calternal-server find-joined-log-lines [<user-id>]`. With no user ID it scans all users. It reads the Daily notes through calternal-fs and writes nothing. It prints one tab-separated row per suspect line: `<user-id> <path> <line> <reason> <text>`. The reasons are `block-id-followed-by-text` and `second-time-in-title`. The file versions keep the text from before the damage. **Not fixed here (follow-ups):** - The parser reads a reversed range (`23:00 - 01:00`) as having no end time. Any rewrite of such a line removes the end text. - `POST /journal/log` writes with a plain write under the user lock and not with `replace_if`. A live editor that writes between the read and the write would lose its change.

kayg referenced this issue

2026-09-25 16:28:37 +00:00

Journal: creating a log entry can overwrite a concurrent live-editor save (data loss) #98

kayg referenced this issue

2026-09-25 16:28:38 +00:00

Journal: overnight ranges (23:00 - 01:00) lose their end time on any rewrite #99

kayg closed this issue

2026-09-25 16:55:32 +00:00

kayg referenced this issue from a commit

2026-09-25 16:55:33 +00:00

test(notes): reproduce Log line join on entry move (#94)

kayg referenced this issue from a commit