kayg/calternal

Watch

Fork 0

DATA: task titles drop non-ASCII letters ("José" → "Jos"); non-Latin tasks are refused #126

New issue

Closed

opened 2026-09-25 22:17:30 +00:00 by kayg · 3 comments

kayg commented

2026-09-25 22:17:30 +00:00

Owner

Severity: high (DATA: text the user typed is silently lost)

Problem

The task NLP tokenizer (crates/calternal-notes-core/src/nlp/tokenizer.rs) knew only ASCII letters. Each other letter became a Symbol token, and residual_title_tasks builds the title from Word tokens only. So:

POST /api/v1/notes/tasks {"text":"Call José about the Zürich café"} saved the title "Call Jos about the Z rich caf" (file Tasks/20260925-call-jos-about-the-z-rich-caf-….md).
{"text":"مهمة عربية"} (any non-Latin task) failed with 400 invalid Task title.

The calendar all-day row then showed "Call Jos about the Z rich caf" (/home/kayg/Developer/calternal/target/breakit/run1/cal-day_320-paper.png).

Repro

POST /api/v1/notes/tasks with {"text":"Call José about the Zürich café","standalone":true}.
Read title in the response.

Expected

The title keeps every letter: "Call José about the Zürich café". Arabic, Persian (with ZWNJ), CJK and decomposed accents work.

Status

Fixed on job/breakit-fixes in 8e4da6e (words take letters of any script, combining diacritics and ZWJ/ZWNJ; regression test nlp::tasks::tests::parse_task_input_keeps_letters_of_every_script). Not merged.

Follow-up to check: the task title also drops ASCII punctuation (Don't → Don t, foo.rs → foo rs, milk & eggs → milk eggs), because only Word/Number tokens are kept. The log-entry path slices the input by byte range and keeps them. Decide if the task path should do the same.

Found by the break-it sweep (#117). Re-run: cd apps/web && bun run build && bun e2e/breakit.mjs --keep <dir> (script on branch job/breakit-fixes). Screenshots: /home/kayg/Developer/calternal/target/breakit (run 1 in run1/, fix checks in verify/).

**Severity:** high (DATA: text the user typed is silently lost) ## Problem The task NLP tokenizer (`crates/calternal-notes-core/src/nlp/tokenizer.rs`) knew only ASCII letters. Each other letter became a `Symbol` token, and `residual_title_tasks` builds the title from `Word` tokens only. So: - `POST /api/v1/notes/tasks {"text":"Call José about the Zürich café"}` saved the title **"Call Jos about the Z rich caf"** (file `Tasks/20260925-call-jos-about-the-z-rich-caf-….md`). - `{"text":"مهمة عربية"}` (any non-Latin task) failed with 400 `invalid Task title`. The calendar all-day row then showed "Call Jos about the Z rich caf" (`/home/kayg/Developer/calternal/target/breakit/run1/cal-day_320-paper.png`). ## Repro 1. `POST /api/v1/notes/tasks` with `{"text":"Call José about the Zürich café","standalone":true}`. 2. Read `title` in the response. ## Expected The title keeps every letter: "Call José about the Zürich café". Arabic, Persian (with ZWNJ), CJK and decomposed accents work. ## Status Fixed on `job/breakit-fixes` in 8e4da6e (words take letters of any script, combining diacritics and ZWJ/ZWNJ; regression test `nlp::tasks::tests::parse_task_input_keeps_letters_of_every_script`). Not merged. Follow-up to check: the task title also drops ASCII punctuation (`Don't` → `Don t`, `foo.rs` → `foo rs`, `milk & eggs` → `milk eggs`), because only Word/Number tokens are kept. The log-entry path slices the input by byte range and keeps them. Decide if the task path should do the same. Found by the break-it sweep (#117). Re-run: `cd apps/web && bun run build && bun e2e/breakit.mjs --keep <dir>` (script on branch `job/breakit-fixes`). Screenshots: `/home/kayg/Developer/calternal/target/breakit` (run 1 in `run1/`, fix checks in `verify/`).

kayg commented

2026-09-25 23:17:50 +00:00

Author

Owner

Second part fixed on job/breakit-fixes in 514a327: the title also dropped every Symbol token ("🎉 emoji task" was saved as "Emoji task", "€5" lost its sign) and put a space between every token ("10am" → "10 am"). residual_title_tasks now keeps symbols and joins tokens that touch in the input, so ZWJ emoji stay whole. Two existing tests changed from "10 am"/"5 pm" to the verbatim "10am"/"5pm". ASCII punctuation (Don't, foo.rs) is still dropped; that is the open follow-up above.

Second part fixed on `job/breakit-fixes` in 514a327: the title also dropped every Symbol token ("🎉 emoji task" was saved as "Emoji task", "€5" lost its sign) and put a space between every token ("10am" → "10 am"). `residual_title_tasks` now keeps symbols and joins tokens that touch in the input, so ZWJ emoji stay whole. Two existing tests changed from "10 am"/"5 pm" to the verbatim "10am"/"5pm". ASCII punctuation (`Don't`, `foo.rs`) is still dropped; that is the open follow-up above.

kayg commented

2026-09-26 01:03:28 +00:00

Author

Owner

Fixed on `job/task-unicode` (not merged, not pushed)

Commits (from dev bff3afb):

5a8d79a notes-core: Task titles keep the text the user typed, in every script
bfd5ac2 notes: count Task title and composer limits in characters
3d9c10a notes: a retitle that keeps the slug still changes the title
e3471b5 notes: API tests for titles in every script
3f6503c tests(adversarial): Unicode Task, Note and Log titles

Root causes

crates/calternal-notes-core/src/nlp/classifier.rs residual_title_tasks (was line 1177): the Task title was the unconsumed Word/Number tokens joined by spaces, so every other byte was dropped.
crates/calternal-notes-core/src/nlp/tokenizer.rs:102: words were ASCII letters only, so each other letter was a Symbol token. Together with (1): "Call José" became "Call Jos", and a Hindi, Arabic or Japanese Task had an empty title, so valid_note_title refused it ("invalid Task title"). Punctuation and emoji were lost too ("Don't" became "Don t"). The 8e4da6e fix on job/breakit-fixes is not enough for Hindi: the virama (U+094D) is not alphabetic, so "हिन्दी" still split.
crates/calternal-notes-core/src/nlp/tasks.rs capitalize_first: "ﬁ" becomes "FI", and with (1) "ﬁle taxes" became "Le taxes".
crates/calternal-notes-core/src/tasks/frontmatter.rs read_scalar/needs_quote: str::trim removed U+00A0 and U+3000 at the edge of a Task title on each read.
crates/plugins/notes/src/tasks_api.rs (PATCH value.len() > 1000, composer text.len() > 4096) and crates/plugins/notes/src/tasks_dav.rs:161 (edit.title.len() > 1000): limits were counted in bytes. A 400-letter Hindi title (1200 bytes) was refused.
crates/plugins/notes/src/store.rs transition (was line 932): a Note retitle returned early when the slug did not change, so the new title was lost. Two non-Latin titles both have the slug untitled, so a retitle from one Hindi title to another did nothing. A change in case or punctuation only was lost too.

Fixes

The Task title is now the input sliced around the consumed spans, byte for byte, in any script. The cleanup only removes a punctuation-only edge word, an edge ,/;/: and runs of ASCII whitespace.
Words hold letters of any script, combining marks and ZWJ/ZWNJ, so a recognizer can no longer claim an ASCII fragment inside a word ("Zürich" held "rich").
Capitalization is applied only when the capital is one letter. Scripts without case are unchanged.
Task frontmatter quotes a value with Unicode whitespace at an edge. The reader trims only ASCII space and tab.
All title and text limits count characters. The refusals are only: control characters, bidi controls, blank after trim, and more than 1000 characters.
A same-slug retitle writes the Note and the referrer label edits in place, each as a checked write. The referrer rewrite is now one helper, shared with the rename replay.

Checked and correct already: filenames (slugify keeps an ASCII slug or untitled, never empty, and the id slice keeps it unique; the filename is not the title source). Also correct: Note create and templates (frontmatter title and H1), Log entries (span removal), attachment link text and targets, linked Notes, and VTODO SUMMARY. fold_ical_lines already folds on UTF-8 boundaries, and the new tests show that unfolding gives back the exact title, joiners included. No title is derived from a filename. extract_task_index reads only the frontmatter title.

Tests

crates/calternal-notes-core/tests/unicode_titles.rs: 16 scripts (Latin with diacritics, Devanagari, CJK, Hangul, Arabic, Persian with ZWNJ, Thai, Cyrillic, Greek, decomposed accents, ZWJ emoji with a skin tone, ASCII punctuation). Also claimed words around Unicode text, capitalization, and Unicode edge spaces. There are 2 proptests: random Unicode titles through parse and the file writer (512 cases), and any printable title through a frontmatter rename.
crates/plugins/notes/src/tests/unicode_titles.rs uses the real router: Tasks create, parse, by-id and rename, then Reminders get, put and the VTODO wire form (at most 75 octets per line). Also a long Devanagari Reminders create, character-counted limits, Note create and retitle, templates, Log entries with a non-Latin attachment, and linked Notes. The property test runs random Unicode titles through create, read, rename and CalDAV.
crates/calternal-dav/src/reminders.rs vtodo_summary_round_trips_every_script_across_folds.
tests/adversarial/attack2.py, new section unicode-titles.
These tests failed before the fix. Two existing tests pinned the old mangling ("10am" became "10 am") and now expect the verbatim "10am".

Gates

cargo fmt --check                                   -> fmt exit 0
cargo clippy --workspace --all-targets -- -D warnings -> Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 39s / clippy exit 0
cargo test --workspace                              -> test exit 0 (66 test binaries, 1127 passed, 0 failed)
bash tests/adversarial/run.sh                       -> adv exit 1

Adversarial: unicode-titles has only SLOW findings. The load average was 50 to 72 while other jobs ran on the same host. No title was lost and every hostile title was refused. The non-SLOW findings are all in journalrace: move timed out (PATCH of a Log entry, not changed here, already tracked as #147). In the second run there was also one sync upload 500 "files Index update failed", in the Files plugin, which is also not changed here. Both look like load. The web did not change, and the OpenAPI did not change, so check-generated.sh and the bun gates were not needed. cargo clean done.

Existing data

Titles cannot be repaired from version history. The characters were lost in parse_task_input, before the first write, so the Task file never held them. Every Version holds a copy of a file that already had the lost title. The Log attachment line (- [ ] → [title](Tasks/…)) was written from the same parsed title. The raw composer text is not stored anywhere. Non-Latin Tasks were refused, so no file exists for them. Affected Tasks can be found but not fixed automatically. They are Task files with no status: done, created before the merge, whose title has single-letter fragments such as "Z rich" or "caf", or whose slug is shorter than expected. The owner must retype them. A Note retitle that was lost (same slug) left the old title in place, and it is safe to retitle again.

## Fixed on `job/task-unicode` (not merged, not pushed) Commits (from `dev` bff3afb): - `5a8d79a` notes-core: Task titles keep the text the user typed, in every script - `bfd5ac2` notes: count Task title and composer limits in characters - `3d9c10a` notes: a retitle that keeps the slug still changes the title - `e3471b5` notes: API tests for titles in every script - `3f6503c` tests(adversarial): Unicode Task, Note and Log titles ### Root causes 1. `crates/calternal-notes-core/src/nlp/classifier.rs` `residual_title_tasks` (was line 1177): the Task title was the unconsumed Word/Number tokens joined by spaces, so every other byte was dropped. 2. `crates/calternal-notes-core/src/nlp/tokenizer.rs:102`: words were ASCII letters only, so each other letter was a Symbol token. Together with (1): "Call José" became "Call Jos", and a Hindi, Arabic or Japanese Task had an empty title, so `valid_note_title` refused it ("invalid Task title"). Punctuation and emoji were lost too ("Don't" became "Don t"). The 8e4da6e fix on `job/breakit-fixes` is not enough for Hindi: the virama (U+094D) is not alphabetic, so "हिन्दी" still split. 3. `crates/calternal-notes-core/src/nlp/tasks.rs` `capitalize_first`: "ﬁ" becomes "FI", and with (1) "ﬁle taxes" became "Le taxes". 4. `crates/calternal-notes-core/src/tasks/frontmatter.rs` `read_scalar`/`needs_quote`: `str::trim` removed U+00A0 and U+3000 at the edge of a Task title on each read. 5. `crates/plugins/notes/src/tasks_api.rs` (PATCH `value.len() > 1000`, composer `text.len() > 4096`) and `crates/plugins/notes/src/tasks_dav.rs:161` (`edit.title.len() > 1000`): limits were counted in bytes. A 400-letter Hindi title (1200 bytes) was refused. 6. `crates/plugins/notes/src/store.rs` `transition` (was line 932): a Note retitle returned early when the slug did not change, so the new title was lost. Two non-Latin titles both have the slug `untitled`, so a retitle from one Hindi title to another did nothing. A change in case or punctuation only was lost too. ### Fixes - The Task title is now the input sliced around the consumed spans, byte for byte, in any script. The cleanup only removes a punctuation-only edge word, an edge `,`/`;`/`:` and runs of ASCII whitespace. - Words hold letters of any script, combining marks and ZWJ/ZWNJ, so a recognizer can no longer claim an ASCII fragment inside a word ("Zürich" held "rich"). - Capitalization is applied only when the capital is one letter. Scripts without case are unchanged. - Task frontmatter quotes a value with Unicode whitespace at an edge. The reader trims only ASCII space and tab. - All title and text limits count characters. The refusals are only: control characters, bidi controls, blank after trim, and more than 1000 characters. - A same-slug retitle writes the Note and the referrer label edits in place, each as a checked write. The referrer rewrite is now one helper, shared with the rename replay. Checked and correct already: filenames (`slugify` keeps an ASCII slug or `untitled`, never empty, and the id slice keeps it unique; the filename is not the title source). Also correct: Note create and templates (frontmatter title and H1), Log entries (span removal), attachment link text and targets, linked Notes, and VTODO SUMMARY. `fold_ical_lines` already folds on UTF-8 boundaries, and the new tests show that unfolding gives back the exact title, joiners included. No title is derived from a filename. `extract_task_index` reads only the frontmatter `title`. ### Tests - `crates/calternal-notes-core/tests/unicode_titles.rs`: 16 scripts (Latin with diacritics, Devanagari, CJK, Hangul, Arabic, Persian with ZWNJ, Thai, Cyrillic, Greek, decomposed accents, ZWJ emoji with a skin tone, ASCII punctuation). Also claimed words around Unicode text, capitalization, and Unicode edge spaces. There are 2 proptests: random Unicode titles through parse and the file writer (512 cases), and any printable title through a frontmatter rename. - `crates/plugins/notes/src/tests/unicode_titles.rs` uses the real router: Tasks create, parse, by-id and rename, then Reminders get, put and the VTODO wire form (at most 75 octets per line). Also a long Devanagari Reminders create, character-counted limits, Note create and retitle, templates, Log entries with a non-Latin attachment, and linked Notes. The property test runs random Unicode titles through create, read, rename and CalDAV. - `crates/calternal-dav/src/reminders.rs` `vtodo_summary_round_trips_every_script_across_folds`. - `tests/adversarial/attack2.py`, new section `unicode-titles`. - These tests failed before the fix. Two existing tests pinned the old mangling ("10am" became "10 am") and now expect the verbatim "10am". ### Gates ``` cargo fmt --check -> fmt exit 0 cargo clippy --workspace --all-targets -- -D warnings -> Finished `dev` profile [unoptimized + debuginfo] target(s) in 7m 39s / clippy exit 0 cargo test --workspace -> test exit 0 (66 test binaries, 1127 passed, 0 failed) bash tests/adversarial/run.sh -> adv exit 1 ``` Adversarial: `unicode-titles` has only SLOW findings. The load average was 50 to 72 while other jobs ran on the same host. No title was lost and every hostile title was refused. The non-SLOW findings are all in `journalrace`: `move` timed out (PATCH of a Log entry, not changed here, already tracked as #147). In the second run there was also one `sync upload` 500 "files Index update failed", in the Files plugin, which is also not changed here. Both look like load. The web did not change, and the OpenAPI did not change, so `check-generated.sh` and the `bun` gates were not needed. `cargo clean` done. ### Existing data Titles cannot be repaired from version history. The characters were lost in `parse_task_input`, before the first write, so the Task file never held them. Every Version holds a copy of a file that already had the lost title. The Log attachment line (`- [ ] → [title](Tasks/…)`) was written from the same parsed title. The raw composer text is not stored anywhere. Non-Latin Tasks were refused, so no file exists for them. Affected Tasks can be found but not fixed automatically. They are Task files with no `status: done`, created before the merge, whose title has single-letter fragments such as "Z rich" or "caf", or whose slug is shorter than expected. The owner must retype them. A Note retitle that was lost (same slug) left the old title in place, and it is safe to retitle again.

kayg commented

2026-09-26 01:28:10 +00:00

Author

Owner

Merged into dev; deploys after the next full gate run.

kayg referenced this issue from a commit

2026-09-26 07:03:12 +00:00

notes-core: Task titles keep the text the user typed, in every script (#126)

kayg referenced this issue from a commit

2026-09-26 07:03:12 +00:00

notes: count Task title and composer limits in characters (#126)

kayg referenced this issue from a commit