Canonically equivalent NFD Search query misses NFC and NFD filenames

kayg commented

2026-10-07 22:25:07 +00:00

Owner

Non-blocking Search consistency finding from #1403's single real release-server adversarial round (runtime ef0aa12b1, unchanged Rust runtime source on the merged branch).

The probe uploads café-nfc.txt and cafe\u0301-nfd.txt with distinct body sentinels. Both uploads and sentinel lookups pass. The NFC query café finds both expected normalized filenames. The canonically equivalent NFD query cafe\u0301 returns 200 but does not include either filename.

Verbatim output (the list contains actual returned titles, not the missing names):

FAIL search chaos: NFD canonical Unicode query missed expected filenames: ['Tag receipt 6990047099474b6a946fa98c621bc72a']

Expected title set: café-nfc.txt, café-nfd.txt. The existing regression is unicode_and_mutation_probes in tests/adversarial/search_chaos.py (#1044). Keep its expectations. Investigate canonical query normalization across filename and body matching; no root cause is claimed yet.

No crash, data loss or authorization hole is observed in this case. Files and sentinel contents remain present, and the rest of the chaos/restart checks continue. File under odd-but-harmless input consistency per the merge contract.

Non-blocking Search consistency finding from #1403's single real release-server adversarial round (runtime ef0aa12b1, unchanged Rust runtime source on the merged branch). The probe uploads `café-nfc.txt` and `cafe\u0301-nfd.txt` with distinct body sentinels. Both uploads and sentinel lookups pass. The NFC query `café` finds both expected normalized filenames. The canonically equivalent NFD query `cafe\u0301` returns 200 but does not include either filename. Verbatim output (the list contains actual returned titles, not the missing names): ``` FAIL search chaos: NFD canonical Unicode query missed expected filenames: ['Tag receipt 6990047099474b6a946fa98c621bc72a'] ``` Expected title set: `café-nfc.txt`, `café-nfd.txt`. The existing regression is `unicode_and_mutation_probes` in `tests/adversarial/search_chaos.py` (#1044). Keep its expectations. Investigate canonical query normalization across filename and body matching; no root cause is claimed yet. No crash, data loss or authorization hole is observed in this case. Files and sentinel contents remain present, and the rest of the chaos/restart checks continue. File under odd-but-harmless input consistency per the merge contract.

search-unicode-1411-click-nfc-390-light.png

72 KiB

search-unicode-1411-click-nfc-390-dark.png

79 KiB

search-unicode-1411-click-nfc-820-dark.png

168 KiB

search-unicode-1411-click-nfc-1440-dark.png

218 KiB

search-unicode-1411-click-nfc-1440-light.png

202 KiB

search-unicode-1411-click-nfc-820-light.png

165 KiB

search-unicode-1411-click-nfd-1440-dark.png

220 KiB

search-unicode-1411-click-nfd-390-dark.png

79 KiB

search-unicode-1411-click-nfd-1440-light.png

204 KiB

search-unicode-1411-click-nfd-390-light.png

77 KiB

search-unicode-1411-click-nfd-820-light.png

167 KiB

search-unicode-1411-click-nfd-820-dark.png

171 KiB

search-unicode-1411-shortcut-nfd-1440-dark.png

220 KiB

search-unicode-1411-shortcut-nfd-1440-light.png

205 KiB

search-unicode-1411-shortcut-nfd-390-dark.png

79 KiB

search-unicode-1411-shortcut-nfd-390-light.png

72 KiB

search-unicode-1411-shortcut-nfd-820-dark.png

171 KiB

search-unicode-1411-shortcut-nfd-820-light.png

167 KiB

search-unicode-1411-click-nfd-390-dark.png

79 KiB

kayg referenced this issue

2026-10-07 22:40:13 +00:00

Merge round 34 #1403

kayg referenced this issue

2026-10-07 22:46:52 +00:00

Semantic Search recall fixture remains absent after its 120-second deadline #1415

kayg referenced this issue

2026-10-07 23:30:43 +00:00

Merge round 34 #1403

kayg commented

2026-10-08 06:49:28 +00:00

Author

Owner

Starting #1411 on branch job/nfd-1411, based on origin/dev d395b6e25d. I read the issue and the search decisions in DESIGN §32. I’m tracing query/index normalization and result highlighting, then I’ll add the two-way NFD/NFC regression and browser coverage.

Starting #1411 on branch job/nfd-1411, based on origin/dev d395b6e25da20d8c67b8b9fc1a593b813939c8a7. I read the issue and the search decisions in DESIGN §32. I’m tracing query/index normalization and result highlighting, then I’ll add the two-way NFD/NFC regression and browser coverage.

kayg commented

2026-10-08 07:06:29 +00:00

Author

Owner

The existing Rust regression could pass from body hits alone. I separated filename-only documents (distinct body sentinels) from body-only documents. The focused calternal-search test passes for NFC and NFD queries against both filename spellings, and snippets retain the original composed/decomposed text. This in-memory search path does not reproduce the reported miss; I’m checking the real upload and per-User index path next.

kayg referenced this issue from a commit

2026-10-08 07:25:46 +00:00

Separate filename and body Unicode search regressions (#1411)

kayg commented

2026-10-08 07:46:01 +00:00

Author

Owner

The real-browser flow now reproduces #1411 on a fresh server and private User Index. After actual Tus uploads, the NFC query shows both indexed filename spellings; switching the same Search field to NFD does not show both expected filename rows. The isolated calternal-search query test passes, so I am tracing the difference through the private-index and HTTP Search path. The NFD browser assertion fails before any fix.

kayg referenced this issue from a commit

2026-10-08 08:22:08 +00:00

Store Search names in NFC and rebuild old indexes (#1411)

kayg commented

2026-10-08 08:29:11 +00:00

Author

Owner

Real-browser finding after NFC index rebuild: the setup wait confirms both NFC and NFD direct keyword queries with limit=50 return all four uploaded fixtures. In the actual Search UI, typing the NFD spelling sends /api/v1/search?q=cafe%CC%81&limit=20&semantic=false and receives zero search plugin hits; the same session immediately gets all four from a direct query with limit=50. The browser test is still failing at the visible NFD filename assertion. I am tracing the default-limit/relevance path before changing the test or implementation.

Real-browser finding after NFC index rebuild: the setup wait confirms both NFC and NFD direct keyword queries with limit=50 return all four uploaded fixtures. In the actual Search UI, typing the NFD spelling sends `/api/v1/search?q=cafe%CC%81&limit=20&semantic=false` and receives zero `search` plugin hits; the same session immediately gets all four from a direct query with limit=50. The browser test is still failing at the visible NFD filename assertion. I am tracing the default-limit/relevance path before changing the test or implementation.

kayg referenced this issue from a commit

2026-10-08 08:50:59 +00:00

Verify Unicode Search in the real browser (#1411)

kayg commented

2026-10-08 08:55:28 +00:00

Author

Owner

Real-browser regression evidence for #1411: using the pre-fix server binary from d395b6e25, the Search palette sends the NFD query with limit=20 and returns zero Search hits; an immediate direct request with the same query and limit=20 returns all four fixtures. The browser test fails on the visible filename rows. With the fixed server, the exact-limit fixture barrier and the User-visible Search assertions pass for 390, 820, and 1440 px in light and dark, after click and after Meta+K.

Real-browser regression evidence for #1411: using the pre-fix server binary from d395b6e25, the Search palette sends the NFD query with limit=20 and returns zero Search hits; an immediate direct request with the same query and limit=20 returns all four fixtures. The browser test fails on the visible filename rows. With the fixed server, the exact-limit fixture barrier and the User-visible Search assertions pass for 390, 820, and 1440 px in light and dark, after click and after Meta+K.

kayg commented

2026-10-08 08:57:28 +00:00

Author

Owner

#1411 complete

Branch: job/nfd-1411
Base: d395b6e25da20d8c67b8b9fc1a593b813939c8a7
Head: ffbf8de562bcc4f32ba5a1fe9b077436ce514a6e

Built

Search now normalizes query text and indexed file names/titles to NFC. Paths and body text keep their original spelling. Search snippets use the original body byte offsets, so highlights cover the correct source text. Old private Search indexes queue the staged generation rebuild.

The browser regression uploads real fixtures and checks the existing Search palette with NFC and NFD queries. It checks results opened by click and ⌘K. The test uses the palette's limit=20 in its index readiness barrier. The test failed against the pre-fix server: the palette returned zero NFD Search hits while a direct limit=20 request in the same session returned all four fixtures. The fixed server passed the full viewport/theme matrix.

The existing Search palette in apps/web/src/lib/components/search-dialog.svelte supplies the visible UI. No new controls or styles were added.

Files

crates/calternal-search/src/index.rs
crates/calternal-search/src/query.rs
crates/calternal-fs/src/root.rs
crates/calternal-fs/src/lib.rs
apps/web/e2e/search-unicode-1411.mjs
apps/web/package.json

Commits: 1b3b28f1f, 7023f90ea, 2e40894b4, ffbf8de56.

Gate output

The final git fetch origin && git merge origin/dev reported Already up to date. Rust gates passed on the unchanged Rust tree; later commits changed only the browser test. cargo fmt --check passed after the merge.

Verbatim output excerpts:

cargo fmt --check
(no output; exit 0)

Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 55s
Finished `dev` profile [unoptimized + debuginfo] target(s) in 21.97s
60 passed; 0 failed; 1 ignored
29 passed
6 passed
2 passed
1 passed; 2 ignored
4 passed
1 passed
1 passed
98 passed; 0 failed; 1 ignored
1 passed
1 passed
52 passed
2 passed

PASS production dependency licences: 759 locked package releases across apps/web, apps/docs and packages
perf-lint: PASS; 0 violations; 22544 scoped exceptions
svelte-check found 0 errors and 4 warnings in 2 files

 Test Files  297 passed (297)
      Tests  2116 passed (2116)
   Duration  640.35s (environment 28%, transform 25%, import 24%, tests 16%, setup 7%)
bun run test exit code: 0

PASS Search NFC/NFD filenames and snippet offsets at 390px light
PASS Search NFC/NFD filenames and snippet offsets at 390px dark
PASS Search NFC/NFD filenames and snippet offsets at 820px light
PASS Search NFC/NFD filenames and snippet offsets at 820px dark
PASS Search NFC/NFD filenames and snippet offsets at 1440px light
PASS Search NFC/NFD filenames and snippet offsets at 1440px dark

svelte-check reported four existing unused-selector warnings in MailView.svelte and routes/money/[budget]/[month]/+page.svelte. node --check apps/web/e2e/search-unicode-1411.mjs and git diff --check exited 0 with no output.

UX gaps closed / left

Closed: NFD and NFC queries return both canonical filename titles; NFD and NFC body matches keep correctly highlighted source text. Click and ⌘K captures show no visible focus ring. The 18 screenshots show loaded results at 390, 820 and 1440 px, in light and dark, with macOS platform emulation.

Left: no known functional gap in this issue. The full E2E shell suite and adversarial matrix are merge-round checks under the owner verification policy.

Decisions

DESIGN §32 does not choose a Unicode form for Search names. I chose NFC for query text and stored Search labels. Stable path identity and body source bytes remain unchanged. I bumped the Search tokenizer schema names and private readiness marker to version 3 so existing indexes rebuild once per User.

Expected cost: NFC validation is linear in query/name length. Already-NFC strings are borrowed; decomposed strings allocate once. Existing body snippet offsets remain source-based.

Screenshots

All screenshots are attached to this issue. Captures use the existing Search palette and have the exact pixel sizes shown in their labels.

390 px:

820 px:

1440 px:

For the merge round

cd apps/web && bun run test:e2e — verify the full shell E2E path against the merged build.
bash tests/adversarial/run.sh — run the hostile-input, authorization and concurrency matrices on the merged server.

#1411 complete Branch: `job/nfd-1411` Base: `d395b6e25da20d8c67b8b9fc1a593b813939c8a7` Head: `ffbf8de562bcc4f32ba5a1fe9b077436ce514a6e` ## Built Search now normalizes query text and indexed file names/titles to NFC. Paths and body text keep their original spelling. Search snippets use the original body byte offsets, so highlights cover the correct source text. Old private Search indexes queue the staged generation rebuild. The browser regression uploads real fixtures and checks the existing Search palette with NFC and NFD queries. It checks results opened by click and ⌘K. The test uses the palette's `limit=20` in its index readiness barrier. The test failed against the pre-fix server: the palette returned zero NFD Search hits while a direct `limit=20` request in the same session returned all four fixtures. The fixed server passed the full viewport/theme matrix. The existing Search palette in `apps/web/src/lib/components/search-dialog.svelte` supplies the visible UI. No new controls or styles were added. ## Files - `crates/calternal-search/src/index.rs` - `crates/calternal-search/src/query.rs` - `crates/calternal-fs/src/root.rs` - `crates/calternal-fs/src/lib.rs` - `apps/web/e2e/search-unicode-1411.mjs` - `apps/web/package.json` Commits: `1b3b28f1f`, `7023f90ea`, `2e40894b4`, `ffbf8de56`. ## Gate output The final `git fetch origin && git merge origin/dev` reported `Already up to date.` Rust gates passed on the unchanged Rust tree; later commits changed only the browser test. `cargo fmt --check` passed after the merge. Verbatim output excerpts: ```text cargo fmt --check (no output; exit 0) Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 55s Finished `dev` profile [unoptimized + debuginfo] target(s) in 21.97s 60 passed; 0 failed; 1 ignored 29 passed 6 passed 2 passed 1 passed; 2 ignored 4 passed 1 passed 1 passed 98 passed; 0 failed; 1 ignored 1 passed 1 passed 52 passed 2 passed PASS production dependency licences: 759 locked package releases across apps/web, apps/docs and packages perf-lint: PASS; 0 violations; 22544 scoped exceptions svelte-check found 0 errors and 4 warnings in 2 files Test Files 297 passed (297) Tests 2116 passed (2116) Duration 640.35s (environment 28%, transform 25%, import 24%, tests 16%, setup 7%) bun run test exit code: 0 PASS Search NFC/NFD filenames and snippet offsets at 390px light PASS Search NFC/NFD filenames and snippet offsets at 390px dark PASS Search NFC/NFD filenames and snippet offsets at 820px light PASS Search NFC/NFD filenames and snippet offsets at 820px dark PASS Search NFC/NFD filenames and snippet offsets at 1440px light PASS Search NFC/NFD filenames and snippet offsets at 1440px dark ``` `svelte-check` reported four existing unused-selector warnings in `MailView.svelte` and `routes/money/[budget]/[month]/+page.svelte`. `node --check apps/web/e2e/search-unicode-1411.mjs` and `git diff --check` exited 0 with no output. ## UX gaps closed / left Closed: NFD and NFC queries return both canonical filename titles; NFD and NFC body matches keep correctly highlighted source text. Click and ⌘K captures show no visible focus ring. The 18 screenshots show loaded results at 390, 820 and 1440 px, in light and dark, with macOS platform emulation. Left: no known functional gap in this issue. The full E2E shell suite and adversarial matrix are merge-round checks under the owner verification policy. ## Decisions DESIGN §32 does not choose a Unicode form for Search names. I chose NFC for query text and stored Search labels. Stable path identity and body source bytes remain unchanged. I bumped the Search tokenizer schema names and private readiness marker to version 3 so existing indexes rebuild once per User. Expected cost: NFC validation is linear in query/name length. Already-NFC strings are borrowed; decomposed strings allocate once. Existing body snippet offsets remain source-based. ## Screenshots All screenshots are attached to this issue. Captures use the existing Search palette and have the exact pixel sizes shown in their labels. 390 px: - [Click · NFC · light](https://git.kayg.org/attachments/5005dd58-2679-4b6f-98c2-207350b981cb) · [Click · NFC · dark](https://git.kayg.org/attachments/f356c72b-659b-4102-8d1d-d09df2701e23) - [Click · NFD · light](https://git.kayg.org/attachments/3d50d871-e83d-4d96-b6e4-0716f746cd91) · [Click · NFD · dark](https://git.kayg.org/attachments/76c2552b-140a-4d73-8537-18c4133100c6) - [⌘K · NFD · light](https://git.kayg.org/attachments/9e602107-e6ab-4ecb-a68b-cf2263b771a6) · [⌘K · NFD · dark](https://git.kayg.org/attachments/ac0b30ca-21d6-4689-b1e6-b65ac406117e) 820 px: - [Click · NFC · light](https://git.kayg.org/attachments/b522b413-0f6e-441e-9070-0a08c6b177ff) · [Click · NFC · dark](https://git.kayg.org/attachments/87e34b75-9a82-40a7-9a4a-c37be9cdcc85) - [Click · NFD · light](https://git.kayg.org/attachments/8fec5c86-9f12-439a-b38a-ef05e7f76bc1) · [Click · NFD · dark](https://git.kayg.org/attachments/2f7c1cf2-ecf8-4d04-8b05-1ce6ddf26ad9) - [⌘K · NFD · light](https://git.kayg.org/attachments/d532294f-0bc8-4573-85cc-6a5f4f4eacfa) · [⌘K · NFD · dark](https://git.kayg.org/attachments/10f1f022-cf63-4164-8377-bb9d2373eb79) 1440 px: - [Click · NFC · light](https://git.kayg.org/attachments/27f82722-bd92-40ae-a0e9-0d0f28399412) · [Click · NFC · dark](https://git.kayg.org/attachments/8d1e490d-a47d-471e-a821-64dfbe71a9e7) - [Click · NFD · light](https://git.kayg.org/attachments/1a55c1cf-e7ec-47c8-9bc6-93b61b99f070) · [Click · NFD · dark](https://git.kayg.org/attachments/e26e9bb7-0443-4466-b0ac-399a6a27e5c1) - [⌘K · NFD · light](https://git.kayg.org/attachments/690569bc-3fe6-4f0a-af4d-8f25b36dee90) · [⌘K · NFD · dark](https://git.kayg.org/attachments/abce74d6-3daa-4cc4-a784-8891885569f9) ## For the merge round - `cd apps/web && bun run test:e2e` — verify the full shell E2E path against the merged build. - `bash tests/adversarial/run.sh` — run the hostile-input, authorization and concurrency matrices on the merged server.

kayg referenced this issue from a commit

2026-10-08 09:18:46 +00:00

Match Search fixture barrier to palette limit (#1411)

kayg referenced this issue from a commit

2026-10-08 10:41:40 +00:00

Merge job/nfd-1411 (#1411)

kayg commented

2026-10-08 18:41:23 +00:00

Author

Owner

Deployed to calternal.cloud in merge round 35a (build cc1af3c12, live 2026-10-09 00:12 IST). Closing.

Deployed to calternal.cloud in merge round 35a (build cc1af3c12, live 2026-10-09 00:12 IST). Closing.