#301 - NOTES: vault maintenance actions — broken links, wikilinks, convert wikilinks → Markdown links, orphaned files (as searches) - kayg/calternal

kayg commented

2026-09-28 07:02:52 +00:00

Owner

Owner (2026-09-28): 'we need more obsidian like features in our block editor… actions: Show all orphaned links; Show all wikilinks; Convert all wikilinks to standard markdown links; Show all orphaned files. Of course all the "show" actions should result in views/searches.'

Definitions (orchestrator; confirm in the report if code or docs disagree):

Orphaned (broken) links: links in notes whose target does not resolve (no file, a deleted heading or block anchor, an ambiguous wikilink).
Wikilinks: every [[…]] still stored as a wikilink (DESIGN §26: wikilinks are input sugar converted only when they resolve uniquely, else kept byte-for-byte, so the ones left are exactly the unresolved or ambiguous ones, plus imported content).
Orphaned files: files in Home (notes, attachments) that no note links to or embeds. Daily notes, Tasks and files in Photos are excluded by default (they are reached by date/mode, not links). The search shows an 'include' toggle.

Build (reuse, one owner):

Show actions are Search operators, not new screens. Extend the S5 operators (DESIGN §32): e.g. is:broken-link (results = the notes that contain one, with the broken link text highlighted in the preview and a jump to the exact block), has:wikilink, is:orphan (files). Each action opens the palette with that query, and 'See all' opens the full results view. The user can save it as a smart folder like any search. The query is deep-linkable (§33).
The data comes from the existing link index (crates/calternal-notes-core links.rs / notes_index.rs, backlinks). Add a resolved/unresolved state and an inbound-link count if they are missing. Keep them incremental (update on write), never a full scan per query. Search latency must stay at the #258 targets.
Convert all wikilinks to Markdown links: a bulk action with a dry-run preview first (file count, every change as before → after, the unresolvable ones listed and left alone unless the user opts in). Then Apply. The server is the single writer; each changed note gets a normal version (Undo = restore the versions in one step, offered in the toast). The link format follows DESIGN §26: a CommonMark relative link, no leading ./, angle brackets when the path has spaces (e.g. [Apple Notes](<../Notes/Apple Notes.md>)). The owner confirmed that [title](../notes/whatever.md) works in Obsidian and ./../ does not. Headings and blocks: [[Note#Heading]] → [Note › Heading](Note.md#heading) using the same slug rules as the rename code; [[Note^id]] → #^id. Aliases [[Note|text]] keep the text. Embeds ![[img.png]] become ![](Attachments/img.png). Also offer it per note (⋯ menu) and per selection.
- Owner decision to confirm: unresolved wikilinks ([[Apple Notes]] with no file). Default proposal: leave them as wikilinks and list them (converting would create a broken Markdown link). The owner uses Obsidian's 'Better Markdown Links', which also skips them.
Where the actions live (chrome rules §34: no new bars): the command palette actions (⌘K), the Notes sidebar ⋯ menu ('Vault health' submenu), and the note ⋯ menu for per-note convert. Each has a Copy link.
Tests: fixtures for each link form; a round-trip test that converted links resolve and open the same target; an adversarial round on the bulk convert (100k notes, concurrent edits during convert, hostile names). The convert must never corrupt a note that is edited concurrently (use the collab session, #292).
Evidence: screenshots of each search, the dry-run preview and the toast with Undo; light and dark.

Owner (2026-09-28): 'we need more obsidian like features in our block editor… actions: Show all orphaned links; Show all wikilinks; Convert all wikilinks to standard markdown links; Show all orphaned files. Of course all the "show" actions should result in views/searches.' Definitions (orchestrator; confirm in the report if code or docs disagree): - **Orphaned (broken) links**: links in notes whose target does not resolve (no file, a deleted heading or block anchor, an ambiguous wikilink). - **Wikilinks**: every `[[…]]` still stored as a wikilink (DESIGN §26: wikilinks are input sugar converted only when they resolve uniquely, else kept byte-for-byte, so the ones left are exactly the unresolved or ambiguous ones, plus imported content). - **Orphaned files**: files in Home (notes, attachments) that no note links to or embeds. Daily notes, Tasks and files in Photos are excluded by default (they are reached by date/mode, not links). The search shows an 'include' toggle. Build (reuse, one owner): 1. **Show actions are Search operators, not new screens.** Extend the S5 operators (DESIGN §32): e.g. `is:broken-link` (results = the notes that contain one, with the broken link text highlighted in the preview and a jump to the exact block), `has:wikilink`, `is:orphan` (files). Each action opens the palette with that query, and 'See all' opens the full results view. The user can save it as a smart folder like any search. The query is deep-linkable (§33). 2. The data comes from the existing link index (crates/calternal-notes-core links.rs / notes_index.rs, backlinks). Add a resolved/unresolved state and an inbound-link count if they are missing. Keep them incremental (update on write), never a full scan per query. Search latency must stay at the #258 targets. 3. **Convert all wikilinks to Markdown links**: a bulk action with a **dry-run preview first** (file count, every change as before → after, the unresolvable ones listed and left alone unless the user opts in). Then Apply. The server is the single writer; each changed note gets a normal version (Undo = restore the versions in one step, offered in the toast). The link format follows DESIGN §26: a CommonMark relative link, no leading `./`, angle brackets when the path has spaces (e.g. `[Apple Notes](<../Notes/Apple Notes.md>)`). The owner confirmed that `[title](../notes/whatever.md)` works in Obsidian and `./../` does not. Headings and blocks: `[[Note#Heading]]` → `[Note › Heading](Note.md#heading)` using the same slug rules as the rename code; `[[Note^id]]` → `#^id`. Aliases `[[Note|text]]` keep the text. Embeds `![[img.png]]` become `![](Attachments/img.png)`. Also offer it per note (⋯ menu) and per selection. - Owner decision to confirm: unresolved wikilinks (`[[Apple Notes]]` with no file). Default proposal: leave them as wikilinks and list them (converting would create a broken Markdown link). The owner uses Obsidian's 'Better Markdown Links', which also skips them. 4. **Where the actions live** (chrome rules §34: no new bars): the command palette actions (⌘K), the Notes sidebar ⋯ menu ('Vault health' submenu), and the note ⋯ menu for per-note convert. Each has a Copy link. 5. Tests: fixtures for each link form; a round-trip test that converted links resolve and open the same target; an adversarial round on the bulk convert (100k notes, concurrent edits during convert, hostile names). The convert must never corrupt a note that is edited concurrently (use the collab session, #292). Evidence: screenshots of each search, the dry-run preview and the toast with Undo; light and dark.

vault-health-broken-links-1440-paper-white.png

91 KiB

vault-health-broken-links-1440-tokyo-night.png

122 KiB

vault-health-conversion-preview-1440-paper-white.png

385 KiB

vault-health-conversion-preview-1440-tokyo-night.png

309 KiB

vault-health-conversion-undo-toast-1440-paper-white.png

388 KiB

vault-health-conversion-undo-toast-1440-tokyo-night.png

313 KiB

vault-health-orphans-1440-paper-white.png

91 KiB

vault-health-orphans-1440-tokyo-night.png

122 KiB

vault-health-orphans-all-1440-paper-white.png

97 KiB

vault-health-orphans-all-1440-tokyo-night.png

128 KiB

vault-health-wikilinks-1440-paper-white.png

77 KiB

vault-health-wikilinks-1440-tokyo-night.png

110 KiB

vault-health-broken-links-1440-paper-white.png

99 KiB

vault-health-broken-links-1440-tokyo-night.png

129 KiB

vault-health-conversion-preview-1440-paper-white.png

387 KiB

vault-health-conversion-preview-1440-tokyo-night.png

312 KiB

vault-health-conversion-undo-toast-1440-paper-white.png

390 KiB

vault-health-conversion-undo-toast-1440-tokyo-night.png

317 KiB

vault-health-orphans-1440-paper-white.png

105 KiB

vault-health-orphans-1440-tokyo-night.png

130 KiB

vault-health-orphans-all-1440-paper-white.png

112 KiB

vault-health-orphans-all-1440-tokyo-night.png

136 KiB

vault-health-wikilinks-1440-paper-white.png

85 KiB

vault-health-wikilinks-1440-tokyo-night.png

116 KiB

vault-health-conversion-undo-toast-1440-paper-white.png

vault-health-conversion-undo-toast-1440-tokyo-night.png

kayg referenced this issue

2026-09-28 07:03:02 +00:00

RESEARCH: convenience and vault-maintenance actions for Notes/Files (Obsidian core + community plugins, other PKMs) — catalogue + grill (NO implementation) #302

kayg commented

2026-09-28 07:04:57 +00:00

Author

Owner

Starting issue #301 on branch job/vault-health, based on c9ec6aff84a9ec810124a2cfbdd07c189eea1b9c (dev merge-base). I am mapping the existing notes link index, search operators, versioned write/undo flows, and UI entry points before implementing the isolated operator/index fields and bulk conversion flow.

Starting issue #301 on branch `job/vault-health`, based on `c9ec6aff84a9ec810124a2cfbdd07c189eea1b9c` (dev merge-base). I am mapping the existing notes link index, search operators, versioned write/undo flows, and UI entry points before implementing the isolated operator/index fields and bulk conversion flow.

kayg referenced this issue

2026-09-28 07:22:09 +00:00

RESEARCH: convenience and vault-maintenance actions for Notes/Files (Obsidian core + community plugins, other PKMs) — catalogue + grill (NO implementation) #302

kayg commented

2026-09-28 07:28:16 +00:00

Author

Owner

Finding: note_links currently stores only (user_id, source, target, kind) and store::index drops embeds from the link_rows_from_note projection; it does not store anchor validity, unresolved references, or exact source lines. Search already has a keyword flags field and SQLite-backed incremental indexing. I am extending the Notes projection with reference form/anchor/resolution and feeding only affected paths into the Search Index, so queries do not parse every Note.

Finding: `note_links` currently stores only `(user_id, source, target, kind)` and `store::index` drops embeds from the `link_rows_from_note` projection; it does not store anchor validity, unresolved references, or exact source lines. Search already has a keyword `flags` field and SQLite-backed incremental indexing. I am extending the Notes projection with reference form/anchor/resolution and feeding only affected paths into the Search Index, so queries do not parse every Note.

kayg commented

2026-09-28 08:18:32 +00:00

Author

Owner

Implemented and committed the detailed link-health projection (c1438c36f31f7fbef19ec6db1a98d32768469a65). It records source text, line, link kind, target path/key, alias, anchors, and resolution state, with indexes for target resolution and inbound lookups. The Notes index refreshes the changed source and wikilink candidates; search attaches broken-link, wikilink, orphan, and orphan-any flags with one grouped lookup per upsert batch.

Evidence: cargo test -p calternal-plugin-notes passed (101 tests); cargo test -p calternal-search passed (28 unit, 1 ask evaluation, 1 citation, 17 indexer, 5 operator, 2 query-property, 1 relevance, 4 retrieval tests; 3 ignored). Added coverage for missing targets, missing headings, and exact source lines. The Notes-only test database has no Files projection, so the resolver detects that optional table once per transaction and still indexes Notes-only links.

Implemented and committed the detailed link-health projection (`c1438c36f31f7fbef19ec6db1a98d32768469a65`). It records source text, line, link kind, target path/key, alias, anchors, and resolution state, with indexes for target resolution and inbound lookups. The Notes index refreshes the changed source and wikilink candidates; search attaches `broken-link`, `wikilink`, `orphan`, and `orphan-any` flags with one grouped lookup per upsert batch. Evidence: `cargo test -p calternal-plugin-notes` passed (101 tests); `cargo test -p calternal-search` passed (28 unit, 1 ask evaluation, 1 citation, 17 indexer, 5 operator, 2 query-property, 1 relevance, 4 retrieval tests; 3 ignored). Added coverage for missing targets, missing headings, and exact source lines. The Notes-only test database has no Files projection, so the resolver detects that optional table once per transaction and still indexes Notes-only links.

kayg referenced this issue

2026-09-28 08:29:37 +00:00

RESEARCH: convenience and vault-maintenance actions for Notes/Files (Obsidian core + community plugins, other PKMs) — catalogue + grill (NO implementation) #302

kayg commented

2026-09-28 08:54:00 +00:00

Author

Owner

The recoverable bulk-conversion API is implemented on job/vault-health. Preview persists exact before/after rows and source hashes; Apply and Undo use compare-and-swap, create normal Note versions, and report concurrent edits without overwriting them. The full Notes crate passed: test result: ok. 102 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 96.24s. The new regression test covers default unresolved-reference handling, version creation, Undo restoration, and a source edit made after preview.

The recoverable bulk-conversion API is implemented on `job/vault-health`. Preview persists exact before/after rows and source hashes; Apply and Undo use compare-and-swap, create normal Note versions, and report concurrent edits without overwriting them. The full Notes crate passed: `test result: ok. 102 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 96.24s`. The new regression test covers default unresolved-reference handling, version creation, Undo restoration, and a source edit made after preview.

kayg commented

2026-09-28 09:39:19 +00:00

Author

Owner

Finding from focused Search regression: broken-link hydration selected the stored body_line field but read a column named line. The regression failed with Database(ColumnNotFound("line")), so matching is:broken-link results could not receive their source snippet or direct-jump URL. The row reader now uses body_line; the regression passes and also checks HTML escaping and per-user isolation.

Finding from focused Search regression: broken-link hydration selected the stored `body_line` field but read a column named `line`. The regression failed with `Database(ColumnNotFound("line"))`, so matching `is:broken-link` results could not receive their source snippet or direct-jump URL. The row reader now uses `body_line`; the regression passes and also checks HTML escaping and per-user isolation.

kayg commented

2026-09-28 10:27:57 +00:00

Author

Owner

Finding from the generated OpenAPI contract: the conversion preview endpoint's cursor and limit were emitted as required path parameters, while the handler reads them from the query string. I added Utoipa's parameter_in = Query annotation and regenerated the contract/client; the generated operation now documents both as optional query parameters. The generator's final diff check reports the expected uncommitted generated-file delta.

Finding from the generated OpenAPI contract: the conversion preview endpoint's `cursor` and `limit` were emitted as required path parameters, while the handler reads them from the query string. I added Utoipa's `parameter_in = Query` annotation and regenerated the contract/client; the generated operation now documents both as optional query parameters. The generator's final diff check reports the expected uncommitted generated-file delta.

kayg commented

2026-09-28 10:29:11 +00:00

Author

Owner

Finding from the orphan flag regression: is:orphan marked an existing target Note as orphaned when its only incoming link had a missing heading, because inbound counts required is_resolved=1. The link index still records the target path, so this did not match the issue definition (“no note links to or embeds” the file). Inbound detection now counts references to that path even when the heading or block is broken; the regression remains in Search tests.

Finding from the orphan flag regression: `is:orphan` marked an existing target Note as orphaned when its only incoming link had a missing heading, because inbound counts required `is_resolved=1`. The link index still records the target path, so this did not match the issue definition (“no note links to or embeds” the file). Inbound detection now counts references to that path even when the heading or block is broken; the regression remains in Search tests.

kayg commented

2026-09-28 11:02:43 +00:00

Author

Owner

Production build screenshots — Vault health, 1440×900.

Broken links

kayg commented

2026-09-28 13:07:17 +00:00

Author

Owner

Full-workspace test finding after merging dev once:

cargo test failed in the Files plugin stress test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm after 315.82s. The write task reported atomic write 924 failed: entry not found; the parent also hit its five-minute completion bound. The Files test is unchanged by this branch and was introduced on dev (cbee4de4).
An isolated run of that exact test passed: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 119 filtered out; finished in 140.43s.

This is load-sensitive based on the isolated pass, but the entry not found error remains unexplained. I did not change Files code because it is outside this job's owned files.

Full-workspace test finding after merging `dev` once: - `cargo test` failed in the Files plugin stress test `tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm` after 315.82s. The write task reported `atomic write 924 failed: entry not found`; the parent also hit its five-minute completion bound. The Files test is unchanged by this branch and was introduced on `dev` (`cbee4de4`). - An isolated run of that exact test passed: `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 119 filtered out; finished in 140.43s`. This is load-sensitive based on the isolated pass, but the `entry not found` error remains unexplained. I did not change Files code because it is outside this job's owned files.

kayg commented

2026-09-28 13:11:17 +00:00

Author

Owner

Completed

Implemented Notes link-health indexing, the is:broken-link, has:wikilink, is:orphan, and is:orphan-all search operators, and an authenticated preview/apply/undo bulk Wikilink conversion API. Added the Vault Health search entry points, conversion preview and unresolved-link controls, context actions, and Undo feedback. Regenerated the OpenAPI contract and API client. Production browser coverage captured 12 screenshots in Paper and Tokyo Night; the gallery comments on this issue have the attachments.

Main files are in crates/calternal-notes-core/{src/links.rs,src/notes_index.rs}, crates/calternal-search/src, crates/plugins/notes/src, apps/web/src/lib/{components/NoteList.svelte,notes,search}, apps/web/src/routes/notes/convert/+page.svelte, contracts/openapi.json, packages/api-client/src/generated.ts, and tests/adversarial/wikilink_conversion.py.

Branch job/vault-health is pushed. Head: 35597600456fa50a4e3466a509f8b410861b55f8.

Gates and verification

cargo fmt --check: exit 0, no stdout.
cargo clippy --all-targets -- -D warnings:
Finished \dev` profile [unoptimized + debuginfo] target(s) in 26m 13s`

cargo test: failed in an unchanged Files plugin stress test. Exact failure excerpt:

test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm has been running for over 60 seconds
test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... FAILED
writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())
atomic write 924 failed: entry not found
test result: FAILED. 119 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 315.82s
error: test failed, to rerun pass `-p calternal-plugin-files --lib`

The test was introduced on dev (cbee4de4) and is unchanged by this branch. An isolated run passed verbatim: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 140.43s. The write error remains unexplained; the full workspace run stopped at this first failing crate.

bun run check:

$ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/vault-health/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run test:

Test Files  110 passed (110)
     Tests  710 passed (710)
   Duration  107.87s (transform 58%, environment 17%, import 15%, tests 6%, setup 3%)

API-client generation check completed, and git diff --exit-code -- contracts/openapi.json packages/api-client/src/generated.ts returned exit 0.
Adversarial endpoint probe: Wikilink conversion adversarial probe passed: validation, user isolation, CAS, concurrency, Apply and Undo.
cargo clean: Removed 22325 files, 19.1GiB total; apps/web/build was deleted.

Decisions not explicit in the design

Applying a conversion leaves unresolved links unchanged by default and lists them in the preview; changing unresolved targets is opt-in.
is:orphan covers Note-like Markdown and excludes tasks, daily notes, and media. is:orphan-all includes those indexed file types. A target with a broken heading is not also an orphan.
Conversion bounds: 1,000 Note IDs per request, scan up to 100,000 vault Notes, 2 MiB per Note, 64 MiB per operation, 20 rows per preview page, 50 Notes per apply batch, and a 24-hour undo expiry.

Known gap

The full Rust workspace test gate is not green because of the unrelated load-sensitive Files stress test described above. No Files changes were made. All 12 production screenshots are attached in the issue gallery comments.

## Completed Implemented Notes link-health indexing, the `is:broken-link`, `has:wikilink`, `is:orphan`, and `is:orphan-all` search operators, and an authenticated preview/apply/undo bulk Wikilink conversion API. Added the Vault Health search entry points, conversion preview and unresolved-link controls, context actions, and Undo feedback. Regenerated the OpenAPI contract and API client. Production browser coverage captured 12 screenshots in Paper and Tokyo Night; the gallery comments on this issue have the attachments. Main files are in `crates/calternal-notes-core/{src/links.rs,src/notes_index.rs}`, `crates/calternal-search/src`, `crates/plugins/notes/src`, `apps/web/src/lib/{components/NoteList.svelte,notes,search}`, `apps/web/src/routes/notes/convert/+page.svelte`, `contracts/openapi.json`, `packages/api-client/src/generated.ts`, and `tests/adversarial/wikilink_conversion.py`. Branch `job/vault-health` is pushed. Head: `35597600456fa50a4e3466a509f8b410861b55f8`. ## Gates and verification - `cargo fmt --check`: exit 0, no stdout. - `cargo clippy --all-targets -- -D warnings`: `Finished \`dev\` profile [unoptimized + debuginfo] target(s) in 26m 13s` - `cargo test`: failed in an unchanged Files plugin stress test. Exact failure excerpt: ``` test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm has been running for over 60 seconds test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... FAILED writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(()) atomic write 924 failed: entry not found test result: FAILED. 119 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 315.82s error: test failed, to rerun pass `-p calternal-plugin-files --lib` ``` The test was introduced on `dev` (`cbee4de4`) and is unchanged by this branch. An isolated run passed verbatim: `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 140.43s`. The write error remains unexplained; the full workspace run stopped at this first failing crate. - `bun run check`: ``` $ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/vault-health/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` - `bun run test`: ``` Test Files 110 passed (110) Tests 710 passed (710) Duration 107.87s (transform 58%, environment 17%, import 15%, tests 6%, setup 3%) ``` - API-client generation check completed, and `git diff --exit-code -- contracts/openapi.json packages/api-client/src/generated.ts` returned exit 0. - Adversarial endpoint probe: `Wikilink conversion adversarial probe passed: validation, user isolation, CAS, concurrency, Apply and Undo.` - `cargo clean`: `Removed 22325 files, 19.1GiB total`; `apps/web/build` was deleted. ## Decisions not explicit in the design - Applying a conversion leaves unresolved links unchanged by default and lists them in the preview; changing unresolved targets is opt-in. - `is:orphan` covers Note-like Markdown and excludes tasks, daily notes, and media. `is:orphan-all` includes those indexed file types. A target with a broken heading is not also an orphan. - Conversion bounds: 1,000 Note IDs per request, scan up to 100,000 vault Notes, 2 MiB per Note, 64 MiB per operation, 20 rows per preview page, 50 Notes per apply batch, and a 24-hour undo expiry. ## Known gap The full Rust workspace test gate is not green because of the unrelated load-sensitive Files stress test described above. No Files changes were made. All 12 production screenshots are attached in the issue gallery comments.

kayg referenced this issue

2026-09-28 13:30:17 +00:00

FILES: atomic write fails with 'entry not found' during a reconcile/watcher storm (stress test flakes under load) #343

kayg referenced this issue from a commit

2026-09-28 13:35:52 +00:00

Merge job/vault-health (#301: link-health search operators, wikilink conversion with preview and undo)

kayg closed this issue

2026-09-28 13:35:56 +00:00

kayg referenced this issue

2026-09-28 13:35:57 +00:00

NOTES: daily notes' previous/next links are wrong-relative and show up as broken links; vault-health copy polish #344