IMPORT: calternal.js vault → calternal Home (Settings → Import + CLI, dry-run first) #300

New issue

Open

opened 2026-09-28 06:57:26 +00:00 by kayg · 3 comments

kayg commented

2026-09-28 06:57:26 +00:00

Owner

Facts:

calternal.js is end-to-end encrypted. The server has only ciphertext, so calternal cannot read it server to server. The calternal.js CLI's calternal sync writes a decrypted working copy of the vault to ~/calternal-vault (or $CALTERNAL_VAULT) on the owner's Mac. That plaintext folder is the import source.
calternal.js layout (lowercase): notes/, journal/ (day files YYYYMMDD-dailynote.md with the ## 📝 Log section), tasks/, attachments/. calternal layout (DESIGN §26): Notes/, Notes/Journal/, Tasks/, Attachments/.
Both products share the note format (CommonMark links, [[wikilinks]] sugar, inline + frontmatter tags, ^block-ids, callouts) and the daily-note format. calternal.js has calternal-id identity (see calternal.js docs/note-identity.md).

Build:

A server import endpoint that takes a zip (streamed; size limits; zip-slip and symlink safe; every write through calternal-fs; never string-concatenate paths). It maps the folders, rewrites relative links and attachment links to the new paths, keeps ^block-ids, tags and frontmatter byte-for-byte otherwise, and keeps calternal-ids so old deep links resolve where possible.
Dry run first: a report of the files, their mapping, rewritten links, name collisions with existing Home files (default: never overwrite; rename to name (calternal.js).md and list it), unknown files (copied as they are into Imported/calternal.js/), and broken links found. Then Import.
Settings → Import (new section, deep-linkable) with an upload (zip, or a folder via the directory picker where the browser supports it), the dry-run report, Import, and a final report with a link to the imported folders. Use the existing Settings components; this is opinionated and easy (owner principle).
CLI: calternal import calternal-js <folder> zips and streams the folder to the same endpoint and prints the same report. (Note for the report: both products name their CLI calternal; say which binary is meant in the docs.)
Idempotent: running it twice imports nothing new (match on calternal-id, then content hash).
After the import, the Index, Search, Calendar (log entries from day files), tags and backlinks see the files through the normal sync/index path. No special index code.
Adversarial round (CLAUDE.md): zip bombs, path traversal, absolute paths, Unicode confusables and NFC/NFD names, huge files, 100k tiny files, symlinks, duplicate names that differ only by case, malformed frontmatter, concurrent imports. Put them in tests/adversarial.
Tests: a fixture vault (tests only) covering every mapping and link rewrite; a round-trip that opens imported notes and checks the links resolve.

Owner (2026-09-28): 'btw how do I now finally import calternal.js data?' DESIGN.md §27 said 'No importers for now', but §26 (Home folders) already says: 'Import from a calternal.js vault translates its lowercase paths.' The owner's request overrides §27 for this one importer. Update DESIGN.md. Facts: - calternal.js is end-to-end encrypted. The server has only ciphertext, so calternal cannot read it server to server. The calternal.js CLI's `calternal sync` writes a decrypted working copy of the vault to `~/calternal-vault` (or `$CALTERNAL_VAULT`) on the owner's Mac. **That plaintext folder is the import source.** - calternal.js layout (lowercase): `notes/`, `journal/` (day files `YYYYMMDD-dailynote.md` with the `## 📝 Log` section), `tasks/`, `attachments/`. calternal layout (DESIGN §26): `Notes/`, `Notes/Journal/`, `Tasks/`, `Attachments/`. - Both products share the note format (CommonMark links, `[[wikilinks]]` sugar, inline + frontmatter tags, `^block-ids`, callouts) and the daily-note format. calternal.js has calternal-id identity (see calternal.js docs/note-identity.md). Build: 1. A server import endpoint that takes a zip (streamed; size limits; zip-slip and symlink safe; every write through calternal-fs; never string-concatenate paths). It maps the folders, **rewrites relative links and attachment links** to the new paths, keeps `^block-ids`, tags and frontmatter byte-for-byte otherwise, and **keeps calternal-ids** so old deep links resolve where possible. 2. **Dry run first**: a report of the files, their mapping, rewritten links, name collisions with existing Home files (default: never overwrite; rename to `name (calternal.js).md` and list it), unknown files (copied as they are into `Imported/calternal.js/`), and broken links found. Then Import. 3. **Settings → Import** (new section, deep-linkable) with an upload (zip, or a folder via the directory picker where the browser supports it), the dry-run report, Import, and a final report with a link to the imported folders. Use the existing Settings components; this is opinionated and easy (owner principle). 4. **CLI**: `calternal import calternal-js <folder>` zips and streams the folder to the same endpoint and prints the same report. (Note for the report: both products name their CLI `calternal`; say which binary is meant in the docs.) 5. Idempotent: running it twice imports nothing new (match on calternal-id, then content hash). 6. After the import, the Index, Search, Calendar (log entries from day files), tags and backlinks see the files through the normal sync/index path. No special index code. 7. Adversarial round (CLAUDE.md): zip bombs, path traversal, absolute paths, Unicode confusables and NFC/NFD names, huge files, 100k tiny files, symlinks, duplicate names that differ only by case, malformed frontmatter, concurrent imports. Put them in tests/adversarial. Tests: a fixture vault (tests only) covering every mapping and link rewrite; a round-trip that opens imported notes and checks the links resolve.

kayg commented

2026-09-28 07:03:33 +00:00

Author

Owner

Owner (2026-09-28): 'honestly we don't need a proper importer for calternal.js, for other apps yes we do. because calternal.js had only one user and that is me: so if we could interactively do it even, that would be great!'
Rescoped: no calternal.js importer is built. The orchestrator migrates the owner's vault once, interactively: a one-off translation script (folder mapping + link rewrite + calternal-id kept) with a dry-run report the owner reviews, then an upload through the calternal CLI/API (the server stays the single writer). This issue stays open for the general importer framework for other apps (Obsidian, Apple Notes, Bear, Notion, Immich… per DESIGN), which needs its own grill first. The dry-run/report/idempotency/adversarial requirements above carry over to it.

Owner (2026-09-28): 'honestly we don't need a proper importer for calternal.js, for other apps yes we do. because calternal.js had only one user and that is me: so if we could interactively do it even, that would be great!' Rescoped: **no calternal.js importer is built.** The orchestrator migrates the owner's vault once, interactively: a one-off translation script (folder mapping + link rewrite + calternal-id kept) with a dry-run report the owner reviews, then an upload through the calternal CLI/API (the server stays the single writer). This issue stays open for the **general importer framework for other apps** (Obsidian, Apple Notes, Bear, Notion, Immich… per DESIGN), which needs its own grill first. The dry-run/report/idempotency/adversarial requirements above carry over to it.

kayg commented

2026-10-02 17:37:14 +00:00

Author

Owner

Stale dedicated-importer proposal: the later owner decision in #719 (2026-10-02) cancels the dedicated calternal.js importer and defines moving in as dropping a Markdown folder through Files, WebDAV/Finder or CLI put. DESIGN §40 also replaces this issue's old Tasks/ and Attachments/ layout. Recommend treating this importer flow as superseded and using #719 for the folder-ingest audit. Do not close this issue in the audit.

kayg referenced this issue

2026-10-02 17:37:19 +00:00

Moving in by dropping Markdown folders: find and fix what breaks (replaces the dedicated importer) #719

kayg referenced this issue

2026-10-02 17:42:29 +00:00

Audit: open issue hygiene (duplicates, already fixed, stale) #866

kayg commented

2026-10-03 11:58:57 +00:00

Author

Owner

The owner decision in #719 supersedes only the dedicated calternal.js importer path. #300 still tracks the general importer framework for other sources, which needs its own grill; leaving that work open.