#335 - SPIKE: Money plain-text format — can an hledger superset hold YNAB/Actual budgets losslessly? Else design a simple calternal format (NO product code) - kayg/calternal

kayg commented

2026-09-28 12:27:26 +00:00

Owner

Owner (2026-09-28, Money grill #316 R1): wants plain text. Proposal on the table: a strict superset of hledger (every file is valid hledger; extras via tags/comments/virtual postings/periodic transactions; app settings such as views, rules and preferences in a small budget.toml). Owner: 'if it doesn't work, let's invent a very simple, extensible, and readable format because hledger is a bitch to read.' Readability is a first-class criterion, not only losslessness.

Do (time-boxed, research + prototype code in spikes/money-format/ only, never product crates):

Feature map: take every YNAB and Actual feature and invariant from docs/research/money-plugin.md (accounts on/off budget, categories + groups + hidden, monthly assignments, Ready to Assign, rollover and overspending rules per category, credit card payment categories, transfers, splits, cleared/reconciled/locked, reconciliation adjustments, scheduled transactions + approve/auto-enter, targets (every month refill/set-aside, by date incl. yearly, keep at least, snooze), payees + payee memory, notes, flags, attachments by stable id, foreign-currency charges with @@ totals, multiple budgets, history/change log ids). For each: show the exact hledger-superset text AND say whether hledger itself parses it (run the real hledger binary, via nix) and whether it is lossless.
Round-trip prototype: a small parser + serializer (Rust or Python in spikes/) that reads and writes the superset; property test: generate random budgets → write → read → write, identical bytes and identical computed budget numbers (Available, Ready to Assign, card payment balances) for 24 months.
Differential check: compute the same months with Actual's MIT budget engine (pinned commit) or a faithful port of its formulas from the research doc, and compare.
Readability test: write the same realistic month (≈60 transactions, 25 categories, 2 cards, 1 foreign charge, a split, a transfer, a reconciliation, 5 targets) in (a) the hledger superset and (b) a clean calternal format designed for humans first (e.g. one line per transaction: 2026-09-28 DMart −450 Groceries @Amex #weekly, indentation for splits, a small header block per month for assignments, simple 'key: value' blocks for targets). (b) must be simple, extensible (versioned header, unknown keys preserved) and parseable with a tiny grammar; give its full grammar. Put both side by side in the report so the owner can judge.
Verdict: (a) superset is lossless AND acceptably readable, or (b) the custom format wins; list the exact reasons. If (b), include the grammar, an hledger export mapping (so the data is still hledger-readable on demand), and a migration note.
Output: docs/research/money-format-spike.md plus the spike code; a summary on this issue. Do not touch product code. The owner's Actual export may be supplied later for a real-data round trip; use synthetic data until then.

Owner (2026-09-28, Money grill #316 R1): wants plain text. Proposal on the table: a strict **superset of hledger** (every file is valid hledger; extras via tags/comments/virtual postings/periodic transactions; app settings such as views, rules and preferences in a small budget.toml). Owner: 'if it doesn't work, let's invent a very simple, extensible, and readable format because hledger is a bitch to read.' **Readability is a first-class criterion, not only losslessness.** Do (time-boxed, research + prototype code in `spikes/money-format/` only, never product crates): 1. **Feature map:** take every YNAB and Actual feature and invariant from docs/research/money-plugin.md (accounts on/off budget, categories + groups + hidden, monthly assignments, Ready to Assign, rollover and overspending rules per category, credit card payment categories, transfers, splits, cleared/reconciled/locked, reconciliation adjustments, scheduled transactions + approve/auto-enter, targets (every month refill/set-aside, by date incl. yearly, keep at least, snooze), payees + payee memory, notes, flags, attachments by stable id, foreign-currency charges with @@ totals, multiple budgets, history/change log ids). For each: show the exact hledger-superset text AND say whether hledger itself parses it (run the real `hledger` binary, via nix) and whether it is lossless. 2. **Round-trip prototype:** a small parser + serializer (Rust or Python in spikes/) that reads and writes the superset; property test: generate random budgets → write → read → write, identical bytes and identical computed budget numbers (Available, Ready to Assign, card payment balances) for 24 months. 3. **Differential check:** compute the same months with Actual's MIT budget engine (pinned commit) or a faithful port of its formulas from the research doc, and compare. 4. **Readability test:** write the same realistic month (≈60 transactions, 25 categories, 2 cards, 1 foreign charge, a split, a transfer, a reconciliation, 5 targets) in (a) the hledger superset and (b) a **clean calternal format** designed for humans first (e.g. one line per transaction: `2026-09-28 DMart −450 Groceries @Amex #weekly`, indentation for splits, a small header block per month for assignments, simple 'key: value' blocks for targets). (b) must be simple, extensible (versioned header, unknown keys preserved) and parseable with a tiny grammar; give its full grammar. Put both side by side in the report so the owner can judge. 5. **Verdict:** (a) superset is lossless AND acceptably readable, or (b) the custom format wins; list the exact reasons. If (b), include the grammar, an hledger **export** mapping (so the data is still hledger-readable on demand), and a migration note. Output: docs/research/money-format-spike.md plus the spike code; a summary on this issue. Do not touch product code. The owner's Actual export may be supplied later for a real-data round trip; use synthetic data until then.

kayg commented

2026-09-28 12:28:46 +00:00

Author

Owner

Starting the money-format spike on branch job/money-format, based on cde2cb813367a8ec616e2d25d3a9b9fd097038d8 (the current dev SHA). I am reviewing #316 and docs/research/money-plugin.md, then I will add only spike code under spikes/money-format/ and the requested research report.

Starting the money-format spike on branch `job/money-format`, based on `cde2cb813367a8ec616e2d25d3a9b9fd097038d8` (the current `dev` SHA). I am reviewing #316 and `docs/research/money-plugin.md`, then I will add only spike code under `spikes/money-format/` and the requested research report.

kayg commented

2026-09-28 13:03:01 +00:00

Author

Owner

Finding: nix shell nixpkgs#hledger --command hledger --version resolves hledger 1.52. hledger print -f spikes/money-format/examples/hledger-superset/main.journal exits 0 for the included account, budget and September transaction journals; hledger check -f .../main.journal exits 0. The sample exercises 60 transaction entries, 25 balanced virtual assignment entries, two periodic schedules and the foreign-currency @@ total. hledger print --forecast=2026-10 emits forecast Rent and Salary entries. Plain hledger print retains transaction tags and virtual postings, but its output does not contain the account declarations, standalone target comments or periodic rules from the source. The hledger dialect parses, but hledger's print output is not a lossless serializer for this superset.

Finding: `nix shell nixpkgs#hledger --command hledger --version` resolves hledger 1.52. `hledger print -f spikes/money-format/examples/hledger-superset/main.journal` exits 0 for the included account, budget and September transaction journals; `hledger check -f .../main.journal` exits 0. The sample exercises 60 transaction entries, 25 balanced virtual assignment entries, two periodic schedules and the foreign-currency `@@` total. `hledger print --forecast=2026-10` emits forecast Rent and Salary entries. Plain `hledger print` retains transaction tags and virtual postings, but its output does not contain the account declarations, standalone target comments or periodic rules from the source. The hledger dialect parses, but hledger's print output is not a lossless serializer for this superset.

kayg commented

2026-09-28 13:13:23 +00:00

Author

Owner

Finding and fix: the first Actual formula port omitted M09's selected-buffer terms. A two-month fixture from the research equations failed first: month 1 expected 3,000 minor units but returned 5,000. The reference port now selects the manual buffer when non-zero, otherwise the automatic buffer, carries the previous selected buffer into the next month's available funds, and subtracts the current selected buffer. The regression now returns 3,000 and 4,500 as expected. The 24-month differential fixture still passes for all six deterministic seeds.

kayg referenced this issue

2026-09-28 13:26:13 +00:00

DOCUMENTS: a Documents plugin with paperless-ngx features — files and plain text, not a database (grill first) #341

kayg commented

2026-09-28 15:26:51 +00:00

Author

Owner

Adversarial round findings from the required local-server run after merging dev (2026-09-28):

The Search probe reported that eight concurrent queries and repeated queries against the old Index missed the committed unicodenfcsentinel result during a staged rebuild.
The editor probe reported that opening an untouched Note rewrote Markdown bytes (seed 25608414). A separate browser undo/redo round trip changed editor text with the same seed.
GET /api/v1/files/download for the .. path returned no response before the 30-second client timeout. This occurred during concurrent indexing load and needs review.
The editor's 10,000-block sync took 3151 ms against a 2000 ms budget; this is a SLOW-only load result.

This job is limited to the money-format spike, so it makes no product behavior changes. I will report the final round result after the runner exits.

Adversarial round findings from the required local-server run after merging dev (2026-09-28): - The Search probe reported that eight concurrent queries and repeated queries against the old Index missed the committed `unicodenfcsentinel` result during a staged rebuild. - The editor probe reported that opening an untouched Note rewrote Markdown bytes (seed 25608414). A separate browser undo/redo round trip changed editor text with the same seed. - `GET /api/v1/files/download` for the `..` path returned no response before the 30-second client timeout. This occurred during concurrent indexing load and needs review. - The editor's 10,000-block sync took 3151 ms against a 2000 ms budget; this is a SLOW-only load result. This job is limited to the money-format spike, so it makes no product behavior changes. I will report the final round result after the runner exits.

kayg commented

2026-09-28 16:01:43 +00:00

Author

Owner

Verdict

Recommend the readable custom calternal text format as the canonical format candidate, with hledger as an import/export adapter. This is a spike recommendation for owner review, not a product decision in docs/DESIGN.md. The real hledger 1.52 fixture parses, but hledger print drops account directives, top-level target comments, periodic rules, and source-file boundaries; forecast output materializes schedule entries instead of preserving rules.

The custom fixture presents the same 60 transactions, 25 categories, two cards, five targets, split, transfer, FX charge, and reconciliation in both forms. The custom version is shorter and easier to scan. The prototype round-trips unknown records and exact decimal values. Its six deterministic seeds cover 144 budget-month results.

Files

docs/research/money-format-spike.md — verdict, feature map, full side-by-side journal, proposed grammar, hledger mapping, migration notes, owner decisions, and gaps.
spikes/money-format/money_format.py — generic nested text parser and canonical serializer.
spikes/money-format/budget_math.py — exact minor-unit envelope calculations and reference checks.
spikes/money-format/test_money_format.py — six prototype tests.
spikes/money-format/examples/ — readable custom month, hledger superset, layout sample, and parser probe.

No product code changed.

Branch and commits

Branch: job/money-format. Merged dev once before final gates. Pushed branch head: 25e88ffccbab8e1cc41a67ddbc08ec9d5ccff050 (matches origin/job/money-format). The spike work is split into atomic commits, followed by the dev merge.

Verification

cargo fmt --check: exit 0, no output.

cargo clippy --all-targets -- -D warnings (successful run after generating the required web build):

   Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/money-format/crates/calternal-server)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 20.69s

cargo test: exit 0; all workspace test binaries passed. Verbatim suite summaries from the output:

test result: ok. 489 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s
test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 220.30s
test result: ok. 66 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 11.04s

bun run check:

Loading Svelte-check in workspace: /home/kayg/Developer/calternal-wt/money-format/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run test:

 Test Files  112 passed (112)
      Tests  726 passed (726)
   Start at  16:56:14
   Duration  98.45s (transform 57%, environment 16%, import 16%, tests 8%, setup 3%)

The spike tests reported Ran 6 tests in 0.395s and OK. Nix supplied hledger 1.52, linux-x86_64; hledger check exited 0 with no output. Final cleanup ran cargo clean (Removed 7951 files, 4.7GiB total) and removed apps/web/build, .svelte-kit, and installed node_modules.

Adversarial round

Ran the required real-server round once. It was stopped at about 48 minutes to keep the pass time-boxed. Search returned no 5xx in its storm (search storm: 0 failures, p50=1389.0ms p95=1814.6ms), but queries missed a committed live hit during staged rebuild; see #336 and #345. The editor undo/redo equality check failed again; see #314. The untouched-note Markdown byte rewrite is filed as #361. Task and other API requests timed out under the concurrent 20,000-file watcher/index load; see #171 and #269. The probes did not establish that traversal inputs were accepted. Requests emitted after the local server was stopped are teardown artifacts and are excluded. A 10,000-block editor sync took 3151 ms against the 2000 ms target; this is SLOW-only and load-confounded.

Decisions not settled in DESIGN.md

Choose custom text as canonical and hledger as adapter, subject to owner review.
Use a versioned money 1 header, stable IDs, a generic record grammar that preserves unknown fields, exact decimal values with ISO currency codes, and date-only transaction dates.
Store posted transactions and assignments as source rows; derive Available, Ready to Assign, card reserve, and uncovered debt. Treat opening balances as dated opening-equity transactions.
The spike uses set-aside, refill, balance-by, and keep-at-least target names, and a five-currency exponent table. Confirm target semantics, full currency support, integer bounds, and history policy before product code.

Known gaps

There is no product code, import/export implementation, Actual or YNAB user export fixture, full record-schema validation, split validation, FX calculation, target or schedule calculation, tracking-transfer calculation, cleared-balance report, or Actual JavaScript engine comparison. These limits and the complete proposed grammar are in the research document.

## Verdict Recommend the readable custom calternal text format as the canonical format candidate, with hledger as an import/export adapter. This is a spike recommendation for owner review, not a product decision in `docs/DESIGN.md`. The real hledger 1.52 fixture parses, but `hledger print` drops account directives, top-level target comments, periodic rules, and source-file boundaries; forecast output materializes schedule entries instead of preserving rules. The custom fixture presents the same 60 transactions, 25 categories, two cards, five targets, split, transfer, FX charge, and reconciliation in both forms. The custom version is shorter and easier to scan. The prototype round-trips unknown records and exact decimal values. Its six deterministic seeds cover 144 budget-month results. ## Files - `docs/research/money-format-spike.md` — verdict, feature map, full side-by-side journal, proposed grammar, hledger mapping, migration notes, owner decisions, and gaps. - `spikes/money-format/money_format.py` — generic nested text parser and canonical serializer. - `spikes/money-format/budget_math.py` — exact minor-unit envelope calculations and reference checks. - `spikes/money-format/test_money_format.py` — six prototype tests. - `spikes/money-format/examples/` — readable custom month, hledger superset, layout sample, and parser probe. No product code changed. ## Branch and commits Branch: `job/money-format`. Merged `dev` once before final gates. Pushed branch head: `25e88ffccbab8e1cc41a67ddbc08ec9d5ccff050` (matches `origin/job/money-format`). The spike work is split into atomic commits, followed by the dev merge. ## Verification `cargo fmt --check`: exit 0, no output. `cargo clippy --all-targets -- -D warnings` (successful run after generating the required web build): ```text Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/money-format/crates/calternal-server) Finished `dev` profile [unoptimized + debuginfo] target(s) in 20.69s ``` `cargo test`: exit 0; all workspace test binaries passed. Verbatim suite summaries from the output: ```text test result: ok. 489 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 220.30s test result: ok. 66 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 11.04s ``` `bun run check`: ```text Loading Svelte-check in workspace: /home/kayg/Developer/calternal-wt/money-format/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run test`: ```text Test Files 112 passed (112) Tests 726 passed (726) Start at 16:56:14 Duration 98.45s (transform 57%, environment 16%, import 16%, tests 8%, setup 3%) ``` The spike tests reported `Ran 6 tests in 0.395s` and `OK`. Nix supplied `hledger 1.52, linux-x86_64`; `hledger check` exited 0 with no output. Final cleanup ran `cargo clean` (`Removed 7951 files, 4.7GiB total`) and removed `apps/web/build`, `.svelte-kit`, and installed `node_modules`. ## Adversarial round Ran the required real-server round once. It was stopped at about 48 minutes to keep the pass time-boxed. Search returned no 5xx in its storm (`search storm: 0 failures, p50=1389.0ms p95=1814.6ms`), but queries missed a committed live hit during staged rebuild; see #336 and #345. The editor undo/redo equality check failed again; see #314. The untouched-note Markdown byte rewrite is filed as #361. Task and other API requests timed out under the concurrent 20,000-file watcher/index load; see #171 and #269. The probes did not establish that traversal inputs were accepted. Requests emitted after the local server was stopped are teardown artifacts and are excluded. A 10,000-block editor sync took 3151 ms against the 2000 ms target; this is SLOW-only and load-confounded. ## Decisions not settled in DESIGN.md 1. Choose custom text as canonical and hledger as adapter, subject to owner review. 2. Use a versioned `money 1` header, stable IDs, a generic record grammar that preserves unknown fields, exact decimal values with ISO currency codes, and date-only transaction dates. 3. Store posted transactions and assignments as source rows; derive Available, Ready to Assign, card reserve, and uncovered debt. Treat opening balances as dated opening-equity transactions. 4. The spike uses `set-aside`, `refill`, `balance-by`, and `keep-at-least` target names, and a five-currency exponent table. Confirm target semantics, full currency support, integer bounds, and history policy before product code. ## Known gaps There is no product code, import/export implementation, Actual or YNAB user export fixture, full record-schema validation, split validation, FX calculation, target or schedule calculation, tracking-transfer calculation, cleared-balance report, or Actual JavaScript engine comparison. These limits and the complete proposed grammar are in the research document.

kayg referenced this issue from a commit

2026-09-28 16:04:51 +00:00

Merge job/money-format (#335: money format spike — custom text canonical, hledger import/export)

kayg referenced this issue

2026-09-29 05:24:27 +00:00

SPIKE v2: Money stored as human-readable Markdown (+ card payment plans and subscriptions) #404

kayg referenced this issue

2026-09-29 05:26:19 +00:00

SPIKE v2: Money stored as human-readable Markdown (+ card payment plans and subscriptions) #404

kayg commented

2026-10-03 11:55:41 +00:00

Author