DATA LOSS: live Note edits not saved ('could not map edited Markdown to the loaded Note source')

kayg commented

2026-10-06 09:31:10 +00:00

Owner

DATA LOSS (production, 2026-10-06): live Note edits not saved

Production log (calternal-cloud, build f5fbced3c then 5301e0208), times CEST:

10:04:15 first WARN calternal_collab::session: live Note flush retry failed error=could not map edited Markdown to the loaded Note source
10:52:39 last-client Note flush failed error=could not map edited Markdown to the loaded Note source
every minute until 11:28:24 live Note not saved at shutdown error=could not map edited Markdown to the loaded Note source (43 occurrences)
Merge round 28 went live 08:35 CEST and its Note date backfill (#1148) added date:/last edited: to 60/55 Notes at 08:40 CEST; #1110 tag rewrites, #1152 block anchors, #1151 link cards also landed. Owner edits Notes and Journal entries daily.

Do, in priority order:

Recover: determine whether the unsaved edits survive anywhere (collab Yrs state/journal under /srv/calternal/data/.system — journal/, cache/, tmp/ — or the browser's local draft store). Read-only on production: copy the relevant files to a scratch dir on production (never print Note content in logs or issues; report counts, Note IDs and byte sizes only). If recoverable, write a recovery tool that saves the recovered text as a NEW sibling Note "

## DATA LOSS (production, 2026-10-06): live Note edits not saved Production log (calternal-cloud, build f5fbced3c then 5301e0208), times CEST: - 10:04:15 first `WARN calternal_collab::session: live Note flush retry failed error=could not map edited Markdown to the loaded Note source` - 10:52:39 `last-client Note flush failed error=could not map edited Markdown to the loaded Note source` - every minute until 11:28:24 `live Note not saved at shutdown error=could not map edited Markdown to the loaded Note source` (43 occurrences) Merge round 28 went live 08:35 CEST and its Note date backfill (#1148) added `date:`/`last edited:` to 60/55 Notes at 08:40 CEST; #1110 tag rewrites, #1152 block anchors, #1151 link cards also landed. Owner edits Notes and Journal entries daily. Do, in priority order: 1. **Recover:** determine whether the unsaved edits survive anywhere (collab Yrs state/journal under /srv/calternal/data/.system — journal/, cache/, tmp/ — or the browser's local draft store). Read-only on production: copy the relevant files to a scratch dir on production (never print Note content in logs or issues; report counts, Note IDs and byte sizes only). If recoverable, write a recovery tool that saves the recovered text as a NEW sibling Note "<title> (recovered 2026-10-06)" through the checked writer, never overwriting the current file; leave running it to the orchestrator. 2. **Root cause:** reproduce: open a Note in the live editor, change its file on disk underneath (date backfill / tag rewrite / block anchor write / external edit), keep typing, close the last client. Find why the bridge cannot map and why it drops instead of preserving. 3. **Fix (data safety first):** a live room must never lose edits. On a source mismatch: rebase the edits onto the current file (three-way merge on blocks), and if that fails, save the room's text as a conflict copy Note and tell the User (notification with Open), never only log a warning. Server-side writers (backfills, tag rewrites, anchors) must go through the collab room when one is open for that Note (apply as a room transaction) instead of writing the file underneath it. 4. Tests: every server-side writer × open live room × concurrent typing; restart with unsaved room state; zero loss. Gates per crate + `bun run check` + `bun run test` quoted. Commit. Do not push. Comment findings on this issue as you go (no Note content).

copy-desktop-dark.png

391 KiB

copy-desktop-light.png

515 KiB

copy-phone-light.png

262 KiB

inbox-desktop-dark.png

118 KiB

inbox-desktop-light.png

135 KiB

copy-phone-dark.png

255 KiB

copy-tablet-dark.png

441 KiB

copy-tablet-light.png

457 KiB

inbox-phone-light.png

52 KiB

inbox-tablet-dark.png

98 KiB

inbox-tablet-light.png

116 KiB

inbox-phone-dark.png

54 KiB

1197-collabloss2-copy-desktop-dark.png

398 KiB

1197-collabloss2-inbox-desktop-light.png

135 KiB

1197-collabloss2-copy-desktop-light.png

522 KiB

1197-collabloss2-copy-phone-dark.png

259 KiB

1197-collabloss2-copy-phone-light.png

268 KiB

1197-collabloss2-copy-tablet-dark.png

444 KiB

1197-collabloss2-copy-tablet-light.png

465 KiB

1197-collabloss2-inbox-desktop-dark.png

118 KiB

1197-collabloss2-inbox-phone-dark.png

54 KiB

1197-collabloss2-inbox-phone-light.png

52 KiB

1197-collabloss2-inbox-tablet-dark.png

98 KiB

1197-collabloss2-inbox-tablet-light.png

116 KiB

1197-final-copy-desktop-dark.png

396 KiB

1197-final-copy-desktop-light.png

519 KiB

1197-final-copy-phone-dark.png

259 KiB

1197-final-copy-phone-light.png

268 KiB

1197-final-copy-tablet-dark.png

441 KiB

1197-final-copy-tablet-light.png

463 KiB

1197-final-inbox-desktop-dark.png

118 KiB

1197-final-inbox-desktop-light.png

135 KiB

1197-final-inbox-phone-dark.png

54 KiB

1197-final-inbox-phone-light.png

52 KiB

1197-final-inbox-tablet-dark.png

98 KiB

1197-final-inbox-tablet-light.png

116 KiB

kayg commented

2026-10-06 09:31:32 +00:00

Author

Owner

Forensics snapshot taken at 15:01 IST: /srv/calternal/forensics-1197-20261006-1131 (journal, cache, tmp). All deploys held until this is fixed.

kayg commented

2026-10-06 09:32:17 +00:00

Author

Owner

Started on job/collabloss-1197, base 5301e020859c1d61b4a7f812a5049a64241203e8. First inspect production recovery sources read-only, then reproduce source mapping failure and add loss-preserving regression coverage. No Note content will be included in reports. No push or deploy.

Started on `job/collabloss-1197`, base `5301e020859c1d61b4a7f812a5049a64241203e8`. First inspect production recovery sources read-only, then reproduce source mapping failure and add loss-preserving regression coverage. No Note content will be included in reports. No push or deploy.

kayg commented

2026-10-06 09:41:40 +00:00

Author

Owner

Recovery evidence: production .system/journal and .system/tmp contain zero files. The copied Index has 18 clean collaboration states and 25 history points across 13 identities; the copied Home histories contain 39 files / 29,548 bytes. Preserved privately at /srv/calternal/recovery-1197-20261006; production source files remain unchanged. Twelve current Notes (13,021 bytes) were copied for comparison. The web Note provider is memory-only; no persistent Note Yjs draft store exists.

Code evidence: flush_document_locked calls patch_source_lines, which rejects a save when the raw body line count differs from its canonical serializer output. Valid adjacent blocks and following-line anchors can change this count. The TypeScript writer already maps source spans by blocks; the Rust flush still assumes aligned lines. A failed flush retains a dirty room in memory, but store_locked explicitly skips dirty rooms and shutdown only logs a failure. Durable collaboration history may retain edits; it needs inspection before a recovery result can be claimed.

Recovery evidence: production `.system/journal` and `.system/tmp` contain zero files. The copied Index has 18 clean collaboration states and 25 history points across 13 identities; the copied Home histories contain 39 files / 29,548 bytes. Preserved privately at `/srv/calternal/recovery-1197-20261006`; production source files remain unchanged. Twelve current Notes (13,021 bytes) were copied for comparison. The web Note provider is memory-only; no persistent Note Yjs draft store exists. Code evidence: `flush_document_locked` calls `patch_source_lines`, which rejects a save when the raw body line count differs from its canonical serializer output. Valid adjacent blocks and following-line anchors can change this count. The TypeScript writer already maps source spans by blocks; the Rust flush still assumes aligned lines. A failed flush retains a dirty room in memory, but `store_locked` explicitly skips dirty rooms and shutdown only logs a failure. Durable collaboration history may retain edits; it needs inspection before a recovery result can be claimed.

kayg commented

2026-10-06 09:54:27 +00:00

Author

Owner

Private recovery inspection completed and committed as 2d1da0759. The tool decoded all 25 preserved points without changing production. Latest durable text differs from current canonical Markdown for two existing Note IDs: 2f9eb5e8-bf12-4be6-b71b-5284e266d069 (7,416 durable bytes; 7,401 raw source bytes), and 31511de1-24b4-4441-a2f3-3fe4c81e821d (761 durable bytes; 762 raw source bytes). Identity d2909aed-47be-49ba-b4da-78537c7e3e9e has three recoverable historical points of 12 / 18 / 12 bytes but no current source. Earlier points remain available for selecting a recovery copy. A difference is a recovery candidate, not proof that it belongs to today's incident. No recovery copy has been written to production.

The focused live-room regression failed with the exact production error: Other("could not map edited Markdown to the loaded Note source"). It used valid adjacent Markdown blocks, an external date-field update, and a subsequent live edit. This confirms the line-count assumption as a flush-failure trigger without needing an external body rewrite. The source-span fix is now under test.

Private recovery inspection completed and committed as `2d1da0759`. The tool decoded all 25 preserved points without changing production. Latest durable text differs from current canonical Markdown for two existing Note IDs: `2f9eb5e8-bf12-4be6-b71b-5284e266d069` (7,416 durable bytes; 7,401 raw source bytes), and `31511de1-24b4-4441-a2f3-3fe4c81e821d` (761 durable bytes; 762 raw source bytes). Identity `d2909aed-47be-49ba-b4da-78537c7e3e9e` has three recoverable historical points of 12 / 18 / 12 bytes but no current source. Earlier points remain available for selecting a recovery copy. A difference is a recovery candidate, not proof that it belongs to today's incident. No recovery copy has been written to production. The focused live-room regression failed with the exact production error: `Other("could not map edited Markdown to the loaded Note source")`. It used valid adjacent Markdown blocks, an external date-field update, and a subsequent live edit. This confirms the line-count assumption as a flush-failure trigger without needing an external body rewrite. The source-span fix is now under test.

kayg referenced this issue from a commit

2026-10-06 09:58:37 +00:00

Add private Note history inspection to locate recoverable edits (#1197)

kayg referenced this issue from a commit

2026-10-06 09:58:37 +00:00

Map live Note edits by source block spans when line counts differ (#1197)

kayg commented

2026-10-06 10:45:09 +00:00

Author

Owner

A second corruption case is confirmed with a synthetic fixture: a heading adds one canonical separator while a following-line block anchor removes one line. Equal raw/canonical line counts can still have different offsets. The old line patch duplicated unchanged prose. Fast patches now validate the requested projection and fall back to source block spans. Focused output:

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 96 filtered out; finished in 0.00s

Independent read-only review also identified missing durable capture on checkpoint rebase, Task writer publication, and unstable copy paths after partial Index publication. These are being fixed with regressions. No production Note content has been posted.

A second corruption case is confirmed with a synthetic fixture: a heading adds one canonical separator while a following-line block anchor removes one line. Equal raw/canonical line counts can still have different offsets. The old line patch duplicated unchanged prose. Fast patches now validate the requested projection and fall back to source block spans. Focused output: ``` test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 96 filtered out; finished in 0.00s ``` Independent read-only review also identified missing durable capture on checkpoint rebase, Task writer publication, and unstable copy paths after partial Index publication. These are being fixed with regressions. No production Note content has been posted.

kayg referenced this issue from a commit

2026-10-06 10:59:49 +00:00

Validate Note source patches when normalized line offsets collide (#1197)

kayg referenced this issue from a commit

2026-10-06 10:59:49 +00:00

Preserve unsaved Note bodies in checked retry-stable sibling copies (#1197)

kayg commented

2026-10-06 11:12:02 +00:00

Author

Owner

Implementation committed through e839f4508 (room safety: 4eb73ce4d; recovery selector: 143cb726f).

Root cause: the live Markdown bridge assumed that normalized editor lines matched source lines. Date metadata changes left a stale source base. Body changes from Tags, anchors, link cards and external writes could invalidate line/block mapping. The old failed flush path logged an error without a durable pending checkpoint or a conflict copy. Last-client close and shutdown could therefore leave the only complete edits in process memory.

The bridge now validates line patches and uses source block spans. Checked Note, Task and Tag writers publish their committed source as a room transaction before acknowledgement. Flush checks current source and rebases disjoint changes. An unmappable edit creates a checked sibling Note and a durable Inbox notification with Open. Failed saves retain the edited Yrs snapshot plus its source base for restart. Recovered browser updates keep their verified author identities.

Focused verification:

test result: ok. 100 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 66.68s

This includes concurrent typing during date backfill, Tag rewrite and undo, anchor writes, link-card writes, Task writes and external writes; quota failure and restart; two successive restarts; and unmappable-source conflict copies. The writer matrix uses a 4 MiB test thread, equal to the real server worker stack budget. The initial default 2 MiB test thread overflowed while polling the existing Tag reconcile path; no production stack limit was raised.

Recovery inspection remains read-only for production source data. The private snapshot now contains 18 cached states, 25 history points across 13 Note IDs, 39 history files (29,548 bytes), and 16 current Note files (21,331 bytes). The old incident process has restarted. The browser live provider has no persistent local Note draft store.

Differences worth review include cached Note 46b85435-6b39-47a4-9db6-4fd84b805add (27 text bytes), cached and durable Note 31511de1-24b4-4441-a2f3-3fe4c81e821d (761 bytes), and durable Note 2f9eb5e8-bf12-4be6-b71b-5284e266d069 (7,416 bytes, versus the current 7,414-byte editor projection). Note d2909aed-47be-49ba-b4da-78537c7e3e9e has 12/18/12-byte history points without a current source file. One legacy cached key also differs (368 bytes versus 369 current bytes); its path is kept private. These are candidates, not proof that the text was typed during today's incident.

The recovery example can select a durable point, a cached UUID state, or a hashed legacy cached key. It creates a new sibling through the checked writer, with a retry-stable recovery plan. It refuses a destination identical to the copied snapshot. No recovery has been applied. The orchestrator must choose the candidate and an offline destination before running it.

Decisions: use the existing SyncConflict Inbox kind and its Open action; use the existing bounded note_collab_state row for marked pending checkpoints rather than add a migration; retain a private retry plan to avoid duplicate or overwritten recovery Notes after partial Index failure. The opt-in benchmark profile extends the existing Note editor series; measurement is deferred to the merge round under the latest verification policy.

Final gates will follow the single origin/dev merge.

Implementation committed through e839f4508 (room safety: 4eb73ce4d; recovery selector: 143cb726f). Root cause: the live Markdown bridge assumed that normalized editor lines matched source lines. Date metadata changes left a stale source base. Body changes from Tags, anchors, link cards and external writes could invalidate line/block mapping. The old failed flush path logged an error without a durable pending checkpoint or a conflict copy. Last-client close and shutdown could therefore leave the only complete edits in process memory. The bridge now validates line patches and uses source block spans. Checked Note, Task and Tag writers publish their committed source as a room transaction before acknowledgement. Flush checks current source and rebases disjoint changes. An unmappable edit creates a checked sibling Note and a durable Inbox notification with Open. Failed saves retain the edited Yrs snapshot plus its source base for restart. Recovered browser updates keep their verified author identities. Focused verification: ``` test result: ok. 100 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 66.68s ``` This includes concurrent typing during date backfill, Tag rewrite and undo, anchor writes, link-card writes, Task writes and external writes; quota failure and restart; two successive restarts; and unmappable-source conflict copies. The writer matrix uses a 4 MiB test thread, equal to the real server worker stack budget. The initial default 2 MiB test thread overflowed while polling the existing Tag reconcile path; no production stack limit was raised. Recovery inspection remains read-only for production source data. The private snapshot now contains 18 cached states, 25 history points across 13 Note IDs, 39 history files (29,548 bytes), and 16 current Note files (21,331 bytes). The old incident process has restarted. The browser live provider has no persistent local Note draft store. Differences worth review include cached Note 46b85435-6b39-47a4-9db6-4fd84b805add (27 text bytes), cached and durable Note 31511de1-24b4-4441-a2f3-3fe4c81e821d (761 bytes), and durable Note 2f9eb5e8-bf12-4be6-b71b-5284e266d069 (7,416 bytes, versus the current 7,414-byte editor projection). Note d2909aed-47be-49ba-b4da-78537c7e3e9e has 12/18/12-byte history points without a current source file. One legacy cached key also differs (368 bytes versus 369 current bytes); its path is kept private. These are candidates, not proof that the text was typed during today's incident. The recovery example can select a durable point, a cached UUID state, or a hashed legacy cached key. It creates a new sibling through the checked writer, with a retry-stable recovery plan. It refuses a destination identical to the copied snapshot. No recovery has been applied. The orchestrator must choose the candidate and an offline destination before running it. Decisions: use the existing SyncConflict Inbox kind and its Open action; use the existing bounded note_collab_state row for marked pending checkpoints rather than add a migration; retain a private retry plan to avoid duplicate or overwritten recovery Notes after partial Index failure. The opt-in benchmark profile extends the existing Note editor series; measurement is deferred to the merge round under the latest verification policy. Final gates will follow the single origin/dev merge.

kayg referenced this issue from a commit

2026-10-06 11:19:18 +00:00

Preserve live Note edits across checked writers and failed flushes (#1197)

kayg referenced this issue from a commit

2026-10-06 11:19:18 +00:00

Recover cached or durable Note text into checked sibling copies (#1197)

kayg referenced this issue from a commit

2026-10-06 11:19:18 +00:00

Profile concurrent Note source writes and last-client durability (#1197)

kayg referenced this issue from a commit

2026-10-06 11:19:18 +00:00

Document live Note recovery and source publication invariants (#1197)

kayg referenced this issue from a commit

2026-10-06 11:38:47 +00:00

Add production UI regression for preserved Note copies and Inbox Open (#1197)

kayg commented

2026-10-06 11:57:13 +00:00

Author

Owner

Recovery retry finding fixed in 23278a584: after file commit with lost Index publication, a parent-folder rename moved the unindexed recovery file. Retry previously recreated the old folder and a second file with the same identity. The new bounded probe checks the original plan and the current source folder plus the frozen basename. Existing identity and exact-body checks still apply; no Home scan or overwrite is added.

The regression failed at assertion failed: moved.path.starts_with("Archive/") before the fix. After it:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 308 filtered out; finished in 1.64s

A read-only review confirms the reported case is resolved.

Final-gate findings: the first full Notes run had 303 passes, two failures and four ignored tests. Both failures came from the shared root filesystem falling below calternal-fs's unchanged free-space reserve. Test fixtures now use this job's private HDD directory through the prescribed worktree target/tmp path. TMPDIR and CARGO_TARGET_DIR values remain unchanged. Those failures will be rerun; no reserve or assertion is weakened.

The collaboration writer matrix also exposed a fixture scheduling race: a timed live save can legitimately make exact Tag Undo refuse its changed source (#1111). The successful-Undo case now pauses that timer before any writer lock, retains its original assertions, and types during Undo publication too. Production timing is unchanged. Focused and final verification are in progress.

The full web gates passed: 274 files and 1,907 tests; svelte-check has zero errors and two existing CSS warnings. Plugin and Tags Clippy/tests passed. Server Clippy passed. The remaining Rust reruns and real-server UI proof are not yet complete.

Recovery retry finding fixed in 23278a584: after file commit with lost Index publication, a parent-folder rename moved the unindexed recovery file. Retry previously recreated the old folder and a second file with the same identity. The new bounded probe checks the original plan and the current source folder plus the frozen basename. Existing identity and exact-body checks still apply; no Home scan or overwrite is added. The regression failed at `assertion failed: moved.path.starts_with("Archive/")` before the fix. After it: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 308 filtered out; finished in 1.64s ``` A read-only review confirms the reported case is resolved. Final-gate findings: the first full Notes run had 303 passes, two failures and four ignored tests. Both failures came from the shared root filesystem falling below calternal-fs's unchanged free-space reserve. Test fixtures now use this job's private HDD directory through the prescribed worktree target/tmp path. TMPDIR and CARGO_TARGET_DIR values remain unchanged. Those failures will be rerun; no reserve or assertion is weakened. The collaboration writer matrix also exposed a fixture scheduling race: a timed live save can legitimately make exact Tag Undo refuse its changed source (#1111). The successful-Undo case now pauses that timer before any writer lock, retains its original assertions, and types during Undo publication too. Production timing is unchanged. Focused and final verification are in progress. The full web gates passed: 274 files and 1,907 tests; svelte-check has zero errors and two existing CSS warnings. Plugin and Tags Clippy/tests passed. Server Clippy passed. The remaining Rust reruns and real-server UI proof are not yet complete.

kayg referenced this issue from a commit

2026-10-06 11:58:04 +00:00

Reuse moved recovery siblings after lost Index publication (#1197)

kayg commented

2026-10-06 12:17:42 +00:00

Author

Owner

The first production UI walk passed preserved text, unchanged source, durable Inbox and Open with pointer, touch and keyboard at 390/820/1440 in light and dark (macOS platform). It exposed a copy-title gap: an original H1 remained the editor title despite the new conflict metadata. A new browser assertion fails against that earlier binary. The fix adds the new managed Note heading and appends the exact room body, including any edited original title. The checked-copy retry test passes. Existing pre-fix plans remain unchanged to avoid overwriting an already-created copy. New final screenshots will replace the earlier review set.

The initial server gate passed 262 tests but timed out after 15 seconds in startup_serves_http_while_upgrade_backfills_wait. That test builds a full app while a Home guard is held. Investigation is focused on the existing startup path and temporary-storage timing; no timeout or assertion was weakened. The Collab library passed all 100 tests; its subsequent cross-language process was terminated before a result. Final crate suites remain in progress.

The first production UI walk passed preserved text, unchanged source, durable Inbox and Open with pointer, touch and keyboard at 390/820/1440 in light and dark (macOS platform). It exposed a copy-title gap: an original H1 remained the editor title despite the new conflict metadata. A new browser assertion fails against that earlier binary. The fix adds the new managed Note heading and appends the exact room body, including any edited original title. The checked-copy retry test passes. Existing pre-fix plans remain unchanged to avoid overwriting an already-created copy. New final screenshots will replace the earlier review set. The initial server gate passed 262 tests but timed out after 15 seconds in startup_serves_http_while_upgrade_backfills_wait. That test builds a full app while a Home guard is held. Investigation is focused on the existing startup path and temporary-storage timing; no timeout or assertion was weakened. The Collab library passed all 100 tests; its subsequent cross-language process was terminated before a result. Final crate suites remain in progress.

kayg referenced this issue from a commit

2026-10-06 12:17:57 +00:00

Validate source capture with Clippy and keep Tag Undo races deterministic (#1197)

kayg referenced this issue from a commit

2026-10-06 12:39:24 +00:00

Keep preserved Notes distinct while retaining the complete edited body (#1197)

kayg commented

2026-10-06 12:54:26 +00:00

Author

Owner

Finding: delayed browser update during server adoption (#1197)

A deterministic regression creates an insert against a live block, holds its update in flight, saves the room as a conflict copy, then delivers the update after the server adopts disk. Before the fix the update vanished into deleted Yrs blocks:

late insert vanished into deleted blocks
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 100 filtered out; finished in 1.18s

Commit 992000f97a retains the pre-delete mirror before a server body delta. Ordinary typing and metadata-only commits do not allocate it. The same regression now verifies the complete insert in the live room and a checked durable save:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 100 filtered out; finished in 1.48s

The first full Collab gate also found that initial clean checkpoints changed the no-history crash epoch contract. The fix stores an initial clean base only when durable history needs it; failed saves still checkpoint independently. Existing restart expectations remain unchanged. The full Collab gate is running again because these are new code changes. Notes passed its full gate: 305 unit tests passed, 4 ignored; both integration tests passed. No production recovery has been applied.

Finding: delayed browser update during server adoption (#1197) A deterministic regression creates an insert against a live block, holds its update in flight, saves the room as a conflict copy, then delivers the update after the server adopts disk. Before the fix the update vanished into deleted Yrs blocks: ``` late insert vanished into deleted blocks test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 100 filtered out; finished in 1.18s ``` Commit 992000f97ace335c86692e9bb6486b2c3ead11cd retains the pre-delete mirror before a server body delta. Ordinary typing and metadata-only commits do not allocate it. The same regression now verifies the complete insert in the live room and a checked durable save: ``` test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 100 filtered out; finished in 1.48s ``` The first full Collab gate also found that initial clean checkpoints changed the no-history crash epoch contract. The fix stores an initial clean base only when durable history needs it; failed saves still checkpoint independently. Existing restart expectations remain unchanged. The full Collab gate is running again because these are new code changes. Notes passed its full gate: 305 unit tests passed, 4 ignored; both integration tests passed. No production recovery has been applied.

kayg commented

2026-10-06 13:10:58 +00:00

Author

Owner

Finding: consecutive source writes and bounded preservation (#1197)

A second regression showed that keeping only the initial mirror was insufficient. A server write created a block, another deleted it, and a delayed browser insert targeted that intermediate block. Before the extension:

new server branch was absent from the retained mirror
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 101 filtered out; finished in 0.53s

The mirror now retains missing structs from each prepared server revision before live mutation, omits deletes there, and advances its comparison base. The live document receives the exact admitted update. The existing 16 MiB encoded-state budget bounds retained branches; exceeding it refuses adoption before either document changes and keeps the existing branches. Its regression checks both trees remain intact.

Both delayed-update cases now pass. The two-write case also checks the durable source and a replay from the real history segment store:

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 100 filtered out; finished in 2.29s

Final Collab Clippy passed. The full suite is running for this last code change; both delayed tests and the budget-refusal test passed within it. The full suite before this extension passed, including all unchanged restart-epoch expectations. The real production UI regression passed and its 12 macOS fixture screenshots have been attached. No production recovery, push, merge or deploy was done.

Finding: consecutive source writes and bounded preservation (#1197) A second regression showed that keeping only the initial mirror was insufficient. A server write created a block, another deleted it, and a delayed browser insert targeted that intermediate block. Before the extension: ``` new server branch was absent from the retained mirror test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 101 filtered out; finished in 0.53s ``` The mirror now retains missing structs from each prepared server revision before live mutation, omits deletes there, and advances its comparison base. The live document receives the exact admitted update. The existing 16 MiB encoded-state budget bounds retained branches; exceeding it refuses adoption before either document changes and keeps the existing branches. Its regression checks both trees remain intact. Both delayed-update cases now pass. The two-write case also checks the durable source and a replay from the real history segment store: ``` test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 100 filtered out; finished in 2.29s ``` Final Collab Clippy passed. The full suite is running for this last code change; both delayed tests and the budget-refusal test passed within it. The full suite before this extension passed, including all unchanged restart-epoch expectations. The real production UI regression passed and its 12 macOS fixture screenshots have been attached. No production recovery, push, merge or deploy was done.

kayg commented

2026-10-06 13:17:01 +00:00

Author

Owner

Job collabloss-1197 — final report

Built

Source-span Note mapping with reparsed-output validation. The permanent save failure and the equal-line-count corruption case both have regression tests.
Checked Note, Task and Tag commits enter an open live room before acknowledgement, under the same User writer guard as room flushes. External revisions use a three-way block merge. Unmappable edits become a checked sibling with a durable Inbox Open action.
Failed flushes retain a marked Yrs checkpoint and source base. Restart restores pending edits, rebases disk changes and keeps verified browser author identities.
A pre-delete Yrs mirror retains late browser updates across server body adoption. It retains later server/client branches from the prepared source update before live mutation. Two before/after regressions check the complete edit, durable source and history replay. A third regression checks that the 16 MiB retention budget refuses adoption without changing either tree. Initial clean checkpoints are restricted to history-backed rooms so the existing no-history crash epoch contract stays unchanged.
Retry-stable recovery copies survive partial Index publication, retitles, midnight and a moved source folder. A new managed heading identifies the copy; the full edited room text remains after it.
A private history/cache inspection and recovery example. It writes only new siblings when the orchestrator applies a selected state to an offline destination.
A focused real-server production UI regression and an opt-in concurrent-write benchmark profile.

Recovery

Production inspection was read-only for source data. A private production scratch copy and a private local snapshot contain 18 cached Yrs states, 25 history points across 13 Note IDs, 39 history files (29,548 bytes; eight of the 25 points dated 2026-10-06), and 16 current Note files (21,331 bytes). Candidate differences were posted as IDs and byte counts in earlier findings. They cannot yet be attributed to the incident. No recovery has been applied. The private copies have mode 0700. The old incident process has restarted; its RAM state is gone. The live browser provider has no persistent local Note draft store. Export any still-open affected tab before reloading it.

The orchestrator must select a candidate from the private snapshot and stop the destination server before applying:

cargo run -p calternal-collab --example recover_note -- artifacts/recovery-1197 --apply <User UUID> <current Note UUID> <point|cached|cached:<legacy-key-hash>> <destination Home root> <destination Index file>

Files

calternal-collab: Cargo.toml, examples/recover_note.rs, src/history/write.rs, src/markdown.rs, src/session.rs, src/stored.rs.
calternal-plugin: src/notes_writer.rs.
calternal-tags: Cargo.toml, src/lib.rs, src/rewrite.rs.
calternal-plugin-notes: src/lib.rs, src/recovery.rs, src/store.rs, src/tasks_store.rs.
bench/notes-editor-series.mjs and apps/web/e2e/notes-preservation-1197.mjs. No dependency versions or migration numbers changed.

Decisions

Keep the checked source writer authoritative. Publish its committed revision into the live room before releasing the User guard or acknowledging the operation. The room uses that same guard and rebases blocks before saving.
Use the existing SyncConflict Inbox kind and stable /n/ links. Reuse InboxPanel, OverlaySurface, Menu, CopyLink, Pill/PillGroup and the existing Note editor/Card surfaces. No shared primitive was restyled.
Keep a bounded mirror of old branches across source adoption. Replay missing structs without deletes; refuse growth before mirror or live mutation when the 16 MiB budget is reached. Keep the existing branches and checkpoints.
Store the initial clean merge base only for history-backed rooms; no-history clean crashes retain their prior fresh-epoch behavior.
Use the existing bounded note_collab_state row for pending checkpoints; add no database migration. A pending envelope includes the source base and is not discarded on an ETag mismatch.
Freeze a private copy plan before the checked create. Reuse only an exact identity/body match. Probe the moved source folder once; never scan Home for a recovery retry.
Give a new copy its managed heading, then preserve the exact room text after it. Pre-fix copy plans stay unchanged rather than overwrite an existing sibling.

Expected cost

One touched Note requires one source read/hash, bounded parse/block merge and the existing checked write/Index publication. Metadata-only writes skip the body merge. A history-backed server body delta already prepares a cloned live document. The first such delta retains one complete Yrs mirror; later deltas add only its missing structs. Retained encoded bytes are capped at 16 MiB. Metadata and ordinary typing do not allocate a new mirror. Failed saves add a checkpoint and, when needed, one copy plan, sibling write and Inbox item. Recovery retry probes at most the planned path and the current sibling path. No new per-request Home scan or unbounded IO loop was added.

Performance measurement is deferred under the latest verification policy because this issue concerns data safety. The existing profile adds a 64-block average case and a 900-block worst case, samples and bursts while typing, p50/p95, CPU and RSS, and last-client durability. There is no existing live-source-write baseline; the older source-locate number is a different operation.

UX gaps closed

Failed mapping now gives the User a saved copy and Open instead of only a log warning.
A copy has a stable new identity and a distinct managed title while retaining all edited text.
Open is checked with pointer, touch and keyboard. Original source bytes remain unchanged in the conflict case. Final macOS screenshots cover 390/820/1440 in light and dark for both Inbox and the saved Note.

UX gaps left / known gaps

The calternal-server gate remains red: wire::tests::live_apps_run_in_separate_processes fails because its startup_serves_http_while_upgrade_backfills_wait child exceeds the unchanged 15-second timeout. It also fails with SSD-backed temporary files. No assertion or timeout was weakened; the cause is not established. The real-server UI fixture starts and completes. This report does not claim all gates passed.
The source-write performance profile is added but unmeasured in this job; the latest policy reserves performance runs for the perf VM and performance issues. The combined release needs the queued measurement.
An unsaved old-process RAM state cannot be reconstructed after that process restarts. Historical candidates need owner review before application.
Existing pre-fix copy plans retain their earlier heading; they need a separate checked repair if their displayed title must change. None of this job's recovery copies were applied to production.
A simultaneous failure of source storage and durable history/checkpoint storage still requires restoring storage; a software retry cannot make unavailable storage durable.

For the merge round

Resolve or verify the startup gate on the combined build with cargo test -p calternal-server wire::tests::startup_serves_http_while_upgrade_backfills_wait -- --exact --ignored --test-threads=4, then cargo test -p calternal-server -- --test-threads=4. It must prove HTTP starts while the Home writer guard remains held.
Full E2E, adversarial matrices, release/staging checks and real Mac interop remain merge-round work under the verification policy. Run ADVERSARIAL_SERVER_BIN=<combined binary> tests/adversarial/run.sh and cd apps/web && NOTES_E2E_EXTERNAL_ONLY=1 NOTES_E2E_MACOS=1 CALTERNAL_SERVER_BIN=<combined binary> bun e2e/notes.mjs --screenshots ../../artifacts/1197-merge-notes to check authorization, protocol abuse, source writers and existing Note actions.
Run cd apps/web && CALTERNAL_SERVER_BIN=<combined binary> bun e2e/notes-preservation-1197.mjs ../../artifacts/1197-merge-preservation on the combined binary if the merge changes its dependencies.
Measure the opt-in bench/notes-editor-series.mjs live-write profile on the perf VM with the shared release binary. Use flock /root/perf.lock bash -c 'uptime; NOTES_EDITOR_PROFILE_LIVE_WRITES=1 CALTERNAL_SERVER_BIN=<shared release binary> bun bench/notes-editor-series.mjs' from the repository on the perf VM; never compile there.

Head: 3f2ec50604. No push, merge or deploy was done. Final gate output and screenshot links follow below. Screenshots use the job server with the managed-copy title implementation and the real production SPA; later branch-mirror changes are covered by the focused and full Rust gates.

Gate output (verbatim)

cargo fmt --check: exit 0; no output. git diff --check: exit 0; no output.

cargo clippy -p calternal-plugin --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 8m 44s

cargo clippy -p calternal-tags --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 28s

cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.90s

cargo clippy -p calternal-collab --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.63s

cargo clippy -p calternal-server --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 09s

cargo test -p calternal-plugin -- --test-threads=4

test result: ok. 48 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-tags -- --test-threads=4

test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 92.12s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-plugin-notes -- --test-threads=4

test result: ok. 305 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 658.19s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.70s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo test -p calternal-server -- --test-threads=4 (SSD-backed comparison; failed)

HTTP startup waited for an upgrade backfill: Elapsed(())
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 272 filtered out; finished in 28.35s
test result: FAILED. 262 passed; 1 failed; 10 ignored; 0 measured; 0 filtered out; finished in 158.64s

cd apps/web && bun run check

perf-lint: PASS; 0 violations; 22359 scoped exceptions
svelte-check found 0 errors and 2 warnings in 2 files

cd apps/web && bun run test

Ran 136 tests in 0.071s
OK
 8 pass
 0 fail
 Test Files  274 passed (274)
      Tests  1907 passed (1907)

CALTERNAL_SERVER_BIN=<job binary> bun e2e/notes-preservation-1197.mjs

PASS #1197 live edit preserved, original unchanged, durable notice, pointer/touch/keyboard Open; 12 macOS screenshots

cargo test -p calternal-collab -- --test-threads=4 (final head, exit 0)

test result: ok. 103 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.92s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.08s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.62s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 83.57s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.71s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.78s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.57s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.95s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.70s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.66s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.31s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 43.13s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.73s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Screenshot evidence (synthetic fixture only; 12 macOS screenshots)

Cleanup

cargo clean completed:

Removed 23785 files, 27.4GiB total

Owned web build output and temporary test data were deleted. The worktree is clean. Private recovery snapshots and review artifacts remain; the local recovery directory is mode 0700.

Job collabloss-1197 — final report Built - Source-span Note mapping with reparsed-output validation. The permanent save failure and the equal-line-count corruption case both have regression tests. - Checked Note, Task and Tag commits enter an open live room before acknowledgement, under the same User writer guard as room flushes. External revisions use a three-way block merge. Unmappable edits become a checked sibling with a durable Inbox Open action. - Failed flushes retain a marked Yrs checkpoint and source base. Restart restores pending edits, rebases disk changes and keeps verified browser author identities. - A pre-delete Yrs mirror retains late browser updates across server body adoption. It retains later server/client branches from the prepared source update before live mutation. Two before/after regressions check the complete edit, durable source and history replay. A third regression checks that the 16 MiB retention budget refuses adoption without changing either tree. Initial clean checkpoints are restricted to history-backed rooms so the existing no-history crash epoch contract stays unchanged. - Retry-stable recovery copies survive partial Index publication, retitles, midnight and a moved source folder. A new managed heading identifies the copy; the full edited room text remains after it. - A private history/cache inspection and recovery example. It writes only new siblings when the orchestrator applies a selected state to an offline destination. - A focused real-server production UI regression and an opt-in concurrent-write benchmark profile. Recovery Production inspection was read-only for source data. A private production scratch copy and a private local snapshot contain 18 cached Yrs states, 25 history points across 13 Note IDs, 39 history files (29,548 bytes; eight of the 25 points dated 2026-10-06), and 16 current Note files (21,331 bytes). Candidate differences were posted as IDs and byte counts in earlier findings. They cannot yet be attributed to the incident. No recovery has been applied. The private copies have mode 0700. The old incident process has restarted; its RAM state is gone. The live browser provider has no persistent local Note draft store. Export any still-open affected tab before reloading it. The orchestrator must select a candidate from the private snapshot and stop the destination server before applying: ``` cargo run -p calternal-collab --example recover_note -- artifacts/recovery-1197 --apply <User UUID> <current Note UUID> <point|cached|cached:<legacy-key-hash>> <destination Home root> <destination Index file> ``` Files - calternal-collab: Cargo.toml, examples/recover_note.rs, src/history/write.rs, src/markdown.rs, src/session.rs, src/stored.rs. - calternal-plugin: src/notes_writer.rs. - calternal-tags: Cargo.toml, src/lib.rs, src/rewrite.rs. - calternal-plugin-notes: src/lib.rs, src/recovery.rs, src/store.rs, src/tasks_store.rs. - bench/notes-editor-series.mjs and apps/web/e2e/notes-preservation-1197.mjs. No dependency versions or migration numbers changed. Decisions - Keep the checked source writer authoritative. Publish its committed revision into the live room before releasing the User guard or acknowledging the operation. The room uses that same guard and rebases blocks before saving. - Use the existing SyncConflict Inbox kind and stable /n/<Note ID> links. Reuse InboxPanel, OverlaySurface, Menu, CopyLink, Pill/PillGroup and the existing Note editor/Card surfaces. No shared primitive was restyled. - Keep a bounded mirror of old branches across source adoption. Replay missing structs without deletes; refuse growth before mirror or live mutation when the 16 MiB budget is reached. Keep the existing branches and checkpoints. - Store the initial clean merge base only for history-backed rooms; no-history clean crashes retain their prior fresh-epoch behavior. - Use the existing bounded note_collab_state row for pending checkpoints; add no database migration. A pending envelope includes the source base and is not discarded on an ETag mismatch. - Freeze a private copy plan before the checked create. Reuse only an exact identity/body match. Probe the moved source folder once; never scan Home for a recovery retry. - Give a new copy its managed heading, then preserve the exact room text after it. Pre-fix copy plans stay unchanged rather than overwrite an existing sibling. Expected cost One touched Note requires one source read/hash, bounded parse/block merge and the existing checked write/Index publication. Metadata-only writes skip the body merge. A history-backed server body delta already prepares a cloned live document. The first such delta retains one complete Yrs mirror; later deltas add only its missing structs. Retained encoded bytes are capped at 16 MiB. Metadata and ordinary typing do not allocate a new mirror. Failed saves add a checkpoint and, when needed, one copy plan, sibling write and Inbox item. Recovery retry probes at most the planned path and the current sibling path. No new per-request Home scan or unbounded IO loop was added. Performance measurement is deferred under the latest verification policy because this issue concerns data safety. The existing profile adds a 64-block average case and a 900-block worst case, samples and bursts while typing, p50/p95, CPU and RSS, and last-client durability. There is no existing live-source-write baseline; the older source-locate number is a different operation. UX gaps closed - Failed mapping now gives the User a saved copy and Open instead of only a log warning. - A copy has a stable new identity and a distinct managed title while retaining all edited text. - Open is checked with pointer, touch and keyboard. Original source bytes remain unchanged in the conflict case. Final macOS screenshots cover 390/820/1440 in light and dark for both Inbox and the saved Note. UX gaps left / known gaps - The calternal-server gate remains red: `wire::tests::live_apps_run_in_separate_processes` fails because its `startup_serves_http_while_upgrade_backfills_wait` child exceeds the unchanged 15-second timeout. It also fails with SSD-backed temporary files. No assertion or timeout was weakened; the cause is not established. The real-server UI fixture starts and completes. This report does not claim all gates passed. - The source-write performance profile is added but unmeasured in this job; the latest policy reserves performance runs for the perf VM and performance issues. The combined release needs the queued measurement. - An unsaved old-process RAM state cannot be reconstructed after that process restarts. Historical candidates need owner review before application. - Existing pre-fix copy plans retain their earlier heading; they need a separate checked repair if their displayed title must change. None of this job's recovery copies were applied to production. - A simultaneous failure of source storage and durable history/checkpoint storage still requires restoring storage; a software retry cannot make unavailable storage durable. For the merge round - Resolve or verify the startup gate on the combined build with `cargo test -p calternal-server wire::tests::startup_serves_http_while_upgrade_backfills_wait -- --exact --ignored --test-threads=4`, then `cargo test -p calternal-server -- --test-threads=4`. It must prove HTTP starts while the Home writer guard remains held. - Full E2E, adversarial matrices, release/staging checks and real Mac interop remain merge-round work under the verification policy. Run `ADVERSARIAL_SERVER_BIN=<combined binary> tests/adversarial/run.sh` and `cd apps/web && NOTES_E2E_EXTERNAL_ONLY=1 NOTES_E2E_MACOS=1 CALTERNAL_SERVER_BIN=<combined binary> bun e2e/notes.mjs --screenshots ../../artifacts/1197-merge-notes` to check authorization, protocol abuse, source writers and existing Note actions. - Run `cd apps/web && CALTERNAL_SERVER_BIN=<combined binary> bun e2e/notes-preservation-1197.mjs ../../artifacts/1197-merge-preservation` on the combined binary if the merge changes its dependencies. - Measure the opt-in bench/notes-editor-series.mjs live-write profile on the perf VM with the shared release binary. Use `flock /root/perf.lock bash -c 'uptime; NOTES_EDITOR_PROFILE_LIVE_WRITES=1 CALTERNAL_SERVER_BIN=<shared release binary> bun bench/notes-editor-series.mjs'` from the repository on the perf VM; never compile there. Head: 3f2ec50604361d6002e73a566b87f4b5bde0eb9e. No push, merge or deploy was done. Final gate output and screenshot links follow below. Screenshots use the job server with the managed-copy title implementation and the real production SPA; later branch-mirror changes are covered by the focused and full Rust gates. Gate output (verbatim) `cargo fmt --check`: exit 0; no output. `git diff --check`: exit 0; no output. `cargo clippy -p calternal-plugin --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 8m 44s ``` `cargo clippy -p calternal-tags --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 28s ``` `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.90s ``` `cargo clippy -p calternal-collab --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.63s ``` `cargo clippy -p calternal-server --all-targets -- -D warnings` ``` Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 09s ``` `cargo test -p calternal-plugin -- --test-threads=4` ``` test result: ok. 48 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-tags -- --test-threads=4` ``` test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 92.12s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-plugin-notes -- --test-threads=4` ``` test result: ok. 305 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 658.19s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.70s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `cargo test -p calternal-server -- --test-threads=4` (SSD-backed comparison; failed) ``` HTTP startup waited for an upgrade backfill: Elapsed(()) test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 272 filtered out; finished in 28.35s test result: FAILED. 262 passed; 1 failed; 10 ignored; 0 measured; 0 filtered out; finished in 158.64s ``` `cd apps/web && bun run check` ``` perf-lint: PASS; 0 violations; 22359 scoped exceptions svelte-check found 0 errors and 2 warnings in 2 files ``` `cd apps/web && bun run test` ``` Ran 136 tests in 0.071s OK 8 pass 0 fail Test Files 274 passed (274) Tests 1907 passed (1907) ``` `CALTERNAL_SERVER_BIN=<job binary> bun e2e/notes-preservation-1197.mjs` ``` PASS #1197 live edit preserved, original unchanged, durable notice, pointer/touch/keyboard Open; 12 macOS screenshots ``` `cargo test -p calternal-collab -- --test-threads=4` (final head, exit 0) ``` test result: ok. 103 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 40.92s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.08s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.62s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 83.57s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.71s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.78s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.57s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.95s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.70s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.66s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.31s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 43.13s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.73s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Screenshot evidence (synthetic fixture only; 12 macOS screenshots) ![copy-desktop-dark](https://git.kayg.org/attachments/17f338f1-d1bc-4bfe-b319-6de8539a162c) ![copy-desktop-light](https://git.kayg.org/attachments/1e6415be-a9c7-410b-b9d1-3ce62f1f5740) ![copy-phone-dark](https://git.kayg.org/attachments/8e333aaf-9767-4903-8910-c2dda22935a4) ![copy-phone-light](https://git.kayg.org/attachments/19594e75-c429-4f5a-bf66-897f273be1ec) ![copy-tablet-dark](https://git.kayg.org/attachments/37f48b14-4e04-4d2c-af8b-da5164ecfddd) ![copy-tablet-light](https://git.kayg.org/attachments/55a53ef1-6da4-4345-83d5-81ee268e05e3) ![inbox-desktop-dark](https://git.kayg.org/attachments/9738cab3-d780-4cf9-ae79-656214f9fe1c) ![inbox-desktop-light](https://git.kayg.org/attachments/9cd52f61-d984-4b9b-92c5-fdfe8e03dee8) ![inbox-phone-dark](https://git.kayg.org/attachments/f7b49ca4-7a24-4a75-ae8c-7c4997277dc7) ![inbox-phone-light](https://git.kayg.org/attachments/60883005-0d7a-453e-83c5-06a16a14a7c2) ![inbox-tablet-dark](https://git.kayg.org/attachments/d1a28cf1-ae90-4a17-b0bc-616f7c1e588b) ![inbox-tablet-light](https://git.kayg.org/attachments/5482fd20-05c2-4143-bfae-357207cafcf7) Cleanup `cargo clean` completed: ``` Removed 23785 files, 27.4GiB total ``` Owned web build output and temporary test data were deleted. The worktree is clean. Private recovery snapshots and review artifacts remain; the local recovery directory is mode 0700.

kayg commented

2026-10-06 13:19:46 +00:00

Author

Owner

Independent review started on branch job/collabloss-1197 at 3f2ec50604. Base origin/dev: 5301e02085. Reviewing the diff and executable failure paths before reading the author report. No push or deployment.

Independent review started on branch job/collabloss-1197 at 3f2ec50604361d6002e73a566b87f4b5bde0eb9e. Base origin/dev: 5301e020859c1d61b4a7f812a5049a64241203e8. Reviewing the diff and executable failure paths before reading the author report. No push or deployment.

kayg commented

2026-10-06 13:27:16 +00:00

Author

Owner

Independent review findings at 3f2ec5060 (before the author report):

Pending recovery cache state begins with CNP1197 plus a Markdown base, but examples/recover_note.rs decodes selected cached rows as raw Yrs V1 bytes. The query also excludes valid pending envelopes above 16 MiB although the writer accepts a 16 MiB update plus a 10 MiB base. Adding shared-envelope decoding and bounded selection tests.
The branch mirror for in-flight browser edits is only in Room.conflict_shadow. store_locked stores only the live Yrs state and baseline. load_room initializes conflict_shadow to None while restoring the prior epoch. A delayed edit to a deleted branch can therefore pass lineage checks after restart without a surviving mirror. A focused regression is compiling to verify the loss.
The offline recovery reader uses SegmentHistoryStore::open on its input copy. That store repairs torn tails by truncating segments and initializes the copied Index. It is not a read-only input reader. The tool documents using a separate private copy, which protects production only when the operator follows that instruction.

Rust build is compiling in the job target with four build workers. Web check finished: svelte-check found 0 errors and 2 warnings in 2 files. No production data, credentials, push or deployment used.

Independent review findings at 3f2ec5060 (before the author report): 1. Pending recovery cache state begins with CNP1197 plus a Markdown base, but examples/recover_note.rs decodes selected cached rows as raw Yrs V1 bytes. The query also excludes valid pending envelopes above 16 MiB although the writer accepts a 16 MiB update plus a 10 MiB base. Adding shared-envelope decoding and bounded selection tests. 2. The branch mirror for in-flight browser edits is only in Room.conflict_shadow. store_locked stores only the live Yrs state and baseline. load_room initializes conflict_shadow to None while restoring the prior epoch. A delayed edit to a deleted branch can therefore pass lineage checks after restart without a surviving mirror. A focused regression is compiling to verify the loss. 3. The offline recovery reader uses SegmentHistoryStore::open on its input copy. That store repairs torn tails by truncating segments and initializes the copied Index. It is not a read-only input reader. The tool documents using a separate private copy, which protects production only when the operator follows that instruction. Rust build is compiling in the job target with four build workers. Web check finished: svelte-check found 0 errors and 2 warnings in 2 files. No production data, credentials, push or deployment used.

kayg commented

2026-10-06 13:33:45 +00:00

Author

Owner

Provisional verdict on reviewed head 3f2ec50604: SAFE TO MERGE: NO.

The branch mirror is not checkpointed across a clean last-client unload or restart. Both restore the prior room epoch but initialize conflict_shadow=None. A browser update authored in a branch deleted by a checked server write is accepted into a collected parent and disappears from visible Markdown.

An independent Yjs wire-format probe (synthetic fixture) reproduced the checkpoint behavior: {"sameEpochCheckpoint":true,"delayedUpdateBytes":29,"delayedInsertRetained":false}. The preservation assertion fails. The Rust regression exercises the actual Hub loader and a quota-failed checkpoint; a second regression exercises the actual checked writer and clean last-client unload. Both are queued behind the fresh dependency build, not yet claimed as executed.

The fix needs to preserve and restore the branch mirror with its comparison tree, or provide a recovery path for stale-epoch browser drafts before changing lineage. Persisting only the already-collected live Yrs state cannot recover those branches. Do not deploy this head based on the passing ordinary typing/restart tests.

Web gates completed independently: svelte-check found 0 errors and 2 warnings in 2 files; Test Files 274 passed (274); Tests 1907 passed (1907). The author report was read after the independent diff assessment and protocol probe design.

Provisional verdict on reviewed head 3f2ec50604361d6002e73a566b87f4b5bde0eb9e: SAFE TO MERGE: NO. The branch mirror is not checkpointed across a clean last-client unload or restart. Both restore the prior room epoch but initialize conflict_shadow=None. A browser update authored in a branch deleted by a checked server write is accepted into a collected parent and disappears from visible Markdown. An independent Yjs wire-format probe (synthetic fixture) reproduced the checkpoint behavior: {"sameEpochCheckpoint":true,"delayedUpdateBytes":29,"delayedInsertRetained":false}. The preservation assertion fails. The Rust regression exercises the actual Hub loader and a quota-failed checkpoint; a second regression exercises the actual checked writer and clean last-client unload. Both are queued behind the fresh dependency build, not yet claimed as executed. The fix needs to preserve and restore the branch mirror with its comparison tree, or provide a recovery path for stale-epoch browser drafts before changing lineage. Persisting only the already-collected live Yrs state cannot recover those branches. Do not deploy this head based on the passing ordinary typing/restart tests. Web gates completed independently: svelte-check found 0 errors and 2 warnings in 2 files; Test Files 274 passed (274); Tests 1907 passed (1907). The author report was read after the independent diff assessment and protocol probe design.

kayg commented

2026-10-06 13:50:11 +00:00

Author

Owner

LIVE REPRO (orchestrator, 2026-10-06 19:14 IST / 15:44 CEST, production 5301e0208): the owner saved a Note; the UI reported "save failed" although the file was written; the server then logged live Note flush deferred ×2, last-client Note flush failed, and live Note flush retry failed every minute since (could not map edited Markdown to the loaded Note source). Second forensics snapshot: /srv/calternal/forensics-1197b-20261006-1549 (journal, cache, tmp; room states also live in note_collab_state.state). Please confirm the fix covers this exact sequence (UI whole-body save succeeds while the live room cannot map → room must rebase onto the saved file, and the UI must not report a failure for a save that committed). Add a regression for it.

LIVE REPRO (orchestrator, 2026-10-06 19:14 IST / 15:44 CEST, production 5301e0208): the owner saved a Note; the UI reported "save failed" although the file was written; the server then logged `live Note flush deferred` ×2, `last-client Note flush failed`, and `live Note flush retry failed` every minute since (could not map edited Markdown to the loaded Note source). Second forensics snapshot: /srv/calternal/forensics-1197b-20261006-1549 (journal, cache, tmp; room states also live in note_collab_state.state). Please confirm the fix covers this exact sequence (UI whole-body save succeeds while the live room cannot map → room must rebase onto the saved file, and the UI must not report a failure for a save that committed). Add a regression for it.

kayg commented

2026-10-06 13:51:10 +00:00

Author

Owner

Owner confirmed nothing was lost (note edd525e5…, 475 bytes, file written 13:45:45 UTC). The write came through a WebDAV upload at that exact second (log: calternal_webdav::profile: WebDAV upload stage), i.e. an external WebDAV write while a live room was open — confirm WebDAV PUT/MOVE writes are routed through the open room (or the room rebases onto them) and that the UI never reports a failed save for a committed write. Regression: open live room → WebDAV PUT of the same Note → no warning loop, no failure message, room converges to the file.

Owner confirmed nothing was lost (note edd525e5…, 475 bytes, file written 13:45:45 UTC). The write came through a WebDAV upload at that exact second (log: `calternal_webdav::profile: WebDAV upload stage`), i.e. an external WebDAV write while a live room was open — confirm WebDAV PUT/MOVE writes are routed through the open room (or the room rebases onto them) and that the UI never reports a failed save for a committed write. Regression: open live room → WebDAV PUT of the same Note → no warning loop, no failure message, room converges to the file.

kayg commented

2026-10-06 13:52:23 +00:00

Author

Owner

CORRECTION (orchestrator): the 13:45:45 UTC write was NOT an external WebDAV client. uploads::profile_stage logs every Files change-event publish (crates/plugins/files/src/lib.rs publish_inner → target calternal_webdav::profile, message "WebDAV upload stage") for ANY writer, including the app's own Note save. So this is the app's own whole-body save landing while its own live room could not map — the same #1197 mismatch, triggered by the app itself. The regression to add: live room open + the app's fallback/whole-body save of the same Note → no warning loop, no "save failed" message, room converges.

CORRECTION (orchestrator): the 13:45:45 UTC write was NOT an external WebDAV client. `uploads::profile_stage` logs every Files change-event publish (crates/plugins/files/src/lib.rs publish_inner → target calternal_webdav::profile, message "WebDAV upload stage") for ANY writer, including the app's own Note save. So this is the app's own whole-body save landing while its own live room could not map — the same #1197 mismatch, triggered by the app itself. The regression to add: live room open + the app's fallback/whole-body save of the same Note → no warning loop, no "save failed" message, room converges.

kayg referenced this issue

2026-10-06 13:52:24 +00:00

Logs: 'WebDAV upload stage' is logged for every file change, not only WebDAV #1204

kayg commented

2026-10-06 13:53:15 +00:00

Author

Owner

Recovery fix committed: d9cac748ca. The recovery example now shares the runtime pending-envelope decoder and the 16 MiB update + 10 MiB source-base selection bound. It streams cached states rather than loading every state at once; legacy-key selection reads identities first and only loads the matching state. Durable history points keep their separate frame limits. No dependency or migration change.

The exact Rust envelope-parser function was tested independently while the full crate dependency build runs. Before: test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s. After: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s. cargo fmt --check and git diff --check exit 0 with no output. Full crate validation is still pending; this comment does not claim its gates passed.

The merge verdict remains NO: this commit does not persist the branch mirror across room unload/restart. The actual Hub preservation regressions and same-block anchor test are in the working tree and queued for execution. No push, deploy or source-data recovery has been performed.

Recovery fix committed: d9cac748cafe1d9fc8bf1dac59fb621a17f07ea6. The recovery example now shares the runtime pending-envelope decoder and the 16 MiB update + 10 MiB source-base selection bound. It streams cached states rather than loading every state at once; legacy-key selection reads identities first and only loads the matching state. Durable history points keep their separate frame limits. No dependency or migration change. The exact Rust envelope-parser function was tested independently while the full crate dependency build runs. Before: test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s. After: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s. cargo fmt --check and git diff --check exit 0 with no output. Full crate validation is still pending; this comment does not claim its gates passed. The merge verdict remains NO: this commit does not persist the branch mirror across room unload/restart. The actual Hub preservation regressions and same-block anchor test are in the working tree and queued for execution. No push, deploy or source-data recovery has been performed.

kayg commented

2026-10-06 14:16:19 +00:00

Author

Owner

Independent same-block finding for requirement (3): assigning a Block ID while the User types in that paragraph duplicates the prose in the three-way plan.

I compiled the exact plan_block_edits / merge_chunk / align / diff functions from calternal-collab/src/lib.rs against the current similar dependency. The fixture has one changed live paragraph and an anchor-only source change in that same paragraph. The preservation assertion reports 2 prose copies instead of 1:

assertion `left == right` failed: anchor-only source change duplicated prose
  left: 2
 right: 1
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The actual Hub regression uses the checked Note writer, live typing in the first block and an added Block ID. It is queued in the crate build. Existing each_source_writer_keeps_concurrent_typing types in another block and does not cover this case. Anchor-only changes need to update the live block metadata; they are not a conflicting prose version. No existing test expectation has been changed.

Verdict remains SAFE TO MERGE: NO at 150c5075d6. The recovery-format fix is committed; branch-mirror persistence and same-block source adoption remain defects. Rust gates remain in progress within the 90-minute review limit.

Independent same-block finding for requirement (3): assigning a Block ID while the User types in that paragraph duplicates the prose in the three-way plan. I compiled the exact plan_block_edits / merge_chunk / align / diff functions from calternal-collab/src/lib.rs against the current similar dependency. The fixture has one changed live paragraph and an anchor-only source change in that same paragraph. The preservation assertion reports 2 prose copies instead of 1: ``` assertion `left == right` failed: anchor-only source change duplicated prose left: 2 right: 1 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` The actual Hub regression uses the checked Note writer, live typing in the first block and an added Block ID. It is queued in the crate build. Existing each_source_writer_keeps_concurrent_typing types in another block and does not cover this case. Anchor-only changes need to update the live block metadata; they are not a conflicting prose version. No existing test expectation has been changed. Verdict remains SAFE TO MERGE: NO at 150c5075d62890b0077ac82718e3a555a6a302cf. The recovery-format fix is committed; branch-mirror persistence and same-block source adoption remain defects. Rust gates remain in progress within the 90-minute review limit.

kayg commented

2026-10-06 14:43:00 +00:00

Author

Owner

SAFE TO MERGE: NO

Independent data-safety review of job/collabloss-1197, original head 3f2ec50604. Final reviewed head: 150c5075d6. origin/dev base 5301e02085; the required fetch and merge found it already up to date. No push, deployment or issue closure.

Blocking findings (actual Hub regression tests, not only model simulations):

Clean room unload/reload silently loses a delayed browser insert into a source block deleted by a server writer. The conflict_shadow holds the pre-delete lineage only in memory; reload initializes it to None while retaining the cached room epoch. The accepted browser update then targets garbage-collected content.
Pending-checkpoint restart has the same loss. Existing restart tests do not cover this deleted-parent update. Both new tests reach apply_client_message and fail their edit-retention assertion.
An anchor-only checked writer during typing in that same paragraph duplicates prose. The merge compares the source block including metadata and retains both variants. The existing writer matrix edits another block and misses this case.

Exact focused output:
clean unload silently discarded the delayed browser edit
restart silently discarded the delayed browser edit
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 107 filtered out; finished in 2.07s
assertion left == right failed: anchor publication duplicated the original prose
left: 2
right: 1
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 108 filtered out; finished in 1.10s

Fix built in two atomic commits:

d9cac748ca: shared bounded clean/pending checkpoint decoder; recovery reads failed-save envelopes rather than raw Yrs, includes the full valid envelope limit (16 MiB update plus 10 MiB source base plus header), rejects missing lineage, streams cache candidates and loads only the selected legacy state. Durable history keeps its separate frame bounds. The decoder does not mutate input bytes.
150c5075d6: documents private recovery input, inactive destination and sibling-only application invariants.
Files: crates/calternal-collab/src/stored.rs, crates/calternal-collab/src/lib.rs, crates/calternal-collab/examples/recover_note.rs. Re-read their changed comments before reporting.

Gate output, verbatim excerpts from this job:

cargo fmt --check: no output, exit 0 (including final clean worktree).
cargo clippy -p calternal-collab --all-targets -- -D warnings:
Finished dev profile [unoptimized + debuginfo] target(s) in 57m 09s
cargo test -p calternal-collab --lib -- --test-threads=4, with review regression patch:
test result: FAILED. 106 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 47.18s
error: test failed, to rerun pass -p calternal-collab --lib
Recovery regressions, including a valid pending state above 16 MiB:
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 107 filtered out; finished in 0.82s
BOM/CRLF, anchors and repeated prose:
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 108 filtered out; finished in 1.59s
cargo clippy -p calternal-plugin --all-targets -- -D warnings:
Finished dev profile [unoptimized + debuginfo] target(s) in 58m 17s
cd apps/web && bun run check:
perf-lint: PASS; 0 violations; 22359 scoped exceptions
svelte-check found 0 errors and 2 warnings in 2 files
cd apps/web && bun run test -- --maxWorkers=2:
Test Files 274 passed (274)
Tests 1907 passed (1907)

Known gaps:

The two edit-loss paths and anchor duplication are NOT fixed. They block this merge.
Strict recovery input read-only behavior is NOT established: SegmentHistoryStore::open is writable and can repair a torn segment in the copied input; opening the Index can initialize it. The documented private-copy requirement protects production only if followed. New sibling creation is checked, but do not treat the input opener as read-only.
Per-User conflict-copy/changed-receipt test was prepared but not successfully executed; do not count it as evidence. The passing 106 collab tests include existing failed-flush, quota, consecutive-write, retry and conflict-shadow coverage. They do not prove the entire requested cross-plugin/starvation matrix.
The initial fresh cargo build took 59m 14s. At the 90-minute boundary, plugin tests and tags/notes/server explicit clippy/test gates remained incomplete; stopped the job's queued runner instead of exceeding the cap. Dependencies being checked by collab Clippy are not substitute per-crate gates. Full collab integration/doc-test gate also remains incomplete. No unrelated expectation was weakened.
Regression source and logs are in worktree artifacts/review-regressions.patch, room-loss-focused.log, anchor-focused.log, source-prose-focused.log, recovery-focused.log, review-collab-tests.log, review-collab-clippy.log and web-test.log. The temporary red regression patch was removed from tracked files after execution; it is preserved for application and repair. No screenshot or review artifact was committed.

For the next verification round, after fixes: apply the saved patch, retain all three retention assertions, then cargo fmt --check; cargo clippy -p --all-targets -- -D warnings and cargo test -p -- --test-threads=4 for calternal-collab, calternal-plugin, calternal-tags, calternal-plugin-notes, calternal-server. Re-run web check/test if dependencies change. Finish focused per-User retry/Open-Inbox receipt proof, fault injection for quota/IO/mapping, server-writer lock order/starvation, and real-server WebDAV/Files overlap. Deployment must remain held.

Decisions: use one shared checkpoint decoder and the existing runtime limits, rather than a second recovery format or weaker bounds. Keep durable-history limits separate. Preserve failing review tests as an artifact rather than commit a knowingly red suite. No new UI or user-visible operation; no UI controls, UX changes, screenshots or performance profile were added. No performance measurement was claimed.

SAFE TO MERGE: NO Independent data-safety review of job/collabloss-1197, original head 3f2ec50604361d6002e73a566b87f4b5bde0eb9e. Final reviewed head: 150c5075d62890b0077ac82718e3a555a6a302cf. origin/dev base 5301e020859c1d61b4a7f812a5049a64241203e8; the required fetch and merge found it already up to date. No push, deployment or issue closure. Blocking findings (actual Hub regression tests, not only model simulations): 1. Clean room unload/reload silently loses a delayed browser insert into a source block deleted by a server writer. The conflict_shadow holds the pre-delete lineage only in memory; reload initializes it to None while retaining the cached room epoch. The accepted browser update then targets garbage-collected content. 2. Pending-checkpoint restart has the same loss. Existing restart tests do not cover this deleted-parent update. Both new tests reach apply_client_message and fail their edit-retention assertion. 3. An anchor-only checked writer during typing in that same paragraph duplicates prose. The merge compares the source block including metadata and retains both variants. The existing writer matrix edits another block and misses this case. Exact focused output: clean unload silently discarded the delayed browser edit restart silently discarded the delayed browser edit test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 107 filtered out; finished in 2.07s assertion `left == right` failed: anchor publication duplicated the original prose left: 2 right: 1 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 108 filtered out; finished in 1.10s Fix built in two atomic commits: - d9cac748cafe1d9fc8bf1dac59fb621a17f07ea6: shared bounded clean/pending checkpoint decoder; recovery reads failed-save envelopes rather than raw Yrs, includes the full valid envelope limit (16 MiB update plus 10 MiB source base plus header), rejects missing lineage, streams cache candidates and loads only the selected legacy state. Durable history keeps its separate frame bounds. The decoder does not mutate input bytes. - 150c5075d62890b0077ac82718e3a555a6a302cf: documents private recovery input, inactive destination and sibling-only application invariants. Files: crates/calternal-collab/src/stored.rs, crates/calternal-collab/src/lib.rs, crates/calternal-collab/examples/recover_note.rs. Re-read their changed comments before reporting. Gate output, verbatim excerpts from this job: - cargo fmt --check: no output, exit 0 (including final clean worktree). - cargo clippy -p calternal-collab --all-targets -- -D warnings: Finished `dev` profile [unoptimized + debuginfo] target(s) in 57m 09s - cargo test -p calternal-collab --lib -- --test-threads=4, with review regression patch: test result: FAILED. 106 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 47.18s error: test failed, to rerun pass `-p calternal-collab --lib` - Recovery regressions, including a valid pending state above 16 MiB: test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 107 filtered out; finished in 0.82s - BOM/CRLF, anchors and repeated prose: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 108 filtered out; finished in 1.59s - cargo clippy -p calternal-plugin --all-targets -- -D warnings: Finished `dev` profile [unoptimized + debuginfo] target(s) in 58m 17s - cd apps/web && bun run check: perf-lint: PASS; 0 violations; 22359 scoped exceptions svelte-check found 0 errors and 2 warnings in 2 files - cd apps/web && bun run test -- --maxWorkers=2: Test Files 274 passed (274) Tests 1907 passed (1907) Known gaps: - The two edit-loss paths and anchor duplication are NOT fixed. They block this merge. - Strict recovery input read-only behavior is NOT established: SegmentHistoryStore::open is writable and can repair a torn segment in the copied input; opening the Index can initialize it. The documented private-copy requirement protects production only if followed. New sibling creation is checked, but do not treat the input opener as read-only. - Per-User conflict-copy/changed-receipt test was prepared but not successfully executed; do not count it as evidence. The passing 106 collab tests include existing failed-flush, quota, consecutive-write, retry and conflict-shadow coverage. They do not prove the entire requested cross-plugin/starvation matrix. - The initial fresh cargo build took 59m 14s. At the 90-minute boundary, plugin tests and tags/notes/server explicit clippy/test gates remained incomplete; stopped the job's queued runner instead of exceeding the cap. Dependencies being checked by collab Clippy are not substitute per-crate gates. Full collab integration/doc-test gate also remains incomplete. No unrelated expectation was weakened. - Regression source and logs are in worktree artifacts/review-regressions.patch, room-loss-focused.log, anchor-focused.log, source-prose-focused.log, recovery-focused.log, review-collab-tests.log, review-collab-clippy.log and web-test.log. The temporary red regression patch was removed from tracked files after execution; it is preserved for application and repair. No screenshot or review artifact was committed. For the next verification round, after fixes: apply the saved patch, retain all three retention assertions, then cargo fmt --check; cargo clippy -p <crate> --all-targets -- -D warnings and cargo test -p <crate> -- --test-threads=4 for calternal-collab, calternal-plugin, calternal-tags, calternal-plugin-notes, calternal-server. Re-run web check/test if dependencies change. Finish focused per-User retry/Open-Inbox receipt proof, fault injection for quota/IO/mapping, server-writer lock order/starvation, and real-server WebDAV/Files overlap. Deployment must remain held. Decisions: use one shared checkpoint decoder and the existing runtime limits, rather than a second recovery format or weaker bounds. Keep durable-history limits separate. Preserve failing review tests as an artifact rather than commit a knowingly red suite. No new UI or user-visible operation; no UI controls, UX changes, screenshots or performance profile were added. No performance measurement was claimed.

kayg commented

2026-10-06 14:45:25 +00:00

Author

Owner

Started collabloss2-1197 on job/collabloss-1197 at 150c5075d6. Base origin/dev: 5301e02085. Applied artifacts/review-regressions.patch unchanged before implementation. Verifying delayed-update lineage, same-block anchor adoption, strict read-only recovery input, receipt/lock order and the app whole-body-save case. No production data, push or deployment.

Started collabloss2-1197 on job/collabloss-1197 at 150c5075d62890b0077ac82718e3a555a6a302cf. Base origin/dev: 5301e020859c1d61b4a7f812a5049a64241203e8. Applied artifacts/review-regressions.patch unchanged before implementation. Verifying delayed-update lineage, same-block anchor adoption, strict read-only recovery input, receipt/lock order and the app whole-body-save case. No production data, push or deployment.

kayg commented

2026-10-06 15:33:28 +00:00

Author

Owner

collabloss2-1197: supplied regressions reproduced unchanged. Clean reload and pending restart both fail with the delayed-edit assertions at review head. A room restores its epoch and live Yrs tree but previously dropped the only surviving pre-delete tree. The checkpoint now stores the bounded pre-delete tree and its comparison point atomically with the live tree; restart source rebases retain that tree before deleting roots. Both supplied reload assertions now pass (2 passed; 0 failed). Anchor regression separately reproduces (original prose count 2, expected 1); anchor-only publication needs an in-place metadata delta, not a root replacement. No assertions weakened. App save path is the production case to verify (not WebDAV). Crate validation is running before the first fix commit.

kayg commented

2026-10-06 15:42:39 +00:00

Author

Owner

Fix commits: 475e5a340 (reload branches), 4efc2c407 (anchor metadata), fbd49f6ed (strict recovery input reads). Supplied anchor assertion now passes unchanged. Recovery segment test passes with a writer lock held, a torn tail, and a staging file: inspection preserves bytes/inode/mode/mtime/ctime and refuses append/truncate/replace/materialized writes. Input Index test also passes with a WAL-only row; writes are refused and the input file set, bytes and identities remain unchanged.

Decision: immutable SQLite alone omitted the WAL-only row in a focused test (RowNotFound). Recovery therefore streams the fixed Index and WAL names through calternal-fs into private scratch, and reads that scratch copy; history and Notes stay in the read-only input. Memory is bounded; copy IO is linear in Index plus WAL size. The input must be an inactive coherent copy. No production input was modified. The Notes addition only exposes the existing identity lookup/read through a supplied reader pool, so recovery does not construct a writable Db on input.

Fix commits: 475e5a340 (reload branches), 4efc2c407 (anchor metadata), fbd49f6ed (strict recovery input reads). Supplied anchor assertion now passes unchanged. Recovery segment test passes with a writer lock held, a torn tail, and a staging file: inspection preserves bytes/inode/mode/mtime/ctime and refuses append/truncate/replace/materialized writes. Input Index test also passes with a WAL-only row; writes are refused and the input file set, bytes and identities remain unchanged. Decision: immutable SQLite alone omitted the WAL-only row in a focused test (RowNotFound). Recovery therefore streams the fixed Index and WAL names through calternal-fs into private scratch, and reads that scratch copy; history and Notes stay in the read-only input. Memory is bounded; copy IO is linear in Index plus WAL size. The input must be an inactive coherent copy. No production input was modified. The Notes addition only exposes the existing identity lookup/read through a supplied reader pool, so recovery does not construct a writable Db on input.

kayg commented

2026-10-06 15:55:59 +00:00

Author

Owner

58b6d6391 commits the production-route and receipt/lock tests. The real local HTTP app-body PUT passes with an unmappable live source projection: HTTP 200, exact saved body, room convergence, three later successful flushes, clean dirty flag, no copy allocated, last-client unload completed with no retry room. The matrix passed 96 cases (8 writer families × 12 locks: User; Hub map/init; room flush/awareness/persisted/baseline/pending Index; bindings/branch mirror/block Index/timezone), with a simultaneous room flush in each case. Real Tag rewrite and Undo, date backfill, Tasks, anchors, link cards, whole-body and recovery-copy writers ran. Source locks were released in a bounded window; no deadlock occurred. Per-User changed-receipt regression passes unchanged.

Nested anchor and replay tests found an extra case: a Block ID belongs to the paragraph inside a quote, rather than its enclosing root. d2602fee8 handles nested carriers, marked prose, delayed inserts and idempotent replay in place. The supplied three blocker assertions remain unchanged.

Recovery was simplified further: its disposable scratch Index now uses the existing Db/Notes reader implementation, so live Notes helpers and their exact performance exceptions are unchanged. Input still uses confined reads only. No ratchet or rule was weakened. One fetch/merge of origin/dev completed (already up to date); final gates are running.

58b6d6391 commits the production-route and receipt/lock tests. The real local HTTP app-body PUT passes with an unmappable live source projection: HTTP 200, exact saved body, room convergence, three later successful flushes, clean dirty flag, no copy allocated, last-client unload completed with no retry room. The matrix passed 96 cases (8 writer families × 12 locks: User; Hub map/init; room flush/awareness/persisted/baseline/pending Index; bindings/branch mirror/block Index/timezone), with a simultaneous room flush in each case. Real Tag rewrite and Undo, date backfill, Tasks, anchors, link cards, whole-body and recovery-copy writers ran. Source locks were released in a bounded window; no deadlock occurred. Per-User changed-receipt regression passes unchanged. Nested anchor and replay tests found an extra case: a Block ID belongs to the paragraph inside a quote, rather than its enclosing root. d2602fee8 handles nested carriers, marked prose, delayed inserts and idempotent replay in place. The supplied three blocker assertions remain unchanged. Recovery was simplified further: its disposable scratch Index now uses the existing Db/Notes reader implementation, so live Notes helpers and their exact performance exceptions are unchanged. Input still uses confined reads only. No ratchet or rule was weakened. One fetch/merge of origin/dev completed (already up to date); final gates are running.

Rows
Columns

DATA LOSS: live Note edits not saved ('could not map edited Markdown to the loaded Note source') #1197

DATA LOSS (production, 2026-10-06): live Note edits not saved