Notes test process crashes with SIGSEGV after all assertions pass in merge round 7c

kayg commented

2026-10-04 19:25:51 +00:00

Owner

Merge round #867, branch job/merge-round-7c, code head da353849b.

Command: CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 OPENSSL_NO_VENDOR=1 cargo test -p calternal-plugin-notes -- --test-threads=4.

Verbatim output:

test result: ok. 278 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 405.98s

error: test failed, to rerun pass `-p calternal-plugin-notes --lib`

Caused by:
  process didn't exit successfully: `/home/kayg/build/targets/merge-round-7c/debug/deps/calternal_plugin_notes-776d209c149f66f4 --test-threads=4` (signal: 11, SIGSEGV: invalid memory reference)

All active test assertions passed before the process crashed. This is a process shutdown failure, not a SLOW-only result. No runtime cause has been identified. Gate log: artifacts/merge-round-7c/gates/calternal-plugin-notes-test-lock-final.log. This blocks READY FOR STAGING.

Merge round #867, branch job/merge-round-7c, code head da353849b. Command: `CARGO_PROFILE_DEV_DEBUG=line-tables-only CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 OPENSSL_NO_VENDOR=1 cargo test -p calternal-plugin-notes -- --test-threads=4`. Verbatim output: ```text test result: ok. 278 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 405.98s error: test failed, to rerun pass `-p calternal-plugin-notes --lib` Caused by: process didn't exit successfully: `/home/kayg/build/targets/merge-round-7c/debug/deps/calternal_plugin_notes-776d209c149f66f4 --test-threads=4` (signal: 11, SIGSEGV: invalid memory reference) ``` All active test assertions passed before the process crashed. This is a process shutdown failure, not a SLOW-only result. No runtime cause has been identified. Gate log: artifacts/merge-round-7c/gates/calternal-plugin-notes-test-lock-final.log. This blocks READY FOR STAGING.

kayg referenced this issue

2026-10-04 19:53:51 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 19:53:52 +00:00

CANVAS core: Excalidraw canvas as a Note, live co-editing, one event path, chrome, element links, export (§60) #976

kayg referenced this issue

2026-10-04 19:53:53 +00:00

CANVAS: files over app — images and files on a canvas are linked Home files, never embedded (§60) #989

kayg referenced this issue

2026-10-04 19:53:53 +00:00

CANVAS: Share and Collaborate — live co-editing with presence, view-only links (§60, headline) #991

kayg referenced this issue

2026-10-04 19:53:54 +00:00

CANVAS: Sketch from the Log composer and the note / menu (§60) #990

kayg referenced this issue

2026-10-04 19:53:55 +00:00

CANVAS: iPad Pencil and touch drawing (§60) #978

kayg referenced this issue

2026-10-04 19:53:56 +00:00

CANVAS: live cards for any calternal item, placeholders, backlinks (§60) #977

kayg commented

2026-10-04 19:57:08 +00:00

Author

Owner

Started repair on branch job/crash-1069, base 094d22e445 (merge round 7c). I will reproduce the post-suite SIGSEGV with direct binary runs, collect a native backtrace, identify the introducing change, and verify five consecutive clean Notes suite exits after the fix. No push or deployment.

Started repair on branch job/crash-1069, base 094d22e44507bf8bdd87dd8ffd460c254cb6329c (merge round 7c). I will reproduce the post-suite SIGSEGV with direct binary runs, collect a native backtrace, identify the introducing change, and verify five consecutive clean Notes suite exits after the fix. No push or deployment.

kayg referenced this issue

2026-10-04 19:57:10 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-04 20:06:54 +00:00

Author

Owner

Rebuilt the exact reported test-binary hash (calternal_plugin_notes-776d209c149f66f4) at base 094d22e445. No-test --list exits 0. A direct full run with --test-threads=4 exits 0: test result: ok. 278 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 79.02s. A second full run under GDB also exits normally (80.82s), so no crash stack yet. Next: serial run with an exit breakpoint to inspect remaining workers, then repeated direct runs. The historical binary was already cleaned; this is the rebuilt base, not a claimed reproduction.

Rebuilt the exact reported test-binary hash (calternal_plugin_notes-776d209c149f66f4) at base 094d22e445. No-test --list exits 0. A direct full run with --test-threads=4 exits 0: `test result: ok. 278 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 79.02s`. A second full run under GDB also exits normally (80.82s), so no crash stack yet. Next: serial run with an exit breakpoint to inspect remaining workers, then repeated direct runs. The historical binary was already cleaned; this is the rebuilt base, not a claimed reproduction.

kayg referenced this issue from a commit

2026-10-04 20:16:59 +00:00

test(notes): require process exit after a passing lib suite for #1069

kayg commented

2026-10-04 20:18:01 +00:00

Author

Owner

Committed the process-exit regression runner as 1b1773386. tests/adversarial/notes_process_exit.py runs the actual Notes lib executable five times and requires both a nonempty passing summary and process exit 0; it stops on a signal, timeout or nonzero exit and keeps the log. Its five Python child-process tests pass, including the exact ordering “278 passed” followed by SIGSEGV.

Serial full-suite GDB run passed all 278 assertions and exited normally in 215.68s. At libc exit, remaining threads were one idle voice-backfill receiver and six idle SQLx receivers. These lifetimes predate 7c; I have not attributed the historical crash to them. Cargo loads the copied Sherpa/ONNX libraries from target/debug whereas direct execution uses the pinned prebuilt directory; the Cargo-launched suite is now attached to GDB. No production fix or root-cause claim yet.

Committed the process-exit regression runner as 1b1773386. `tests/adversarial/notes_process_exit.py` runs the actual Notes lib executable five times and requires both a nonempty passing summary and process exit 0; it stops on a signal, timeout or nonzero exit and keeps the log. Its five Python child-process tests pass, including the exact ordering “278 passed” followed by SIGSEGV. Serial full-suite GDB run passed all 278 assertions and exited normally in 215.68s. At libc exit, remaining threads were one idle voice-backfill receiver and six idle SQLx receivers. These lifetimes predate 7c; I have not attributed the historical crash to them. Cargo loads the copied Sherpa/ONNX libraries from target/debug whereas direct execution uses the pinned prebuilt directory; the Cargo-launched suite is now attached to GDB. No production fix or root-cause claim yet.

kayg commented

2026-10-04 20:21:33 +00:00

Author

Owner

Starting test-lifecycle audit on job/crash-1069 at base 1b1773386b.

Starting test-lifecycle audit on job/crash-1069 at base 1b1773386b3bcb21892e18bb4731b499876dd864.

kayg commented

2026-10-04 20:24:44 +00:00

Author

Owner

Finding: crates/calternal-server/src/main.rs sends the startup worker stop signal but does not join its std::thread; LiveApp does not retain the production Db for Db::close(). The startup runtime is owned by that thread, so its Notes/reconcile and job workers can still be shutting down when main returns. In the Notes Plugin, the shared setup() creates a Db per test but nearly all tests omit close(). Fixing both lifecycle gaps next.

kayg commented

2026-10-04 20:47:02 +00:00

Author

Owner

Notes test hygiene: added a shared close_test_db helper and applied it to every Notes test fixture that opens a Db, including databases owned through route/provider state. IMAP session tasks, the reconcile observer, job workers, and the contention writer are stopped and joined before cleanup. Evidence: built both test binaries; the focused files_open_foreign_and_missing_paths_have_identical_responses test passed (1 passed); Diff in /home/kayg/Developer/calternal-wt/crash-1069/crates/calternal-server/src/wire.rs:1216:
impl LiveApp {
/// Stop and join startup workers before closing their shared Index (#1069).
pub async fn shutdown_background(&mut self) -> Result<(), String> {

   stop_background_worker(self.background_shutdown.take(), self.background_thread.take()).await

```
   stop_background_worker(
```

       self.background_shutdown.take(),

```
       self.background_thread.take(),
```
```
   )
```
```
   .await
```
}
}
and passed.

Notes test hygiene: added a shared close_test_db helper and applied it to every Notes test fixture that opens a Db, including databases owned through route/provider state. IMAP session tasks, the reconcile observer, job workers, and the contention writer are stopped and joined before cleanup. Evidence: built both test binaries; the focused files_open_foreign_and_missing_paths_have_identical_responses test passed (1 passed); Diff in /home/kayg/Developer/calternal-wt/crash-1069/crates/calternal-server/src/wire.rs:1216: impl LiveApp { /// Stop and join startup workers before closing their shared Index (#1069). pub async fn shutdown_background(&mut self) -> Result<(), String> { - stop_background_worker(self.background_shutdown.take(), self.background_thread.take()).await + stop_background_worker( + self.background_shutdown.take(), + self.background_thread.take(), + ) + .await } } and passed.

kayg commented

2026-10-04 21:19:46 +00:00

Author

Owner

Server shutdown audit for #1069: server_main keeps the startup shutdown sender and OS thread handle while serve consumes the Router, then signals and joins the startup runtime, shuts down the Search Indexer, and awaits Db::close() before returning. Db::close() closes the reader, ordinary writer and authority pools, then joins the WAL checkpointer. The focused join test passed. The full server unit suite passed (240 passed, 10 ignored). The enclosing cargo test -p calternal-server command failed only in tests/perf_guards.rs: perf-lint reports an unused exception ('cache.raw-read', 'apps/web/src/lib/calendar/data.ts', 'createEvent:0bcbe54637cddeb4'). This is outside the files changed for #1069; git fetch origin && git merge origin/dev reported Already up to date.

Server shutdown audit for #1069: `server_main` keeps the startup shutdown sender and OS thread handle while `serve` consumes the Router, then signals and joins the startup runtime, shuts down the Search Indexer, and awaits `Db::close()` before returning. `Db::close()` closes the reader, ordinary writer and authority pools, then joins the WAL checkpointer. The focused join test passed. The full server unit suite passed (240 passed, 10 ignored). The enclosing `cargo test -p calternal-server` command failed only in `tests/perf_guards.rs`: `perf-lint` reports an unused exception `('cache.raw-read', 'apps/web/src/lib/calendar/data.ts', 'createEvent:0bcbe54637cddeb4')`. This is outside the files changed for #1069; `git fetch origin && git merge origin/dev` reported `Already up to date.`

kayg commented

2026-10-04 21:35:52 +00:00

Author

Owner

Notes lifecycle finding for #1069: the first full-suite batch exposed a teardown hang in imap::tests::empty_mailbox_reads_do_not_need_user_or_database_writer. The test kept its explicit PoolConnection alive when close_test_db closed the pool; SQLx waits for checked-out connections to return. The test now drops the writer lease after ROLLBACK and before Db::close(). The focused test passes (1 passed, 0 failed, 0.36 s), and Notes formatting plus Clippy pass. I stopped the pre-fix batch after this hang and am starting ten complete post-fix runs; the interrupted attempt is not counted.

Notes lifecycle finding for #1069: the first full-suite batch exposed a teardown hang in `imap::tests::empty_mailbox_reads_do_not_need_user_or_database_writer`. The test kept its explicit `PoolConnection` alive when `close_test_db` closed the pool; SQLx waits for checked-out connections to return. The test now drops the writer lease after ROLLBACK and before `Db::close()`. The focused test passes (1 passed, 0 failed, 0.36 s), and Notes formatting plus Clippy pass. I stopped the pre-fix batch after this hang and am starting ten complete post-fix runs; the interrupted attempt is not counted.

kayg referenced this issue

2026-10-04 21:42:09 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 21:58:36 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-04 22:15:40 +00:00

Author

Owner

Finished #1069 on job/crash-1069.

Built:

Added shared Notes test cleanup helpers and closed test-owned Db pools after fixtures, providers, workers, reconcile loops, IMAP sessions and thread work complete.
Fixed an IMAP test cleanup deadlock: the test now returns its checked-out SQLx writer connection before Db::close() waits for pool leases.
Made LiveApp retain the startup runtime thread handle. Normal server shutdown signals and joins that thread after HTTP drain, shuts down Search, then awaits Db::close() for reader, writer, authority and checkpoint resources.
Added a focused test that proves shutdown returns only after the OS thread exits.

Files: crates/plugins/notes/src/{bookmarks.rs,imap.rs,lib.rs,reminders_tests.rs,tasks_dav.rs,voice.rs}, crates/plugins/notes/tests/apple_replay.rs, crates/calternal-server/src/{main.rs,wire.rs}.

Commits:

eea3381d6 Close Notes test Index pools before exit
04e137c78 Join startup workers before closing server Index
78da93306 Release IMAP writer lease before test cleanup

Head: 78da933067d141193b4a4d62f5141c1b6f96c2f8

Gate output:

cargo fmt --all --check: exit 0, no output.
cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings: Finished dev profile [unoptimized + debuginfo] target(s) in 23.23s
cargo clippy -p calternal-server --all-targets -- -D warnings: Finished dev profile [unoptimized + debuginfo] target(s) in 37.09s
Focused cleanup test: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 279 filtered out; finished in 0.36s
Ten sequential cargo test -p calternal-plugin-notes -- --test-threads=4 runs, all exit 0:

run 01 exit 0
run 02 exit 0
run 03 exit 0
run 04 exit 0
run 05 exit 0
run 06 exit 0
run 07 exit 0
run 08 exit 0
run 09 exit 0
run 10 exit 0

Each run reported test result: ok. 278 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out.

cargo test -p calternal-server: unit tests passed, then the integration perf guard failed. Verbatim relevant output:

test result: ok. 240 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out; finished in 248.65s

test deterministic_performance_guards ... FAILED

performance guard failed (exit status: 2):

perf-lint: parse Rust product sources
perf-lint: parse browser product sources
perf-lint: validate coverage and architecture
perf-lint: INVALID: ('cache.raw-read', 'apps/web/src/lib/calendar/data.ts', 'createEvent:0bcbe54637cddeb4')

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.43s
error: test failed, to rerun pass `-p calternal-server --test perf_guards`

bun run build in apps/web completed before the server gates. The web build printed existing unused CSS and Excalidraw Rollup warnings. git fetch origin && git merge origin/dev reported Already up to date. cargo clean completed and web build output was deleted. The worktree is clean.

Decisions not set by DESIGN: LiveApp owns the startup JoinHandle; shutdown signals it and performs the blocking join through Tokio before closing shared Index pools. Tests call the shared async close helper after all checked-out connections and worker tasks are released. The perf-lint exception is unrelated to these files and remains a follow-up recorded here.

Known gap: the full server test command is not green because the existing cache.raw-read exception is unused. No files in that calendar path or perf-lint configuration were changed.

READY FOR MERGE: no (the server crate integration gate above fails).

Finished #1069 on `job/crash-1069`. Built: - Added shared Notes test cleanup helpers and closed test-owned Db pools after fixtures, providers, workers, reconcile loops, IMAP sessions and thread work complete. - Fixed an IMAP test cleanup deadlock: the test now returns its checked-out SQLx writer connection before `Db::close()` waits for pool leases. - Made `LiveApp` retain the startup runtime thread handle. Normal server shutdown signals and joins that thread after HTTP drain, shuts down Search, then awaits `Db::close()` for reader, writer, authority and checkpoint resources. - Added a focused test that proves shutdown returns only after the OS thread exits. Files: `crates/plugins/notes/src/{bookmarks.rs,imap.rs,lib.rs,reminders_tests.rs,tasks_dav.rs,voice.rs}`, `crates/plugins/notes/tests/apple_replay.rs`, `crates/calternal-server/src/{main.rs,wire.rs}`. Commits: - `eea3381d6` Close Notes test Index pools before exit - `04e137c78` Join startup workers before closing server Index - `78da93306` Release IMAP writer lease before test cleanup Head: `78da933067d141193b4a4d62f5141c1b6f96c2f8` Gate output: - `cargo fmt --all --check`: exit 0, no output. - `cargo clippy -p calternal-plugin-notes --all-targets -- -D warnings`: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 23.23s` - `cargo clippy -p calternal-server --all-targets -- -D warnings`: `Finished `dev` profile [unoptimized + debuginfo] target(s) in 37.09s` - Focused cleanup test: `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 279 filtered out; finished in 0.36s` - Ten sequential `cargo test -p calternal-plugin-notes -- --test-threads=4` runs, all exit 0: ```text run 01 exit 0 run 02 exit 0 run 03 exit 0 run 04 exit 0 run 05 exit 0 run 06 exit 0 run 07 exit 0 run 08 exit 0 run 09 exit 0 run 10 exit 0 ``` Each run reported `test result: ok. 278 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out`. - `cargo test -p calternal-server`: unit tests passed, then the integration perf guard failed. Verbatim relevant output: ```text test result: ok. 240 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out; finished in 248.65s test deterministic_performance_guards ... FAILED performance guard failed (exit status: 2): perf-lint: parse Rust product sources perf-lint: parse browser product sources perf-lint: validate coverage and architecture perf-lint: INVALID: ('cache.raw-read', 'apps/web/src/lib/calendar/data.ts', 'createEvent:0bcbe54637cddeb4') test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.43s error: test failed, to rerun pass `-p calternal-server --test perf_guards` ``` `bun run build` in `apps/web` completed before the server gates. The web build printed existing unused CSS and Excalidraw Rollup warnings. `git fetch origin && git merge origin/dev` reported `Already up to date.` `cargo clean` completed and web build output was deleted. The worktree is clean. Decisions not set by DESIGN: `LiveApp` owns the startup `JoinHandle`; shutdown signals it and performs the blocking join through Tokio before closing shared Index pools. Tests call the shared async close helper after all checked-out connections and worker tasks are released. The perf-lint exception is unrelated to these files and remains a follow-up recorded here. Known gap: the full server test command is not green because the existing `cache.raw-read` exception is unused. No files in that calendar path or perf-lint configuration were changed. READY FOR MERGE: no (the server crate integration gate above fails).

kayg referenced this issue

2026-10-04 22:32:51 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 23:46:08 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 23:48:50 +00:00

Plan: merge order and conflicts for merge round 7b #867

Rows
Columns

Notes test process crashes with SIGSEGV after all assertions pass in merge round 7c #1069