Flaky: calternal-collab crash simulation exits 101 instead of 137 under load

kayg commented

2026-09-29 10:38:01 +00:00

Owner

Flaky test

In calternal-collab, the crash simulation crash_before_debounce_reloads_only_complete_markdown fails intermittently in full cargo test runs on a loaded host (load 20+). The child exits with 101 (a panic) instead of the intended 137 (SIGKILL). It passes when run alone. It was seen in job editor-385 (#385) and job ab-384-luna (#384) on 2026-09-29.

Work

Find why the child panics before it is killed under load. Likely a timing assumption: the kill arrives after a timeout or deadline assertion fires inside the child. Make the test deterministic: synchronise on a readiness signal from the child (a pipe or file) instead of sleeps, and never race a wall-clock deadline. Run it 200 times under stress-ng --cpu 8 and show 0 failures. Gates: fmt, and clippy and test for calternal-collab.

## Flaky test In `calternal-collab`, the crash simulation `crash_before_debounce_reloads_only_complete_markdown` fails intermittently in full `cargo test` runs on a loaded host (load 20+). The child exits with 101 (a panic) instead of the intended 137 (SIGKILL). It passes when run alone. It was seen in job editor-385 (#385) and job ab-384-luna (#384) on 2026-09-29. ## Work Find why the child panics before it is killed under load. Likely a timing assumption: the kill arrives after a timeout or deadline assertion fires inside the child. Make the test deterministic: synchronise on a readiness signal from the child (a pipe or file) instead of sleeps, and never race a wall-clock deadline. Run it 200 times under `stress-ng --cpu 8` and show 0 failures. Gates: fmt, and clippy and test for `calternal-collab`.

kayg commented

2026-09-29 10:38:53 +00:00

Author

Owner

Starting flaky-426 on job/flaky-426; base dev SHA 191b179baa.

Starting flaky-426 on job/flaky-426; base dev SHA 191b179baac3ef4f5bebfe07ce91c4b7a887ace2.

kayg commented

2026-09-29 10:40:52 +00:00

Author

Owner

Finding: crates/calternal-collab/tests/two_clients.rs applies the durable update, sleeps 900 ms, then asserts that Markdown was flushed. The debounce task targets 750 ms, so the assertion can run before the flush completes when both timers are delayed under load; that panic exits the test child with 101 before its process::exit(137). The current child also exits with code 137 rather than receiving SIGKILL. I’m replacing the sleep with finish()-confirmed persistence and a pipe readiness handshake, then the parent will kill the blocked child.

kayg commented

2026-09-29 12:33:01 +00:00

Author

Owner

Stress finding: the readiness handshake did not fail, but iteration 6 of the 200-run round reached the parent recovery check and returned SQLite error code 5 () at crates/calternal-collab/tests/two_clients.rs:145. I’m tracing the parent/child pool lifetime and transaction cleanup before changing the harness.

kayg commented

2026-09-29 12:38:26 +00:00

Author

Owner

Correction to the stress finding: iteration 6 reached the parent recovery check, where Db::connect returned SQLite error code 5 (database is locked) at crates/calternal-collab/tests/two_clients.rs:145. The readiness handshake and SIGKILL completed. I’m tracing the parent and child database pool lifetime before changing the harness.

Correction to the stress finding: iteration 6 reached the parent recovery check, where `Db::connect` returned SQLite error code 5 (`database is locked`) at `crates/calternal-collab/tests/two_clients.rs:145`. The readiness handshake and SIGKILL completed. I’m tracing the parent and child database pool lifetime before changing the harness.

kayg commented

2026-09-29 12:39:02 +00:00

Author

Owner

The lock occurred after the child had exited and the parent had waited for its status, so it is separate from the readiness race. The parent creates the setup Db on a current-thread runtime, drops it implicitly, then leaves that runtime undriven while it blocks on the child pipe. Db::close() is available to await closure of both SQLx pools; I’m testing whether explicitly closing the setup pool before spawning the child removes the intermittent lock.

The lock occurred after the child had exited and the parent had waited for its status, so it is separate from the readiness race. The parent creates the setup `Db` on a current-thread runtime, drops it implicitly, then leaves that runtime undriven while it blocks on the child pipe. `Db::close()` is available to await closure of both SQLx pools; I’m testing whether explicitly closing the setup pool before spawning the child removes the intermittent lock.

kayg commented

2026-09-29 12:46:09 +00:00

Author

Owner

The parent setup now awaits db.close() before it pauses the current-thread runtime. This prevents pool shutdown from depending on that runtime being polled while the test waits on the child pipe. The targeted test passed, then the requested stress-ng --cpu 8 run completed 200 iterations with zero failures.

The parent setup now awaits `db.close()` before it pauses the current-thread runtime. This prevents pool shutdown from depending on that runtime being polled while the test waits on the child pipe. The targeted test passed, then the requested `stress-ng --cpu 8` run completed 200 iterations with zero failures.

kayg commented

2026-09-29 14:11:31 +00:00

Author

Owner

Finished flaky-426 (#426).

Built: deterministic crash recovery simulation. The child flushes the durable Note with AgentTurn::finish(), applies a pending edit, writes a readiness line, and blocks its current-thread runtime on a pipe. The parent waits for readiness, sends SIGKILL, verifies signal 9, then checks recovery. The parent closes its setup DB pools before it blocks on the pipe.

Files: crates/calternal-collab/tests/two_clients.rs

Head SHA: 3c5082f039d38cd8143dcca8340414b218d5c622

Stress result: Completed 200 iterations under stress-ng --cpu 8: 0 failures. This run completed before the required dev merge. After that merge, the targeted crash test and the full calternal-collab suite passed on the final tree.

Gate output:

cargo fmt --check
No stdout or stderr; exit status 0.

cargo clippy -p calternal-collab --all-targets -- -D warnings

Finished `dev` profile [unoptimized + debuginfo] target(s) in 70m 52s

cargo test -p calternal-collab

test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.79s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.34s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.51s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 52.91s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.26s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.27s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.83s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 13.54s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.29s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.83s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 28.96s

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Known gaps: no code gaps identified. No web build output was created in this worktree.

Decisions: DESIGN §9 requires crash recovery to keep complete Markdown but does not define a test handshake. The test uses stdout readiness and a blocking stdin pipe. Blocking the current-thread runtime prevents the debounce task from running if the parent is delayed. Db::close() explicitly releases the setup pools before that block because the test runtime is otherwise idle. No production behavior or API changed.

Finished flaky-426 (#426). Built: deterministic crash recovery simulation. The child flushes the durable Note with `AgentTurn::finish()`, applies a pending edit, writes a readiness line, and blocks its current-thread runtime on a pipe. The parent waits for readiness, sends SIGKILL, verifies signal 9, then checks recovery. The parent closes its setup DB pools before it blocks on the pipe. Files: `crates/calternal-collab/tests/two_clients.rs` Head SHA: `3c5082f039d38cd8143dcca8340414b218d5c622` Stress result: `Completed 200 iterations under stress-ng --cpu 8: 0 failures.` This run completed before the required `dev` merge. After that merge, the targeted crash test and the full `calternal-collab` suite passed on the final tree. Gate output: `cargo fmt --check` No stdout or stderr; exit status 0. `cargo clippy -p calternal-collab --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 70m 52s ``` `cargo test -p calternal-collab` ```text test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.79s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.34s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.51s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 52.91s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.26s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.27s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.83s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 13.54s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.29s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.83s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 28.96s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` Known gaps: no code gaps identified. No web build output was created in this worktree. Decisions: DESIGN §9 requires crash recovery to keep complete Markdown but does not define a test handshake. The test uses stdout readiness and a blocking stdin pipe. Blocking the current-thread runtime prevents the debounce task from running if the parent is delayed. `Db::close()` explicitly releases the setup pools before that block because the test runtime is otherwise idle. No production behavior or API changed.

kayg commented

2026-09-29 14:58:11 +00:00

Author

Owner

Another timing flake under host load (≈30): calternal-search tests/indexer.rs reindex_keeps_the_previous_index_searchable_until_the_new_scan_finishes failed once in a full crate run and passed 3/3 alone. Same class as the collab crash test: replace wall-clock waits with explicit synchronisation. Queue for the flaky-426 follow-up.

Another timing flake under host load (≈30): calternal-search tests/indexer.rs `reindex_keeps_the_previous_index_searchable_until_the_new_scan_finishes` failed once in a full crate run and passed 3/3 alone. Same class as the collab crash test: replace wall-clock waits with explicit synchronisation. Queue for the flaky-426 follow-up.

kayg referenced this issue from a commit

2026-09-29 15:34:48 +00:00

Merge job/flaky-426 (#426: deterministic collab crash test via readiness handshake)

kayg commented

2026-09-29 15:47:37 +00:00

Author

Owner

Merged into dev and pushed (9bf3d549) after the web gates (816/816) and the relevant crate gates. Deploy follows the #356/#393 Apple GO/NO-GO.

Merged into dev and pushed (9bf3d549) after the web gates (816/816) and the relevant crate gates. Deploy follows the #356/#393 Apple GO/NO-GO.

kayg closed this issue

2026-09-29 15:47:37 +00:00

Rows
Columns

Flaky: calternal-collab crash simulation exits 101 instead of 137 under load #426

Flaky test

Work