kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Fix functional test flakes and duplicate move usage accounting #825

Open
kayg wants to merge 1 commit from fix/parallel-test-flakes into dev
pull from: fix/parallel-test-flakes
merge into: kayg:dev
Conversation 2 Commits 1 Files changed 1 +26
kayg commented 2026-11-01 08:18:10 +00:00
Owner
Copy link

Functional tests now prove progress and ordering during parked work, with documented 120-second deadlock watchdogs for positive completion. They no longer impose notes/auth speed budgets on fsync. Fresh verification at d0f89de360a964f5f718c2a3b9faa1025c05decb passed all eight default-parallel workspace commands: dev-01 5/5, dev-02 3/3, zero failures.

Original commit a55e9af9 is a product fix, rebased as 04c6f09e. A recount can cache both names across a rename. The move notification must subtract the destination's cached contribution before replacing it, or logical Home usage is double-counted. The fix adds O(1) work per cached moved path; existing directory traversal/recount remains O(N), with no persistent format change.

Test, cause, fix, run results

Test / group Cause Fix Current-head results
usage move/recount regression and concurrent mutation/restart cached destination bytes counted twice after rename/recount overlap subtract destination contribution before replacement, exact-byte regression dev-01: 5/5; dev-02: 3/3
multiwriter and single-writer crash checkpoints fixed sleeps assumed enough confirmed writes under parallel load wait for acknowledged appends; single writer signals next attempt dev-01: 5/5; dev-02: 3/3
sign-in during recount relative sign-in/walk speed determined outcome park recount at explicit checkpoint; complete sign-in before release dev-01: 5/5; dev-02: 3/3
refused-anchor snapshot automatic save inserted metadata between full reads manual-save fixture preserves full snapshot comparison dev-01: 5/5; dev-02: 3/3
notes shutdown/full-history/settlement elapsed-under-five-seconds assertions measured storage speed retain joins, durable history/file/replay/lock assertions; 120 s deadlock watchdog dev-01: 5/5; dev-02: 3/3
notes fan-out, Saved receipt, retained-byte pressure, event visibility silence intervals treated as evidence of blocking sync round trips, explicit crash checkpoint, one-poll checks and lifecycle mutex state dev-01: 5/5; dev-02: 3/3
notes crash during rename/delete timeout expiry used as crash trigger await existing applied-batch checkpoint, then stop the runtime/Home dev-01: 5/5; dev-02: 3/3
invite visibility silence interval used to assert excluded recipient observe later admin publication in explicit recipient id order, then inspect unrelated queue dev-01: 5/5; dev-02: 3/3
auth admission/app-password queue/general blocking pool short silence/latency windows and timed worker release poll queued work while dependency parked; release workers only after auth completes dev-01: 5/5; dev-02: 3/3
auth idle uploads / directory recount fixed 31 s reader check and two-second /proc observation raced I/O await idle cleanup under deadlock watchdog; existing after-walk hook dev-01: 5/5; dev-02: 3/3
admitted cold app-password lookup 200 concurrent reads assumed to drain within product COLD_WAIT (250 ms) move assertion beside parked DB worker, reserve valid cold lookup first, exhaust source budget before completing it; existing queue/admission tests retained dev-01: 5/5; dev-02: 3/3
markdown-view containment recovery recovery success assertion implicitly used the production 50 ms parse budget explicit test limits with 120 s watchdog; production budget unchanged and pure deadline boundary test retained dev-01: 5/5; dev-02: 3/3
search stalled rebuild / writer and new readers three-second durable action and one-second snapshot guards measured completion speed explicitly parked provider remains parked until writer and new readers finish; shared deadlock watchdogs only dev-01: 5/5; dev-02: 3/3
fsio temporary-file fixtures wall-clock names plus create_dir_all permit parallel fixture sharing and cleanup of open directories shared process-unique atomic counter and exclusive create_dir dev-01: 5/5; dev-02: 3/3
live note protocol (join/sync/edit/missing note) five-second response guards expired during storage stalls shared 120 s deadlock watchdog; protocol ordering, content and not-found assertions retained dev-01: 5/5; dev-02: 3/3
server duplicate-upload admission 10 s guard expired during fsync stalls shared 120 s watchdog only detects deadlock; quota assertions retained dev-01: 5/5; dev-02: 3/3
server randomized Home shutdown 5 s guard expired during fsync stalls shared 120 s watchdog only detects deadlock; storage release assertions retained dev-01: 5/5; dev-02: 3/3

No test retries were added. Functional watchdogs were deliberately increased to 120 s per maintainer direction, with a shared comment explaining that they catch deadlocks, not speed regressions. Ignored perf tests retain their performance budgets on perf VMs; frame budgets belong on the Mac VM.

Workspace wall-clock audit

Full workspace inventory and follow-up table lists paths, lines and functions found by Instant::now, elapsed(), literal timeout/recv_timeout and millisecond-assert searches. Notes/auth completion guards were converted; their remaining clocks are documented liveness, expiry/backoff/cache-age policy, watcher polling, or ignored perf measurements.

Follow-up candidates outside this scope include tasks, backdrops uploads, contacts actions, daily, prefs, server live/signout, search, fsio usage and the remaining calternal live/startup/storage/admin/shutdown-I/O tests. Policy grace bounds need virtual-clock review; functional speed assertions should become ordering checks or deadlock watchdogs. The inventory also lists frontend performance.now candidates, engine/search measurement helpers and markdown scaling. This PR does not claim those follow-ups are fixed.

Exact verification

Each VM uses its dedicated worktree, locked Bun dependencies, formatting check, and disk check before heavy work:

export PATH=/nix/var/nix/profiles/default/bin:$PATH
export CARGO_TARGET_DIR="$PWD/target-liveness"
nix develop -c bash -c '~/bin/slot cargo test --workspace --locked'

No -j limit or test-thread override. Repository default RUST_TEST_THREADS=4 applies. Logs: ~/parallel-flakes/liveness-d0f89de3/ on each VM.

VM Run 1 Run 2 Run 3 Run 4 Run 5
dev-01 x86_64 0 0 0 0 0
dev-02 aarch64 0 0 0 n/a n/a

Storage evidence for maintainer

On idle dev-02, three independent 4 KiB writes plus fsync took 2.012433 s, 2.034076 s and 16.228720 s. Full I/O pressure averaged 84.98% over 10 seconds, with ext4 journal/flush workers blocked and 73 GB free. A single durable write exceeded both old five-second shutdown and ten-second admission guards. Functional correctness must tolerate these stalls; speed belongs in the perf harness. During investigation, dev-02's generated prefs and task workers were also observed waiting in jbd2_log_wait_commit; increasing process write counters confirmed progress. No durability requirement or concurrency setting was weakened.

Commit review and rebase

Rebased onto 55f35fcde3bc91ca3781055577b93e93819f6cd4, still the current remote dev at final verification. There were five branch-only commits at the supplied head, all reviewed:

Original → rebased Assessment
c3b90bef → f65c492c Deterministic exact-byte regression for both names appearing in the usage snapshot
a55e9af9 → 04c6f09e Product fix at the accounting root. One rebase conflict adapted the insertion to dev's Files::insert(&Path, ...) API
1a4f1572 → 26783f65 Multi-writer checkpoints wait for confirmed appends; content, restart, progress and idempotency assertions remain
fd814050 → 46bbe220 Scoped readiness/release channels control recount overlap. Callback runs without the snapshot lock; dropping the sender releases a worker during unwinding. Performance cases retain unparked walks
d1310c40 → fd9c9586 Single writer confirms 3–8 appends, then signals the next attempt before killing the server; exact-content and progress assertions remain

Resume fix e15507ab isolates refused-anchor snapshots from automatic saves. Temporary settlement diagnostics were reverted.

Earlier evidence retained

At 074aac50, all eight scheduled commands finished: dev-01 5/5, dev-02 exits 101, 0, 0. The failed command contained the two server guard failures above. This failed epoch is retained, not counted as successful current-head verification. Logs: ~/parallel-flakes/final-074aac50/.

During replacement-check development, focused fan-out tests exposed two incorrect assumptions: API reads could precede edit application, and a parked actor does not close the client channel. The final checks use a sync acknowledgement and an explicit reached fan-out checkpoint. At 8723264a, dev-01 runs 1–2 passed; run 3 failed the old cold-valid concurrency test with 429 versus 200. The bounded cold admission window had expired. This false speed requirement was replaced with explicit admitted-lookup ordering. At a3b68c6e, dev-01 passed all five commands and dev-02 run 1 passed; dev-02 run 2 failed markdown-view containment recovery at the implicit production 50 ms parser budget. That recovery check now uses explicit test limits. At 3a93293b, dev-01 passed all five commands and dev-02 run 1 passed; dev-02 run 2 failed the search stalled-rebuild test at its three-second action guard. It now checks completion while the provider stays parked with deadlock watchdogs. At ba2367f4, dev-01 passed all five commands and dev-02 run 1 passed; run 2 failed the missing-note live join at its five-second response guard. Live note completion guards now use the shared deadlock watchdog. At 9e6582a6, dev-01 completed three commands before its fourth was stopped; dev-02 run 1 failed fsio temporary-file creation with NotFound. The fixture helper used timestamps without exclusive creation; process-unique fixture names now prevent parallel cleanup from sharing a root. Other in-progress development batches were stopped before completion and are not counted. Earlier diagnostic and previous-head results remain in PR history. Never merge.

Final gate: 8 workspace commands, 8 passes, 0 failures at the head above. Formatting and locked dependency preparation passed on both VMs. All task jobs finished; both VM worktrees and the control worktree are clean. Removed the task target directories after verification; small run logs remain at the path above.

Functional tests now prove progress and ordering during parked work, with documented 120-second deadlock watchdogs for positive completion. They no longer impose notes/auth speed budgets on fsync. Fresh verification at `d0f89de360a964f5f718c2a3b9faa1025c05decb` passed all eight default-parallel workspace commands: **dev-01 5/5, dev-02 3/3, zero failures**. **Original commit `a55e9af9` is a product fix**, rebased as `04c6f09e`. A recount can cache both names across a rename. The move notification must subtract the destination's cached contribution before replacing it, or logical Home usage is double-counted. The fix adds O(1) work per cached moved path; existing directory traversal/recount remains O(N), with no persistent format change. ## Test, cause, fix, run results | Test / group | Cause | Fix | Current-head results | | --- | --- | --- | --- | | usage move/recount regression and concurrent mutation/restart | cached destination bytes counted twice after rename/recount overlap | subtract destination contribution before replacement, exact-byte regression | dev-01: 5/5; dev-02: 3/3 | | multiwriter and single-writer crash checkpoints | fixed sleeps assumed enough confirmed writes under parallel load | wait for acknowledged appends; single writer signals next attempt | dev-01: 5/5; dev-02: 3/3 | | sign-in during recount | relative sign-in/walk speed determined outcome | park recount at explicit checkpoint; complete sign-in before release | dev-01: 5/5; dev-02: 3/3 | | refused-anchor snapshot | automatic save inserted metadata between full reads | manual-save fixture preserves full snapshot comparison | dev-01: 5/5; dev-02: 3/3 | | notes shutdown/full-history/settlement | elapsed-under-five-seconds assertions measured storage speed | retain joins, durable history/file/replay/lock assertions; 120 s deadlock watchdog | dev-01: 5/5; dev-02: 3/3 | | notes fan-out, Saved receipt, retained-byte pressure, event visibility | silence intervals treated as evidence of blocking | sync round trips, explicit crash checkpoint, one-poll checks and lifecycle mutex state | dev-01: 5/5; dev-02: 3/3 | | notes crash during rename/delete | timeout expiry used as crash trigger | await existing applied-batch checkpoint, then stop the runtime/Home | dev-01: 5/5; dev-02: 3/3 | | invite visibility | silence interval used to assert excluded recipient | observe later admin publication in explicit recipient id order, then inspect unrelated queue | dev-01: 5/5; dev-02: 3/3 | | auth admission/app-password queue/general blocking pool | short silence/latency windows and timed worker release | poll queued work while dependency parked; release workers only after auth completes | dev-01: 5/5; dev-02: 3/3 | | auth idle uploads / directory recount | fixed 31 s reader check and two-second `/proc` observation raced I/O | await idle cleanup under deadlock watchdog; existing after-walk hook | dev-01: 5/5; dev-02: 3/3 | | admitted cold app-password lookup | 200 concurrent reads assumed to drain within product COLD_WAIT (250 ms) | move assertion beside parked DB worker, reserve valid cold lookup first, exhaust source budget before completing it; existing queue/admission tests retained | dev-01: 5/5; dev-02: 3/3 | | markdown-view containment recovery | recovery success assertion implicitly used the production 50 ms parse budget | explicit test limits with 120 s watchdog; production budget unchanged and pure deadline boundary test retained | dev-01: 5/5; dev-02: 3/3 | | search stalled rebuild / writer and new readers | three-second durable action and one-second snapshot guards measured completion speed | explicitly parked provider remains parked until writer and new readers finish; shared deadlock watchdogs only | dev-01: 5/5; dev-02: 3/3 | | fsio temporary-file fixtures | wall-clock names plus create_dir_all permit parallel fixture sharing and cleanup of open directories | shared process-unique atomic counter and exclusive create_dir | dev-01: 5/5; dev-02: 3/3 | | live note protocol (join/sync/edit/missing note) | five-second response guards expired during storage stalls | shared 120 s deadlock watchdog; protocol ordering, content and not-found assertions retained | dev-01: 5/5; dev-02: 3/3 | | server duplicate-upload admission | 10 s guard expired during fsync stalls | shared 120 s watchdog only detects deadlock; quota assertions retained | dev-01: 5/5; dev-02: 3/3 | | server randomized Home shutdown | 5 s guard expired during fsync stalls | shared 120 s watchdog only detects deadlock; storage release assertions retained | dev-01: 5/5; dev-02: 3/3 | No test retries were added. Functional watchdogs were deliberately increased to 120 s per maintainer direction, with a shared comment explaining that they catch deadlocks, not speed regressions. Ignored perf tests retain their performance budgets on perf VMs; frame budgets belong on the Mac VM. ## Workspace wall-clock audit [Full workspace inventory and follow-up table](https://git.kayg.org/kayg/calternal/pulls/825#issuecomment-73338) lists paths, lines and functions found by `Instant::now`, `elapsed()`, literal timeout/recv_timeout and millisecond-assert searches. Notes/auth completion guards were converted; their remaining clocks are documented liveness, expiry/backoff/cache-age policy, watcher polling, or ignored perf measurements. Follow-up candidates outside this scope include tasks, backdrops uploads, contacts actions, daily, prefs, server live/signout, search, fsio usage and the remaining calternal live/startup/storage/admin/shutdown-I/O tests. Policy grace bounds need virtual-clock review; functional speed assertions should become ordering checks or deadlock watchdogs. The inventory also lists frontend `performance.now` candidates, engine/search measurement helpers and markdown scaling. This PR does not claim those follow-ups are fixed. ## Exact verification Each VM uses its dedicated worktree, locked Bun dependencies, formatting check, and disk check before heavy work: ```sh export PATH=/nix/var/nix/profiles/default/bin:$PATH export CARGO_TARGET_DIR="$PWD/target-liveness" nix develop -c bash -c '~/bin/slot cargo test --workspace --locked' ``` No `-j` limit or test-thread override. Repository default `RUST_TEST_THREADS=4` applies. Logs: `~/parallel-flakes/liveness-d0f89de3/` on each VM. | VM | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | | --- | --- | --- | --- | --- | --- | | dev-01 x86_64 | 0 | 0 | 0 | 0 | 0 | | dev-02 aarch64 | 0 | 0 | 0 | n/a | n/a | ## Storage evidence for maintainer On idle dev-02, three independent 4 KiB writes plus fsync took **2.012433 s, 2.034076 s and 16.228720 s**. Full I/O pressure averaged **84.98% over 10 seconds**, with ext4 journal/flush workers blocked and 73 GB free. A single durable write exceeded both old five-second shutdown and ten-second admission guards. Functional correctness must tolerate these stalls; speed belongs in the perf harness. During investigation, dev-02's generated prefs and task workers were also observed waiting in `jbd2_log_wait_commit`; increasing process write counters confirmed progress. No durability requirement or concurrency setting was weakened. ## Commit review and rebase Rebased onto `55f35fcde3bc91ca3781055577b93e93819f6cd4`, still the current remote `dev` at final verification. There were five branch-only commits at the supplied head, all reviewed: | Original → rebased | Assessment | | --- | --- | | `c3b90bef` → `f65c492c` | Deterministic exact-byte regression for both names appearing in the usage snapshot | | `a55e9af9` → `04c6f09e` | Product fix at the accounting root. One rebase conflict adapted the insertion to dev's `Files::insert(&Path, ...)` API | | `1a4f1572` → `26783f65` | Multi-writer checkpoints wait for confirmed appends; content, restart, progress and idempotency assertions remain | | `fd814050` → `46bbe220` | Scoped readiness/release channels control recount overlap. Callback runs without the snapshot lock; dropping the sender releases a worker during unwinding. Performance cases retain unparked walks | | `d1310c40` → `fd9c9586` | Single writer confirms 3–8 appends, then signals the next attempt before killing the server; exact-content and progress assertions remain | Resume fix `e15507ab` isolates refused-anchor snapshots from automatic saves. Temporary settlement diagnostics were reverted. ## Earlier evidence retained At `074aac50`, all eight scheduled commands finished: dev-01 5/5, dev-02 exits 101, 0, 0. The failed command contained the two server guard failures above. This failed epoch is retained, not counted as successful current-head verification. Logs: `~/parallel-flakes/final-074aac50/`. During replacement-check development, focused fan-out tests exposed two incorrect assumptions: API reads could precede edit application, and a parked actor does not close the client channel. The final checks use a sync acknowledgement and an explicit reached fan-out checkpoint. At `8723264a`, dev-01 runs 1–2 passed; run 3 failed the old cold-valid concurrency test with 429 versus 200. The bounded cold admission window had expired. This false speed requirement was replaced with explicit admitted-lookup ordering. At `a3b68c6e`, dev-01 passed all five commands and dev-02 run 1 passed; dev-02 run 2 failed markdown-view containment recovery at the implicit production 50 ms parser budget. That recovery check now uses explicit test limits. At `3a93293b`, dev-01 passed all five commands and dev-02 run 1 passed; dev-02 run 2 failed the search stalled-rebuild test at its three-second action guard. It now checks completion while the provider stays parked with deadlock watchdogs. At `ba2367f4`, dev-01 passed all five commands and dev-02 run 1 passed; run 2 failed the missing-note live join at its five-second response guard. Live note completion guards now use the shared deadlock watchdog. At `9e6582a6`, dev-01 completed three commands before its fourth was stopped; dev-02 run 1 failed fsio temporary-file creation with NotFound. The fixture helper used timestamps without exclusive creation; process-unique fixture names now prevent parallel cleanup from sharing a root. Other in-progress development batches were stopped before completion and are not counted. Earlier diagnostic and previous-head results remain in PR history. Never merge. Final gate: **8 workspace commands, 8 passes, 0 failures at the head above.** Formatting and locked dependency preparation passed on both VMs. All task jobs finished; both VM worktrees and the control worktree are clean. Removed the task target directories after verification; small run logs remain at the path above.
test: reproduce usage move after recount snapshot
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
c3b90bef7c
kayg changed title from fix: reconcile parallel test usage accounting and bridge diagnostics to fix: avoid counting moved files twice during usage recount 2026-11-01 08:25:11 +00:00
kayg changed title from fix: avoid counting moved files twice during usage recount to fix: remove usage recount race and crash-test throughput assumption 2026-11-01 08:57:57 +00:00
kayg changed title from fix: remove usage recount race and crash-test throughput assumption to fix: remove parallel-test accounting and timing races 2026-11-01 10:58:37 +00:00
kayg changed title from fix: remove parallel-test accounting and timing races to Fix parallel crash/auth test flakes and duplicate move usage accounting 2026-11-01 16:43:50 +00:00
kayg commented 2026-11-01 18:12:41 +00:00
Author
Owner
Copy link

Workspace wall-clock guard audit at d0f89de3

Searched Rust tests and inline test modules for Instant::now, elapsed(), literal timeout / recv_timeout, and Duration::from_millis inside assertions. Production clocks/logging, expiry inputs, and dedicated perf binaries/examples are excluded. This is a static inventory: policy timers and measurements are identified separately from speed requirements.

Notes and auth disposition

  • Shutdown and full-history tests retain writer-join, saved-history, file-content, replay, and lock-ownership assertions. Removed elapsed-under-five-seconds requirements.
  • Positive functional waits use calternal_testkit::LIVENESS (120 s), documented as a deadlock watchdog, never a speed budget. This includes the two failing server guards and crash-test waits.
  • Parked-history receipt checks use sync round trips or an explicitly reached fan-out crash checkpoint; admission/retained-byte/DB queue checks poll parked work instead of waiting 10–250 ms for silence.
  • Auth completes while general blocking workers remain explicitly parked. Idle uploads are awaited to completion rather than inspected after 31 s; their natural expiry is not a speed bound on authentication.
  • Rename/recount regression uses the existing after-walk checkpoint rather than racing /proc/self/fd over two seconds.
  • Remaining Instant uses in notes clients, storm, writers, tag crash recovery, and native watcher tests are the documented 120-second liveness deadlines.
  • Notes index::reload_speed, tag keystroke tests, save/slow-disk/group-commit measurement helpers and auth sign-in/source-churn/invite timing budgets are ignored perf-harness cases. Their speed budgets remain in the perf harness.
  • Auth limiter backoff/expiry and app-password cache-age tests use clocks to test policy state, not completion speed. Session/setup expiry inputs and intentional idle/debounce timers remain policy fixtures.

Other Rust functional candidates, left for follow-up

Convert short completion waits to explicit ordering or the shared liveness watchdog. Review true expiry/grace-policy bounds with virtual clocks. Move actual throughput/latency requirements to perf harnesses. Entries are grouped by file; line numbers identify candidate sites, including helper functions used by tests.

File Candidate lines Functions
crates/backdrops/src/uploads.rs 269 upload_admission_is_bounded_and_cancellation_restores_capacity
crates/calternal/tests/notes_crash_typing.rs 63, 65, 194, 306, 308, 316, 394, 396, 398 join, kill_nine_while_people_type_loses_and_doubles_nothing, run, start
crates/calternal/tests/prefs_live.rs 41, 43 start
crates/calternal/tests/search_live.rs 44, 47, 55, 95, 108 eventually, http, start
crates/calternal/tests/startup.rs 49, 71, 76, 143 a_new_server_prints_a_setup_token, launch_and_wait_for_health
crates/calternal/tests/attachments_crash.rs 46, 48, 167, 169, 275, 285 a_big_upload_streams_in_bounded_memory, start, wait_for
crates/calternal/tests/contacts_live.rs 54, 56, 162, 164, 165, 207, 215, 222, 233, 281, 302, 315, 320 a_live_h1_collision_keeps_contact_and_incoming_link_identity, a_phone_renames_a_contact_while_someone_types_in_a_linking_note, start
crates/calternal/tests/storage_startup.rs 168, 170, 584, 589 a_lock_held_past_the_longest_grace_still_fails_startup, starts
crates/calternal/tests/fix_tz_live.rs 55, 57, 62, 135, 137, 138, 268, 274 join, start, strip_tz_markers_on_a_live_home_with_a_client_typing
crates/calternal/src/admin.rs 345, 381, 385, 393, 417, 443 a_restart_waits_out_an_admin_request_longer_than_ten_seconds, a_stop_sent_before_subscribing_is_still_seen, an_active_request_gets_a_graceful_shutdown_window, shutdown_stops_a_client_waiting_on_a_partial_request
crates/calternal/src/shutdown_io.rs 175, 177, 193, 197 shutdown_bounds_response_writes_to_a_peer_that_stops_reading, shutdown_finishes_an_idle_read_and_keeps_response_writes_open
crates/server/tests/signout.rs 61 next
crates/server/tests/live.rs 75 next
crates/server/src/rooms.rs 556 assembly_is_bounded_per_connection_and_globally_and_expiry_releases_it
crates/tasks/src/status/executor_tests.rs 64 kill_at
crates/tasks/src/tests.rs 93, 191, 192 eventually, run
crates/daily/src/tests.rs 237, 239 eventually
crates/prefs/src/store/tests.rs 376 interrupted_conflict_exchange_restores_outside_file_on_restart
crates/prefs/src/tests.rs 330 a_change_on_one_device_reaches_the_other
crates/tasks/src/create_tests.rs 224 opening_today_then_creating_tasks_scopes_keys_and_retries
crates/search/tests/search.rs 332, 336, 339, 345, 352, 407, 409 behind_answers_at_once_and_catching_up_is_announced_once, searches_answer_while_indexing_runs
crates/search/tests/common/mod.rs 236, 241 settle
crates/contacts-actions/src/tests/break_links.rs 234, 238 five_thousand_links_across_a_thousand_notes
crates/contacts-actions/src/tests/correctness.rs 91, 101, 241 bounded_shutdown_joins_the_card_write_and_refuses_later_copies, kill_delete_at
crates/contacts-actions/src/tests/review.rs 498, 638, 650, 820, 980 kill9_import_during_file_placement, kill_delete_restore, kill_import_phase_in_folder, kill_restore_links
crates/contacts-actions/src/tests/kill9.rs 153 killed_at
crates/contacts-actions/src/tests/env.rs 337, 339 eventually
crates/fsio/src/usage.rs 421, 428, 531 reads_do_not_wait_for_the_counter_writer, stale_metadata_cannot_replace_a_newer_update

A subsequent workspace run also exposed an implicit 50 ms production parser budget in markdown-view's containment-recovery success assertion. That check now passes explicit 120-second test limits to extraction inside containment; the production limit and deterministic deadline boundary test are unchanged. Other functional callers of production parser defaults need follow-up review.

Engine's interleave_with_huge_job, search's ignored ordinary_save_p95_during_executor_and_rebuild timing, and markdown-view extraction scaling are measurement helpers or ignored perf cases, not new functional speed gates.

Frontend follow-up candidates

These test files also call performance.now(). Review speed/frame assertions separately: frames must run on the Mac VM. Date-based content/fixture timestamps are not speed requirements.

File Clock call lines
apps/mockup/tests/break.spec.ts 63, 68
apps/web/e2e/dialog-stable.spec.ts 37, 42
apps/web/e2e/list-fit.spec.ts 96
apps/web/e2e/page-chrome.spec.ts 15, 17, 32
apps/web/e2e/search-corpus.spec.ts 22, 24
apps/web/e2e/swipe-rows.spec.ts 95, 141
apps/web/e2e/switching.spec.ts 48
apps/web/src/composer/recognise.test.ts 186, 188
packages/editor/test/branch.test.ts 47, 49
packages/editor/test/link-rank.test.ts 160, 162
packages/editor/test/refusal-performance.test.ts 36, 55, 117, 119, 123, 124, 149, 150, 168, 171, 229, 230
packages/editor/test/tag-rank.test.ts 74, 76
packages/editor/test/tree-cost.test.ts 21, 23

Maintainer evidence: dev-02 independently took 2.012433 s, 2.034076 s, and 16.228720 s for 4 KiB writes followed by fsync, with 73 GB free; full I/O stall pressure reached 84.98% over 10 seconds. Functional correctness must tolerate that storage latency. Fresh verification at d0f89de360a964f5f718c2a3b9faa1025c05decb passed: dev-01 5/5, dev-02 3/3 default-parallel workspace commands, zero failures.

The functional search case stalled_rebuild_uses_old_sources_while_writer_and_new_readers_finish subsequently failed its three-second action guard at 3a93293b. It now waits under the shared liveness watchdog and explicitly asserts the rebuild remains parked until writer and new-reader completion. Its one-second snapshot guard was converted too. Other search guard candidates in the inventory remain follow-ups.

Live note follow-up resolved in 9e6582a6: crates/calternal/tests/notes_live.rs startup, sync, edit visibility and missing-note response guards now use shared 120 s LIVENESS; protocol ordering/content/error assertions remain. At ba2367f4, dev-02 run 2 failed the five-second missing-note response guard.

## Workspace wall-clock guard audit at `d0f89de3` Searched Rust tests and inline test modules for `Instant::now`, `elapsed()`, literal `timeout` / `recv_timeout`, and `Duration::from_millis` inside assertions. Production clocks/logging, expiry inputs, and dedicated perf binaries/examples are excluded. This is a static inventory: policy timers and measurements are identified separately from speed requirements. ### Notes and auth disposition - Shutdown and full-history tests retain writer-join, saved-history, file-content, replay, and lock-ownership assertions. Removed elapsed-under-five-seconds requirements. - Positive functional waits use `calternal_testkit::LIVENESS` (120 s), documented as a deadlock watchdog, never a speed budget. This includes the two failing server guards and crash-test waits. - Parked-history receipt checks use sync round trips or an explicitly reached fan-out crash checkpoint; admission/retained-byte/DB queue checks poll parked work instead of waiting 10–250 ms for silence. - Auth completes while general blocking workers remain explicitly parked. Idle uploads are awaited to completion rather than inspected after 31 s; their natural expiry is not a speed bound on authentication. - Rename/recount regression uses the existing after-walk checkpoint rather than racing `/proc/self/fd` over two seconds. - Remaining `Instant` uses in notes `clients`, `storm`, `writers`, tag crash recovery, and native watcher tests are the documented 120-second liveness deadlines. - Notes `index::reload_speed`, tag keystroke tests, save/slow-disk/group-commit measurement helpers and auth sign-in/source-churn/invite timing budgets are ignored perf-harness cases. Their speed budgets remain in the perf harness. - Auth limiter backoff/expiry and app-password cache-age tests use clocks to test policy state, not completion speed. Session/setup expiry inputs and intentional idle/debounce timers remain policy fixtures. ### Other Rust functional candidates, left for follow-up Convert short completion waits to explicit ordering or the shared liveness watchdog. Review true expiry/grace-policy bounds with virtual clocks. Move actual throughput/latency requirements to perf harnesses. Entries are grouped by file; line numbers identify candidate sites, including helper functions used by tests. | File | Candidate lines | Functions | | --- | --- | --- | | [crates/backdrops/src/uploads.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/backdrops/src/uploads.rs#L269) | 269 | `upload_admission_is_bounded_and_cancellation_restores_capacity` | | [crates/calternal/tests/notes_crash_typing.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/notes_crash_typing.rs#L63) | 63, 65, 194, 306, 308, 316, 394, 396, 398 | `join`, `kill_nine_while_people_type_loses_and_doubles_nothing`, `run`, `start` | | [crates/calternal/tests/prefs_live.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/prefs_live.rs#L41) | 41, 43 | `start` | | [crates/calternal/tests/search_live.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/search_live.rs#L44) | 44, 47, 55, 95, 108 | `eventually`, `http`, `start` | | [crates/calternal/tests/startup.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/startup.rs#L49) | 49, 71, 76, 143 | `a_new_server_prints_a_setup_token`, `launch_and_wait_for_health` | | [crates/calternal/tests/attachments_crash.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/attachments_crash.rs#L46) | 46, 48, 167, 169, 275, 285 | `a_big_upload_streams_in_bounded_memory`, `start`, `wait_for` | | [crates/calternal/tests/contacts_live.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/contacts_live.rs#L54) | 54, 56, 162, 164, 165, 207, 215, 222, 233, 281, 302, 315, 320 | `a_live_h1_collision_keeps_contact_and_incoming_link_identity`, `a_phone_renames_a_contact_while_someone_types_in_a_linking_note`, `start` | | [crates/calternal/tests/storage_startup.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/storage_startup.rs#L168) | 168, 170, 584, 589 | `a_lock_held_past_the_longest_grace_still_fails_startup`, `starts` | | [crates/calternal/tests/fix_tz_live.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/tests/fix_tz_live.rs#L55) | 55, 57, 62, 135, 137, 138, 268, 274 | `join`, `start`, `strip_tz_markers_on_a_live_home_with_a_client_typing` | | [crates/calternal/src/admin.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/src/admin.rs#L345) | 345, 381, 385, 393, 417, 443 | `a_restart_waits_out_an_admin_request_longer_than_ten_seconds`, `a_stop_sent_before_subscribing_is_still_seen`, `an_active_request_gets_a_graceful_shutdown_window`, `shutdown_stops_a_client_waiting_on_a_partial_request` | | [crates/calternal/src/shutdown_io.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/calternal/src/shutdown_io.rs#L175) | 175, 177, 193, 197 | `shutdown_bounds_response_writes_to_a_peer_that_stops_reading`, `shutdown_finishes_an_idle_read_and_keeps_response_writes_open` | | [crates/server/tests/signout.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/server/tests/signout.rs#L61) | 61 | `next` | | [crates/server/tests/live.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/server/tests/live.rs#L75) | 75 | `next` | | [crates/server/src/rooms.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/server/src/rooms.rs#L556) | 556 | `assembly_is_bounded_per_connection_and_globally_and_expiry_releases_it` | | [crates/tasks/src/status/executor_tests.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/tasks/src/status/executor_tests.rs#L64) | 64 | `kill_at` | | [crates/tasks/src/tests.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/tasks/src/tests.rs#L93) | 93, 191, 192 | `eventually`, `run` | | [crates/daily/src/tests.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/daily/src/tests.rs#L237) | 237, 239 | `eventually` | | [crates/prefs/src/store/tests.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/prefs/src/store/tests.rs#L376) | 376 | `interrupted_conflict_exchange_restores_outside_file_on_restart` | | [crates/prefs/src/tests.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/prefs/src/tests.rs#L330) | 330 | `a_change_on_one_device_reaches_the_other` | | [crates/tasks/src/create_tests.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/tasks/src/create_tests.rs#L224) | 224 | `opening_today_then_creating_tasks_scopes_keys_and_retries` | | [crates/search/tests/search.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/search/tests/search.rs#L332) | 332, 336, 339, 345, 352, 407, 409 | `behind_answers_at_once_and_catching_up_is_announced_once`, `searches_answer_while_indexing_runs` | | [crates/search/tests/common/mod.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/search/tests/common/mod.rs#L236) | 236, 241 | `settle` | | [crates/contacts-actions/src/tests/break_links.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/contacts-actions/src/tests/break_links.rs#L234) | 234, 238 | `five_thousand_links_across_a_thousand_notes` | | [crates/contacts-actions/src/tests/correctness.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/contacts-actions/src/tests/correctness.rs#L91) | 91, 101, 241 | `bounded_shutdown_joins_the_card_write_and_refuses_later_copies`, `kill_delete_at` | | [crates/contacts-actions/src/tests/review.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/contacts-actions/src/tests/review.rs#L498) | 498, 638, 650, 820, 980 | `kill9_import_during_file_placement`, `kill_delete_restore`, `kill_import_phase_in_folder`, `kill_restore_links` | | [crates/contacts-actions/src/tests/kill9.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/contacts-actions/src/tests/kill9.rs#L153) | 153 | `killed_at` | | [crates/contacts-actions/src/tests/env.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/contacts-actions/src/tests/env.rs#L337) | 337, 339 | `eventually` | | [crates/fsio/src/usage.rs](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/crates/fsio/src/usage.rs#L421) | 421, 428, 531 | `reads_do_not_wait_for_the_counter_writer`, `stale_metadata_cannot_replace_a_newer_update` | A subsequent workspace run also exposed an implicit 50 ms production parser budget in markdown-view's containment-recovery success assertion. That check now passes explicit 120-second test limits to extraction inside containment; the production limit and deterministic deadline boundary test are unchanged. Other functional callers of production parser defaults need follow-up review. Engine's `interleave_with_huge_job`, search's ignored `ordinary_save_p95_during_executor_and_rebuild` timing, and markdown-view extraction scaling are measurement helpers or ignored perf cases, not new functional speed gates. ### Frontend follow-up candidates These test files also call `performance.now()`. Review speed/frame assertions separately: frames must run on the Mac VM. Date-based content/fixture timestamps are not speed requirements. | File | Clock call lines | | --- | --- | | [apps/mockup/tests/break.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/mockup/tests/break.spec.ts#L63) | 63, 68 | | [apps/web/e2e/dialog-stable.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/e2e/dialog-stable.spec.ts#L37) | 37, 42 | | [apps/web/e2e/list-fit.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/e2e/list-fit.spec.ts#L96) | 96 | | [apps/web/e2e/page-chrome.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/e2e/page-chrome.spec.ts#L15) | 15, 17, 32 | | [apps/web/e2e/search-corpus.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/e2e/search-corpus.spec.ts#L22) | 22, 24 | | [apps/web/e2e/swipe-rows.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/e2e/swipe-rows.spec.ts#L95) | 95, 141 | | [apps/web/e2e/switching.spec.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/e2e/switching.spec.ts#L48) | 48 | | [apps/web/src/composer/recognise.test.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/apps/web/src/composer/recognise.test.ts#L186) | 186, 188 | | [packages/editor/test/branch.test.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/packages/editor/test/branch.test.ts#L47) | 47, 49 | | [packages/editor/test/link-rank.test.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/packages/editor/test/link-rank.test.ts#L160) | 160, 162 | | [packages/editor/test/refusal-performance.test.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/packages/editor/test/refusal-performance.test.ts#L36) | 36, 55, 117, 119, 123, 124, 149, 150, 168, 171, 229, 230 | | [packages/editor/test/tag-rank.test.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/packages/editor/test/tag-rank.test.ts#L74) | 74, 76 | | [packages/editor/test/tree-cost.test.ts](https://git.kayg.org/kayg/calternal/src/branch/fix/parallel-test-flakes/packages/editor/test/tree-cost.test.ts#L21) | 21, 23 | Maintainer evidence: dev-02 independently took 2.012433 s, 2.034076 s, and 16.228720 s for 4 KiB writes followed by fsync, with 73 GB free; full I/O stall pressure reached 84.98% over 10 seconds. Functional correctness must tolerate that storage latency. Fresh verification at `d0f89de360a964f5f718c2a3b9faa1025c05decb` passed: dev-01 5/5, dev-02 3/3 default-parallel workspace commands, zero failures. The functional search case `stalled_rebuild_uses_old_sources_while_writer_and_new_readers_finish` subsequently failed its three-second action guard at `3a93293b`. It now waits under the shared liveness watchdog and explicitly asserts the rebuild remains parked until writer and new-reader completion. Its one-second snapshot guard was converted too. Other search guard candidates in the inventory remain follow-ups. Live note follow-up resolved in `9e6582a6`: `crates/calternal/tests/notes_live.rs` startup, sync, edit visibility and missing-note response guards now use shared 120 s LIVENESS; protocol ordering/content/error assertions remain. At `ba2367f4`, dev-02 run 2 failed the five-second missing-note response guard.
kayg changed title from Fix parallel crash/auth test flakes and duplicate move usage accounting to Fix functional test flakes and duplicate move usage accounting 2026-11-01 21:18:55 +00:00
kayg commented 2026-11-07 20:26:03 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+failing, left open for review.

rebased test flake and move-accounting fixes. used functional liveness/join_all checks in place of obsolete wall-clock probes; removed stale diagnostic variable references.

head: a4a316206a359f1ea9c4453aa9d34ed7c7686d8f; based on dev cd7f84432febfc63b788fbb9b9f9950452982d74, pushed with the original exact force-with-lease.

frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B).

Rust gate results (Mac VM):

exit command
0 cargo fmt -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit -- --check
101 cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit --all-targets
101 cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit --all-targets -- -D warnings
101 cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit

server-wide all-target checks/tests are blocked on macOS by storage_startup.rs:147 borrowing Child immutably; clippy also reports unused state in storage.rs:265. both reproduce on dev.

all executed gates used the VM Nix environment, shared Mac lock and slot. exact commands, source heads, logs and dev baseline. frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head.

original-head cherry/patch inventory, pr-825.md; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.

housekeeping #1009: rebased+failing, left open for review. rebased test flake and move-accounting fixes. used functional liveness/join_all checks in place of obsolete wall-clock probes; removed stale diagnostic variable references. head: `a4a316206a359f1ea9c4453aa9d34ed7c7686d8f`; based on dev `cd7f84432febfc63b788fbb9b9f9950452982d74`, pushed with the original exact force-with-lease. frontend frozen install, check, lint, build and unit tests passed. bundle gate passed. first-paint brotli vs dev: JS 230777 B (+0 B), CSS 18386 B (+0 B). Rust gate results (Mac VM): | exit | command | |---|---| | 0 | `cargo fmt -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit -- --check` | | 101 | `cargo check -j4 --locked -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit --all-targets` | | 101 | `cargo clippy -j4 --locked -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit --all-targets -- -D warnings` | | 101 | `cargo test -j4 --locked -p calternal-auth -p calternal -p calternal-fsio -p calternal-markdown-view -p calternal-notes -p calternal-search -p calternal-testkit` | server-wide all-target checks/tests are blocked on macOS by `storage_startup.rs:147` borrowing Child immutably; clippy also reports unused `state` in `storage.rs:265`. both reproduce on dev. all executed gates used the VM Nix environment, shared Mac lock and slot. [exact commands, source heads, logs and dev baseline](https://git.kayg.org/attachments/bdb8d284-b80d-4185-b0be-6fa74f48a191). frontend inputs are identical to the tested frontend tree when a Rust-only repair changed the head. [original-head cherry/patch inventory](https://git.kayg.org/attachments/e05a38dc-c028-4d77-8655-efa9d5c8ea50), `pr-825.md`; unmatched entries are missing-patch candidates, with semantic overlap/replacements noted above.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
This pull request has changes conflicting with the target branch.
  • crates/auth/tests/reliability_round3.rs
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/parallel-test-flakes:fix/parallel-test-flakes
git switch fix/parallel-test-flakes

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff fix/parallel-test-flakes
git switch fix/parallel-test-flakes
git rebase dev
git switch dev
git merge --ff-only fix/parallel-test-flakes
git switch fix/parallel-test-flakes
git rebase dev
git switch dev
git merge --no-ff fix/parallel-test-flakes
git switch dev
git merge --squash fix/parallel-test-flakes
git switch dev
git merge --ff-only fix/parallel-test-flakes
git switch dev
git merge fix/parallel-test-flakes
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!825
No description provided.