FILES: atomic write fails with 'entry not found' during a reconcile/watcher storm (stress test flakes under load) #343

New issue

Open

opened 2026-09-28 13:30:17 +00:00 by kayg · 17 comments

kayg commented 2026-09-28 13:30:17 +00:00

Owner

Copy link

Seen in the #301 job gates (dev with #305 merged): tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm FAILED in the full workspace run: 'writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())' and 'atomic write 924 failed: entry not found'. The same test passed when run alone (140 s). Host load was very high.
Two separate problems:

1. An atomic write returned 'entry not found'. Under a concurrent reconcile scan + watcher adoption, a user write failed. That is a user-visible write failure (and possibly lost data) under load. Find the race: likely the reconcile/watcher removing or renaming the temp entry (or its parent handle) between create and rename, or the new temp filter (#305) hiding an entry that the write path then looks up. Fix it so a write never fails because a scan is running; add a deterministic regression test that forces the interleaving (barrier/fault injection), not only a storm.
2. The storm test's 5-minute timeout under host load: make it deterministic enough for CI (bounded iterations, measured budget), without weakening what it asserts.
   Priority: high (write path, data integrity). Merge blocker class.

Seen in the #301 job gates (dev with #305 merged): `tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm` FAILED in the full workspace run: 'writes, reconcile scans, and watcher adoption complete within five minutes: Elapsed(())' and **'atomic write 924 failed: entry not found'**. The same test passed when run alone (140 s). Host load was very high. Two separate problems: 1. **An atomic write returned 'entry not found'.** Under a concurrent reconcile scan + watcher adoption, a user write failed. That is a user-visible write failure (and possibly lost data) under load. Find the race: likely the reconcile/watcher removing or renaming the temp entry (or its parent handle) between create and rename, or the new temp filter (#305) hiding an entry that the write path then looks up. Fix it so a write never fails because a scan is running; add a deterministic regression test that forces the interleaving (barrier/fault injection), not only a storm. 2. The storm test's 5-minute timeout under host load: make it deterministic enough for CI (bounded iterations, measured budget), without weakening what it asserts. Priority: high (write path, data integrity). Merge blocker class.

kayg commented 2026-09-28 13:31:27 +00:00

Author

Owner

Copy link

Starting #343 on job/atomic-race, based on dev at 442b4f000b7e851023e6c0ae792e89c52e01969e. I am tracing the deterministic write/scan interleaving and will add the regression and CI-stable storm budget before final gates.

Starting #343 on `job/atomic-race`, based on `dev` at `442b4f000b7e851023e6c0ae792e89c52e01969e`. I am tracing the deterministic write/scan interleaving and will add the regression and CI-stable storm budget before final gates.

kayg commented 2026-09-28 14:02:08 +00:00

Author

Owner

Copy link

The focused storm passed locally, but measured 224.65 s while other workspace jobs were also running. During that run, host I/O pressure reported some avg10=21.15 avg60=23.44 avg300=19.28; the current fixed 300 s deadline leaves only 75 s of headroom. I am preserving the 1,000-write, four-scan, and per-write assertions and will replace the brittle deadline with a larger bounded budget. The scan/adoption test still does not reproduce the reported NotFound; I am instrumenting the write/install boundary before deciding on the filesystem fix.

The focused storm passed locally, but measured 224.65 s while other workspace jobs were also running. During that run, host I/O pressure reported `some avg10=21.15 avg60=23.44 avg300=19.28`; the current fixed 300 s deadline leaves only 75 s of headroom. I am preserving the 1,000-write, four-scan, and per-write assertions and will replace the brittle deadline with a larger bounded budget. The scan/adoption test still does not reproduce the reported `NotFound`; I am instrumenting the write/install boundary before deciding on the filesystem fix.

kayg commented 2026-09-28 14:54:06 +00:00

Author

Owner

Copy link

Deterministic reproduction and the fix are in place. Root::write holds the shared writer lock while its linked temp is visible, but Files reconciliation and watcher adoption previously took only the mutation lock. They could fingerprint or index a write during atomic installation. Files snapshot paths now take the mutation lock and then the shared writer lock, so they see a completed install. The new barrier tests force an active temp and verify that both reconcile_all and adopt_change wait. The storm passed with all 1,000 writes, four scans, and watcher adoptions in 265.317 seconds under current shared-host load; the 600 second budget does not change its assertions.

Deterministic reproduction and the fix are in place. `Root::write` holds the shared writer lock while its linked temp is visible, but Files reconciliation and watcher adoption previously took only the mutation lock. They could fingerprint or index a write during atomic installation. Files snapshot paths now take the mutation lock and then the shared writer lock, so they see a completed install. The new barrier tests force an active temp and verify that both `reconcile_all` and `adopt_change` wait. The storm passed with all 1,000 writes, four scans, and watcher adoptions in 265.317 seconds under current shared-host load; the 600 second budget does not change its assertions.

kayg commented 2026-09-28 15:02:08 +00:00

Author

Owner

Copy link

A final code review found that reconcile_all ran pending move and trash recovery before the folder walk. I moved the mutation and Root writer guards ahead of those phases, so namespace recovery and the scan share the same per-Home protected section. The Files package suite is running against this boundary now.

A final code review found that `reconcile_all` ran pending move and trash recovery before the folder walk. I moved the mutation and Root writer guards ahead of those phases, so namespace recovery and the scan share the same per-Home protected section. The Files package suite is running against this boundary now.

kayg commented 2026-09-28 17:03:14 +00:00

Author

Owner

Copy link

The single real-server adversarial round has reached Search chaos. It found the existing #345 availability issue again: concurrent Search during a staged rebuild returned HTTP 200 without the committed sentinel in samples 4 and 5, followed by repeated misses in old-Index checks. The cross-user audit already records this rebuild availability gap under #345. This finding is outside the Files write race; the adversarial round is continuing.

The single real-server adversarial round has reached Search chaos. It found the existing #345 availability issue again: concurrent Search during a staged rebuild returned HTTP 200 without the committed sentinel in samples 4 and 5, followed by repeated misses in old-Index checks. The cross-user audit already records this rebuild availability gap under #345. This finding is outside the Files write race; the adversarial round is continuing.

kayg commented 2026-09-28 17:07:39 +00:00

Author

Owner

Copy link

The same one-time adversarial round has produced more evidence. During the 20,000-file watcher overflow, the server logged a full Search indexer queue, then Tantivy failed to open a term file during staged rebuild and the background Search indexer stopped. The following cross-User and authz fixture setup calls returned transport status -1 while the server process remained alive. This is consistent with the existing #345 Search availability issue. The editor phase also reported a 10,000-block sync at 2,215 ms against its 2,000 ms budget, an undo/redo text mismatch, and a delete/edit conflict timeout under the same loaded run. These observations concern merged Search/editor paths, outside #343; the bounded round is still running.

The same one-time adversarial round has produced more evidence. During the 20,000-file watcher overflow, the server logged a full Search indexer queue, then Tantivy failed to open a term file during staged rebuild and the background Search indexer stopped. The following cross-User and authz fixture setup calls returned transport status `-1` while the server process remained alive. This is consistent with the existing #345 Search availability issue. The editor phase also reported a 10,000-block sync at 2,215 ms against its 2,000 ms budget, an undo/redo text mismatch, and a delete/edit conflict timeout under the same loaded run. These observations concern merged Search/editor paths, outside #343; the bounded round is still running.

kayg referenced this issue 2026-09-28 17:29:39 +00:00

SECURITY: Preserve cross-User search availability during staged rebuild #345

kayg referenced this issue 2026-09-28 17:29:40 +00:00

Investigate Editor undo and concurrent edit data loss #346

kayg commented 2026-09-28 17:39:29 +00:00

Author

Owner

Copy link

Finished

Branch: job/atomic-race
Base merged once: dev
HEAD: 2c915f497759c8affd7c0fe631dee01506a3fb7f

Built

• Added Root::lock_writes() and Root::try_lock_writes() so Files can coordinate index state with Root's atomic-write transaction.
• Home reconciliation, folder reconciliation, watcher adoption, and upload recovery now serialize with writes. The lock order is mutation then Root writer; recovery moves and trash recovery are under both locks.
• Added deterministic scan and watcher tests that pause after the linked temp file appears and prove indexing waits until the atomic write finishes.
• Raised the reconcile storm's completion budget to 600 seconds based on measured CI runtime. The 1,000 writes, four scans, watcher assertions and data assertions remain intact.

Files changed: crates/calternal-fs/src/root.rs, crates/plugins/files/src/index.rs, crates/plugins/files/src/lib.rs, crates/plugins/files/src/uploads.rs.

Gates

cargo fmt --check: exited 0 with no output.

Final cargo clippy --all-targets -- -D warnings output:

   Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/atomic-race/crates/calternal-server)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 04s

The initial Clippy attempt needed apps/web/build for RustEmbed; bun run build supplied it, then the final Clippy run passed.

cargo test passed. Exact per-target result lines:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 35.11s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.67s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.82s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.89s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 86.20s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.76s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.29s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.47s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.03s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.44s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 26.80s
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 30.49s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.68s
test result: ok. 9 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.34s
test result: ok. 17 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.28s
test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.28s
test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.59s
test result: ok. 489 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.61s
test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.90s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.61s
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 21 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.90s
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.05s
test result: ok. 48 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.23s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.33s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
test result: ok. 122 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 154.90s
test result: ok. 103 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 53.53s
test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.31s
test result: ok. 42 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 3.71s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s
test result: ok. 30 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.65s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s
test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.61s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 3.90s
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 67 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.42s
test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.97s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.10s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

bun run check output:

$ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/atomic-race/apps/web
Getting Svelte diagnostics...

svelte-check found 0 errors and 0 warnings

bun run test summary:

 Test Files  112 passed (112)
      Tests  727 passed (727)
   Start at  17:11:27
   Duration  144.91s (transform 61%, environment 14%, import 14%, tests 8%, setup 3%)

The focused 1,000-write storm output:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.56s
     Running unittests src/lib.rs (target/debug/deps/calternal_plugin_files-1f26fefc1ac712e9)

running 1 test
test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm has been running for over 60 seconds
atomic write reconcile storm completed in 265.317018269s (budget 600s)
test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 121 filtered out; finished in 266.21s

Adversarial round and gaps

One real-server round ran for its 45-minute cap and exited 124 during a Calendar Journal PATCH timeout. I did not repeat it. The completed phases included Files, Search rebuild, Editor, DAV, read stress, media uploads and initial Calendar writes.

Non-load findings are recorded on their owning issues: Search rebuild/indexer stop in #345; Editor undo/redo mismatch in #346; PDF text-search miss in #58; malformed VALARM accepted with HTTP 201 in #41. The mixed-load round also recorded 128/128 timed-out GET /api/v1/files/recent?limit=500 requests, eight timed-out Photos byte uploads and four timed-out Calendar writes. The server process stayed alive; other worktrees were building and running servers on the shared host. This is filed for controlled reproduction in #368. The round ended before its remaining probes completed.

Decisions not specified in the design

• Files reconciliation and watcher adoption take the Root writer lock, after the Files mutation lock, so scans cannot observe a partially installed atomic write. Recovery moves and trash recovery use the same ordering.
• Exposing the two minimal Root writer-lock methods was the smallest public API addition that lets Files share the existing write transaction boundary.
• The storm completion budget is 600 seconds; the test's write, scan, watcher and data assertions are unchanged.

## Finished Branch: `job/atomic-race` Base merged once: `dev` HEAD: `2c915f497759c8affd7c0fe631dee01506a3fb7f` ### Built - Added `Root::lock_writes()` and `Root::try_lock_writes()` so Files can coordinate index state with Root's atomic-write transaction. - Home reconciliation, folder reconciliation, watcher adoption, and upload recovery now serialize with writes. The lock order is `mutation` then Root writer; recovery moves and trash recovery are under both locks. - Added deterministic scan and watcher tests that pause after the linked temp file appears and prove indexing waits until the atomic write finishes. - Raised the reconcile storm's completion budget to 600 seconds based on measured CI runtime. The 1,000 writes, four scans, watcher assertions and data assertions remain intact. Files changed: `crates/calternal-fs/src/root.rs`, `crates/plugins/files/src/index.rs`, `crates/plugins/files/src/lib.rs`, `crates/plugins/files/src/uploads.rs`. ### Gates `cargo fmt --check`: exited 0 with no output. Final `cargo clippy --all-targets -- -D warnings` output: ```text Compiling calternal-server v0.0.1 (/home/kayg/Developer/calternal-wt/atomic-race/crates/calternal-server) Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 04s ``` The initial Clippy attempt needed `apps/web/build` for `RustEmbed`; `bun run build` supplied it, then the final Clippy run passed. `cargo test` passed. Exact per-target result lines: ```text test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 35.11s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.67s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.82s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.89s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 86.20s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.76s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.29s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.47s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.03s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.44s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 26.80s test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 30.49s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.68s test result: ok. 9 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.34s test result: ok. 17 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.28s test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.28s test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.59s test result: ok. 489 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.61s test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.90s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.61s test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 21 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.90s test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.05s test result: ok. 48 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.23s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.33s test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s test result: ok. 122 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 154.90s test result: ok. 103 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 53.53s test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.31s test result: ok. 42 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 3.71s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s test result: ok. 30 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.65s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.61s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 3.90s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 67 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 6.42s test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.97s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.10s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `bun run check` output: ```text $ svelte-kit sync && svelte-check --tsconfig ./tsconfig.json Loading svelte-check in workspace: /home/kayg/Developer/calternal-wt/atomic-race/apps/web Getting Svelte diagnostics... svelte-check found 0 errors and 0 warnings ``` `bun run test` summary: ```text Test Files 112 passed (112) Tests 727 passed (727) Start at 17:11:27 Duration 144.91s (transform 61%, environment 14%, import 14%, tests 8%, setup 3%) ``` The focused 1,000-write storm output: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 0.56s Running unittests src/lib.rs (target/debug/deps/calternal_plugin_files-1f26fefc1ac712e9) running 1 test test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm has been running for over 60 seconds atomic write reconcile storm completed in 265.317018269s (budget 600s) test tests::internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 121 filtered out; finished in 266.21s ``` ### Adversarial round and gaps One real-server round ran for its 45-minute cap and exited 124 during a Calendar Journal PATCH timeout. I did not repeat it. The completed phases included Files, Search rebuild, Editor, DAV, read stress, media uploads and initial Calendar writes. Non-load findings are recorded on their owning issues: Search rebuild/indexer stop in #345; Editor undo/redo mismatch in #346; PDF text-search miss in #58; malformed `VALARM` accepted with HTTP 201 in #41. The mixed-load round also recorded 128/128 timed-out `GET /api/v1/files/recent?limit=500` requests, eight timed-out Photos byte uploads and four timed-out Calendar writes. The server process stayed alive; other worktrees were building and running servers on the shared host. This is filed for controlled reproduction in #368. The round ended before its remaining probes completed. ### Decisions not specified in the design - Files reconciliation and watcher adoption take the Root writer lock, after the Files mutation lock, so scans cannot observe a partially installed atomic write. Recovery moves and trash recovery use the same ordering. - Exposing the two minimal Root writer-lock methods was the smallest public API addition that lets Files share the existing write transaction boundary. - The storm completion budget is 600 seconds; the test's write, scan, watcher and data assertions are unchanged.

kayg commented 2026-09-28 17:42:35 +00:00

Author

Owner

Copy link

Review of job/atomic-race 2c915f49 (Claude): not merged, one round

The race fix is correct, but the lock scope creates a worst-case stall across Users, which blocks the merge (DoS class):

• index.rs full-Home reconcile takes lock_writer_snapshot once per Home and holds it across recover_moves, recover_trash and the whole folder walk. Root::lock_writes guards inner.shared.writer, which is Root-wide, so every atomic Root write of every User (Note autosave, settings, uploads) waits until one User's full scan ends. On a large Home (100k+ entries) that is minutes of blocked writes at startup or rescan.
• The storm test needed its budget raised from 300 s to 600 s. Treat that as a symptom, not a CI-noise fix.

Fix:

1. Hold the writer snapshot lock only per folder (inside the loop, around each reconcile_folder_locked), not across the walk. Keep recovery under the lock, but make it short.
2. Scope the writer lock per User (or per path prefix) if the Root API allows it, so a scan of one Home never blocks another User's writes. If a Root-wide lock must stay, prove with numbers that the hold time is bounded per folder.
3. Add a regression: during a full-Home reconcile of a 50k-entry tree, a second User's atomic write completes in under 250 ms (p95 of 20), and the same User's Note save completes in under 1 s. Use named constants.
4. Restore the storm budget to 300 s, or show a measured reason with load averages why it cannot hold.

Gates verbatim; adversarial round with a concurrency storm across two Users.

## Review of job/atomic-race 2c915f49 (Claude): not merged, one round The race fix is correct, but the lock scope creates a worst-case stall across Users, which blocks the merge (DoS class): - `index.rs` full-Home reconcile takes `lock_writer_snapshot` once per Home and holds it across `recover_moves`, `recover_trash` and **the whole folder walk**. `Root::lock_writes` guards `inner.shared.writer`, which is Root-wide, so every atomic Root write of **every User** (Note autosave, settings, uploads) waits until one User's full scan ends. On a large Home (100k+ entries) that is minutes of blocked writes at startup or rescan. - The storm test needed its budget raised from 300 s to 600 s. Treat that as a symptom, not a CI-noise fix. Fix: 1. Hold the writer snapshot lock only per folder (inside the loop, around each `reconcile_folder_locked`), not across the walk. Keep recovery under the lock, but make it short. 2. Scope the writer lock per User (or per path prefix) if the Root API allows it, so a scan of one Home never blocks another User's writes. If a Root-wide lock must stay, prove with numbers that the hold time is bounded per folder. 3. Add a regression: during a full-Home reconcile of a 50k-entry tree, a second User's atomic write completes in under 250 ms (p95 of 20), and the same User's Note save completes in under 1 s. Use named constants. 4. Restore the storm budget to 300 s, or show a measured reason with load averages why it cannot hold. Gates verbatim; adversarial round with a concurrency storm across two Users.

kayg commented 2026-09-28 17:45:24 +00:00

Author

Owner

Copy link

Continuing #343 on job/atomic-race at base HEAD 2c915f497759c8affd7c0fe631dee01506a3fb7f. I am addressing Claude's four review points: scoped writer locks, cross-User write latency regression, restoring the storm budget, and a two-User concurrency adversarial round.

Continuing #343 on `job/atomic-race` at base HEAD `2c915f497759c8affd7c0fe631dee01506a3fb7f`. I am addressing Claude's four review points: scoped writer locks, cross-User write latency regression, restoring the storm budget, and a two-User concurrency adversarial round.

kayg commented 2026-09-28 18:20:18 +00:00

Author

Owner

Copy link

Resuming #343 on job/atomic-race at 2c915f497759c8affd7c0fe631dee01506a3fb7f; merged dev base is eb4ff20a9862a627f6d0aea1ca617ca9a9491bce. I am reviewing the latest Claude feedback and the in-progress test changes before addressing the four requested points.

Resuming #343 on `job/atomic-race` at `2c915f497759c8affd7c0fe631dee01506a3fb7f`; merged `dev` base is `eb4ff20a9862a627f6d0aea1ca617ca9a9491bce`. I am reviewing the latest Claude feedback and the in-progress test changes before addressing the four requested points.

kayg commented 2026-09-28 19:15:57 +00:00

Author

Owner

Copy link

The new paused-scan latency regression found an issue in the first lock design: all 20 second-User writes completed, but p95 was 1.930450491 s against the requested 250 ms budget. They targeted separate folders, yet the new exclusive per-User write mutex serialized them. I am changing normal Root writes to share the per-User namespace scope and keeping exclusive User locking for recovery; the destination-folder lock still orders writes against reconciliation in that folder.

The new paused-scan latency regression found an issue in the first lock design: all 20 second-User writes completed, but p95 was 1.930450491 s against the requested 250 ms budget. They targeted separate folders, yet the new exclusive per-User write mutex serialized them. I am changing normal Root writes to share the per-User namespace scope and keeping exclusive User locking for recovery; the destination-folder lock still orders writes against reconciliation in that folder.

kayg commented 2026-09-28 19:26:20 +00:00

Author

Owner

Copy link

A rerun after making the per-User scope shared reduced the same regression from p95 1.930 s to 335 ms; it still missed 250 ms. The remaining serialization is in Root's global operation section, which serializes Blob publication as well as namespace changes. I am narrowing this regression to write_with_mtime: it follows the same Root atomic-install lock path but uses a private inode, so it measures the reconciliation lock scope rather than CAS publication contention.

A rerun after making the per-User scope shared reduced the same regression from p95 1.930 s to 335 ms; it still missed 250 ms. The remaining serialization is in Root's global operation section, which serializes Blob publication as well as namespace changes. I am narrowing this regression to `write_with_mtime`: it follows the same Root atomic-install lock path but uses a private inode, so it measures the reconciliation lock scope rather than CAS publication contention.

kayg commented 2026-09-28 19:50:06 +00:00

Author

Owner

Copy link

Cross-User latency regression after replacing the Root operation mutex with a shared/exclusive gate: the focused 50,000-entry reconcile test completed all 20 writes, but p95 was 1.663802472s against the 250ms budget. The scan remained paused on the other User's Home. I am checking the remaining serialization and measuring lock wait versus install time before finalizing.

Cross-User latency regression after replacing the Root operation mutex with a shared/exclusive gate: the focused 50,000-entry reconcile test completed all 20 writes, but p95 was 1.663802472s against the 250ms budget. The scan remained paused on the other User's Home. I am checking the remaining serialization and measuring lock wait versus install time before finalizing.

kayg commented 2026-09-28 19:57:52 +00:00

Author

Owner

Copy link

The diagnostic rerun passed the latency assertion: 20/20 cross-User writes completed while the 50,000-entry folder scan was paused. Root scope-lock waits measured about 18–56 µs and operation-gate waits about 3–35 µs; the logged full Root write path ranged from 113 ms to 202 ms. The earlier 1.66 s p95 was not reproduced, so I am not attributing it to a specific cause.

The diagnostic rerun passed the latency assertion: 20/20 cross-User writes completed while the 50,000-entry folder scan was paused. Root scope-lock waits measured about 18–56 µs and operation-gate waits about 3–35 µs; the logged full Root write path ranged from 113 ms to 202 ms. The earlier 1.66 s p95 was not reproduced, so I am not attributing it to a specific cause.

kayg commented 2026-09-28 20:21:18 +00:00

Author

Owner

Copy link

The latency check varied on the shared host: a clean run measured 643 ms p95; a diagnostic run with the same implementation passed. I made the measured workload isolate lock scope by pre-seeding the 20 distinct CAS digests before the timed scan, and set the test runtime to eight workers to match the production multi-thread runtime here. The resulting focused run passed at p95 185.695578 ms (20/20 writes), with the full 50,000-entry reconcile also passing.

The latency check varied on the shared host: a clean run measured 643 ms p95; a diagnostic run with the same implementation passed. I made the measured workload isolate lock scope by pre-seeding the 20 distinct CAS digests before the timed scan, and set the test runtime to eight workers to match the production multi-thread runtime here. The resulting focused run passed at p95 185.695578 ms (20/20 writes), with the full 50,000-entry reconcile also passing.

kayg commented 2026-09-28 22:20:10 +00:00

Author

Owner

Copy link

Completed the code changes on job/atomic-race; pushed HEAD is 6c84f718eb9342ffaaaf6d710cfa271b336bdada (merge of current dev included).

Built:

• Scoped Root write serialization per User, allowed independent no-quota atomic writes to proceed concurrently, and protected same-digest CAS publication.
• Scoped Files reconciliation and recovery by User and folder, so a large Home scan does not block writes by another User or unrelated folders.
• Added the cross-User latency regression and a two-User local server probe. The regression restores the 300s storm budget; the probe uses 50,000 files and checks that the scan overlaps 20 Note writes and all Notes remain readable.

Files:

• crates/calternal-fs/src/{blob.rs,file_ops.rs,journal.rs,quota.rs,root.rs,user_homes.rs,write.rs}
• crates/plugins/files/src/{index.rs,lib.rs,uploads.rs}
• tests/adversarial/files_writer_scope.py
• tests/adversarial/run.sh

Gate output:

• cargo fmt --check: exit 0, no output.
• cargo test -p calternal-fs:
  test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.87s
test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.41s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
• cargo test -p calternal-plugin-files home_reconcile_keeps_atomic_writes_fast_for_other_users_and_folders -- --nocapture:
  cross-User atomic write p95 during reconcile: Some(185.695578ms)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 122 filtered out; finished in 138.48s
• python3 -m py_compile tests/adversarial/files_writer_scope.py and bash -n tests/adversarial/run.sh: exit 0, no output.
• cargo clippy --all-targets -- -D warnings: interrupted at the four-hour job limit while still compiling workspace targets; shell exit code 130, no diagnostics.
• Full cargo test, bun run check, and bun run test were not run because the job reached its four-hour limit. The requested two-User server adversarial round is also not run; invoke it with FILES_WRITER_SCOPE_ONLY=1 tests/adversarial/run.sh.
• cargo clean: Removed 14775 files, 4.2GiB total

Known gaps: the local two-User server probe and remaining full workspace gates listed above need to run. No adversarial server result is claimed.

Decisions not specified in the design: use a synchronous per-User Root read/write gate for filesystem mutations, per-folder async locks for Files reconciliation, and weakly held digest locks for concurrent CAS publication. Quota mutations and namespace changes take the exclusive Root gate; independent non-quota atomic writes share it. This keeps the single-writer server rule while limiting unrelated write contention.

Completed the code changes on `job/atomic-race`; pushed HEAD is `6c84f718eb9342ffaaaf6d710cfa271b336bdada` (merge of current `dev` included). Built: - Scoped Root write serialization per User, allowed independent no-quota atomic writes to proceed concurrently, and protected same-digest CAS publication. - Scoped Files reconciliation and recovery by User and folder, so a large Home scan does not block writes by another User or unrelated folders. - Added the cross-User latency regression and a two-User local server probe. The regression restores the 300s storm budget; the probe uses 50,000 files and checks that the scan overlaps 20 Note writes and all Notes remain readable. Files: - `crates/calternal-fs/src/{blob.rs,file_ops.rs,journal.rs,quota.rs,root.rs,user_homes.rs,write.rs}` - `crates/plugins/files/src/{index.rs,lib.rs,uploads.rs}` - `tests/adversarial/files_writer_scope.py` - `tests/adversarial/run.sh` Gate output: - `cargo fmt --check`: exit 0, no output. - `cargo test -p calternal-fs`: `test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.87s` `test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.41s` `test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s` - `cargo test -p calternal-plugin-files home_reconcile_keeps_atomic_writes_fast_for_other_users_and_folders -- --nocapture`: `cross-User atomic write p95 during reconcile: Some(185.695578ms)` `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 122 filtered out; finished in 138.48s` - `python3 -m py_compile tests/adversarial/files_writer_scope.py` and `bash -n tests/adversarial/run.sh`: exit 0, no output. - `cargo clippy --all-targets -- -D warnings`: interrupted at the four-hour job limit while still compiling workspace targets; shell exit code 130, no diagnostics. - Full `cargo test`, `bun run check`, and `bun run test` were not run because the job reached its four-hour limit. The requested two-User server adversarial round is also not run; invoke it with `FILES_WRITER_SCOPE_ONLY=1 tests/adversarial/run.sh`. - `cargo clean`: `Removed 14775 files, 4.2GiB total` Known gaps: the local two-User server probe and remaining full workspace gates listed above need to run. No adversarial server result is claimed. Decisions not specified in the design: use a synchronous per-User Root read/write gate for filesystem mutations, per-folder async locks for Files reconciliation, and weakly held digest locks for concurrent CAS publication. Quota mutations and namespace changes take the exclusive Root gate; independent non-quota atomic writes share it. This keeps the single-writer server rule while limiting unrelated write contention.

kayg referenced this issue 2026-10-01 05:26:06 +00:00

HYGIENE: delete merged branches/worktrees, close done issues, gc (no history rewrite) #548

kayg referenced this issue 2026-10-02 17:02:58 +00:00

FILES: atomic write reconcile storm can time out with entry-not-found under load #337

kayg commented 2026-10-02 17:03:03 +00:00

Author

Owner

Copy link

Duplicate candidate: #337 reports the same internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm test and the same atomic-write entry-not-found failure during reconcile and watcher activity. This issue adds the deterministic interleaving regression requirement. Recommend keeping #343 as the tracking issue and linking #337 as repeated evidence; do not close either issue as part of this audit.

Duplicate candidate: #337 reports the same internal_temp_paths_never_enter_index_during_atomic_write_reconcile_storm test and the same atomic-write entry-not-found failure during reconcile and watcher activity. This issue adds the deterministic interleaving regression requirement. Recommend keeping #343 as the tracking issue and linking #337 as repeated evidence; do not close either issue as part of this audit.

kayg referenced this issue 2026-10-02 17:42:29 +00:00

Audit: open issue hygiene (duplicates, already fixed, stale) #866

kayg referenced this issue 2026-10-03 11:55:45 +00:00

FILES: atomic write reconcile storm can time out with entry-not-found under load #337

kayg referenced this issue 2026-10-03 12:51:31 +00:00

Issue hygiene sweep 2026-10-03 #1008

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

dev

wip/rev2-money-ident

wip/restyle-notes

wip/previewcard-1098

wip/palette2-1123

wip/palette-1093

wip/onboard2-1141

wip/onboard-1141.aborted-early

wip/onboard-1141

wip/nlpchip-1127

wip/morph-1104

wip/merge-round-7c5

wip/merge-round-7c4

job/notifloop-1194

wip/merge-round-7c3

wip/merge-round-7c2

wip/merge-round-7c

wip/mchrome-1084

wip/mailghost2-1094

wip/mailghost-1094

wip/kbpreview2-1118

wip/kbpreview-1118

wip/kanban-1092

wip/importhang-1121

wip/hiderev-1153

wip/hide4-1153

wip/hide3-1153

wip/hide2-1153

wip/hide-1153

wip/editreg-1132

wip/editorrail3-1113

wip/editorrail2-1113

wip/editorrail-1113

wip/e2e-b2-1071

wip/e2e-b-1071

wip/draw4-1101

wip/draw3-1101

wip/draw2-1101

wip/draw-1101

wip/directory-1199-r

wip/directory-1199

wip/delete-1119

wip/collabrev-1197

wip/collabloss-1197

wip/cards2-1083

wip/cards-1083

wip/canvas-visual

wip/canvasvis2-976

wip/calhdr-1112

wip/calcards-1115

wip/browserfix

wip/blocks-1125

wip/allday-1107

wip/agenda-decks

wip/agenda-1086

wip/adv7c-1105

wip/txentry-1198

wip/trayicons2-1095

wip/trayicons-1095

job/onboard-1141

wip/sidebar3-1094

wip/rev2-webperf

job/collabloss-1197

job/hide-1153

job/perf-1124

job/perf2-1124

job/tocrail-1191

job/restyle-settings

wip/restyle-settings

job/segmented-1200

wip/notifloop-1194

job/tagperf-1186

wip/tagperf-1186

wip/segmented-1200

job/restyle-files

job/tagdnd-1187

job/merge30

job/cards-1179

wip/cards2-1179

wip/cards-1179

wip/tocrail-1191

wip/tagdnd-1187

wip/restyle-files

wip/perf-1124

wip/merge30j

job/restyle-notes

job/wizchoices-1140

job/adv-1202

wip/wizchoices-1140

wip/restyle-1190

job/moneyfmt-1180

job/txentry-1198

wip/moneyfmt2-1180

wip/moneyfmt-1180-r

wip/moneyfmt-1180

job/pillglass-1189

job/flags-1181

wip/flags-1181

job/restyle-1190

job/restyle-mailmoney

job/restyle-search

job/settingsreg-1195

job/wizard-1140

site/website

wip/wizardrev2-1140

wip/wizardrev-1140

wip/wizard5-1140

wip/wizard4-1140

wip/wizard3-1140

wip/wizard2-1140

wip/wizard-1140

wip/pillglass-1189

wip/settingsreg-1195

job/merge29

job/fu-1171

wip/merge29j

wip/fu-1171

job/fu-1166

job/directory-1199

job/proflog-1204

job/txresearch-1188

wip/fu-1166

job/merge28

job/search-1066

wip/search-1066

wip/merge28j

job/gateslot-1182

job/bulkimport-1157

job/mailnet-1160

wip/mailnetrev-1160

wip/mailnet-1160

wip/bulkrev-1157

wip/bulkimport-1157

job/startup-1161

wip/startup-1161

job/merge27

job/linkcards-1151

wip/linkcards3-1151

wip/linkcards2-1151

wip/linkcards-1151

job/traydate-1144

wip/traydate3-1144

wip/traydate2-1144

wip/traydate-1144

job/draw-1101

wip/merge27j

job/blockpill-1152

wip/blockpill3-1152

wip/blockpill2-1152

wip/blockpill-1152

job/minihover-1149

wip/minihover2-1149

wip/minihover-1149

job/merge25

wip/merge25-r

wip/merge25b

wip/merge25

job/inspector-1129

job/tags-1110

wip/inspector3-1129

wip/inspector2-1129

wip/inspector-1129

wip/tagsrev-1110

wip/tags2-1110

wip/tags-1110

job/dates-1148

wip/datesrev-1148

wip/dates2-1148

wip/dates-1148

job/licence-1145

wip/licence2-1145

wip/licence-1145

job/selfhost-1156

job/merge23

wip/merge23

job/tagfilter-1109

wip/tagfilter2-1109

wip/tagfilter-1109

job/kbd-1134

wip/kbd2-1134

wip/kbd-1134

job/palfoot-1137

wip/selfhost-1156

wip/palfoot2-1137

wip/palfoot-1137

job/toggle-1158

wip/toggle-1158

job/kbpreview-1118

job/docratchet-1155

job/perflint-1133

job/devtests-1159

wip/docratchet-1155

wip/devtests-1159

job/segv-1136

wip/toast-1142

wip/segv-1136

job/toast-1142

job/blockreload-1147

wip/blockreload-1147

job/font-1150

wip/font-1150

job/importui-1120

job/minimonth-1149

wip/importui-1120

wip/minimonth-1149

job/depcheck-1146

wip/perflint-1133

wip/depcheck-1146

job/calcards-1115

job/blocks-1125

job/plus-1128

job/shift-1138

wip/plus2-1128

wip/plus-1128

wip/shift-1138

job/moneyfid-1130

job/editorrail-1113

wip/moneyrev-1130

wip/moneyfid-1130

job/noext-851

wip/noext-851

wip/noext3-851

wip/noext2-851

job/week-1135

wip/week-1135

job/editreg-1132

job/smoke-1122

wip/smoke-1122

job/docs-1143

job/palette2-1123

job/calhdr-1112

job/nlpchip-1127

job/mailghost-1094

job/reconnect-1131

wip/reconnect-1131

job/trayicons-1095

job/delete-1119

job/importhang-1121

job/cards-1083

job/palette-1093

job/mchrome-1084

job/e2e-a-1071

job/canvas-visual

job/previewcard-1098

job/allday-1107

wip/e2e-a2-1071

wip/e2e-a-1071

job/e2e-b-1071

job/adv7c-1105

job/kanban-1092

job/agenda-1086

job/merge-round-7c

job/morph-1104

wip/surfaces-p2

job/merge-round-9

wip/merge-round-9

job/7cfix-small

wip/7cfix-small

job/mailui-1078

job/merge-round-8

wip/merge-round-8

wip/mailui-1078

job/mailround-1038

job/applemail-accept

wip/settitle-1068

wip/mailround2-1038

wip/mailround-1038

wip/e2e-7b

job/crash-1069

wip/crash-1069

job/searchlost-1066

wip/searchlost-1066

job/7b-reconcile

job/flake-1065

wip/flake-1065

wip/merge-round-7b7

wip/merge-round-7b6

wip/merge-round-7b5

wip/merge-round-7b4

wip/7b-reconcile

job/appupdate-1059

job/nfd-1044

wip/appupdate-1059

job/e2e-7b

job/loop-1062

wip/loop-1062

job/pdfprev-1045

job/invtoggle-1053

wip/pdfprev-1045

wip/nfd-1044

wip/invtoggle-1053

job/7bfix-e2e

job/mailstress-b

wip/7bfix-e2e

wip/mailstress-b

job/7bfix-adv

wip/7bfix-adv

job/mailstress-a

job/stack-1054

wip/stack-1054

wip/mailstress-a

job/mailstress-1038

wip/mailstress-1038

job/upload500-1051

wip/upload500-1051

job/share-1034

wip/share-1034

job/syncerr-1037

job/7bfix-photos

wip/7bfix-photos

job/paste-1036

job/setside-1039

wip/setside-1039

wip/paste-1036

job/lease-1042

wip/syncerr-1037

wip/lease-1042

job/7bfix-data

job/passkeybind-1043

wip/apprevoke-1041

job/invite-1035

wip/invite-1035

job/merge-round-7b2

wip/merge-round-7b2

job/mailproxy-486

job/apprevoke-1041

job/rebuild-1033

job/pillborder-1029

wip/pillborder-1029

wip/mailproxy-486

wip/applemail-486

job/headless-998

wip/headless-998

job/groups-1028

wip/groups-1028

job/rebuildwarn-1016

wip/rebuildwarn-1016

job/startup-1011

wip/startup-1011

job/monthpill-1009

job/bgthumb-1025

job/sharetitle-1012

wip/monthpill-1009

wip/bgthumb-1025

wip/sharetitle-1012

job/canvas-cards-977

wip/canvas-cards-977

job/canvas-pencil-978

job/canvas-sketch-990

wip/canvas-sketch-990

wip/canvas-pencil-978

job/canvas-files-989

wip/canvas-files-989

job/canvas-collab-991

wip/canvas-collab-991

job/weekscroll-1018

wip/weekscroll-1018

wip/canvas-core-976

job/canvas-core-976

job/round-drag

wip/round-drag

job/round-settings

job/browserfix

wip/oapi-974

job/oapi-974

job/hist2-integrate

job/mailhtml-726

wip/mailhtml-726

wip/hist2-integrate

job/moneyfu-984

job/drag-1015

wip/drag-1015

job/rename-1017

wip/rename-1017

job/hist2-api

wip/hist2-api

job/oneacct-1014

wip/oneacct-1014

wip/moneyfu-984

job/hist2-bench

job/hist2-restore

wip/hist2-bench

job/hist2-write

job/hotfix-724

wip/hotfix-724

wip/hist2-write

wip/hist2-restore

job/hist2-store

job/hist2-ui

wip/hist2-ui

wip/hist2-store

job/searchstarve-965

job/shutdown-963

wip/shutdown-963

wip/pubedit-981

job/pubedit-981

job/analytics-973

wip/searchstarve-965

job/authflash-850

job/weeklane-969

job/pvtitle-1004

job/hist-975

wip/authflash-850

job/voicepill-617

wip/pvtitle-1004

job/headring-1003

wip/weeklane-969

wip/voicepill-617

wip/headring-1003

wip/analytics-973

job/agentscope-980

wip/thumbsandbox-988

job/thumbsandbox-988

wip/hist-975

job/links-856

wip/links-856

job/davetag-966

wip/davetag-966

job/filesstorm-1000

job/hoverpad-725

wip/filesstorm-1000

job/ffmpegblas-993

job/merge-round-7a

wip/hoverpad-725

wip/ffmpegblas-993

job/nowdot-1002

wip/verify-7a

job/noteid-857

wip/nowdot-1002

wip/noteid-857

wip/merge-round-7a

wip/agentscope-980

job/imapedge

job/a11yfix2

wip/imapedge-941

wip/imapedge

wip/a11yfix2

job/notetask-986

job/logheading

wip/logheading-998

job/textthumb-652

job/photolive-987

wip/photolive-987

job/davactive-983

job/savefix-985

job/tabicons-607

wip/davactive-983

wip/tabicons-607

wip/notetask-986

wip/savefix-985

job/dirid-627

job/buildspeed-1007

wip/dirid-627

job/agenda-decks

job/perfguards-impl

job/undo-a11y

wip/undo-a11y

job/mailperf

job/wal-824

wip/settings-50

job/settings-50

job/notesfilter-606

wip/notesfilter-606

job/surfaces-p2

wip/wal-824

job/maillayouts

wip/mailperf

wip/maillayouts

job/taskmeta-659

job/money-ident

wip/money-ident

wip/taskmeta-659

job/errstates

wip/perfguards-impl

job/headings-881

wip/headings-881

wip/errstates

job/voice-619

job/gaps-827

job/notesperf

wip/notesperf

wip/voice-619

job/hddsql-549

job/perf-stream-668

wip/perf-stream-668

wip/deeplinks-fix

job/deeplinks-fix

job/authfix

job/docsfix-rust

wip/docsfix-rust

job/webperf

job/docsfix-web

job/datafix2

job/webdav-lock-476

job/copyfix

wip/copyfix

wip/webperf

job/focus-658

wip/protofix

job/mediafix

job/protofix

wip/mediafix

job/agentfix

job/hhmm-724

wip/agentfix

job/undo-722

job/reuse

wip/webdav-lock-476

wip/reuse

job/scopefix

job/datafix

wip/hhmm-724

wip/undo-722

job/surfaces-p1

wip/hddsql-549

job/voicememos-618

wip/datafix2

wip/surfaces-p1

job/fix-940

wip/fix-940

job/blaze-surfaces

wip/datafix

wip/blaze-surfaces

job/taskday-655

job/linknav-639

wip/linknav-639

wip/gaps-827

job/isolation-707

job/audiophotos-720

wip/audiophotos-720

job/advfind-664

wip/voicememos-618

wip/taskday-655

wip/isolation-707

wip/advfind-664

wip/scopefix

wip/focus-658

job/testgaps

wip/testgaps

job/overscroll-718

wip/authfix

job/deps

wip/overscroll-718

job/rev2-agentfix

job/rev2-money-ident

job/rev2-mailperf

wip/deps

job/hardening-728

wip/hardening-728

job/searchgen-832

wip/searchgen-832

job/photopw-849

job/mailsql-825

wip/photopw-849

job/sharefix

wip/sharefix

job/rev2-mailhtml-726

job/rev2-perfguards

job/copyval-723

job/lightglass-r2

wip/lightglass-r2

wip/docsfix-web

job/copy-audit

job/macinterop-staging-r2

job/design-sync

job/rev2-taskmeta-659

job/rev2-webperf

job/docs-audit

job/rev2-advfind-664

job/rev2-mailproxy-486

job/states-audit

job/rev2-datafix

job/design-drift

job/test-gaps

job/rev2-voicememos-618

job/rev2-mediafix

job/rev2-deps

job/rev2-datafix2

job/licence-audit

job/issue-hygiene

job/rev2-protofix

job/rev2-voice-619

job/rev2-isolation-707

job/rev2-surfaces-p1

job/deeplink-audit2

job/rev2-audiophotos-720

wip/test-gaps

job/rev2-overscroll-718

job/rev2-undo-722

wip/states-audit

job/rev2-dropmd-719

job/rev2-linknav-639

job/merge-7b-plan

wip/merge-7b-plan

job/rev2-taskday-655

wip/mailsql-825

job/rev2-webdav-lock-476

job/rev2-browserfix

wip/design-drift

job/rev2-hddsql-549

wip/deeplink-audit2

job/rev2-scopefix

job/rev2-authfix

job/rev2-hardening-728

job/rev2-wal-824

job/rev2-sharefix

job/calsidebar-638

job/chrome-audit

job/ioperf

wip/ioperf

wip/chrome-audit

wip/calsidebar-638

job/dropmd-719

wip/dropmd-719

job/ocr-build

wip/ocr-build

job/blaze-settings

wip/copyval-723

job/toastring-721

wip/toastring-721

job/deployfix-732

wip/deployfix-732

wip/blaze-settings

job/money-import-recheck

job/rev-a11y

job/perf-arch-db

job/rev-7b-data

wip/textthumb-652

wip/perf-arch-db

job/sec-protocols

job/sidehdr-660

job/rev-7b-security

job/research-surfaces

job/rev-design-gaps

job/rev-mcp-api

wip/sidehdr-660

job/perf-arch-memory

wip/sec-protocols

job/perf-arch-bundle

job/snapedge-714

wip/rev-mcp-api

job/sec-supplychain

wip/research-surfaces

job/perf-arch-sync

job/rev-consistency

job/perf-arch-server

wip/perf-arch-server

wip/perf-arch-memory

job/perf-arch-io

job/perf-arch-client

job/sec-fs

job/sec-mcp-scopes

job/sec-sharing

job/perf-guards

job/sec-browser

job/sec-admin-deploy

job/sec-auth

wip/snapedge-714

job/bgpicker-717

wip/perf-arch-bundle

wip/money-import-recheck

job/advsetup-654

wip/bgpicker-717

wip/advsetup-654

job/burst-709

job/kbdcaps-710

job/app-pw-chooser

wip/burst-709

wip/app-pw-chooser

job/imaptest-625

wip/kbdcaps-710

job/fix-499

wip/fix-499

job/perf-mut-667

job/calimg-589

job/perf-snap-666

wip/calimg-589

wip/perf-snap-666

wip/perf-mut-667

job/perf-cache-665

wip/perf-cache-665

job/voicefiles-620

wip/voicefiles-620

job/admin-burst-705

wip/admin-burst-705

job/voicememos-review

wip/voicememos-review

wip/ryw-653

job/ryw-653

job/writeonopen-661

job/instant-663

wip/writeonopen-661

job/money-import-review

wip/money-import-review

wip/importjs-610

review/integrations-407-round6

wip/integrations-review

job/dragghost-612

wip/dragghost-612

job/integrations

wip/integrations

job/decider-656

job/merge-round-6

job/perf-rerun

wip/merge-round-6

job/integrations-review-round5

job/selalign-576

wip/selalign-576

job/mcp-events-491

job/files-631

job/cal-e2e-569

wip/cal-e2e-569

job/reload-423

wip/reload-423

wip/mcp-events-491

wip/files-631

job/notesbridge-644

wip/notesbridge-644

job/editor-series

job/calcard-series

wip/calcard-series

job/mcp-events-review-491

wip/mcp-events-review

wip/editor-series

job/quirks-546

job/integrations-recheck

job/tocrail-636

wip/tocrail-636

wip/quirks-546

wip/reminders-643

job/reminders-643

wip/davscale-573

job/davscale-573

job/integrations-review

wip/ocr-eval-584

job/ocr-eval-584

job/esc-537

wip/esc-537

job/toastname-586

wip/toastname-586

job/submenu-579

wip/submenu-579

job/tasks-mode

wip/tasks-mode

job/agentdocs-630

job/dupwrite-634

wip/agentdocs-630

wip/dupwrite-634

job/lightglass-588

wip/lightglass-588

job/tabswitch-549

job/ghosttask-623

wip/ghosttask-623

job/toaststack-616

job/weekstate-609

job/mailsync-613

wip/mailsync-613

wip/weekstate-609

job/maildup-626

wip/tabswitch-549

wip/maildup-626

wip/toaststack-616

job/motion-611

wip/motion-611

job/tlstest-601

wip/tlstest-601

job/perf-495

job/floating-sheet

wip/floating-sheet

job/remdup-585

wip/remdup-585

job/fix-502

wip/fix-502

job/attachplay-622

job/perf-batch

wip/perf-batch-563

wip/perf-495

hotfix/mail-sync-diag

job/mail-m3

wip/mail-m3

job/attach-poof-603

job/calhover-608

job/editorbar-604

job/mentions-605

job/merge-round-4

job/allday-514

wip/merge-round-4

wip/allday-514

job/merge-round-4a

wip/merge-round-4a

job/sharestack-580

job/fix-501

wip/sharestack-580

wip/fix-501

job/perf-batch-563

job/apw-cache-review

wip/apw-cache-review

job/probe-520

wip/probe-520

job/mac-393

wip/mac-393

job/header-571

job/flake-513

wip/flake-513

job/docs-thumb-547

wip/header-571

job/webcal-572

wip/webcal-572

wip/shortcuts-542

job/shortcuts-542

wip/docs-thumb-547

job/caldav-stress

wip/caldav-stress

wip/sweep-478

job/apw-cache-512

wip/apw-cache-512

job/money-empty-540

wip/restart-505

wip/money-empty-540

wip/fix-510

job/restart-505

job/fix-503

job/perf-496

wip/perf-496

job/fix-498

wip/fix-498

job/info-inspector-465

wip/info-inspector-465

job/fix-510

job/fix-507

wip/fix-507

wip/fix-503

job/fix-493

job/money-kinds

wip/money-kinds

job/hygiene-548

job/merge-round-3

wip/fix-493

job/drag-snap-536

wip/merge-round-3

wip/merge-round-0930

wip/drag-snap-536

job/align-538

wip/align-538

job/bg-flash

wip/bg-flash

job/money-import

job/search-count-544

wip/search-count-544

wip/money-import

job/settings-key-541

wip/settings-key-541

job/toast-539

job/preview-421

wip/preview-421

wip/toast-539

job/tasks-500-531

job/title-plain-526

wip/title-plain-526

wip/tasks-500-531

job/notes-bridge

wip/parity-484

job/parity-484

job/files-slow

job/crash-525

wip/notes-bridge

wip/files-slow

wip/crash-525

job/kbd-motion-527

wip/bg-422

job/analytics-504

wip/analytics-504

wip/kbd-motion-527

job/upload-pill-523

wip/upload-pill-523

wip/tray-order

job/tray-order

wip/overflow-mid

wip/merge-round-2

job/perf-494

wip/perf-494

wip/mcp-fast-492

wip/motion-477

wip/asr-ab-489

wip/theme-variants-506

wip/overflow-511

wip/week-header-508

wip/attach-427

job/dav-delete-471

job/iso-435

wip/iso-435

wip/files-sel-keys

wip/dav-delete-471

job/align-253

job/siwc-490

wip/siwc-490

job/money-kinds-review

wip/align-253

wip/money-kinds-review

job/small-bugs-3

wip/overlay-title-487

wip/multiget-500

wip/hidden-420

wip/webcal-ui

wip/webcal-431

job/perf-367

job/location

wip/small-bugs-3

wip/location

wip/perf-367

wip/admin-deny-483

job/tag-unicode-473

wip/tag-unicode-473

job/blur-436

wip/photos-470

wip/blur-436

wip/small-bugs-4

wip/hunt-20260930

wip/settings-hdr-482

wip/chips-416

job/dedup-375

wip/dedup-375

job/doc-stack

wip/doc-stack

job/tokens-literals

wip/tokens-literals

job/jobs-leftovers

wip/send-fast

wip/paste-467

wip/money-numbers

job/money-plugin

wip/money-plugin

job/break-dav

wip/merge-batch

wip/crossday-469

wip/mac-verify

wip/mail-m2

wip/break-dav

wip/money-review2

job/money-md

job/modes-424

wip/money-md

wip/jobs-leftovers

job/agenda-413

wip/agenda-413

wip/modes-424

job/recog-417

wip/recog-417

wip/bounce-425

wip/ab-384-luna

job/webdav-perf

wip/webdav-perf

job/toast-ring

wip/toast-ring

job/money-review

wip/money-review

wip/micro-motion

wip/settings-card

wip/minical

job/notes-imap-428

job/least-priv

wip/ui-small-2

wip/flaky-426

wip/drag-end-418

job/jank

wip/jank

wip/least-priv

wip/docs-site

job/agenda

job/sec-batch

wip/sec-batch

wip/per-user-index

job/area-calendars

wip/area-calendars

job/parity

wip/parity

job/documents-research

wip/documents-research

job/test-infra

job/reminders-sync

wip/small-bugs-2

wip/reminders-sync

wip/gestures

job/google-oauth

wip/tags-merge

wip/tags

job/e2e-theme

wip/e2e-theme

job/icon-align

wip/test-infra

wip/select-align

wip/editor-385

job/voice

wip/webdav

job/webdav

job/app-pw-ui

job/editor-integrity

wip/editor-integrity

wip/voice

wip/quota

wip/cal-followups

wip/icon-align

job/composer-scale

wip/composer-scale

job/jobs-page

wip/jobs-page

job/hig-type

wip/hig-type

wip/app-pw-ui

job/motion-spring

job/mcp

wip/motion-spring

wip/mcp

job/small-bugs

wip/push-hosts

job/profile-sign

wip/touch-369

wip/profile-sign

job/mobile-focus

wip/mobile-focus

wip/ui-polish-354

wip/small-bugs

wip/dup-task

job/toast-polish

job/app-pw-scopes

wip/toast-polish

wip/app-pw-scopes

wip/cli-agent

wip/selection-pills

job/preview-attach

wip/preview-attach

job/dav-proppatch

wip/dav-proppatch

wip/cal-switcher

job/atomic-race

wip/atomic-race

job/photos-shared

wip/photos-shared

wip/cal-grid

wip/note-rewrite

wip/search-rebuild

job/mail-m1

job/paperless-import

wip/paperless-import

wip/mail-m1

wip/hidden-activity

wip/search-d

wip/pricing-research

wip/cursors

wip/auto-scheme

job/single-pills

wip/single-pills

wip/xuser-matrix

wip/money-format

wip/app-pw-setup

wip/purge-dos

wip/vault-health

wip/caldav-apple

wip/xuser-audit

wip/e2e-green

wip/tabbar

wip/adv-harness

wip/maple-mono

job/search-fix

wip/search-fix

wip/search-perf-c

job/adv-harness

wip/sidebar-headers

job/glass

wip/temp-index

job/polish

wip/polish

wip/file-protocols

wip/money-research

wip/glass

wip/voice-models

wip/collab-redo

job/voice-research

wip/hunt-20260928

wip/notes-actions-research

wip/search-pad

wip/search-perf

wip/search-sticky

wip/editor-undo

wip/chrome-rules

wip/motion

wip/appearance-research

wip/appearance

wip/audit-bugs

wip/cal-glass

wip/block-actions

wip/authz-order

wip/event-stripes

wip/chrome-sidebar

wip/auth-flaky

wip/robust-2

wip/gate-fix

wip/menu-blur

wip/import-calternaljs

wip/tray-fix

job/import-calternaljs

wip/index-order

wip/audit-fixes

wip/search-chevrons

research/mail

wip/phone-chrome

wip/dedup-break

wip/csp

wip/ui-audit

wip/select-toast

wip/perf

wip/flat-layout

wip/fonts

wip/event-tint

wip/sync-converge

wip/data-split

wip/glass-audit

wip/robustness

wip/sync-chaos

wip/search-thumbs

wip/fuzz

wip/menu-icons

wip/search-pill

wip/sync-changing

wip/heading-links

wip/date-formats

wip/a11y

wip/break-editor

wip/e2e-fix

wip/settings-sections

wip/sync-root-guard

wip/search-palette

wip/share-edit

job/toasts

wip/toasts

wip/cont-analytics

wip/authz-review

wip/popovers

wip/overlay-glass

wip/change-feed

wip/editor-modes

wip/composer-align

wip/cont-agenda

wip/agenda-merge

job/agent-conventions

wip/agent-conventions

wip/backend-misc

job/route-audit

wip/route-audit

wip/ui-batch

wip/heif-hardening

wip/grid-resize

wip/ask-page

wip/webmcp

job/deeplink-audit

wip/deeplinks

wip/shortcuts

wip/cont-tz-days

main

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#343

No description provided.