kayg/calternal

Watch

Fork 0

Investigate Calendar and Task projection timeouts in adversarial bursts #709

New issue

Closed

opened 2026-10-02 09:21:35 +00:00 by kayg · 3 comments

kayg commented

2026-10-02 09:21:35 +00:00

Owner

Found during the one local adversarial round for calternal at head 9968f4d3f. The server stayed alive, but several non-SLOW fixture operations timed out or returned incomplete projections during the burst phase:

Block Reminder setup timed out at item 7 after items 0–6 returned 201.
Calendar duplicate concurrency returned 12 of 20 requests; the remaining 8 timed out, and the probe saw only 12 distinct IDs.
Calendar duplicate Task source timed out.
Bookmark Calendar projection timed out three times and the probe could not find the Note on its saved date.
Calendar items burst expected 120 Photos Items but saw 0.

The host load average was about 22 during this run, with other Cargo and browser jobs active. The probe also reported many successful responses as SLOW. Please reproduce these consistency and timeout results on a quiet host before assigning a product cause. The one-time adversarial run was stopped in dedup setup because this invocation had no built CLI binary.

Found during the one local adversarial round for calternal at head `9968f4d3f`. The server stayed alive, but several non-SLOW fixture operations timed out or returned incomplete projections during the burst phase: - Block Reminder setup timed out at item 7 after items 0–6 returned 201. - Calendar duplicate concurrency returned 12 of 20 requests; the remaining 8 timed out, and the probe saw only 12 distinct IDs. - Calendar duplicate Task source timed out. - Bookmark Calendar projection timed out three times and the probe could not find the Note on its saved date. - Calendar items burst expected 120 Photos Items but saw 0. The host load average was about 22 during this run, with other Cargo and browser jobs active. The probe also reported many successful responses as SLOW. Please reproduce these consistency and timeout results on a quiet host before assigning a product cause. The one-time adversarial run was stopped in dedup setup because this invocation had no built CLI binary.

kayg referenced this issue

2026-10-02 09:37:02 +00:00

Calendar: new Task shows on the previous day and in the all-day lane; Tasks without dates should appear at the time they were added #655

kayg commented

2026-10-02 11:13:16 +00:00

Author

Owner

Starting #709 on branch job/burst-709, based on origin/dev at c4a61e8cf090170f35b1bed3350d9de20c83ecd5 (HEAD c4a61e8cf090170f35b1bed3350d9de20c83ecd5). I am reproducing the listed projection cases on the locked perf VM, then I will classify each result and add regressions for any product bugs found.

Starting #709 on branch `job/burst-709`, based on `origin/dev` at `c4a61e8cf090170f35b1bed3350d9de20c83ecd5` (HEAD `c4a61e8cf090170f35b1bed3350d9de20c83ecd5`). I am reproducing the listed projection cases on the locked perf VM, then I will classify each result and add regressions for any product bugs found.

kayg referenced this issue from a commit

2026-10-02 12:01:20 +00:00

test(adversarial): isolate #709 projection bursts

kayg commented

2026-10-02 12:38:57 +00:00

Author

Owner

Final report — #709

Branch: job/burst-709
Base: c4a61e8cf090170f35b1bed3350d9de20c83ecd5
Head: c421756ac9af5a9b653b7c9f53c41b3aca24de89

Built

Added tests/adversarial/issue_709_projection.mjs. It exercises the real API and Index for block reminders 0–7, 20 concurrent Calendar Journal duplicates followed by a standalone Task source and duplicate, three bookmark captures and parallel Calendar Items reads, and 120 TUS Photos uploads with immediate and settled Calendar Items checks. It reports status and latency without logging credentials or response bodies. It does not retry writes; an idempotent read retries once only after 503 Retry-After: 1.

Quiet perf VM results

Each case ran while holding /root/perf.lock. The full /proc/loadavg value is shown before and after each measured run.

Block reminders, items 0–7: all 8 returned 201; the reminder projection contained all 8. p50 94.2 ms, p95/max 247.6 ms. Load: 0.00 0.02 0.20 1/208 77114 → 0.00 0.02 0.20 1/208 77135.
Calendar duplicates, concurrency 20: all 20 returned 201; IDs were distinct and visible in the Journal projection. p50 645.8 ms, p95 1355.8 ms, max 1415.8 ms. The Task source and duplicate immediately afterward both returned 201 in 111.4/94.9 ms. Load: 0.00 0.02 0.20 1/209 77150 → 0.00 0.02 0.19 3/215 77177.
Bookmark projection: all 3 captures returned 201; all 3 Calendar Items reads returned 200 and contained the saved Notes. Capture p50 96.6 ms, p95/max 179.2 ms; read p50 126.2 ms, p95 145.8 ms, max 145.8 ms. Load: 0.04 0.07 0.17 1/218 77379 → 0.10 0.08 0.17 1/218 77394.
Photos burst: all 120 uploads returned 204. Calendar Items showed 120 immediately and 120 after a 0.2 ms poll; upload wall time was 11205.3 ms. Upload p50 449.6 ms, p95 844.5 ms, max 1000.4 ms. Load: 0.00 0.01 0.18 1/213 77219 → 0.35 0.11 0.20 1/217 77341.

No request timed out, no projection was incomplete, and no 503 backpressure response was needed. I classify the reported symptoms as load artefacts for this quiet-host reproduction. The original issue records host load around 22; this run does not establish behavior at that load.

Gates

cargo build --release -p calternal-server: Finished release profile [optimized] target(s) in 39m 15s
cargo fmt --check: no output; exit 0.
node --check tests/adversarial/issue_709_projection.mjs: no output; exit 0.
git diff --check: no output; exit 0.
cargo clippy and cargo test: not run; no Rust crate source changed.
cargo clean: Removed 7721 files, 2.6GiB total

Known gaps

This pass confirms behavior on the quiet perf VM. It does not reproduce the original load-22 condition. No UI or screenshot check applied to this API and projection probe.

Decisions

No production behavior change was justified because none of the reported failures reproduced. The probe treats only an explicit 503 Retry-After: 1 (or the bookmark route's explicit 429 limit) as controlled backpressure; write requests are never retried.

# Final report — #709 Branch: `job/burst-709` Base: `c4a61e8cf090170f35b1bed3350d9de20c83ecd5` Head: `c421756ac9af5a9b653b7c9f53c41b3aca24de89` ## Built Added `tests/adversarial/issue_709_projection.mjs`. It exercises the real API and Index for block reminders 0–7, 20 concurrent Calendar Journal duplicates followed by a standalone Task source and duplicate, three bookmark captures and parallel Calendar Items reads, and 120 TUS Photos uploads with immediate and settled Calendar Items checks. It reports status and latency without logging credentials or response bodies. It does not retry writes; an idempotent read retries once only after `503 Retry-After: 1`. ## Quiet perf VM results Each case ran while holding `/root/perf.lock`. The full `/proc/loadavg` value is shown before and after each measured run. - Block reminders, items 0–7: all 8 returned 201; the reminder projection contained all 8. p50 94.2 ms, p95/max 247.6 ms. Load: `0.00 0.02 0.20 1/208 77114` → `0.00 0.02 0.20 1/208 77135`. - Calendar duplicates, concurrency 20: all 20 returned 201; IDs were distinct and visible in the Journal projection. p50 645.8 ms, p95 1355.8 ms, max 1415.8 ms. The Task source and duplicate immediately afterward both returned 201 in 111.4/94.9 ms. Load: `0.00 0.02 0.20 1/209 77150` → `0.00 0.02 0.19 3/215 77177`. - Bookmark projection: all 3 captures returned 201; all 3 Calendar Items reads returned 200 and contained the saved Notes. Capture p50 96.6 ms, p95/max 179.2 ms; read p50 126.2 ms, p95 145.8 ms, max 145.8 ms. Load: `0.04 0.07 0.17 1/218 77379` → `0.10 0.08 0.17 1/218 77394`. - Photos burst: all 120 uploads returned 204. Calendar Items showed 120 immediately and 120 after a 0.2 ms poll; upload wall time was 11205.3 ms. Upload p50 449.6 ms, p95 844.5 ms, max 1000.4 ms. Load: `0.00 0.01 0.18 1/213 77219` → `0.35 0.11 0.20 1/217 77341`. No request timed out, no projection was incomplete, and no 503 backpressure response was needed. I classify the reported symptoms as load artefacts for this quiet-host reproduction. The original issue records host load around 22; this run does not establish behavior at that load. ## Gates - `cargo build --release -p calternal-server`: `Finished `release` profile [optimized] target(s) in 39m 15s` - `cargo fmt --check`: no output; exit 0. - `node --check tests/adversarial/issue_709_projection.mjs`: no output; exit 0. - `git diff --check`: no output; exit 0. - `cargo clippy` and `cargo test`: not run; no Rust crate source changed. - `cargo clean`: ` Removed 7721 files, 2.6GiB total` ## Known gaps This pass confirms behavior on the quiet perf VM. It does not reproduce the original load-22 condition. No UI or screenshot check applied to this API and projection probe. ## Decisions No production behavior change was justified because none of the reported failures reproduced. The probe treats only an explicit `503 Retry-After: 1` (or the bookmark route's explicit 429 limit) as controlled backpressure; write requests are never retried.

kayg commented

2026-10-02 12:40:29 +00:00

Author

Owner

Correction to the Gates section above: these command outputs were exactly:

cargo build --release -p calternal-server:

Finished `release` profile [optimized] target(s) in 39m 15s

cargo clean:

     Removed 7721 files, 2.6GiB total

Correction to the Gates section above: these command outputs were exactly: `cargo build --release -p calternal-server`: ``` Finished `release` profile [optimized] target(s) in 39m 15s ``` `cargo clean`: ``` Removed 7721 files, 2.6GiB total ```

kayg closed this issue

2026-10-02 12:44:46 +00:00

kayg referenced this issue

2026-10-03 13:13:42 +00:00

Issue hygiene sweep 2026-10-03 #1008

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#709

No description provided.

Rows
Columns