Task creation stalls during a concurrent request storm #185

New issue

Open

opened 2026-09-26 15:02:12 +00:00 by kayg · 2 comments

kayg commented

2026-09-26 15:02:12 +00:00

Owner

Finding from the local real-server adversarial run on 2026-09-26.

tests/adversarial/attack.py measured a Task create baseline p50 of 9.100 seconds (5/5 successful). It then sent 24 POST /api/v1/notes/tasks requests with 12 concurrent workers and a 60 second client timeout. Requests numbered 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, and 23 returned NO RESPONSE (timed out). The suite continued into DAV requests afterward.

The shared host was heavily loaded during this probe (load average 52–58, with multiple Calternal servers and adversarial runs active). This is a non-SLOW result because the clients received no response, so it needs a quiet-host reproduction. Check that the Task create storm completes without stalled requests and that every accepted task is persisted exactly once.

Finding from the local real-server adversarial run on 2026-09-26. `tests/adversarial/attack.py` measured a Task create baseline p50 of 9.100 seconds (5/5 successful). It then sent 24 `POST /api/v1/notes/tasks` requests with 12 concurrent workers and a 60 second client timeout. Requests numbered 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, and 23 returned `NO RESPONSE (timed out)`. The suite continued into DAV requests afterward. The shared host was heavily loaded during this probe (load average 52–58, with multiple Calternal servers and adversarial runs active). This is a non-SLOW result because the clients received no response, so it needs a quiet-host reproduction. Check that the Task create storm completes without stalled requests and that every accepted task is persisted exactly once.

kayg referenced this issue

2026-09-26 15:02:22 +00:00

Perf: concurrent Appearance/settings writes serialize on one settings.json (5-6 s under a 31-PUT burst) #114

kayg commented

2026-09-27 06:13:25 +00:00

Author

Owner

Post-merge adversarial update from 2026-09-27 (branch job/appearance, merge head b115ddf34863b0f3759d856208ce8a93013bed75):

Five sequential POST /api/v1/notes/tasks requests had p50 1.817s (5/5 success); the probe's concurrency SLOW threshold was 45.41s.
With 12 workers and a 60s client timeout, requests 20–23 returned no response. The server remained alive at the end of the round.
The probe did not verify whether those four timed-out requests committed, so this run does not establish whether task files were persisted exactly once.
The same run's journal race answered all 40 task creates with HTTP 201 and reported 46 entries including the six seed entries; no request timed out in that section.

This repeats the existing concurrent Task-create finding. It ran on the shared, heavily contended host, so it still needs a quiet-host reproduction. No server behavior was changed in this appearance job.

Post-merge adversarial update from 2026-09-27 (branch `job/appearance`, merge head `b115ddf34863b0f3759d856208ce8a93013bed75`): - Five sequential `POST /api/v1/notes/tasks` requests had p50 1.817s (5/5 success); the probe's concurrency SLOW threshold was 45.41s. - With 12 workers and a 60s client timeout, requests 20–23 returned no response. The server remained alive at the end of the round. - The probe did not verify whether those four timed-out requests committed, so this run does not establish whether task files were persisted exactly once. - The same run's journal race answered all 40 task creates with HTTP 201 and reported 46 entries including the six seed entries; no request timed out in that section. This repeats the existing concurrent Task-create finding. It ran on the shared, heavily contended host, so it still needs a quiet-host reproduction. No server behavior was changed in this appearance job.

kayg referenced this issue

2026-09-27 06:13:26 +00:00

Appearance: one Theme card with per-mode theme dropdowns and backgrounds; shared borderless segmented pill everywhere #183

kayg referenced this issue

2026-09-27 07:11:55 +00:00

Appearance: one Theme card with per-mode theme dropdowns and backgrounds; shared borderless segmented pill everywhere #183

kayg referenced this issue

2026-10-02 17:26:14 +00:00

Task create requests can time out during concurrent load #171

kayg commented

2026-10-02 17:26:15 +00:00

Author

Owner

Duplicate run to #171: both use 24 concurrent POST /api/v1/notes/tasks requests with 12 workers and report client timeouts during the same shared-host workload. #171 verifies that all 24 Task files and Index rows existed afterward. Recommend keeping #185 as the current storm report and linking #171 as its persistence check.