Task create requests can time out during concurrent load #171

New issue

Closed

opened 2026-09-26 12:59:10 +00:00 by kayg · 4 comments

kayg commented

2026-09-26 12:59:10 +00:00

Owner

A post-merge adversarial pass found Task-create requests that exceeded the client timeout during concurrent load.

Evidence from 2026-09-26:

tests/adversarial/attack.py measured five sequential Task creates at p50 4.047 seconds, then issued 24 POST /api/v1/notes/tasks requests with 12 workers and a 60-second per-request timeout.
Requests 14–19 and 23 returned NO RESPONSE (timed out) to the probe. This was not reported as SLOW because each exceeded the client timeout.
The server remained available. A count-only check after the storm found all 24 Task files and all 24 distinct Task source paths in the Index; no data loss or missing projection was observed.
The shared host also had several concurrent Rust builds, local servers, and adversarial probes. Resource contention is a plausible cause, but this run did not isolate the Task endpoint.

Please profile Task creation under controlled concurrent load and decide whether to reduce service latency or document the capacity limit. Reproduce with the Task storm in tests/adversarial/attack.py. No server behavior change is proposed from this single shared-host run.

A post-merge adversarial pass found Task-create requests that exceeded the client timeout during concurrent load. Evidence from 2026-09-26: - `tests/adversarial/attack.py` measured five sequential Task creates at p50 4.047 seconds, then issued 24 `POST /api/v1/notes/tasks` requests with 12 workers and a 60-second per-request timeout. - Requests 14–19 and 23 returned `NO RESPONSE (timed out)` to the probe. This was not reported as `SLOW` because each exceeded the client timeout. - The server remained available. A count-only check after the storm found all 24 Task files and all 24 distinct Task source paths in the Index; no data loss or missing projection was observed. - The shared host also had several concurrent Rust builds, local servers, and adversarial probes. Resource contention is a plausible cause, but this run did not isolate the Task endpoint. Please profile Task creation under controlled concurrent load and decide whether to reduce service latency or document the capacity limit. Reproduce with the Task storm in `tests/adversarial/attack.py`. No server behavior change is proposed from this single shared-host run.

kayg referenced this issue

2026-09-26 13:35:41 +00:00

Overlays: one shared glass surface for menus, dropdowns, popovers, sheets and the composer; more transparent #157

kayg commented

2026-09-26 22:20:16 +00:00

Author

Owner

Additional evidence from the post-merge real-server adversarial pass on 2026-09-27:

The Task storm requests numbered 8 through 11 returned HTTP 503 with service_unavailable and Authentication database is busy; retry shortly; the probe expected HTTP 201 for each. The server stayed alive. This is a defined overload response instead of the no-response timeout recorded earlier, but the four writes did not complete during this shared-host run. The run had other adversarial jobs active and does not isolate Task capacity.

Additional evidence from the post-merge real-server adversarial pass on 2026-09-27: The Task storm requests numbered 8 through 11 returned HTTP 503 with `service_unavailable` and `Authentication database is busy; retry shortly`; the probe expected HTTP 201 for each. The server stayed alive. This is a defined overload response instead of the no-response timeout recorded earlier, but the four writes did not complete during this shared-host run. The run had other adversarial jobs active and does not isolate Task capacity.

kayg commented

2026-09-28 15:58:31 +00:00

Author

Owner

Additional evidence from the one time-boxed real-server adversarial round on job/money-format at merged head 25e88ffccbab8e1cc41a67ddbc08ec9d5ccff050: the Task-create single-request baseline reported p50 0.200s with 0/5 successful; during a 24-request Task storm, one request returned 201 after 54.4s and the other 23 timed out without a response. The server was under a 20,000-file watcher/index load, so this needs a controlled-load replay. No task loss or duplicate was verified in this run.

Additional evidence from the one time-boxed real-server adversarial round on `job/money-format` at merged head `25e88ffccbab8e1cc41a67ddbc08ec9d5ccff050`: the Task-create single-request baseline reported p50 `0.200s` with `0/5` successful; during a 24-request Task storm, one request returned `201` after `54.4s` and the other 23 timed out without a response. The server was under a 20,000-file watcher/index load, so this needs a controlled-load replay. No task loss or duplicate was verified in this run.

kayg referenced this issue

2026-09-28 16:01:43 +00:00

SPIKE: Money plain-text format — can an hledger superset hold YNAB/Actual budgets losslessly? Else design a simple calternal format (NO product code) #335

kayg commented

2026-10-02 17:26:14 +00:00

Author

Owner

Duplicate run to #185: both use 24 concurrent POST /api/v1/notes/tasks requests with 12 workers and report client timeouts during the same shared-host workload. This issue also verifies that all 24 Task files and Index rows existed afterward. Recommend keeping #185 as the current storm report and linking this persistence check as evidence.