Investigate API request timeouts under adversarial concurrency

kayg commented

2026-09-27 21:10:04 +00:00

Owner

Evidence

During the one local adversarial run on 2026-09-27, several requests timed out under mixed API concurrency:

The bookmark capture storm sent 16 concurrent requests with a 10 second client timeout. Four returned no response; the other 12 returned 429.
The Journal conditional PATCH storm sent 24 concurrent requests. Sixteen timed out and eight returned 412. The probe expected one 200 and 23 conflicts.
DAV initial sync and the second recurring Calendar occurrence Log write each timed out.
An upload during User purge took 3.30 seconds against a 2 second probe limit. Twenty mkdir requests with 64 open SSE streams took 7.2 seconds.

Most other requests were reported as SLOW. The server stayed alive and the run reported no 5xx crash. Several other worktrees were running Cargo builds and adversarial suites at the same time; the server log also showed SQLx pool acquisition warnings over 3 seconds.

Follow-up

Please determine whether the no-response cases came from shared-host contention or server-side lock and queue behavior. Check that writes remain safe when the client times out and retries. This run was not repeated.

## Evidence During the one local adversarial run on 2026-09-27, several requests timed out under mixed API concurrency: - The bookmark capture storm sent 16 concurrent requests with a 10 second client timeout. Four returned no response; the other 12 returned 429. - The Journal conditional PATCH storm sent 24 concurrent requests. Sixteen timed out and eight returned 412. The probe expected one 200 and 23 conflicts. - DAV initial sync and the second recurring Calendar occurrence Log write each timed out. - An upload during User purge took 3.30 seconds against a 2 second probe limit. Twenty mkdir requests with 64 open SSE streams took 7.2 seconds. Most other requests were reported as SLOW. The server stayed alive and the run reported no 5xx crash. Several other worktrees were running Cargo builds and adversarial suites at the same time; the server log also showed SQLx pool acquisition warnings over 3 seconds. ## Follow-up Please determine whether the no-response cases came from shared-host contention or server-side lock and queue behavior. Check that writes remain safe when the client times out and retries. This run was not repeated.

kayg referenced this issue

2026-09-27 21:10:43 +00:00

RESEARCH: mail plugin (from mailternal) — decisions, architecture options, grill questions (NO implementation) #257

kayg referenced this issue

2026-09-27 21:39:18 +00:00

RESEARCH: mail plugin (from mailternal) — decisions, architecture options, grill questions (NO implementation) #257

kayg commented

2026-09-27 22:03:30 +00:00

Author

Owner

Evidence from the single chrome-sidebar adversarial round on 2026-09-27: DAV initial sync timed out at the probe's 30-second request timeout, and Calendar Event-from-Log also timed out. The server remained alive; follow-on DAV REPORTs returned 207/412 but took 5.2–8.9 seconds, and the remaining Calendar/Journal requests completed with expected statuses but several 8–23 second SLOW latencies. Multiple other adversarial/build/performance jobs were active on the shared host. No 5xx or server crash was observed in this round. This run was not repeated.

Evidence from the single chrome-sidebar adversarial round on 2026-09-27: `DAV initial sync` timed out at the probe's 30-second request timeout, and Calendar Event-from-Log also timed out. The server remained alive; follow-on DAV REPORTs returned 207/412 but took 5.2–8.9 seconds, and the remaining Calendar/Journal requests completed with expected statuses but several 8–23 second SLOW latencies. Multiple other adversarial/build/performance jobs were active on the shared host. No 5xx or server crash was observed in this round. This run was not repeated.

kayg referenced this issue

2026-09-27 22:16:27 +00:00

Chrome: sidebar and H1 never repeat each other; Files breadcrumb; Pinned first with drag-to-pin; calm H1 transitions #156

kayg referenced this issue

2026-09-28 11:01:23 +00:00

Adversarial attack2-only run uses deleted users and skips restart coordination #285

kayg commented

2026-09-28 11:24:00 +00:00

Author

Owner

Additional evidence from the single post-merge adversarial round for #319:

Branch: job/single-pills, head 46497b6a7c03eb02a7cbbbfac3f36b563c906af7.
While tests/adversarial/attack.py ran against the local server, it reported GET /api/v1/files/download ['..'] and POST /api/v1/files/mkdir ['..'] as NO RESPONSE (client timeout).
GET /api/v1/files/versions ['..'] returned 400 after 18.3 seconds; the probe classified this as SLOW.
Other cargo and adversarial processes were active on the shared host during the round, so the no-response results need a quiet-host reproduction.

This is a further no-response case for the timeout investigation already tracked here.

Additional evidence from the single post-merge adversarial round for #319: - Branch: `job/single-pills`, head `46497b6a7c03eb02a7cbbbfac3f36b563c906af7`. - While `tests/adversarial/attack.py` ran against the local server, it reported `GET /api/v1/files/download ['..']` and `POST /api/v1/files/mkdir ['..']` as `NO RESPONSE` (client timeout). - `GET /api/v1/files/versions ['..']` returned 400 after 18.3 seconds; the probe classified this as SLOW. - Other cargo and adversarial processes were active on the shared host during the round, so the no-response results need a quiet-host reproduction. This is a further no-response case for the timeout investigation already tracked here.

kayg commented

2026-09-28 11:27:31 +00:00

Author

Owner

Additional evidence from the single-pills adversarial run at 46497b6a7c03eb02a7cbbbfac3f36b563c906af7: tests/adversarial/authz_matrix.py stopped during fixture setup because its upload request returned -1 (no HTTP response). The worktree was running alongside other build and adversarial jobs on the shared host, so this does not identify a product cause. The same run also recorded the two API request timeouts noted in my earlier comment.

Additional evidence from the single-pills adversarial run at `46497b6a7c03eb02a7cbbbfac3f36b563c906af7`: `tests/adversarial/authz_matrix.py` stopped during fixture setup because its upload request returned `-1` (no HTTP response). The worktree was running alongside other build and adversarial jobs on the shared host, so this does not identify a product cause. The same run also recorded the two API request timeouts noted in my earlier comment.

kayg commented

2026-09-28 11:40:00 +00:00

Author

Owner

Additional evidence from the same single-pills adversarial run at head 46497b6a7c03eb02a7cbbbfac3f36b563c906af7: attack.py reported DAV incremental sync: NO RESPONSE (timed out). Nearby DAV discovery, initial sync, invalid token, create, query, multiget and move probes returned their expected statuses, with several marked SLOW. Other builds and adversarial runners were active on the shared host, so this is another timeout observation without cause attribution.

Additional evidence from the same single-pills adversarial run at head `46497b6a7c03eb02a7cbbbfac3f36b563c906af7`: `attack.py` reported `DAV incremental sync: NO RESPONSE (timed out)`. Nearby DAV discovery, initial sync, invalid token, create, query, multiget and move probes returned their expected statuses, with several marked SLOW. Other builds and adversarial runners were active on the shared host, so this is another timeout observation without cause attribution.

kayg referenced this issue

2026-09-28 13:04:18 +00:00

UI: every lone icon button is a pill-in-a-pill (shared single PillGroup) — sidebar toggle first, then hunt all instances #319

kayg commented

2026-09-28 15:58:32 +00:00

Author

Owner

Additional evidence from the one time-boxed real-server adversarial round on job/money-format at merged head 25e88ffccbab8e1cc41a67ddbc08ec9d5ccff050: while a 20,000-file watcher/index fixture and request storms were active, Files download for .., Files mkdir for .., Files versions for .., and encoded traversal variants timed out without a response. Cross-plugin Note and Task create requests also timed out. An items/.. read returned 503 service_unavailable with Authentication database is busy; retry shortly. The probe did not establish that any hostile path was accepted; these observations are confounded by the concurrent indexing load and need a controlled replay.

Additional evidence from the one time-boxed real-server adversarial round on `job/money-format` at merged head `25e88ffccbab8e1cc41a67ddbc08ec9d5ccff050`: while a 20,000-file watcher/index fixture and request storms were active, Files download for `..`, Files mkdir for `..`, Files versions for `..`, and encoded traversal variants timed out without a response. Cross-plugin Note and Task create requests also timed out. An `items/..` read returned `503 service_unavailable` with `Authentication database is busy; retry shortly`. The probe did not establish that any hostile path was accepted; these observations are confounded by the concurrent indexing load and need a controlled replay.

kayg referenced this issue

2026-09-28 16:01:43 +00:00

SPIKE: Money plain-text format — can an hledger superset hold YNAB/Actual budgets losslessly? Else design a simple calternal format (NO product code) #335

kayg referenced this issue

2026-10-02 17:35:52 +00:00

Investigate Calendar Event from Log timeout under shared load #250

kayg commented

2026-10-02 17:36:00 +00:00

Author

Owner

Partial duplicate symptom with #250: both report Calendar Event-from-Log creation timing out at the 30-second client deadline during a shared-host adversarial run. This issue also reports distinct bookmark, Journal PATCH, DAV and SSE latency results. Recommend linking only the Event-from-Log observation to #250 and keeping the other findings separate.

kayg referenced this issue

2026-10-02 17:42:29 +00:00

Audit: open issue hygiene (duplicates, already fixed, stale) #866

kayg commented

2026-10-03 09:32:14 +00:00

Author

Owner

Merge-round 7a follow-up on head 516faaa698. The real local release server stayed up during the full adversarial API round, but several write paths produced client timeouts under the shared-host load (load average around 40–50):

Block reminder creation timed out at request 90 while filling the configured limit.
A Files contention TUS install timed out on item 7.
A DAV child-delete fixture timed out, and one authorized PUT in the DAV lock storm timed out.
Seven of 20 Calendar Journal duplicate writes and 18 of 40 saved-search creates timed out; those are also reported on the existing subsystem issues.

The server log showed SQLx pool-acquisition waits up to 7.7 seconds. Some probes only counted returned IDs and did not read back every timed-out write, so this evidence does not establish data loss. No server crash was observed. This is additional shared-load evidence for the open concurrency investigation.

Merge-round 7a follow-up on head 516faaa698570bdb468626cf6cd75d9c81b33ac2. The real local release server stayed up during the full adversarial API round, but several write paths produced client timeouts under the shared-host load (load average around 40–50): - Block reminder creation timed out at request 90 while filling the configured limit. - A Files contention TUS install timed out on item 7. - A DAV child-delete fixture timed out, and one authorized PUT in the DAV lock storm timed out. - Seven of 20 Calendar Journal duplicate writes and 18 of 40 saved-search creates timed out; those are also reported on the existing subsystem issues. The server log showed SQLx pool-acquisition waits up to 7.7 seconds. Some probes only counted returned IDs and did not read back every timed-out write, so this evidence does not establish data loss. No server crash was observed. This is additional shared-load evidence for the open concurrency investigation.

kayg commented

2026-10-03 10:28:25 +00:00

Author

Owner

Merge-round 7a full adversarial evidence: round 2 recorded 60 no-response composer atomic log storm requests, a worker timeout at 120 s, and only 96/100 completions; the analytics burst recorded 11 no-response requests. The same round saw a Sync initial-upload deadline miss for a.txt and keep/k0..k3, and a Files TUS PATCH returned 503 Index is busy; retry shortly (reported separately on #960). Host load during these phases was high (47.19/44.71/43.12), so these timings are load-contaminated; no committed Search hits or cross-user data leaked. Full outputs are in #427's target/tmp/verify-7a-adversarial.log.

Merge-round 7a full adversarial evidence: round 2 recorded 60 no-response `composer atomic log storm` requests, a worker timeout at 120 s, and only 96/100 completions; the analytics burst recorded 11 no-response requests. The same round saw a Sync initial-upload deadline miss for `a.txt` and `keep/k0..k3`, and a Files TUS PATCH returned 503 `Index is busy; retry shortly` (reported separately on #960). Host load during these phases was high (47.19/44.71/43.12), so these timings are load-contaminated; no committed Search hits or cross-user data leaked. Full outputs are in #427's `target/tmp/verify-7a-adversarial.log`.

kayg commented

2026-10-03 10:46:19 +00:00

Author

Owner

Additional full-round evidence: the Notes IMAP abuse probe aborted on a 15-second TLS read timeout while waiting for the tagged response to a valid APPEND after the server had accepted the literal continuation. The probe did not read the mailbox afterward, so commit/duplicate behavior is unknown. This is a non-SLOW timeout under the same heavily loaded run; it is recorded as an inconclusive API timeout, not a confirmed data loss.

Additional full-round evidence: the Notes IMAP abuse probe aborted on a 15-second TLS read timeout while waiting for the tagged response to a valid `APPEND` after the server had accepted the literal continuation. The probe did not read the mailbox afterward, so commit/duplicate behavior is unknown. This is a non-SLOW timeout under the same heavily loaded run; it is recorded as an inconclusive API timeout, not a confirmed data loss.

kayg commented

2026-10-03 10:58:55 +00:00

Author

Owner

The final #957 authorization matrix completed 2,216 requests but logged no responses for three non-Search operations under host load 37.50/40.48/40.86: GET /api/v1/admin/user-archives using a valid Upload-Only App Password, POST /api/v1/notes/linked-notes as a standard User, and DELETE /api/v1/notifications/subscriptions/{installation_id} as a standard User. These are availability timeouts; the matrix found no authorization mismatch. Full output is in #427's run log.

The final #957 authorization matrix completed 2,216 requests but logged no responses for three non-Search operations under host load 37.50/40.48/40.86: `GET /api/v1/admin/user-archives` using a valid Upload-Only App Password, `POST /api/v1/notes/linked-notes` as a standard User, and `DELETE /api/v1/notifications/subscriptions/{installation_id}` as a standard User. These are availability timeouts; the matrix found no authorization mismatch. Full output is in #427's run log.

kayg referenced this issue

2026-10-03 12:19:06 +00:00

Log attachments: deleted file still shown, attachments not live, batch send not live, Log order, raw camera names #427

kayg referenced this issue

2026-10-03 12:56:40 +00:00

Issue hygiene sweep 2026-10-03 #1008

Rows
Columns

Investigate API request timeouts under adversarial concurrency #269

Evidence

Follow-up