Bookmarks: capture storm request got no response within 30 s (possible hang/DoS under load)

kayg commented

2026-09-26 01:27:45 +00:00

Owner

Seen by job/public-dotfiles' full adversarial run (2026-09-26, load 50-70): round 1 'bookmark capture storm' reported -1 = no HTTP response within 30 s. A request that never answers is a potential DoS (owner rule: crashes and DoS block merges), unlike a SLOW response. Find what bookmark capture does per request (outbound fetch of the page? readability extraction? screenshot? lock held across network I/O?) and make it bounded: outbound timeouts, size caps, a concurrency limit per user with queueing or 429, and no lock held across I/O. Reproduce with the storm section under synthetic load; acceptance: every request answers (success or 429/503) within its deadline, server stays responsive to other routes during the storm.

kayg commented

2026-09-26 07:06:15 +00:00

Author

Owner

Starting #149 on branch job/robustness, based on dev SHA 7c1d6c82ea98ac3772117a534fbb923825a60bab.

I will trace bookmark capture I/O and locking, reproduce the capture storm, then bound outbound work and per-user request concurrency with a regression in tests/adversarial.

Starting #149 on branch `job/robustness`, based on dev SHA `7c1d6c82ea98ac3772117a534fbb923825a60bab`. I will trace bookmark capture I/O and locking, reproduce the capture storm, then bound outbound work and per-user request concurrency with a regression in `tests/adversarial`.

kayg commented

2026-09-26 07:12:12 +00:00

Author

Owner

Finding: bookmark capture runs fetch_clip before lock_user, so no per-user mutex is held during network I/O. The save path then waits on an unbounded per-user Mutex; the current CLIP_LIMIT only rejects excess Clip fetches and does not bound bookmark capture requests. I am adding a per-user request admission limit that returns 429 before the save queue grows, while keeping all mutex work after fetch completion.

Finding: bookmark capture runs `fetch_clip` before `lock_user`, so no per-user mutex is held during network I/O. The save path then waits on an unbounded per-user `Mutex`; the current `CLIP_LIMIT` only rejects excess Clip fetches and does not bound bookmark capture requests. I am adding a per-user request admission limit that returns 429 before the save queue grows, while keeping all mutex work after fetch completion.

kayg commented

2026-09-26 08:49:35 +00:00

Author

Owner

Regression evidence for the bookmark storm: a new unit test launches 16 capture handlers for one User while that User's write lock is held. Before the admission limit, the requests remained queued until the test timeout. With the limit, excess handlers return 429 before joining the lock queue; cargo test -p calternal-plugin-notes now passes bookmark_capture_storm_rejects_excess_requests_before_the_user_lock_queue.

Regression evidence for the bookmark storm: a new unit test launches 16 capture handlers for one User while that User's write lock is held. Before the admission limit, the requests remained queued until the test timeout. With the limit, excess handlers return 429 before joining the lock queue; `cargo test -p calternal-plugin-notes` now passes `bookmark_capture_storm_rejects_excess_requests_before_the_user_lock_queue`.

kayg referenced this issue

2026-09-26 10:50:17 +00:00

Adversarial API requests time out during concurrent Notes and sync probes #166

kayg commented

2026-09-26 10:53:59 +00:00

Author

Owner

The job/agenda full adversarial run observed the same POST /api/v1/notes/bookmarks storm behavior: 6 of 16 concurrent requests had no response by the runner's 30-second deadline (10/16 unique IDs returned), and the server process stayed alive. The host was heavily loaded, with concurrent builds/probes and load averages reaching about 85, so this run does not establish whether the endpoint also stalls under controlled load. Related cross-endpoint observations are tracked in #166.

The `job/agenda` full adversarial run observed the same `POST /api/v1/notes/bookmarks` storm behavior: 6 of 16 concurrent requests had no response by the runner's 30-second deadline (10/16 unique IDs returned), and the server process stayed alive. The host was heavily loaded, with concurrent builds/probes and load averages reaching about 85, so this run does not establish whether the endpoint also stalls under controlled load. Related cross-endpoint observations are tracked in #166.

kayg referenced this issue

2026-09-26 10:56:13 +00:00

Agenda view: flat list, attachment deck that fans on hover and expands with titles on select #112

kayg commented

2026-09-26 13:50:34 +00:00

Author

Owner

Additional evidence from the ask-page branch's one post-merge adversarial run on 2026-09-26 (head 32f9d1a9):

tests/adversarial/run.sh ran against a real local server.
In attack.py, the bookmark capture storm returned statuses [201, 201, 201, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]; only 3 of 16 unique IDs were returned.
The server process was alive at the end of the round.
Several independent builds and adversarial probes were active on the shared host. This is not a controlled-load replay and does not establish product cause, but it reproduces the no-response condition recorded here.

Additional evidence from the `ask-page` branch's one post-merge adversarial run on 2026-09-26 (head `32f9d1a9`): - `tests/adversarial/run.sh` ran against a real local server. - In `attack.py`, the bookmark capture storm returned statuses `[201, 201, 201, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]`; only 3 of 16 unique IDs were returned. - The server process was alive at the end of the round. - Several independent builds and adversarial probes were active on the shared host. This is not a controlled-load replay and does not establish product cause, but it reproduces the no-response condition recorded here.

kayg referenced this issue

2026-09-26 14:17:40 +00:00

Ask mode: a real Ask page (5th tray mode) with conversation history #150

kayg referenced this issue from a commit

2026-09-27 14:45:56 +00:00

Merge job/robustness (#148, #149)

kayg commented

2026-09-27 14:45:58 +00:00

Author

Owner

Merged in 57d1752c: 30 s pool timeout for all callers, 5 s writer-checkout deadline at HTTP boundary → 503 + Retry-After, bounded BUSY/LOCKED retries.

Merged in 57d1752c: 30 s pool timeout for all callers, 5 s writer-checkout deadline at HTTP boundary → 503 + Retry-After, bounded BUSY/LOCKED retries.

kayg closed this issue

2026-09-27 14:45:58 +00:00

Rows
Columns

Bookmarks: capture storm request got no response within 30 s (possible hang/DoS under load) #149