[adversarial] Review API probe timeouts under mixed load #387

Closed
opened 2026-09-29 02:51:29 +00:00 by kayg · 6 comments
Owner

Evidence

One real-server adversarial API round on 2026-09-29, after merging dev into job/app-pw-ui at de4531ce:

ADVERSARIAL_SERVER_BIN=/mnt/hdd/targets/jobs/app-pw-ui/debug/calternal-server \
ADVERSARIAL_SKIP_WEB_BUILD=1 ADVERSARIAL_API_ONLY=1 \
tests/adversarial/run.sh

The round finished with exit code 1. The server reported alive at the end. The runner removed its temporary fixture directory. It printed no credentials.

  • tests/adversarial/attack.py:894: a 3 MiB PUT /api/v1/appearance was expected to return 413. The local proxy returned 502 with local adversarial server is unavailable.
  • tests/adversarial/attack.py:3307-3320: two POST /api/v1/calendar/events/{id}/log-this requests timed out during the mixed-load run, after the 120-photo upload burst.
  • The Journal PATCH concurrency storm returned 200 once and 412 for 17 requests, but 6 of 24 requests timed out. A later Journal read returned 200.
  • tests/adversarial/attack.py:2552-2555: DAV discovery returned 207, but the probe did not find calendar-home-set in the response.

The round also reports that a DAV Event containing VALARM returned 201. The merged dev branch includes the Apple Reminders work in #358, which intentionally accepts and persists alarms. Review that existing adversarial expectation before changing behavior; this UI job did not change the API or its assertions.

Latency-only findings are omitted. These results came from the mixed hostile-input and concurrency run on a shared build host, so please check the response and logs before deciding whether they show a product issue or a load-sensitive probe result.

## Evidence One real-server adversarial API round on 2026-09-29, after merging `dev` into `job/app-pw-ui` at `de4531ce`: ```sh ADVERSARIAL_SERVER_BIN=/mnt/hdd/targets/jobs/app-pw-ui/debug/calternal-server \ ADVERSARIAL_SKIP_WEB_BUILD=1 ADVERSARIAL_API_ONLY=1 \ tests/adversarial/run.sh ``` The round finished with exit code 1. The server reported alive at the end. The runner removed its temporary fixture directory. It printed no credentials. - `tests/adversarial/attack.py:894`: a 3 MiB `PUT /api/v1/appearance` was expected to return 413. The local proxy returned 502 with `local adversarial server is unavailable`. - `tests/adversarial/attack.py:3307-3320`: two `POST /api/v1/calendar/events/{id}/log-this` requests timed out during the mixed-load run, after the 120-photo upload burst. - The Journal PATCH concurrency storm returned 200 once and 412 for 17 requests, but 6 of 24 requests timed out. A later Journal read returned 200. - `tests/adversarial/attack.py:2552-2555`: DAV discovery returned 207, but the probe did not find `calendar-home-set` in the response. The round also reports that a DAV Event containing `VALARM` returned 201. The merged `dev` branch includes the Apple Reminders work in #358, which intentionally accepts and persists alarms. Review that existing adversarial expectation before changing behavior; this UI job did not change the API or its assertions. Latency-only findings are omitted. These results came from the mixed hostile-input and concurrency run on a shared build host, so please check the response and logs before deciding whether they show a product issue or a load-sensitive probe result.
Author
Owner

Initial harness finding: the API probe in tests/adversarial/attack.py sends requests through editor-proxy.mjs. The proxy maps any backend connection error to a synthetic 502 body (local adversarial server is unavailable), so the reported 3 MiB PUT result does not show whether the API sent 413 or the proxy lost its upstream connection. I will route the API-only round directly to the real backend to separate API behavior from this proxy failure mode. The Journal and Calendar timeouts remain unclassified until that round completes.

Initial harness finding: the API probe in `tests/adversarial/attack.py` sends requests through `editor-proxy.mjs`. The proxy maps any backend connection error to a synthetic 502 body (`local adversarial server is unavailable`), so the reported 3 MiB PUT result does not show whether the API sent 413 or the proxy lost its upstream connection. I will route the API-only round directly to the real backend to separate API behavior from this proxy failure mode. The Journal and Calendar timeouts remain unclassified until that round completes.
Author
Owner

Additional mixed-load evidence from job/sec-batch at HEAD edce1e84: the real-server API campaign completed the xuser matrix and DAV Journal probes while the server stayed alive, then the 120-photo upload burst reported NO RESPONSE (b'timed out') for upload requests around burst-062 through burst-064. Other uploads returned 201/204 but took 5–19 s. This happened while the shared host ran multiple builds; the server remained running. The same round saw a proxy-generated 502 on a 200 KB Appearance body, which I fixed in edce1e84 with a focused old-code 502/new-code 413 regression. This does not resolve the separate 3 MiB Appearance case or classify the photo timeouts; #387 remains the follow-up for mixed-load transport behavior.

Additional mixed-load evidence from job/sec-batch at HEAD edce1e84: the real-server API campaign completed the xuser matrix and DAV Journal probes while the server stayed alive, then the 120-photo upload burst reported `NO RESPONSE (b'timed out')` for upload requests around burst-062 through burst-064. Other uploads returned 201/204 but took 5–19 s. This happened while the shared host ran multiple builds; the server remained running. The same round saw a proxy-generated 502 on a 200 KB Appearance body, which I fixed in edce1e84 with a focused old-code 502/new-code 413 regression. This does not resolve the separate 3 MiB Appearance case or classify the photo timeouts; #387 remains the follow-up for mixed-load transport behavior.
Author
Owner

Final note from the 20-minute sec-batch round: the server remained alive through the xuser matrix (280 operations classified, 455 comparisons) and the DAV Journal alarm checks. After the 120-photo upload burst, attack.py also logged timeouts for Calendar Event-from-Log, the second Log this first write, Journal patch target creation, and Journal DELETE with stale condition. I ended the broad probe at its time box while those unrelated mixed-load requests were still running; the runner exited 1 from that termination and cleaned up the server. These observations extend #387 and are not classified as a product crash or a successful hostile write.

Final note from the 20-minute sec-batch round: the server remained alive through the xuser matrix (280 operations classified, 455 comparisons) and the DAV Journal alarm checks. After the 120-photo upload burst, `attack.py` also logged timeouts for Calendar Event-from-Log, the second `Log this` first write, Journal patch target creation, and Journal DELETE with stale condition. I ended the broad probe at its time box while those unrelated mixed-load requests were still running; the runner exited 1 from that termination and cleaned up the server. These observations extend #387 and are not classified as a product crash or a successful hostile write.
Author
Owner

Update for #387 from the one API-only adversarial round on head 0eaa6ab4e4d18f2e0dcc60e7ed0245a9cdcd2dd5:

  • The API-only runner now targets the real backend directly (127.0.0.1:16223 in this run). It bypasses the Node frontend proxy, which converted any upstream connection error into a synthetic 502. The oversized Appearance PUT had no finding; its expected 413 check passed.
  • The two Calendar Log this retry paths and the Journal PATCH race completed without no-response findings. The Log this retry storm returned 200, with several 6–19s responses.
  • Other direct backend timeouts did occur: the Task create baseline and 8 Task storm requests timed out at 30s; POST /api/v1/calendar/events/from-log timed out at 30s after the 120-photo burst. I reported these mixed-load results to #367 with the shared-host load evidence.
  • The server reported alive at the end. The API round exited 1 with 163 findings, mostly SLOW results under active host load.

The round also repeated the missing calendar-home-set result and accepted a DAV alarm. #389 already notes the former; #358 intentionally accepts and stores VALARM. The Reminders unsupported-field probe returned 201 where its existing assertion expects 422, then the following delete returned 412. I left all existing API and test expectations unchanged for review.

Update for #387 from the one API-only adversarial round on head `0eaa6ab4e4d18f2e0dcc60e7ed0245a9cdcd2dd5`: - The API-only runner now targets the real backend directly (`127.0.0.1:16223` in this run). It bypasses the Node frontend proxy, which converted any upstream connection error into a synthetic 502. The oversized Appearance PUT had no finding; its expected 413 check passed. - The two Calendar `Log this` retry paths and the Journal PATCH race completed without no-response findings. The Log this retry storm returned 200, with several 6–19s responses. - Other direct backend timeouts did occur: the Task create baseline and 8 Task storm requests timed out at 30s; `POST /api/v1/calendar/events/from-log` timed out at 30s after the 120-photo burst. I reported these mixed-load results to #367 with the shared-host load evidence. - The server reported alive at the end. The API round exited 1 with 163 findings, mostly `SLOW` results under active host load. The round also repeated the missing `calendar-home-set` result and accepted a DAV alarm. #389 already notes the former; #358 intentionally accepts and stores `VALARM`. The Reminders unsupported-field probe returned 201 where its existing assertion expects 422, then the following delete returned 412. I left all existing API and test expectations unchanged for review.
Author
Owner

Additional real-server reproduction on 2026-09-29 from job/toast-ring after merging dev at d8c62e42 (ADVERSARIAL_API_ONLY=1): the 3 MiB PUT /api/v1/appearance returned 502 local adversarial server is unavailable through the local Node editor proxy. The direct-backend oversized Appearance probe passed its expected 413, and the adversarial harness reported server alive at end: True. This reproduces the proxy transport finding without evidence of a Rust server crash. The complete run log is in the ignored worktree file target/tmp/toast-ring-adversarial.log.

Additional real-server reproduction on 2026-09-29 from `job/toast-ring` after merging `dev` at `d8c62e42` (`ADVERSARIAL_API_ONLY=1`): the 3 MiB `PUT /api/v1/appearance` returned `502 local adversarial server is unavailable` through the local Node editor proxy. The direct-backend oversized Appearance probe passed its expected `413`, and the adversarial harness reported `server alive at end: True`. This reproduces the proxy transport finding without evidence of a Rust server crash. The complete run log is in the ignored worktree file `target/tmp/toast-ring-adversarial.log`.
Author
Owner

Fixed in d06030b5ca (origin/dev); the API probe runner now uses cancellable loopback requests and load-aware timeouts.

Fixed in d06030b5ca (origin/dev); the API probe runner now uses cancellable loopback requests and load-aware timeouts.
kayg closed this issue 2026-10-03 11:55:36 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#387
No description provided.