Investigate API transport timeouts during mixed read/write stress

kayg commented

2026-09-28 17:36:28 +00:00

Owner

During the single 45-minute local real-server adversarial round on 2026-09-28, the probe issued 128 GET requests to /api/v1/files/recent?limit=500 through a 32-worker pool. All 128 returned a client timeout with no HTTP response. Eight Photos fixture byte uploads and four Calendar write requests also timed out during the same mixed Search/media/Calendar load. The server process remained alive and the captured API requests did not return 5xx responses. Other worktrees were building and running servers on the shared host at the time, so the cause is not isolated to this server. Reproduce under controlled load and make overload behavior bounded and observable.

During the single 45-minute local real-server adversarial round on 2026-09-28, the probe issued 128 GET requests to `/api/v1/files/recent?limit=500` through a 32-worker pool. All 128 returned a client timeout with no HTTP response. Eight Photos fixture byte uploads and four Calendar write requests also timed out during the same mixed Search/media/Calendar load. The server process remained alive and the captured API requests did not return 5xx responses. Other worktrees were building and running servers on the shared host at the time, so the cause is not isolated to this server. Reproduce under controlled load and make overload behavior bounded and observable.

kayg referenced this issue

2026-09-28 17:39:29 +00:00

FILES: atomic write fails with 'entry not found' during a reconcile/watcher storm (stress test flakes under load) #343

kayg commented

2026-09-28 20:36:13 +00:00

Author

Owner

Additional run evidence for the existing transport-timeout investigation: during the same adversarial round, several Photos fixture upload requests (burst-082 through burst-087 byte uploads and uploads for burst-088, 089, 095, 096, and 097) timed out without an HTTP response. One Calendar recurring-log write also timed out; the next occurrence write returned 201, and the later identity check found the same Event identity for both occurrences. At the end of the DAV probe, an oversized appearance PUT returned 502 local adversarial server is unavailable, while the runner reported the server process alive. The shared-host load prevents assigning a cause. The search storm requests mostly returned 200 but were SLOW (about 20–26 s).

Additional run evidence for the existing transport-timeout investigation: during the same adversarial round, several Photos fixture upload requests (`burst-082` through `burst-087` byte uploads and uploads for `burst-088`, `089`, `095`, `096`, and `097`) timed out without an HTTP response. One Calendar recurring-log write also timed out; the next occurrence write returned 201, and the later identity check found the same Event identity for both occurrences. At the end of the DAV probe, an oversized appearance PUT returned 502 `local adversarial server is unavailable`, while the runner reported the server process alive. The shared-host load prevents assigning a cause. The search storm requests mostly returned 200 but were SLOW (about 20–26 s).

kayg commented

2026-09-28 20:52:32 +00:00

Author

Owner

During the same bounded real-server round, xuser_matrix.py got HTTP -1 while creating its fixture Folder. The 1122-request authorization matrix then reported POST /api/v1/admin/search/rebuild as admin/valid timed out; the runner continued into editor probes, so the server stayed available for later requests. This occurred alongside the watcher overflow and search rebuild load, and is evidence for the existing transport-timeout issue, not a UI code change.

During the same bounded real-server round, `xuser_matrix.py` got HTTP -1 while creating its fixture Folder. The 1122-request authorization matrix then reported `POST /api/v1/admin/search/rebuild as admin/valid` timed out; the runner continued into editor probes, so the server stayed available for later requests. This occurred alongside the watcher overflow and search rebuild load, and is evidence for the existing transport-timeout issue, not a UI code change.

kayg referenced this issue

2026-09-28 21:00:11 +00:00

ALIGN: one shared icon+label primitive with cap-height centring + automated alignment check #253

kayg commented

2026-09-28 21:38:01 +00:00

Author

Owner

Additional evidence from the #188 real-server adversarial run at HEAD 9bd81553: a valid PUT /api/v1/notifications/settings/quiet-hours for 22:30–07:15 timed out without a response. The next GET returned the previous 23:00–06:00 values. A Calendar Event-from-Note write also timed out. Later, an oversized Appearance PUT returned HTTP 502 local adversarial server is unavailable, while the runner reported the server process alive. These requests ran after the 20,000-write watcher fixture and under the broad API storm; this run does not assign a cause. The selected Photos two-user isolation check passed.

Additional evidence from the #188 real-server adversarial run at HEAD 9bd81553: a valid `PUT /api/v1/notifications/settings/quiet-hours` for 22:30–07:15 timed out without a response. The next GET returned the previous 23:00–06:00 values. A Calendar Event-from-Note write also timed out. Later, an oversized Appearance PUT returned HTTP 502 `local adversarial server is unavailable`, while the runner reported the server process alive. These requests ran after the 20,000-write watcher fixture and under the broad API storm; this run does not assign a cause. The selected Photos two-user isolation check passed.

kayg referenced this issue

2026-09-28 22:50:18 +00:00

Shared Photos timeline stays empty after member opt-in #188

kayg referenced this issue

2026-09-28 23:08:15 +00:00

MOTION: extend the tab-bar spring language app-wide (sheets, sidebar, segmented, popovers, toasts, lists) #291

kayg commented

2026-09-28 23:42:40 +00:00

Author

Owner

The #354 adversarial round observed two API calls that did not receive a response within the probe's 30-second request timeout: GET /api/v1/files/download with the hostile path .., and POST /api/v1/files/mkdir with ... These appeared after the Search and Editor stress phases. This may be load-related and does not identify a root cause, but it is a transport timeout rather than a SLOW response, so I am recording it against the existing API timeout investigation. No API or file behavior was changed in this UI job.

The #354 adversarial round observed two API calls that did not receive a response within the probe's 30-second request timeout: `GET /api/v1/files/download` with the hostile path `..`, and `POST /api/v1/files/mkdir` with `..`. These appeared after the Search and Editor stress phases. This may be load-related and does not identify a root cause, but it is a transport timeout rather than a `SLOW` response, so I am recording it against the existing API timeout investigation. No API or file behavior was changed in this UI job.

kayg referenced this issue

2026-09-28 23:47:37 +00:00

UI polish batch: theme variants as submenus, bigger sidebar section headers, one overlay transparency with more-opaque inner cards, toasts above overlays #354

kayg commented

2026-09-29 00:42:22 +00:00

Author

Owner

In the one-time local adversarial round on job/touch-369, XUser fixture Folder creation and authz fixture upload both returned HTTP -1 (connection error). The server log showed SQLite pool acquire waits of 2–10 seconds around that stage. Later requests in the API storm returned successful statuses but took 5–26 seconds and were marked SLOW by the harness. The server process remained alive while processing the storm. Other worktrees were also building/running on the shared host, so this evidence does not isolate the cause.

In the one-time local adversarial round on `job/touch-369`, XUser fixture Folder creation and authz fixture upload both returned HTTP `-1` (connection error). The server log showed SQLite pool acquire waits of 2–10 seconds around that stage. Later requests in the API storm returned successful statuses but took 5–26 seconds and were marked SLOW by the harness. The server process remained alive while processing the storm. Other worktrees were also building/running on the shared host, so this evidence does not isolate the cause.

kayg commented

2026-09-29 00:54:33 +00:00

Author

Owner

The one adversarial pass hit its 45-minute cap with exit 124 during the Journal PATCH storm; it was not repeated. More evidence under the same shared-host load: several Photos uploads, Calendar linked Note/Log writes, and a Journal newline-title update timed out with no HTTP response. The Journal PATCH storm returned one 200, eight 412 conflicts, and 15 no-response timeouts; the follow-up Journal read and Note creation also timed out. The server process remained alive at the time. This round did not reach the later media and round-two phases.

The one adversarial pass hit its 45-minute cap with exit `124` during the Journal PATCH storm; it was not repeated. More evidence under the same shared-host load: several Photos uploads, Calendar linked Note/Log writes, and a Journal newline-title update timed out with no HTTP response. The Journal PATCH storm returned one `200`, eight `412` conflicts, and 15 no-response timeouts; the follow-up Journal read and Note creation also timed out. The server process remained alive at the time. This round did not reach the later media and round-two phases.

kayg referenced this issue

2026-09-29 02:00:11 +00:00

TOUCH: hide shortcut hints on touch, one 'Info' title in sheets, menu icon gap, phone breadcrumb #369

kayg referenced this issue

2026-09-29 02:02:25 +00:00

SECURITY: Prevent one User from exhausting Instance storage #333

kayg referenced this issue

2026-10-03 12:51:31 +00:00

Issue hygiene sweep 2026-10-03 #1008

kayg commented

2026-10-04 08:34:28 +00:00

Author

Owner

Round 7b #867 evidence: artifacts/from-7b2/round3-adversarial.log:97 records block reminder create 126 with no response after the client deadline. Reminder creation enforces a 128-record cap in crates/plugins/notes/src/reminders_api.rs; store::index_note_projection_once rewrites the reminder projection for every edit. These paths also exist in production 6074f71d1. No lifecycle or write-stage log is included in the supplied artifacts. This is not established as a crash or as SLOW-only load.

The same log at line 103 records a Photos oversized Undo 502 whose body is emitted by tests/adversarial/editor-proxy.mjs, and line 134 records an AppleDouble upload Broken pipe. Synthetic proxy peers return 413 successfully before and after Continue for 200 KB, 5 MiB and 8 MiB+1 bodies. This does not prove the real-server finding is fixed. The merge round must retain backend exit status and content-free socket/write-stage diagnostics and distinguish an early 413 from a process failure. No live hostile-input/DoS reproduction ran in 7bfix-data.

Round 7b #867 evidence: `artifacts/from-7b2/round3-adversarial.log:97` records block reminder create 126 with no response after the client deadline. Reminder creation enforces a 128-record cap in `crates/plugins/notes/src/reminders_api.rs`; `store::index_note_projection_once` rewrites the reminder projection for every edit. These paths also exist in production `6074f71d1`. No lifecycle or write-stage log is included in the supplied artifacts. This is not established as a crash or as SLOW-only load. The same log at line 103 records a Photos oversized Undo 502 whose body is emitted by `tests/adversarial/editor-proxy.mjs`, and line 134 records an AppleDouble upload Broken pipe. Synthetic proxy peers return 413 successfully before and after Continue for 200 KB, 5 MiB and 8 MiB+1 bodies. This does not prove the real-server finding is fixed. The merge round must retain backend exit status and content-free socket/write-stage diagnostics and distinguish an early 413 from a process failure. No live hostile-input/DoS reproduction ran in 7bfix-data.

kayg referenced this issue

2026-10-04 08:36:21 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 10:06:35 +00:00

Plan: merge order and conflicts for merge round 7b #867

Rows
Columns

Investigate API transport timeouts during mixed read/write stress #368