GATES: dev failures blocking the first calternal.cloud deploy

kayg commented

2026-09-27 12:37:38 +00:00

Owner

Full gates on dev 1701cef5 (after #220 and #224 merged; host under load from ~13 jobs). These block the first production deploy to calternal.cloud.

Failures

Web unit: src/lib/components/analytics/BklitAnalytics.test.ts > Bklit analytics data adapters > formats minutes and accessible mark labels consistently fails (578/579).
E2E calendar fails in both passes: getByRole('dialog', { name: 'Event' }).getByText(/23:00 – .* 02:30/) never visible. Suspect the date/time-format merge (#179) changed the label; decide whether the app or the test is wrong against DESIGN.
E2E files fails in both passes (also in the previous run at 65265f87): locator('.fc-item').filter({ hasText: 'from-elsewhere.txt' }) never visible → a file created on disk outside the app never appears. Likely a real regression in watch_home_changes / the change feed (#123) / live refresh. Find the root cause.
Adversarial round 1:
- slowloris silent: connection still open after 35 s and slowloris partial header: connection still open after 35 s → DoS class (blocks merges). Add header-read and idle timeouts in the server (hyper/axum), per-IP connection caps; keep long-lived WebSockets/SSE and slow large uploads working (body read timeout is separate from header timeout).
- sync changing file: remote bytes differ from the final local bytes (len 3145728) even after #220 merged (2d84737b). Find why the probe still diverges; data correctness class.
- AI undo rejects oversized body: 502 'local adversarial server is unavailable' → was the server down (crash/OOM/restart)? Check server logs from the run; any crash blocks.
Round 2: 11 findings; classify each (SLOW-only under load vs real), fix the real ones.

Rules

Regression test for each fix. Re-run the full /mnt/hdd/targets/gates-full.sh equivalent at the end (in your worktree) and quote the summary. The full log of the failing run is in the issue comment below.

Full gates on `dev` 1701cef5 (after #220 and #224 merged; host under load from ~13 jobs). These block the first production deploy to calternal.cloud. ## Failures 1. **Web unit**: `src/lib/components/analytics/BklitAnalytics.test.ts > Bklit analytics data adapters > formats minutes and accessible mark labels consistently` fails (578/579). 2. **E2E calendar** fails in both passes: `getByRole('dialog', { name: 'Event' }).getByText(/23:00 – .* 02:30/)` never visible. Suspect the date/time-format merge (#179) changed the label; decide whether the app or the test is wrong against DESIGN. 3. **E2E files** fails in both passes (also in the previous run at 65265f87): `locator('.fc-item').filter({ hasText: 'from-elsewhere.txt' })` never visible → a file created on disk outside the app never appears. Likely a real regression in `watch_home_changes` / the change feed (#123) / live refresh. Find the root cause. 4. **Adversarial round 1**: - `slowloris silent: connection still open after 35 s` and `slowloris partial header: connection still open after 35 s` → DoS class (blocks merges). Add header-read and idle timeouts in the server (hyper/axum), per-IP connection caps; keep long-lived WebSockets/SSE and slow large uploads working (body read timeout is separate from header timeout). - `sync changing file: remote bytes differ from the final local bytes (len 3145728)` even after #220 merged (2d84737b). Find why the probe still diverges; data correctness class. - `AI undo rejects oversized body: 502 'local adversarial server is unavailable'` → was the server down (crash/OOM/restart)? Check server logs from the run; any crash blocks. 5. **Round 2**: 11 findings; classify each (SLOW-only under load vs real), fix the real ones. ## Rules Regression test for each fix. Re-run the full `/mnt/hdd/targets/gates-full.sh` equivalent at the end (in your worktree) and quote the summary. The full log of the failing run is in the issue comment below.

editor-before-history.png

641 KiB

editor-after-history.png

718 KiB

kayg commented

2026-09-27 12:37:45 +00:00

Author

Owner

Gate log (ANSI stripped, tail):

head 1701cef5
deps-ok
web-built
fmt-ok
GEN-FAIL
+        /**
+         * @description The clock cycle used by human-readable time labels.
+         * @enum {string}
+         */
+        TimeFormat: "system" | "24_hour" | "12_hour";
         Timeline: {
             days: components["schemas"]["TimelineDay"][];
             next_before?: string | null;
svelte-check found 0 errors and 0 warnings
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
 FAIL   unit  src/lib/components/analytics/BklitAnalytics.test.ts > Bklit analytics data adapters > formats minutes and accessible mark labels consistently
      Tests  1 failed | 578 passed (579)
clippy-done
tests-done
server-built
E2E calendar FAIL

  log: [ "  - waiting for getByRole('dialog', { name: 'Event' }).getByText(/23:00 \\u2013 .* 02:30/) to be visible" ],

      at /home/kayg/Developer/calternal-wt/gates/tests/adversarial/node_modules/playwright-core/lib/coreBundle.js:57694:13

e2e composer ok
e2e search ok
E2E files FAIL

      at main (/home/kayg/Developer/calternal-wt/gates/apps/web/e2e/files.mjs:114:41)
      at processTicksAndRejections (native:7:39)

Bun v1.4.2 (Linux x64)
e2e share ok
E2E calendar FAIL
Call log:
  - waiting for getByRole('dialog', { name: 'Event' }).getByText(/23:00 \u2013 .* 02:30/) to be visible

  log: [ "  - waiting for getByRole('dialog', { name: 'Event' }).getByText(/23:00 \\u2013 .* 02:30/) to be visible" ],


e2e composer ok
e2e search ok
E2E files FAIL
  - waiting for locator('.fc-item').filter({ hasText: 'from-elsewhere.txt' }) to be visible

  log: [ "  - waiting for locator('.fc-item').filter({ hasText: 'from-elsewhere.txt' }) to be visible" ],


Bun v1.4.2 (Linux x64)
e2e share ok
==== FINDINGS 3
==== HOSTILE BYTES FINDINGS 0
!! slowloris silent: connection still open after 35 s
!! slowloris partial header: connection still open after 35 s
!! sync changing file: remote bytes differ from the final local bytes (len 3145728)
!! AI undo rejects oversized body: 502 b'local adversarial server is unavailable'
==== ROUND 2 FINDINGS 11
adv-done

Gate log (ANSI stripped, tail): ``` head 1701cef5 deps-ok web-built fmt-ok GEN-FAIL + /** + * @description The clock cycle used by human-readable time labels. + * @enum {string} + */ + TimeFormat: "system" | "24_hour" | "12_hour"; Timeline: { days: components["schemas"]["TimelineDay"][]; next_before?: string | null; svelte-check found 0 errors and 0 warnings ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ FAIL unit src/lib/components/analytics/BklitAnalytics.test.ts > Bklit analytics data adapters > formats minutes and accessible mark labels consistently Tests 1 failed | 578 passed (579) clippy-done tests-done server-built E2E calendar FAIL log: [ " - waiting for getByRole('dialog', { name: 'Event' }).getByText(/23:00 \\u2013 .* 02:30/) to be visible" ], at /home/kayg/Developer/calternal-wt/gates/tests/adversarial/node_modules/playwright-core/lib/coreBundle.js:57694:13 e2e composer ok e2e search ok E2E files FAIL at main (/home/kayg/Developer/calternal-wt/gates/apps/web/e2e/files.mjs:114:41) at processTicksAndRejections (native:7:39) Bun v1.4.2 (Linux x64) e2e share ok E2E calendar FAIL Call log: - waiting for getByRole('dialog', { name: 'Event' }).getByText(/23:00 \u2013 .* 02:30/) to be visible log: [ " - waiting for getByRole('dialog', { name: 'Event' }).getByText(/23:00 \\u2013 .* 02:30/) to be visible" ], e2e composer ok e2e search ok E2E files FAIL - waiting for locator('.fc-item').filter({ hasText: 'from-elsewhere.txt' }) to be visible log: [ " - waiting for locator('.fc-item').filter({ hasText: 'from-elsewhere.txt' }) to be visible" ], Bun v1.4.2 (Linux x64) e2e share ok ==== FINDINGS 3 ==== HOSTILE BYTES FINDINGS 0 !! slowloris silent: connection still open after 35 s !! slowloris partial header: connection still open after 35 s !! sync changing file: remote bytes differ from the final local bytes (len 3145728) !! AI undo rejects oversized body: 502 b'local adversarial server is unavailable' ==== ROUND 2 FINDINGS 11 adv-done ```

kayg commented

2026-09-27 12:38:32 +00:00

Author

Owner

Starting gate-fix on branch job/gate-fix, based on dev at f1c0766907. I am retrieving the full gate and round-2 evidence, then I will fix each real failure with a regression test and commit green slices.

Starting gate-fix on branch job/gate-fix, based on dev at f1c0766907d5d1a0a3a7ea0ae807e43dd2caf9f0. I am retrieving the full gate and round-2 evidence, then I will fix each real failure with a regression test and commit green slices.

kayg referenced this issue

2026-09-27 12:42:14 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 13:31:17 +00:00

Author

Owner

Round 2 findings confirmed from the saved gate logs:

e2e-files.log shows the Tus upload returned 201, then the visible .fc-item row timed out. That case uses the Files API, so I kept it as a separate regression while adding a real outside-Files write to exercise the Home watcher.
watch_home_changes calls FilesState::adopt_if_missing; the current method records an Index row but does not insert a files_events row or wake Files SSE subscribers. I am fixing adoption to publish one event after a newly indexed item, and testing that duplicate watcher notifications do not publish twice.
The adversarial probe reports the oversized AI undo request as a proxy 502 / upstream unavailable. The AI routes do not use the shared oversized-JSON drain middleware that Files, Notes and Calendar use. The captured server log remains alive through probe cleanup and has no panic/exit, so the evidence points to the in-flight request connection being reset, not a confirmed process crash. I am adding the shared drain and a route regression for 413.
The prior slowloris run reported silent and partial-header sockets. The current server has the Hyper header-read timeout; I added a per-IP active-connection cap and regressions for silent/partial headers, slow request bodies, streaming responses and permit release.

The 7 round-2 entries marked SLOW are load observations under the shared host; I will classify the fresh round by its own output.

Round 2 findings confirmed from the saved gate logs: - `e2e-files.log` shows the Tus upload returned 201, then the visible `.fc-item` row timed out. That case uses the Files API, so I kept it as a separate regression while adding a real outside-Files write to exercise the Home watcher. - `watch_home_changes` calls `FilesState::adopt_if_missing`; the current method records an Index row but does not insert a `files_events` row or wake Files SSE subscribers. I am fixing adoption to publish one event after a newly indexed item, and testing that duplicate watcher notifications do not publish twice. - The adversarial probe reports the oversized AI undo request as a proxy 502 / upstream unavailable. The AI routes do not use the shared oversized-JSON drain middleware that Files, Notes and Calendar use. The captured server log remains alive through probe cleanup and has no panic/exit, so the evidence points to the in-flight request connection being reset, not a confirmed process crash. I am adding the shared drain and a route regression for 413. - The prior slowloris run reported silent and partial-header sockets. The current server has the Hyper header-read timeout; I added a per-IP active-connection cap and regressions for silent/partial headers, slow request bodies, streaming responses and permit release. The 7 round-2 entries marked `SLOW` are load observations under the shared host; I will classify the fresh round by its own output.

kayg commented

2026-09-27 13:49:40 +00:00

Author

Owner

Note from orchestrator: #161 (merged into dev as 74d60726) moved the slowloris and oversized-URI probes to target the Rust server directly; the gate findings were measuring the Node editor proxy. Merge dev before working on the slowloris item and re-verify against the Rust server (production has no Node proxy).

Note from orchestrator: #161 (merged into dev as 74d60726) moved the slowloris and oversized-URI probes to target the Rust server directly; the gate findings were measuring the Node editor proxy. Merge dev before working on the slowloris item and re-verify against the Rust server (production has no Node proxy).

kayg commented

2026-09-27 14:05:22 +00:00

Author

Owner

Calendar E2E evidence: the production Event dialog rendered the cross-midnight interval as Sun, 20 Sept 23:00 – Mon, 21 Sept 2:30. The event correctly names both days; the test expected a zero-padded 02:30, which en-GB formatting does not emit here. I changed the assertion to derive the two day labels in the browser locale and accept the rendered 2:30 form. I will rerun the real-server E2E when the shared browser slot is free.

Calendar E2E evidence: the production Event dialog rendered the cross-midnight interval as `Sun, 20 Sept 23:00 – Mon, 21 Sept 2:30`. The event correctly names both days; the test expected a zero-padded `02:30`, which en-GB formatting does not emit here. I changed the assertion to derive the two day labels in the browser locale and accept the rendered `2:30` form. I will rerun the real-server E2E when the shared browser slot is free.

kayg commented

2026-09-27 14:07:03 +00:00

Author

Owner

Sync divergence regression completed on the current checkout: changing_file_campaign.py held a 3 MiB upload after staging, rewrote the local file 20 times at the same size and restored its mtime within one second, then confirmed the remote bytes converged to the final SHA-256. It printed PASS same-size writes in one second during upload converged to final remote bytes. The stable-read fix and campaign are already in the dev base, so this job needs no additional Sync behavior change.

Sync divergence regression completed on the current checkout: `changing_file_campaign.py` held a 3 MiB upload after staging, rewrote the local file 20 times at the same size and restored its mtime within one second, then confirmed the remote bytes converged to the final SHA-256. It printed `PASS same-size writes in one second during upload converged to final remote bytes`. The stable-read fix and campaign are already in the `dev` base, so this job needs no additional Sync behavior change.

kayg commented

2026-09-27 14:20:51 +00:00

Author

Owner

After the corrected midnight assertion, the calendar E2E reached its photo preview check and failed because it waited for Photo · taken. ItemPreview.svelte renders Photo · <localized day>; §39 requires kind/title/time or range and does not require the EXIF source in the popover. The API assertion already checks time_source === 'taken'. I changed the UI assertion to verify the localized day label shown by the real preview.

After the corrected midnight assertion, the calendar E2E reached its photo preview check and failed because it waited for `Photo · taken`. `ItemPreview.svelte` renders `Photo · <localized day>`; §39 requires kind/title/time or range and does not require the EXIF source in the popover. The API assertion already checks `time_source === 'taken'`. I changed the UI assertion to verify the localized day label shown by the real preview.

kayg commented

2026-09-27 14:28:17 +00:00

Author

Owner

The next full calendar run passed the repaired Event and photo preview checks, then stopped at the 100-photo pile label. The rendered accessibility label was 100 photos saved at 8:05; the test expected 08:05. This is the same en-GB zero-padding mismatch, so I made the assertion accept either rendered hour width while keeping the minute and 100-photo checks exact.

The next full calendar run passed the repaired Event and photo preview checks, then stopped at the 100-photo pile label. The rendered accessibility label was `100 photos saved at 8:05`; the test expected `08:05`. This is the same en-GB zero-padding mismatch, so I made the assertion accept either rendered hour width while keeping the minute and 100-photo checks exact.

kayg commented

2026-09-27 14:34:04 +00:00

Author

Owner

The next calendar run passed the Event, photo and 100-photo checks, then timed out waiting for a server-backed zoom reset. Evidence in +page.svelte: Calendar flushes a pending zoomSave during component teardown; the E2E was writing 48 directly to the server while the Calendar component still held 160. I moved the reset to the Settings route after Calendar unmounts, so teardown cannot overwrite the server value. The following Calendar load still verifies that the server's 48 is applied.

The next calendar run passed the Event, photo and 100-photo checks, then timed out waiting for a server-backed zoom reset. Evidence in `+page.svelte`: Calendar flushes a pending `zoomSave` during component teardown; the E2E was writing 48 directly to the server while the Calendar component still held 160. I moved the reset to the Settings route after Calendar unmounts, so teardown cannot overwrite the server value. The following Calendar load still verifies that the server's 48 is applied.

kayg commented

2026-09-27 15:13:56 +00:00

Author

Owner

Files round-2 root cause and fix (commit f2db8500): the server Home watcher and the Files plugin router each constructed a FilesState over the same Root and database, but each state had its own SSE wakeup channel. The watcher persisted Archive/from-disk.txt in files_events; the open Files stream remained asleep on the router state's separate channel. The Files E2E showed the file in /api/v1/files/entries after the timeout while the visible folder did not refresh. Wakeups now share a process-wide bounded channel, while the durable event rows remain user-scoped in SQLite. Added server_watcher_state_wakes_files_route_subscribers to cover the two-state server arrangement. cargo test -p calternal-plugin-files: 102 passed, 0 failed.

Files round-2 root cause and fix (commit f2db8500): the server Home watcher and the Files plugin router each constructed a `FilesState` over the same Root and database, but each state had its own SSE wakeup channel. The watcher persisted `Archive/from-disk.txt` in `files_events`; the open Files stream remained asleep on the router state's separate channel. The Files E2E showed the file in `/api/v1/files/entries` after the timeout while the visible folder did not refresh. Wakeups now share a process-wide bounded channel, while the durable event rows remain user-scoped in SQLite. Added `server_watcher_state_wakes_files_route_subscribers` to cover the two-state server arrangement. `cargo test -p calternal-plugin-files`: 102 passed, 0 failed.

kayg commented

2026-09-27 15:37:02 +00:00

Author

Owner

Post-merge rechecks on dev 895d3479: cargo test -p calternal-server serve::tests passed 4/4 against the direct Hyper server (header timeout, body timeout distinction, per-IP cap, stream shutdown); cargo test -p calternal-plugin-ai oversized_undo_request_returns_payload_too_large passed 1/1; changing_file_campaign.py printed PASS same-size writes in one second during upload converged to final remote bytes. I have not run the fresh E2Es or final adversarial round yet; both are queued for the shared browser slot.

Post-merge rechecks on `dev` 895d3479: `cargo test -p calternal-server serve::tests` passed 4/4 against the direct Hyper server (header timeout, body timeout distinction, per-IP cap, stream shutdown); `cargo test -p calternal-plugin-ai oversized_undo_request_returns_payload_too_large` passed 1/1; `changing_file_campaign.py` printed `PASS same-size writes in one second during upload converged to final remote bytes`. I have not run the fresh E2Es or final adversarial round yet; both are queued for the shared browser slot.

kayg referenced this issue

2026-09-27 16:30:21 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 16:33:30 +00:00

Author

Owner

Root-cause evidence from the production-build probe: the Home watcher indexed Archive/from-disk.txt and emitted its SSE event. The browser received the event and scheduled a folder reload, but the API client reused its 750 ms GET snapshot. The listing therefore stayed stale and no later event caused another request. Live Files reloads now bypass that snapshot. A new SSE connection also starts after existing user events because its initial page load is authoritative; explicit Last-Event-ID reconnects still replay.

Regression coverage added: an api-client test for fresh-read snapshot invalidation, a Files SSE test for skipping old events while waking for new changes, and a Files E2E case that writes a file directly under the registered Home. Verified so far: api-client 8/8, bun run check 0 errors/0 warnings, Files SSE regression 1/1, focused production watcher probe passed. Full Files/Calendar E2E and final gates remain pending.

Root-cause evidence from the production-build probe: the Home watcher indexed `Archive/from-disk.txt` and emitted its SSE event. The browser received the event and scheduled a folder reload, but the API client reused its 750 ms GET snapshot. The listing therefore stayed stale and no later event caused another request. Live Files reloads now bypass that snapshot. A new SSE connection also starts after existing user events because its initial page load is authoritative; explicit Last-Event-ID reconnects still replay. Regression coverage added: an api-client test for fresh-read snapshot invalidation, a Files SSE test for skipping old events while waking for new changes, and a Files E2E case that writes a file directly under the registered Home. Verified so far: api-client 8/8, `bun run check` 0 errors/0 warnings, Files SSE regression 1/1, focused production watcher probe passed. Full Files/Calendar E2E and final gates remain pending.

kayg commented

2026-09-27 16:55:15 +00:00

Author

Owner

The post-merge adversarial round has produced a non-SLOW editor finding: the browser anchor test timed out after the 500-step undo/redo storm. Its captured saved Note body still contained repeated ^duplicate-id anchors and did not contain the three normalized anchors copied by the UI. I am tracing whether undo/redo can restore duplicate block IDs or whether persistence is delayed under host load; I will classify and report the evidence before finishing the round.

The post-merge adversarial round has produced a non-SLOW editor finding: the browser anchor test timed out after the 500-step undo/redo storm. Its captured saved Note body still contained repeated `^duplicate-id` anchors and did not contain the three normalized anchors copied by the UI. I am tracing whether undo/redo can restore duplicate block IDs or whether persistence is delayed under host load; I will classify and report the evidence before finishing the round.

kayg commented

2026-09-27 16:59:12 +00:00

Author

Owner

The second editor finding is a cascade from the browser-anchor failure, not evidence of a server crash: the anchor probe closes its Browser context only on its success path. When the earlier assertion timed out, that page stayed open through the orchestrated server restart and its API requests recorded transient proxy 502s. The restart was requested by the probe. I will make the browser probe close its context on failure and classify the anchor persistence timeout separately.

kayg referenced this issue

2026-09-27 17:08:52 +00:00

Owner test tracker: every request and when it is testable on o2 #203

kayg commented

2026-09-27 17:18:04 +00:00

Author

Owner

Round-two non-SLOW finding: the Analytics storm produced multiple request timeouts (not marked SLOW), then the probe reported that two reads after the storm disagreed. I am checking the probe's completion/consistency condition and will verify whether accepted writes were lost or whether a stale read occurred while timed-out requests were still in flight.

kayg commented

2026-09-27 17:59:40 +00:00

Author

Owner

Round-2 Editor finding is now deterministic. I changed the browser adversarial probe to write rich HTML through the granted Clipboard API and to assert the paste was accepted. One image paste followed by 500 Ctrl+Z and 500 Ctrl+Shift+Z produced 9 image blocks; the same history storm without the accepted paste preserved the document. This is a real editor/Yjs undo-redo data corruption path, not a timeout. I am testing the Yjs protected-paragraph filter as the root cause.

kayg commented

2026-09-27 18:12:03 +00:00

Author

Owner

Starting #235 continuation on job/gate-fix; base SHA from git merge-base HEAD dev: 895714eb. Current HEAD: d7b3ceac54de71903879b0cdcce8a9fa750a7768.

Starting #235 continuation on `job/gate-fix`; base SHA from `git merge-base HEAD dev`: `895714eb`. Current HEAD: `d7b3ceac54de71903879b0cdcce8a9fa750a7768`.

kayg commented

2026-09-27 18:29:39 +00:00

Author

Owner

Focused post-merge editor repro confirms the undo/redo finding. Against the production web build and real local server, the rich-text paste produced 1 .cal-image-block; after 500 Ctrl+Z and 500 Ctrl+Y, the editor had 3 image blocks and 13 paragraphs (up from 1 image and 11 paragraphs). The live DOM also returned to duplicate ^duplicate-id anchors during undo, then restored the three normalized anchors during redo. This reproduces with Ctrl+Y, so the result is not limited to the earlier Ctrl+Shift+Z probe. I am testing the Y-Tiptap default protected-paragraph delete filter as the source; no implementation fix is verified yet.

Focused post-merge editor repro confirms the undo/redo finding. Against the production web build and real local server, the rich-text paste produced 1 `.cal-image-block`; after 500 Ctrl+Z and 500 Ctrl+Y, the editor had 3 image blocks and 13 paragraphs (up from 1 image and 11 paragraphs). The live DOM also returned to duplicate `^duplicate-id` anchors during undo, then restored the three normalized anchors during redo. This reproduces with Ctrl+Y, so the result is not limited to the earlier Ctrl+Shift+Z probe. I am testing the Y-Tiptap default protected-paragraph delete filter as the source; no implementation fix is verified yet.

kayg commented

2026-09-27 18:49:50 +00:00

Author

Owner

The single full adversarial run has also recorded FINDING editor 10,000 top-level blocks: collaboration sync timed out (the probe's 20-second sync bound). The 1 MiB paragraph, mixed Markdown stability, concurrent same-block writes, offline reconciliation, delete/edit, external Note write, and restart areas passed in this run. I will classify this timeout from the completed server/probe log; no byte divergence has been observed in this area.

The single full adversarial run has also recorded `FINDING editor 10,000 top-level blocks: collaboration sync timed out` (the probe's 20-second sync bound). The 1 MiB paragraph, mixed Markdown stability, concurrent same-block writes, offline reconciliation, delete/edit, external Note write, and restart areas passed in this run. I will classify this timeout from the completed server/probe log; no byte divergence has been observed in this area.

kayg commented

2026-09-27 18:57:14 +00:00

Author

Owner

The one-time full round has also recorded DAV initial sync: NO RESPONSE (timed out). The surrounding DAV discovery/query/multiget/stale/move/delete requests returned 207/412/200/204 but were marked SLOW under the concurrent API round. The server remained available for later photo setup and round-2 traffic; no crash is evidenced. I will keep the initial-sync timeout in remaining work because it was not classified SLOW by the probe.

The one-time full round has also recorded `DAV initial sync: NO RESPONSE (timed out)`. The surrounding DAV discovery/query/multiget/stale/move/delete requests returned 207/412/200/204 but were marked SLOW under the concurrent API round. The server remained available for later photo setup and round-2 traffic; no crash is evidenced. I will keep the initial-sync timeout in remaining work because it was not classified SLOW by the probe.

kayg commented

2026-09-27 18:58:30 +00:00

Author

Owner

The same round has recorded a second unclassified timeout: calendar Event from Log: NO RESPONSE (b'timed out'). Its preceding linked Note and Log creates returned 201 but were marked SLOW (6.0s, 21.0s and 14.5s); the event-from-Log request did not return before the probe timeout. The round is still active. No crash or data-loss evidence is present so far; I am keeping this endpoint timeout in remaining work.

The same round has recorded a second unclassified timeout: `calendar Event from Log: NO RESPONSE (b'timed out')`. Its preceding linked Note and Log creates returned 201 but were marked SLOW (6.0s, 21.0s and 14.5s); the event-from-Log request did not return before the probe timeout. The round is still active. No crash or data-loss evidence is present so far; I am keeping this endpoint timeout in remaining work.

kayg commented

2026-09-27 19:02:24 +00:00

Author

Owner

Adversarial finding: Journal PATCH concurrency under load

In the single full round, tests/adversarial/attack.py sent 24 concurrent PATCH requests to one Journal entry using the same If-Match value. The probe expects one 200 and 23 412 responses. Requests 0–15 timed out; requests 16–23 returned 412 after 12.0 seconds. The probe did not observe the expected successful writer response. The server remained alive. A later Journal child-attachment read returned 200 after 29.1 seconds.

This is an unresolved availability/concurrency result under the round's photo and calendar load. It needs follow-up on whether a write committed despite its client timeout and whether the request queue is bounded. It is not a SLOW-only observation.

Adversarial finding: Journal PATCH concurrency under load In the single full round, `tests/adversarial/attack.py` sent 24 concurrent PATCH requests to one Journal entry using the same `If-Match` value. The probe expects one `200` and 23 `412` responses. Requests 0–15 timed out; requests 16–23 returned `412` after 12.0 seconds. The probe did not observe the expected successful writer response. The server remained alive. A later Journal child-attachment read returned `200` after 29.1 seconds. This is an unresolved availability/concurrency result under the round's photo and calendar load. It needs follow-up on whether a write committed despite its client timeout and whether the request queue is bounded. It is not a SLOW-only observation.

kayg commented

2026-09-27 19:05:03 +00:00

Author

Owner

Adversarial finding: template creation timed out under load

The full round's valid POST /api/v1/notes/from-template request for a Unicode title did not receive a response within the client timeout. The probe expected 201. Subsequent invalid and oversized template requests returned their expected 400 responses, though several took 12–20 seconds. This is an unresolved availability finding under concurrent photo and calendar load; the request may have committed despite the timeout and should be checked for duplicate creation on retry.

Adversarial finding: template creation timed out under load The full round's valid `POST /api/v1/notes/from-template` request for a Unicode title did not receive a response within the client timeout. The probe expected `201`. Subsequent invalid and oversized template requests returned their expected `400` responses, though several took 12–20 seconds. This is an unresolved availability finding under concurrent photo and calendar load; the request may have committed despite the timeout and should be checked for duplicate creation on retry.

kayg commented

2026-09-27 19:06:19 +00:00

Author

Owner

Adversarial finding: Reminders CalDAV discovery timed out

During the full round, PROPFIND /dav/calendars/<user>/reminders/ for supported components and PROPFIND /dav/calendars/<user>/ for collection discovery both timed out. The probe expects 207 for each and verifies the VTODO component and Reminders collection. These are additional non-SLOW availability failures under concurrent API and photo load. Follow-up should check the next Reminders sync request and whether the timeouts leave DAV clients with inconsistent discovery state.

Adversarial finding: Reminders CalDAV discovery timed out During the full round, `PROPFIND /dav/calendars/<user>/reminders/` for supported components and `PROPFIND /dav/calendars/<user>/` for collection discovery both timed out. The probe expects `207` for each and verifies the `VTODO` component and Reminders collection. These are additional non-SLOW availability failures under concurrent API and photo load. Follow-up should check the next Reminders sync request and whether the timeouts leave DAV clients with inconsistent discovery state.

kayg commented

2026-09-27 19:07:31 +00:00

Author

Owner

Additional non-SLOW timeouts in the same adversarial round

The probe also timed out on the valid initial Reminders CalDAV REPORT (expected 207 with a sync token) and on a valid bookmark capture (expected 201). These timeouts occurred after Reminders discovery had already timed out. The API and DAV server process remained alive. The round's current high concurrent load is part of the evidence; check for committed writes and idempotency before any client retry.

Additional non-SLOW timeouts in the same adversarial round The probe also timed out on the valid initial Reminders CalDAV `REPORT` (expected `207` with a sync token) and on a valid bookmark capture (expected `201`). These timeouts occurred after Reminders discovery had already timed out. The API and DAV server process remained alive. The round's current high concurrent load is part of the evidence; check for committed writes and idempotency before any client retry.

kayg commented

2026-09-27 19:08:55 +00:00

Author

Owner

Additional write-storm timeouts from the full adversarial round

The concurrent template creation storm produced timeout (-1) responses among the 16 requests; the probe reports 0/16 client-visible Note IDs. The bookmark capture storm also returned timeout responses, mixed with 429 rate-limit responses. The tests accept the documented rate-limit status, but treat transport timeouts as failures. These occurred in the same sustained load window as the Journal, DAV, template, and bookmark single-request timeouts. Persisted results and safe retry behavior remain unverified.

Additional write-storm timeouts from the full adversarial round The concurrent template creation storm produced timeout (`-1`) responses among the 16 requests; the probe reports `0/16` client-visible Note IDs. The bookmark capture storm also returned timeout responses, mixed with `429` rate-limit responses. The tests accept the documented rate-limit status, but treat transport timeouts as failures. These occurred in the same sustained load window as the Journal, DAV, template, and bookmark single-request timeouts. Persisted results and safe retry behavior remain unverified.

kayg commented

2026-09-27 19:53:11 +00:00

Author

Owner

#235 final report

Branch: job/gate-fix
HEAD: 62b91e4e70cb7fc4b9ea8a17d6224c1cbbc1a502
Merge: dev was merged once at fe0a6977. The branch is pushed; origin already had this HEAD.

Built

Added a per-peer-IP open connection cap to reject excess slow clients.
Drain oversized AI undo request bodies before returning an error.
Publish watcher adoption and share live-event wakeups; refresh file listings when a live cursor cannot safely continue from a stale snapshot.
Correct the localized calendar probe and extend editor adversarial coverage for pasted image undo/redo.

Files: crates/calternal-server/src/serve.rs, crates/plugins/ai/src/routes.rs, crates/plugins/files/src/lib.rs, apps/web/src/lib/files/FilesBrowser.svelte, apps/web/src/lib/files/api.ts, packages/api-client/src/index.ts, packages/api-client/src/index.test.ts, apps/web/e2e/calendar.mjs, apps/web/e2e/files.mjs, apps/web/src/lib/notes/editorHost.ts, tests/adversarial/editor.mjs.

Gates

cargo fmt --check: no stdout or stderr. The exec wrapper did not retain this command's exit code.

cargo clippy --all-targets -- -D warnings:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 15m 51s
CARGO_CLIPPY_EXIT_CODE=0

cargo test:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 8m 00s
CARGO_TEST_EXIT_CODE=0

Parsed 72 test-result lines: 1,299 passed, 0 failed, 12 ignored.

bun run check:

svelte-check found 0 errors and 0 warnings
WEB_CHECK_EXIT_CODE=0

bun run test:

 Test Files  82 passed (82)
      Tests  593 passed (593)
   Start at  20:54:54
   Duration  110.63s (transform 63%, environment 14%, import 13%, tests 8%, setup 2%)
WEB_TEST_EXIT_CODE=0

Vitest also printed a CSS parse notice and jsdom scrollTo() notices; the suite passed.

Adversarial round and remaining work

One full round ran for 30 minutes and exited with ADVERSARIAL_EXIT_CODE=124 during isolation/collaboration checks. Those checks remain incomplete. The hostile-byte probe reported zero findings. Shared-host Cargo builds and a separate adversarial runner were active during this round, so the availability observations below occurred under concurrent load; SLOW-only observations are non-blocking.

The round found additional non-SLOW timeouts: DAV initial sync; Reminders discovery and initial sync; Calendar Event-from-Log; valid template creation and its concurrent create storm; bookmark capture and its concurrent storm. In the 24-request Journal PATCH storm, 16 requests timed out and 8 returned 412; the probe expected one 200 and 23 412. The server stayed alive. These were filed in this issue. Focused editor probing also found pasted-image undo/redo duplication after 500 undo/redo cycles, and a 10,000-block collaboration sync timeout; those findings are recorded above in the issue thread. Investigate committed state and safe retry behavior for timed-out writes, then rerun the unfinished isolation/collaboration checks.

Decisions not specified in the design

Set the open TCP connection limit to 256 per peer IP. The server counts its socket peer address; clients behind one reverse proxy share that limit.
A new listing stream without a resume cursor triggers a full listing refresh. A stream with Last-Event-ID resumes event replay.

Screenshots

## #235 final report **Branch:** `job/gate-fix` **HEAD:** `62b91e4e70cb7fc4b9ea8a17d6224c1cbbc1a502` **Merge:** `dev` was merged once at `fe0a6977`. The branch is pushed; origin already had this HEAD. ### Built - Added a per-peer-IP open connection cap to reject excess slow clients. - Drain oversized AI undo request bodies before returning an error. - Publish watcher adoption and share live-event wakeups; refresh file listings when a live cursor cannot safely continue from a stale snapshot. - Correct the localized calendar probe and extend editor adversarial coverage for pasted image undo/redo. **Files:** `crates/calternal-server/src/serve.rs`, `crates/plugins/ai/src/routes.rs`, `crates/plugins/files/src/lib.rs`, `apps/web/src/lib/files/FilesBrowser.svelte`, `apps/web/src/lib/files/api.ts`, `packages/api-client/src/index.ts`, `packages/api-client/src/index.test.ts`, `apps/web/e2e/calendar.mjs`, `apps/web/e2e/files.mjs`, `apps/web/src/lib/notes/editorHost.ts`, `tests/adversarial/editor.mjs`. ### Gates - `cargo fmt --check`: no stdout or stderr. The exec wrapper did not retain this command's exit code. - `cargo clippy --all-targets -- -D warnings`: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 15m 51s CARGO_CLIPPY_EXIT_CODE=0 ``` - `cargo test`: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 8m 00s CARGO_TEST_EXIT_CODE=0 ``` Parsed 72 test-result lines: 1,299 passed, 0 failed, 12 ignored. - `bun run check`: ```text svelte-check found 0 errors and 0 warnings WEB_CHECK_EXIT_CODE=0 ``` - `bun run test`: ```text Test Files 82 passed (82) Tests 593 passed (593) Start at 20:54:54 Duration 110.63s (transform 63%, environment 14%, import 13%, tests 8%, setup 2%) WEB_TEST_EXIT_CODE=0 ``` Vitest also printed a CSS parse notice and jsdom `scrollTo()` notices; the suite passed. ### Adversarial round and remaining work One full round ran for 30 minutes and exited with `ADVERSARIAL_EXIT_CODE=124` during isolation/collaboration checks. Those checks remain incomplete. The hostile-byte probe reported zero findings. Shared-host Cargo builds and a separate adversarial runner were active during this round, so the availability observations below occurred under concurrent load; SLOW-only observations are non-blocking. The round found additional non-SLOW timeouts: DAV initial sync; Reminders discovery and initial sync; Calendar Event-from-Log; valid template creation and its concurrent create storm; bookmark capture and its concurrent storm. In the 24-request Journal PATCH storm, 16 requests timed out and 8 returned `412`; the probe expected one `200` and 23 `412`. The server stayed alive. These were filed in this issue. Focused editor probing also found pasted-image undo/redo duplication after 500 undo/redo cycles, and a 10,000-block collaboration sync timeout; those findings are recorded above in the issue thread. Investigate committed state and safe retry behavior for timed-out writes, then rerun the unfinished isolation/collaboration checks. ### Decisions not specified in the design - Set the open TCP connection limit to 256 per peer IP. The server counts its socket peer address; clients behind one reverse proxy share that limit. - A new listing stream without a resume cursor triggers a full listing refresh. A stream with `Last-Event-ID` resumes event replay. ### Screenshots ![Editor before history stress](https://git.kayg.org/attachments/47fcbedf-7b8b-46b1-8495-ae718cd809c5) ![Editor after history stress](https://git.kayg.org/attachments/b3129762-f8f9-4723-b94c-1e6195c2c045)

kayg referenced this issue

2026-09-27 19:55:50 +00:00

FOLLOW-UP from #235 adversarial round: DAV/Journal/bookmark timeouts, pasted-image undo duplication, 10k-block collab timeout #265

kayg commented

2026-09-27 19:58:21 +00:00

Author

Owner

Started the #235 gate-fix follow-up on branch job/gate-fix at HEAD 62b91e4e70cb7fc4b9ea8a17d6224c1cbbc1a502; merge-base with dev is ac048aa4b8828ada23e2e6145739a70e146af1df (the requested dev merge is already reflected in this worktree; I will not merge again). Initial finding: crates/calternal-server/src/serve.rs applies the 256 cap to every TCP peer, while calternal_auth::client_ip is currently used only by auth rate limiting. This makes all clients behind one trusted proxy share the socket cap.

Started the #235 gate-fix follow-up on branch `job/gate-fix` at HEAD `62b91e4e70cb7fc4b9ea8a17d6224c1cbbc1a502`; merge-base with `dev` is `ac048aa4b8828ada23e2e6145739a70e146af1df` (the requested dev merge is already reflected in this worktree; I will not merge again). Initial finding: `crates/calternal-server/src/serve.rs` applies the 256 cap to every TCP peer, while `calternal_auth::client_ip` is currently used only by auth rate limiting. This makes all clients behind one trusted proxy share the socket cap.

kayg commented

2026-09-27 20:05:40 +00:00

Author

Owner

Second finding with implementation evidence: Hyper 1.11.1's UpgradeableConnection::poll resolves after pending.fulfill(Upgraded::new(...)) (registry hyper-1.11.1/src/server/conn/http1.rs, lines 565-575). The accept-loop permit would therefore drop at the WebSocket handshake unless the collaboration task retains it. I will add a cloneable opaque connection lease through request extensions and retain it in the WebSocket callback. Decision for the unspecified per-client stream value: 256, shared by SSE and WebSocket permits keyed by the IP resolved by the existing trusted-proxy middleware. Proxy-wide connection cap: 4,096 by default, configurable as CALTERNAL_SERVER__MAX_TRUSTED_PROXY_CONNECTIONS. Over-limit stream responses will use 503 with Retry-After: 1.

Second finding with implementation evidence: Hyper 1.11.1's `UpgradeableConnection::poll` resolves after `pending.fulfill(Upgraded::new(...))` (registry `hyper-1.11.1/src/server/conn/http1.rs`, lines 565-575). The accept-loop permit would therefore drop at the WebSocket handshake unless the collaboration task retains it. I will add a cloneable opaque connection lease through request extensions and retain it in the WebSocket callback. Decision for the unspecified per-client stream value: 256, shared by SSE and WebSocket permits keyed by the IP resolved by the existing trusted-proxy middleware. Proxy-wide connection cap: 4,096 by default, configurable as `CALTERNAL_SERVER__MAX_TRUSTED_PROXY_CONNECTIONS`. Over-limit stream responses will use 503 with `Retry-After: 1`.

kayg commented

2026-09-27 20:48:21 +00:00

Author

Owner

Implementation detail: I added a small shared ConnectionLease and client-IP stream permit API in calternal-plugin. The server uses the permit for SSE; the three collaboration WebSocket handlers use the same permit and retain the lease until the upgraded socket task ends. I also exposed is_trusted_proxy from calternal-auth so the TCP accept loop and client_ip use the same trusted-network check. This keeps the 256 cap shared across both stream types and keeps proxy TCP permits counted after Hyper transfers an upgrade.

Implementation detail: I added a small shared `ConnectionLease` and client-IP stream permit API in `calternal-plugin`. The server uses the permit for SSE; the three collaboration WebSocket handlers use the same permit and retain the lease until the upgraded socket task ends. I also exposed `is_trusted_proxy` from `calternal-auth` so the TCP accept loop and `client_ip` use the same trusted-network check. This keeps the 256 cap shared across both stream types and keeps proxy TCP permits counted after Hyper transfers an upgrade.

kayg commented

2026-09-27 20:57:59 +00:00

Author

Owner

Completed #235 on branch job/gate-fix.

Head SHA: 8615c69a494ea8a61a82d80fdc8ae24d2b24b0f9
Push: origin/job/gate-fix

Built: trusted TCP peers bypass the per-peer 256 socket cap and share a configurable global cap (default 4,096). SSE and WebSocket streams use a shared 256-per-resolved-client-IP cap. Over-limit streams return 503 with Retry-After: 1. The accepted-connection lease remains held for the WebSocket task lifetime after Hyper transfers the upgrade.

Files:

crates/calternal-server/src/serve.rs
crates/calternal-server/src/main.rs
crates/calternal-server/src/wire.rs
crates/calternal-auth/src/api.rs
crates/calternal-auth/src/lib.rs
crates/calternal-plugin/src/lib.rs
crates/calternal-collab/src/session.rs
docs/DESIGN.md

Gate output:

cargo test -p calternal-server

test result: ok. 50 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 9.30s

cargo clippy -p calternal-auth -p calternal-plugin -p calternal-collab -p calternal-server --all-targets -- -D warnings

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 21s

cargo fmt --check: no output; exit code 0.

Focused test output:

calternal-auth: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 49 filtered out; finished in 0.00s
calternal-plugin: test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 18 filtered out; finished in 0.01s
calternal-collab: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out; finished in 0.01s

Known gaps: The full workspace test suite was not run; the server suite and focused tests for other touched crates passed. Two existing process-isolation server tests remain ignored by their test marker. Cargo artifacts and generated web build output were removed after verification.

Decisions recorded in docs/DESIGN.md §21: the per-client-IP long-lived stream cap is 256, shared across SSE and WebSocket; the trusted-proxy global TCP cap defaults to 4,096 and uses CALTERNAL_SERVER__MAX_TRUSTED_PROXY_CONNECTIONS.

Completed #235 on branch `job/gate-fix`. Head SHA: `8615c69a494ea8a61a82d80fdc8ae24d2b24b0f9` Push: `origin/job/gate-fix` Built: trusted TCP peers bypass the per-peer 256 socket cap and share a configurable global cap (default 4,096). SSE and WebSocket streams use a shared 256-per-resolved-client-IP cap. Over-limit streams return 503 with `Retry-After: 1`. The accepted-connection lease remains held for the WebSocket task lifetime after Hyper transfers the upgrade. Files: - `crates/calternal-server/src/serve.rs` - `crates/calternal-server/src/main.rs` - `crates/calternal-server/src/wire.rs` - `crates/calternal-auth/src/api.rs` - `crates/calternal-auth/src/lib.rs` - `crates/calternal-plugin/src/lib.rs` - `crates/calternal-collab/src/session.rs` - `docs/DESIGN.md` Gate output: `cargo test -p calternal-server` ```text test result: ok. 50 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 9.30s ``` `cargo clippy -p calternal-auth -p calternal-plugin -p calternal-collab -p calternal-server --all-targets -- -D warnings` ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 21s ``` `cargo fmt --check`: no output; exit code 0. Focused test output: ```text calternal-auth: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 49 filtered out; finished in 0.00s calternal-plugin: test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 18 filtered out; finished in 0.01s calternal-collab: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out; finished in 0.01s ``` Known gaps: The full workspace test suite was not run; the server suite and focused tests for other touched crates passed. Two existing process-isolation server tests remain ignored by their test marker. Cargo artifacts and generated web build output were removed after verification. Decisions recorded in `docs/DESIGN.md` §21: the per-client-IP long-lived stream cap is 256, shared across SSE and WebSocket; the trusted-proxy global TCP cap defaults to 4,096 and uses `CALTERNAL_SERVER__MAX_TRUSTED_PROXY_CONNECTIONS`.

kayg referenced this issue

2026-09-27 20:58:44 +00:00

FOLLOW-UP from #235 adversarial round: DAV/Journal/bookmark timeouts, pasted-image undo duplication, 10k-block collab timeout #265

kayg referenced this issue

2026-09-27 21:03:04 +00:00

FOLLOW-UP from #235 adversarial round: DAV/Journal/bookmark timeouts, pasted-image undo duplication, 10k-block collab timeout #265

kayg referenced this issue from a commit

2026-09-27 21:04:48 +00:00

Merge job/gate-fix (#235)

kayg commented

2026-09-27 21:06:48 +00:00

Author

Owner

Merged in a2dca85b (import conflict resolved). Orchestrator ran full clippy (clean) and the full workspace tests (1,320 passed, 0 failed) on the merge. Proxy-aware connection caps are in; follow-ups in #265.

Merged in a2dca85b (import conflict resolved). Orchestrator ran full clippy (clean) and the full workspace tests (1,320 passed, 0 failed) on the merge. Proxy-aware connection caps are in; follow-ups in #265.

kayg closed this issue

2026-09-27 21:06:49 +00:00

kayg referenced this issue

2026-09-28 01:22:16 +00:00

Owner test tracker: every request and when it is testable on o2 #203

Rows
Columns

GATES: dev failures blocking the first calternal.cloud deploy #235

Failures

Rules

#235 final report

Built

Gates

Adversarial round and remaining work

Decisions not specified in the design

Screenshots