kayg commented

Owner

Problem and evidence

One valid fixture-after-start run found 306 native Podman rows but only 289 API rows, unchanged for 60 seconds across remote and embedded inventory checks; load1 was about 20–23. Later pre-created runs correctly returned all 306. A browser fixture also failed to enter the rendered list within 30 seconds. This is a 17-row stale-inventory correctness bug.

Current event handling publishes a native change before the debounced refresh completes (crates/provider-bollard/src/capabilities.rs:264-307, 1165-1183). That supports the audit's event-publication/cache-ordering hypothesis but does not prove it is the only cause. Evidence is in audit-results.json and continuation-results.json; no CPU profile was collected for this correctness observation.

Recommendation

Coordinate with the existing fix-inventory-mixed2 job already investigating remote Podman inventory staleness. Reproduce the fixture-after-start sequence and ensure the server publishes/notifies only after the complete generation is ready, or forces reconciliation when a reader observes an older generation. Add real Podman coverage for embedded and remote paths; preserve explicit resync on gaps.

Payoff, effort and risk

Expected gain: all 306 rows become visible within the intended refresh bound instead of leaving 17 rows absent for 60 seconds.
Quick win vs deeper change: reproduce and compare publication order in ≤½ day; integrate the fix and real-provider regression in about one day.
Risk: medium; event ordering and resync changes must not publish partial generations or hide true gaps.

Priority: Now (≤1 day; inventory correctness blocker).

2026-10-04 owner decisions: Matches the infraternal agent as source of truth, event-driven server cache, unified embedded/remote path, and 500-host / 20,000-container scale target. Related: #63 (shared event route and resync contract); coordinate with fix-inventory-mixed2 rather than duplicating its implementation.

Report: perf-audit report
Source: perf-audit 2026-10-04

## Problem and evidence One valid fixture-after-start run found 306 native Podman rows but only 289 API rows, unchanged for 60 seconds across remote and embedded inventory checks; load1 was about 20–23. Later pre-created runs correctly returned all 306. A browser fixture also failed to enter the rendered list within 30 seconds. This is a 17-row stale-inventory correctness bug. Current event handling publishes a native change before the debounced refresh completes (crates/provider-bollard/src/capabilities.rs:264-307, 1165-1183). That supports the audit's event-publication/cache-ordering hypothesis but does not prove it is the only cause. Evidence is in [audit-results.json](/mnt/hdd/infraternal/jobs/perf-audit/audit-results.json) and [continuation-results.json](/mnt/hdd/infraternal/jobs/perf-audit/continuation-results.json); no CPU profile was collected for this correctness observation. ## Recommendation Coordinate with the existing fix-inventory-mixed2 job already investigating remote Podman inventory staleness. Reproduce the fixture-after-start sequence and ensure the server publishes/notifies only after the complete generation is ready, or forces reconciliation when a reader observes an older generation. Add real Podman coverage for embedded and remote paths; preserve explicit resync on gaps. ## Payoff, effort and risk **Expected gain:** all 306 rows become visible within the intended refresh bound instead of leaving 17 rows absent for 60 seconds. **Quick win vs deeper change:** reproduce and compare publication order in ≤½ day; integrate the fix and real-provider regression in about one day. **Risk:** medium; event ordering and resync changes must not publish partial generations or hide true gaps. **Priority:** Now (≤1 day; inventory correctness blocker). **2026-10-04 owner decisions:** Matches the infraternal agent as source of truth, event-driven server cache, unified embedded/remote path, and 500-host / 20,000-container scale target. Related: #63 (shared event route and resync contract); coordinate with fix-inventory-mixed2 rather than duplicating its implementation. **Report:** [perf-audit report](/mnt/hdd/infraternal/jobs/perf-audit/report.md) Source: perf-audit 2026-10-04

kayg referenced this issue

2026-10-04 19:04:03 +00:00

Performance audit: release end-to-end hot spots at d1eefa9 #51

kayg referenced this issue from a commit

2026-10-04 20:13:12 +00:00

Merge fix-inventory-mixed: no stale inventory on unavailable lists, coalesced staging (#49, #87)

kayg commented

2026-10-04 23:44:29 +00:00

Author

Owner

Starting fix-containers-503 on branch fix-containers-503, base/head 056ef45435. Reproduce full browser gate, compare 069059d first parent, repair inventory completeness/availability handling and add regression coverage. Requested gates: fast and full browser; no merge, no push.

Starting fix-containers-503 on branch fix-containers-503, base/head 056ef454356fde7488a207bc1ad735c1b04e6b81. Reproduce full browser gate, compare 069059d first parent, repair inventory completeness/availability handling and add regression coverage. Requested gates: fast and full browser; no merge, no push.

kayg commented

2026-10-05 00:00:41 +00:00

Author

Owner

Base 056ef45435 full browser reproduction: release build, Chromium install and terminal fixture passed. Full app step stopped at an existing cold-JS budget: container detail 119410 bytes, maximum 117760. No budget changed. During checkInfrastructureTree, selecting the SSH-only host sends GET /api/v1/containers?host=h_01M44NNN9NGMD40ZC55M70M4RH. It returns typed host_offline, request req_01M44NPYC5VM8EKK1HA8NAJ0G2; the real server log independently records GET /api/v1/containers status 503 under that same request ID. This route has no workload owner, so no complete engine inventory can be served. The outer page-clean collector currently retains this expected SSH-only failure. This request is not a provider refresh/coalescing failure. Further comparison with 069059d first parent and regression tests are in progress. Evidence: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503-before/.

Base 056ef454356fde7488a207bc1ad735c1b04e6b81 full browser reproduction: release build, Chromium install and terminal fixture passed. Full app step stopped at an existing cold-JS budget: container detail 119410 bytes, maximum 117760. No budget changed. During checkInfrastructureTree, selecting the SSH-only host sends GET /api/v1/containers?host=h_01M44NNN9NGMD40ZC55M70M4RH. It returns typed host_offline, request req_01M44NPYC5VM8EKK1HA8NAJ0G2; the real server log independently records GET /api/v1/containers status 503 under that same request ID. This route has no workload owner, so no complete engine inventory can be served. The outer page-clean collector currently retains this expected SSH-only failure. This request is not a provider refresh/coalescing failure. Further comparison with 069059d first parent and regression tests are in progress. Evidence: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503-before/.

kayg commented

2026-10-05 00:08:21 +00:00

Author

Owner

Regression commit 15a0ad6: coalesced_unavailable_and_partial_refreshes_preserve_healthy_inventory passed against dev production logic. Both ProviderError::Unavailable and a partial native generation (ResourceExhausted at publication) coalesce to one attempt, preserve healthy Docker rows, remove Podman completeness/stale rows, expose Podman unavailable metadata, reuse the partial cache safely and recover on a later complete refresh. Exact output:

running 1 test
test host_route::inventory_tests::coalesced_unavailable_and_partial_refreshes_preserve_healthy_inventory ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 237 filtered out; finished in 0.34s

No production runtime/API code or memory limits changed. The full-browser first-parent comparison is still building; fast is in progress.

Regression commit 15a0ad6: coalesced_unavailable_and_partial_refreshes_preserve_healthy_inventory passed against dev production logic. Both ProviderError::Unavailable and a partial native generation (ResourceExhausted at publication) coalesce to one attempt, preserve healthy Docker rows, remove Podman completeness/stale rows, expose Podman unavailable metadata, reuse the partial cache safely and recover on a later complete refresh. Exact output: ```text running 1 test test host_route::inventory_tests::coalesced_unavailable_and_partial_refreshes_preserve_healthy_inventory ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 237 filtered out; finished in 0.34s ``` No production runtime/API code or memory limits changed. The full-browser first-parent comparison is still building; fast is in progress.

kayg commented

2026-10-05 00:15:55 +00:00

Author

Owner

Comparison at 069059d first parent 8c16b59ec0 is NOT green. The release build and terminal fixture pass. Before the candidate inventory change, checkInfrastructureTree emits the same retained SSH-only GET /api/v1/containers?host=... HTTP503 with body code host_offline: request req_01M44PGSJW66D911Y285VKHEMW is independently recorded as status503 by the real server log. This excludes 069059d as the introduction of this page-clean503. The parent cold detail JS is119411 bytes>117760. In the diagnostic archive only, cold-budget violations were deferred into the existing final failure list (not removed) to continue the matrix; it subsequently timed out focusing the lifecycle container Stop button before reaching the late page-clean assertion. Thus there is no claim that the exact parent full gate passed, nor a full late401 parent comparison. Artifacts: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503-parent/.

The current fast retry passed Clippy and the new server regression but failed an existing native_http diagnostic logging test introduced by069059d: native_http.rs:686 unwraps an empty capture. Investigating/retrying this separately; no runtime code or bounds were changed.

Comparison at 069059d first parent 8c16b59ec0ddb164c99394c4f01cda58ac8bea08 is NOT green. The release build and terminal fixture pass. Before the candidate inventory change, checkInfrastructureTree emits the same retained SSH-only GET /api/v1/containers?host=... HTTP503 with body code host_offline: request req_01M44PGSJW66D911Y285VKHEMW is independently recorded as status503 by the real server log. This excludes 069059d as the introduction of this page-clean503. The parent cold detail JS is119411 bytes>117760. In the diagnostic archive only, cold-budget violations were deferred into the existing final failure list (not removed) to continue the matrix; it subsequently timed out focusing the lifecycle container Stop button before reaching the late page-clean assertion. Thus there is no claim that the exact parent full gate passed, nor a full late401 parent comparison. Artifacts: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503-parent/. The current fast retry passed Clippy and the new server regression but failed an existing native_http diagnostic logging test introduced by069059d: native_http.rs:686 unwraps an empty capture. Investigating/retrying this separately; no runtime code or bounds were changed.

kayg commented

2026-10-05 00:25:03 +00:00

Author

Owner

The first-parent diagnostic lifecycle focus timeout is a browser fixture viewport issue, not an inventory error: earlier SSH and sidebar screenshot matrices leave the main page at390px; the legacy keyboard flow then expects an inline Stop control that exists only at desktop width. The separate web-perf-now harness had already adapted that action flow to compact menus, explaining why that branch reached the later page-clean assertion. This branch now explicitly sets1440x900 for its desktop keyboard action flow and restores the infrastructure screenshot helper incoming viewport. No UI/product files changed. Current fast with RUST_TEST_THREADS=1 passes Rust workspace tests (296.079s), including coalesced-unavailable/partial regression; remaining JS checks are in progress. Serialized execution also passes the native diagnostic capture test that failed in the parallel run and passed alone. No test was skipped.

kayg commented

2026-10-05 00:28:13 +00:00

Author

Owner

Correction to the previous viewport finding: the current containers UI uses an Actions row menu at ALL widths (packages/web/src/routes/containers/+page.svelte:455), so the legacy inline Stop/Start selectors time out regardless of viewport. The prior viewport assumption was wrong and those changes have been removed. The existing web-perf-now harness already drives the real row menu by keyboard and uses the correct Stop container confirmation label. This branch reuses that harness repair to reach the later 401 diagnosis; product UI code remains untouched.

kayg commented

2026-10-05 00:29:27 +00:00

Author

Owner

Fast passed with exit0 using INFRATERNAL_VERIFY_RUN_ID=fix-containers-503, CARGO_BUILD_JOBS=2, RUST_TEST_THREADS=1, nice10 and the sccache preset. Test serialization avoids the native diagnostic logging capture failure seen in the parallel run; the same diagnostic test passed alone, and no test was skipped. Verbatim stages:

PASS fast: migrations are append-only (0.242s)
PASS fast: workspace package dependencies (0.260s)
PASS fast: gate target-directory and launch diagnostics tests (0.299s)
PASS fast: JavaScript dependencies (0.005s)
PASS fast: advisory review policy tests (0.148s)
PASS fast: supply-chain checks (5.371s)
PASS fast: build web app (10.955s)
PASS fast: cargo fmt --check (1.725s)
PASS fast: cargo clippy (38.545s)
PASS fast: cargo test (296.079s)
PASS fast: svelte-check (26.551s)
PASS fast: UI package svelte-check (5.538s)
PASS fast: stream package typecheck (7.913s)
PASS fast: Vitest (10.830s)
PASS fast: UI package Vitest (13.332s)
PASS fast: stream package Vitest (2.068s)
PASS fast: OpenAPI and TypeScript client drift (107.702s)
PASS fast: terminal TypeScript (4.735s)
PASS fast: terminal Vitest (2.209s)

Strict full browser gate remains blocked by existing#84 cold-JS:119410 bytes against117760. Release build, Chromium install, terminal fixture and the new exact typed SSH503 assertion passed. No budget changed. Final standalone browser output: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503/browser.log. The late401 diagnostic continues separately with deferred budget failures; it is not claimed as a passing gate.

Fast passed with exit0 using INFRATERNAL_VERIFY_RUN_ID=fix-containers-503, CARGO_BUILD_JOBS=2, RUST_TEST_THREADS=1, nice10 and the sccache preset. Test serialization avoids the native diagnostic logging capture failure seen in the parallel run; the same diagnostic test passed alone, and no test was skipped. Verbatim stages: ```text PASS fast: migrations are append-only (0.242s) PASS fast: workspace package dependencies (0.260s) PASS fast: gate target-directory and launch diagnostics tests (0.299s) PASS fast: JavaScript dependencies (0.005s) PASS fast: advisory review policy tests (0.148s) PASS fast: supply-chain checks (5.371s) PASS fast: build web app (10.955s) PASS fast: cargo fmt --check (1.725s) PASS fast: cargo clippy (38.545s) PASS fast: cargo test (296.079s) PASS fast: svelte-check (26.551s) PASS fast: UI package svelte-check (5.538s) PASS fast: stream package typecheck (7.913s) PASS fast: Vitest (10.830s) PASS fast: UI package Vitest (13.332s) PASS fast: stream package Vitest (2.068s) PASS fast: OpenAPI and TypeScript client drift (107.702s) PASS fast: terminal TypeScript (4.735s) PASS fast: terminal Vitest (2.209s) ``` Strict full browser gate remains blocked by existing#84 cold-JS:119410 bytes against117760. Release build, Chromium install, terminal fixture and the new exact typed SSH503 assertion passed. No budget changed. Final standalone browser output: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503/browser.log. The late401 diagnostic continues separately with deferred budget failures; it is not claimed as a passing gate.

kayg commented

2026-10-05 00:34:09 +00:00

Author

Owner

Late401 reproduced in the retained real dev056ef45 release binary using the corrected menu harness (budget failures deferred only in the diagnostic copy). Main-page1 continues polling GET /api/v1/containers while main-page9 terminal pagination fixture rotates the same refresh cookie. At2026-10-05T00:31:52.492963Z refresh req_01M44QJRNBHJ2JJZC674H9EJSY commits200; old-token GET req_01M44QJRNDHWXKP04HVD3FAMP8 returns401 at00:31:52.493625Z, then exact-URL retry req_01M44QJS0CYXKG51VXYW3Y2TMQ returns200 at00:31:52.845409Z. Second case: refresh req_01M44QJWEK0FJBRY7Q9CR7H471200 at00:31:56.385799Z; main GET req_01M44QJWJEFTW7P7Y8187YGSWT401 at00:31:56.496086Z; exact-URL retry req_01M44QJWKJ75WTEPY4ZHRSZHZ4200 at00:31:56.531719Z. This is shared-cookie fixture choreography; client recovery succeeds and no auth production code changes are justified.

Chosen harness fix: unmount the inactive main document at about:blank while secondary terminal fixtures own the shared session, then restore its original route. Preserve all dedicated supported/fallback multi-tab tests, including the intentional real401/200 recovery probes, and keep unexpected401/503 page-clean checks strict. Fix will be rechecked in the diagnostic full matrix. Evidence: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503-diagnostic/diagnostic.log and server.log.

Late401 reproduced in the retained real dev056ef45 release binary using the corrected menu harness (budget failures deferred only in the diagnostic copy). Main-page1 continues polling GET /api/v1/containers while main-page9 terminal pagination fixture rotates the same refresh cookie. At2026-10-05T00:31:52.492963Z refresh req_01M44QJRNBHJ2JJZC674H9EJSY commits200; old-token GET req_01M44QJRNDHWXKP04HVD3FAMP8 returns401 at00:31:52.493625Z, then exact-URL retry req_01M44QJS0CYXKG51VXYW3Y2TMQ returns200 at00:31:52.845409Z. Second case: refresh req_01M44QJWEK0FJBRY7Q9CR7H471200 at00:31:56.385799Z; main GET req_01M44QJWJEFTW7P7Y8187YGSWT401 at00:31:56.496086Z; exact-URL retry req_01M44QJWKJ75WTEPY4ZHRSZHZ4200 at00:31:56.531719Z. This is shared-cookie fixture choreography; client recovery succeeds and no auth production code changes are justified. Chosen harness fix: unmount the inactive main document at about:blank while secondary terminal fixtures own the shared session, then restore its original route. Preserve all dedicated supported/fallback multi-tab tests, including the intentional real401/200 recovery probes, and keep unexpected401/503 page-clean checks strict. Fix will be rechecked in the diagnostic full matrix. Evidence: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503-diagnostic/diagnostic.log and server.log.

kayg commented

2026-10-05 00:47:47 +00:00

Author

Owner

At head 82816b2, the diagnostic copy against the retained real dev release has passed the original light/desktop page-clean assertion and reached font evidence and subsequent viewport work. Only the two early cold JavaScript assertions are deferred into the final failure list in this ignored diagnostic copy; committed budgets remain strict. No unexpected containers 401 from main-page-1 during secondary terminal/pagination fixtures; dedicated locks-only/neither fixtures still exercise real 401/200 recovery. SSH-only sidebar response is explicitly asserted: req_01M44R4J3J44BY471PF7FDC3XW, 503 host_offline. Full diagnostic is still running; this is not a green browser gate.

At head 82816b2, the diagnostic copy against the retained real dev release has passed the original light/desktop page-clean assertion and reached font evidence and subsequent viewport work. Only the two early cold JavaScript assertions are deferred into the final failure list in this ignored diagnostic copy; committed budgets remain strict. No unexpected containers 401 from main-page-1 during secondary terminal/pagination fixtures; dedicated locks-only/neither fixtures still exercise real 401/200 recovery. SSH-only sidebar response is explicitly asserted: req_01M44R4J3J44BY471PF7FDC3XW, 503 host_offline. Full diagnostic is still running; this is not a green browser gate.

kayg commented

2026-10-05 00:49:47 +00:00

Author

Owner

fix-containers-503 (#87, #49)

Head: 82816b2a9531b54df8b16238cec90da9a2afc6bf. Base: 056ef454356fde7488a207bc1ad735c1b04e6b81. No push or merge.

Root cause and comparison

The suspected inventory merge 069059d did not introduce the observed SSH-only host 503. Dev and its first parent 8c16b59ec0ddb164c99394c4f01cda58ac8bea08 both return 503 host_offline for an SSH-only host selected by the sidebar test. That host has no workload owner: crates/server/src/host_route.rs:368 returns ProviderError::Unavailable; crates/server/src/resources.rs:2381 maps it to the typed HTTP response. This is a whole-request failure with nothing to serve, not an engine failure in a combined list. Request IDs: dev req_01M44NPYC5VM8EKK1HA8NAJ0G2; parent req_01M44PGSJW66D911Y285VKHEMW. Saved server logs confirm 503. The browser collector retained the expected console error and later failed page-clean. Parent did not pass the full gate; both versions also violate the existing cold JavaScript budget, and the parent diagnostic encountered obsolete lifecycle markup. This is a negative suspect comparison, not a completed passing/failing binary bisect.

The late containers 401 is separate fixture interference: the primary page keeps polling while a secondary tab rotates the shared session cookie. The old access token is revoked before the replacement reaches the primary tab; the client retries successfully. The saved refresh-race.json records refresh req_01M44QJRNBHJ2JJZC674H9EJSY 200, GET req_01M44QJRNDHWXKP04HVD3FAMP8 401, retry req_01M44QJS0CYXKG51VXYW3Y2TMQ 200; a second sequence repeats this behavior. No production authentication change is warranted by these traces.

Changes

scripts/browser-check.mjs:4672 asserts the exact SSH-only host response is 503 with host_offline, requires the SSH UI to remain usable, and consumes only that exact expected console error. Other unexpected HTTP/console failures remain strict.
scripts/browser-check.mjs:5354 and the second session-list fixture park the primary document on about:blank during secondary-tab terminal fixtures, then restore its URL. Dedicated real simultaneous-tab and stale-token 401/200 tests remain intact.
The browser lifecycle check follows the current keyboard Actions menu and Stop confirmation label; the Hosts sticky-header assertion follows the shared .table-header primitive. These are obsolete test selectors, separate from inventory behavior.
HTTP inventory diagnostics print status, path and request ID synchronously without awaiting bodies or security headers. Playwright documents synchronous response.headers().
Both Docker and Podman fixtures accept INFRATERNAL_TEST_LABEL; this job uses infraternal.test=fix-containers-503.
crates/server/src/host_route/inventory_tests.rs:302 adds a blocked, coalesced refresh regression for unavailable and partial Podman collection. It proves a single refresh attempt, healthy Docker rows plus unavailable metadata, exclusion of stale/partial Podman rows, cached repeat behavior, and later recovery. Production code and bounded limits are unchanged: the backend already handles per-engine failures correctly.

Verification

Targeted regression: 1 passed; 0 failed; 237 filtered out.

INFRATERNAL_VERIFY_RUN_ID=fix-containers-503 scripts/verify fast: PASS with RUST_TEST_THREADS=1, nice 10 and the sccache preset. All tests ran. An earlier parallel run hit the existing native diagnostic capture test at crates/provider-bollard/src/native_http.rs:686; it passed alone, then the full serial fast gate passed. No product change was made for that test.

Fast output (verbatim):

PASS fast: migrations are append-only (0.242s)
PASS fast: workspace package dependencies (0.260s)
PASS fast: gate target-directory and launch diagnostics tests (0.299s)
PASS fast: JavaScript dependencies (0.005s)
PASS fast: advisory review policy tests (0.148s)
PASS fast: supply-chain checks (5.371s)
PASS fast: build web app (10.955s)
PASS fast: cargo fmt --check (1.725s)
PASS fast: cargo clippy (38.545s)
PASS fast: cargo test (296.079s)
PASS fast: svelte-check (26.551s)
PASS fast: UI package svelte-check (5.538s)
PASS fast: stream package typecheck (7.913s)
PASS fast: Vitest (10.830s)
PASS fast: UI package Vitest (13.332s)
PASS fast: stream package Vitest (2.068s)
PASS fast: OpenAPI and TypeScript client drift (107.702s)
PASS fast: terminal TypeScript (4.735s)
PASS fast: terminal Vitest (2.209s)

Strict full scripts/verify browser: FAIL at the unchanged container-detail cold JavaScript budget: 119410 bytes > 117760 bytes (#84). Budgets were not relaxed in committed code. Gate output (verbatim):

PASS browser: JavaScript dependencies (0.015s)
PASS browser: build web app (16.032s)
PASS browser: build release binary (312.154s)
PASS browser: install Chromium and system dependencies (6.358s)
PASS browser: terminal fixture Playwright (84.691s)
FAIL browser: Playwright against release server (79.917s)

The strict browser run above predates the final fixture-parking/header diagnostics commits. A separate diagnostic copy uses the retained real dev release, moves the two early cold-JavaScript failures to the final failure list, and exercises the later assertions; it still fails on those budget findings and is not a passing gate. Final diagnostic outcome and cleanup are recorded below after completion.

No verify merge, performance-VM benchmark, push, or merge ran. No new production hot path was introduced. The requested passing strict full browser gate remains outstanding because of #84.

Final diagnostic outcome and cleanup

At final head, the diagnostic passed the original light/desktop page-clean assertion, real 51-session pagination, terminal stream and reattachment checks, and the light desktop/medium/tablet/mobile route captures. It recorded zero primary-page containers 401 responses. The dedicated stale-token fixtures still recorded their real 401 responses and successful recovery. It did not complete the dark matrix: at light/narrow (320px), scripts/browser-check.mjs:1336 rejected a clipped SSH key-rotation fixture label extending to x=331.125px. This is an additional layout/fixture-name-sensitive blocker; it was not hidden by shortening the diagnostic fixture name. Diagnostic exit 1, not a passing browser gate.

Final browser script and verification shell syntax checks and git diff --check pass. Docker and rootless Podman contain zero objects with infraternal.test=fix-containers-503 after cleanup. The owned diagnostic server shut down and its temporary database directory was removed. Saved server setup-token lines are redacted. The earlier interrupted diagnostic's seven unremoved API objects belonged to its removed temporary database; owned runtime containers were cleaned.

Artifacts: /mnt/hdd/infraternal/artifacts/verify/fix-containers-503/fast.log, browser.log; before and parent gate logs under fix-containers-503-before and fix-containers-503-parent; final diagnostic log, server log, refresh-race JSON and screenshots under fix-containers-503-diagnostic.

Remaining requirement: a passing strict full browser gate. #84 is the first strict blocker; after that, the narrow fixture-label layout assertion needs follow-up. This branch is ready for independent review of the scoped harness/regression changes, not for a merge claim.

# fix-containers-503 (#87, #49) Head: `82816b2a9531b54df8b16238cec90da9a2afc6bf`. Base: `056ef454356fde7488a207bc1ad735c1b04e6b81`. No push or merge. ## Root cause and comparison The suspected inventory merge `069059d` did not introduce the observed SSH-only host 503. Dev and its first parent `8c16b59ec0ddb164c99394c4f01cda58ac8bea08` both return `503 host_offline` for an SSH-only host selected by the sidebar test. That host has no workload owner: `crates/server/src/host_route.rs:368` returns `ProviderError::Unavailable`; `crates/server/src/resources.rs:2381` maps it to the typed HTTP response. This is a whole-request failure with nothing to serve, not an engine failure in a combined list. Request IDs: dev `req_01M44NPYC5VM8EKK1HA8NAJ0G2`; parent `req_01M44PGSJW66D911Y285VKHEMW`. Saved server logs confirm 503. The browser collector retained the expected console error and later failed page-clean. Parent did not pass the full gate; both versions also violate the existing cold JavaScript budget, and the parent diagnostic encountered obsolete lifecycle markup. This is a negative suspect comparison, not a completed passing/failing binary bisect. The late containers 401 is separate fixture interference: the primary page keeps polling while a secondary tab rotates the shared session cookie. The old access token is revoked before the replacement reaches the primary tab; the client retries successfully. The saved `refresh-race.json` records refresh `req_01M44QJRNBHJ2JJZC674H9EJSY` 200, GET `req_01M44QJRNDHWXKP04HVD3FAMP8` 401, retry `req_01M44QJS0CYXKG51VXYW3Y2TMQ` 200; a second sequence repeats this behavior. No production authentication change is warranted by these traces. ## Changes - `scripts/browser-check.mjs:4672` asserts the exact SSH-only host response is 503 with `host_offline`, requires the SSH UI to remain usable, and consumes only that exact expected console error. Other unexpected HTTP/console failures remain strict. - `scripts/browser-check.mjs:5354` and the second session-list fixture park the primary document on `about:blank` during secondary-tab terminal fixtures, then restore its URL. Dedicated real simultaneous-tab and stale-token 401/200 tests remain intact. - The browser lifecycle check follows the current keyboard Actions menu and Stop confirmation label; the Hosts sticky-header assertion follows the shared `.table-header` primitive. These are obsolete test selectors, separate from inventory behavior. - HTTP inventory diagnostics print status, path and request ID synchronously without awaiting bodies or security headers. Playwright documents synchronous [response.headers()](https://playwright.dev/docs/api/class-response#response-headers). - Both Docker and Podman fixtures accept `INFRATERNAL_TEST_LABEL`; this job uses `infraternal.test=fix-containers-503`. - `crates/server/src/host_route/inventory_tests.rs:302` adds a blocked, coalesced refresh regression for unavailable and partial Podman collection. It proves a single refresh attempt, healthy Docker rows plus unavailable metadata, exclusion of stale/partial Podman rows, cached repeat behavior, and later recovery. Production code and bounded limits are unchanged: the backend already handles per-engine failures correctly. ## Verification Targeted regression: `1 passed; 0 failed; 237 filtered out`. `INFRATERNAL_VERIFY_RUN_ID=fix-containers-503 scripts/verify fast`: PASS with `RUST_TEST_THREADS=1`, nice 10 and the sccache preset. All tests ran. An earlier parallel run hit the existing native diagnostic capture test at `crates/provider-bollard/src/native_http.rs:686`; it passed alone, then the full serial fast gate passed. No product change was made for that test. Fast output (verbatim): ``` PASS fast: migrations are append-only (0.242s) PASS fast: workspace package dependencies (0.260s) PASS fast: gate target-directory and launch diagnostics tests (0.299s) PASS fast: JavaScript dependencies (0.005s) PASS fast: advisory review policy tests (0.148s) PASS fast: supply-chain checks (5.371s) PASS fast: build web app (10.955s) PASS fast: cargo fmt --check (1.725s) PASS fast: cargo clippy (38.545s) PASS fast: cargo test (296.079s) PASS fast: svelte-check (26.551s) PASS fast: UI package svelte-check (5.538s) PASS fast: stream package typecheck (7.913s) PASS fast: Vitest (10.830s) PASS fast: UI package Vitest (13.332s) PASS fast: stream package Vitest (2.068s) PASS fast: OpenAPI and TypeScript client drift (107.702s) PASS fast: terminal TypeScript (4.735s) PASS fast: terminal Vitest (2.209s) ``` Strict full `scripts/verify browser`: FAIL at the unchanged container-detail cold JavaScript budget: **119410 bytes > 117760 bytes** (#84). Budgets were not relaxed in committed code. Gate output (verbatim): ``` PASS browser: JavaScript dependencies (0.015s) PASS browser: build web app (16.032s) PASS browser: build release binary (312.154s) PASS browser: install Chromium and system dependencies (6.358s) PASS browser: terminal fixture Playwright (84.691s) FAIL browser: Playwright against release server (79.917s) ``` The strict browser run above predates the final fixture-parking/header diagnostics commits. A separate diagnostic copy uses the retained real dev release, moves the two early cold-JavaScript failures to the final failure list, and exercises the later assertions; it still fails on those budget findings and is not a passing gate. Final diagnostic outcome and cleanup are recorded below after completion. No `verify merge`, performance-VM benchmark, push, or merge ran. No new production hot path was introduced. The requested passing strict full browser gate remains outstanding because of #84. ## Final diagnostic outcome and cleanup At final head, the diagnostic passed the original light/desktop page-clean assertion, real 51-session pagination, terminal stream and reattachment checks, and the light desktop/medium/tablet/mobile route captures. It recorded zero primary-page containers 401 responses. The dedicated stale-token fixtures still recorded their real 401 responses and successful recovery. It did **not** complete the dark matrix: at light/narrow (320px), `scripts/browser-check.mjs:1336` rejected a clipped SSH key-rotation fixture label extending to x=331.125px. This is an additional layout/fixture-name-sensitive blocker; it was not hidden by shortening the diagnostic fixture name. Diagnostic exit 1, not a passing browser gate. Final browser script and verification shell syntax checks and `git diff --check` pass. Docker and rootless Podman contain zero objects with `infraternal.test=fix-containers-503` after cleanup. The owned diagnostic server shut down and its temporary database directory was removed. Saved server setup-token lines are redacted. The earlier interrupted diagnostic's seven unremoved API objects belonged to its removed temporary database; owned runtime containers were cleaned. Artifacts: `/mnt/hdd/infraternal/artifacts/verify/fix-containers-503/fast.log`, `browser.log`; before and parent gate logs under `fix-containers-503-before` and `fix-containers-503-parent`; final diagnostic log, server log, refresh-race JSON and screenshots under `fix-containers-503-diagnostic`. Remaining requirement: a passing strict full browser gate. #84 is the first strict blocker; after that, the narrow fixture-label layout assertion needs follow-up. This branch is ready for independent review of the scoped harness/regression changes, not for a merge claim.

kayg referenced this issue

2026-10-05 00:49:47 +00:00

Provider plugins: Docker, Podman, Kubernetes, LXC, Proxmox as plugins (architecture refactor) #49

kayg commented

2026-10-05 01:09:19 +00:00

Author

Owner

The inherited 320px diagnostic recorded light/narrow: container table controls do not fit: the DataTable host-filter button SSH key rotation fix-containers-503-diagnostic measured 315.125px wide and ended at x=331.125px on a 320px viewport. The shared Pill chip fixes its height at 24px and keeps labels on one line. I changed only DataTable filter chips to allow their dynamic labels to wrap and grow while preserving the existing strict fit assertion.

The inherited 320px diagnostic recorded `light/narrow: container table controls do not fit`: the DataTable host-filter button `SSH key rotation fix-containers-503-diagnostic` measured 315.125px wide and ended at x=331.125px on a 320px viewport. The shared Pill chip fixes its height at 24px and keeps labels on one line. I changed only DataTable filter chips to allow their dynamic labels to wrap and grow while preserving the existing strict fit assertion.

kayg referenced this issue

2026-10-05 01:34:01 +00:00

Handover: start here (orchestrator on the dev VM) #16

kayg referenced this issue from a commit

2026-10-05 04:33:42 +00:00

Merge browser-green: precompressed assets with identity default, cold-JS under budget, search cache, narrow-screen wrapping, browser harness fixes (#82, #84, #79, #87)