Performance audit: release end-to-end hot spots at d1eefa9

kayg commented

2026-10-04 16:09:16 +00:00

Owner

Starting owner-requested performance audit. Worktree /mnt/hdd/infraternal/wt-lp/perf-audit, detached HEAD/base d1eefa9a1bbd68b1ff75fdfb394aebb094d64cd7. No product changes or pushes. Artifacts and report under /mnt/hdd/infraternal/jobs/perf-audit/. Measure release server, embedded/remote infraternal agents, Docker/rootless Podman with 300+ real labeled fixtures, streams, SQLite and Chromium. This explicitly requested shared-dev-VM audit records load and does not certify perf-test-VM budgets. Previous architectural review contains estimates only.

Starting owner-requested performance audit. Worktree `/mnt/hdd/infraternal/wt-lp/perf-audit`, detached HEAD/base `d1eefa9a1bbd68b1ff75fdfb394aebb094d64cd7`. No product changes or pushes. Artifacts and report under `/mnt/hdd/infraternal/jobs/perf-audit/`. Measure release server, embedded/remote infraternal agents, Docker/rootless Podman with 300+ real labeled fixtures, streams, SQLite and Chromium. This explicitly requested shared-dev-VM audit records load and does not certify perf-test-VM budgets. Previous architectural review contains estimates only.

kayg commented

2026-10-04 16:19:49 +00:00

Author

Owner

Release measurements are waiting at the existing scripts/verify perf admission loop: load1 has ranged 14.3–22.7 on this 8-vCPU shared dev VM. Hardware perf stat instructions/cycles are available (953,138 instructions for the availability probe /bin/true); Debian linux-perf and heaptrack installed. Code index built at d1eefa9. The older review at 450c297 is stale in three areas: DATA batches now share one SQLite fence; provider events are debounced and summary-irrelevant actions skipped; remote snapshot publication sends epoch notifications rather than every container. These are source observations, not claimed performance gains. A production web build has 2,291,200 total static bytes, of which JS is 1,744,641 raw / 533,182 gzip across all chunks, including lazy chunks; this is not cold-route size. Test scripts outside the worktree prepare 305 stopped fixtures per provider plus live streams, repeated embedded/mTLS HTTP and stream measurements, owned-process instruction counters and timelines. Every fixture is labeled infraternal.test=perf-audit.

Release measurements are waiting at the existing `scripts/verify perf` admission loop: load1 has ranged 14.3–22.7 on this 8-vCPU shared dev VM. Hardware `perf stat` instructions/cycles are available (953,138 instructions for the availability probe `/bin/true`); Debian linux-perf and heaptrack installed. Code index built at d1eefa9. The older review at 450c297 is stale in three areas: DATA batches now share one SQLite fence; provider events are debounced and summary-irrelevant actions skipped; remote snapshot publication sends epoch notifications rather than every container. These are source observations, not claimed performance gains. A production web build has 2,291,200 total static bytes, of which JS is 1,744,641 raw / 533,182 gzip across all chunks, including lazy chunks; this is not cold-route size. Test scripts outside the worktree prepare 305 stopped fixtures per provider plus live streams, repeated embedded/mTLS HTTP and stream measurements, owned-process instruction counters and timelines. Every fixture is labeled infraternal.test=perf-audit.

kayg commented

2026-10-04 16:29:44 +00:00

Author

Owner

The audit release build is progressing with nice -n 15 cargo build --release -p infraternal -j 2 in the owned worktree/target after load1 fell below 10. The unmodified perf gate retains its full admission checks and will reuse the resulting artifacts when admitted. No product changes. Process observation correction: inspecting /proc/<pid>/task/<pid>/children alone misses compiler children spawned by Cargo worker threads, so empty main-thread children do not prove inactive builds. The diagnostic source-import closure finds /hosts/:id at 122,798 gzip bytes, 82 bytes below the absolute 122,880-byte budget before browser-only dynamic imports; browser evidence pending.

The audit release build is progressing with `nice -n 15 cargo build --release -p infraternal -j 2` in the owned worktree/target after load1 fell below 10. The unmodified perf gate retains its full admission checks and will reuse the resulting artifacts when admitted. No product changes. Process observation correction: inspecting `/proc/<pid>/task/<pid>/children` alone misses compiler children spawned by Cargo worker threads, so empty main-thread children do not prove inactive builds. The diagnostic source-import closure finds `/hosts/:id` at 122,798 gzip bytes, 82 bytes below the absolute 122,880-byte budget before browser-only dynamic imports; browser evidence pending.

kayg commented

2026-10-04 16:41:45 +00:00

Author

Owner

New source finding at d1eefa9: production ContainerLogPanel, exec-session and TerminalPane explicitly request recv_window=256 KiB. StreamClient returns one coalesced CREDIT globally every 6 ms (packages/stream/src/index.ts:810). The stream benchmark gives logs 4 MiB (packages/stream/src/benchmark_workload.ts:31) while exec gets 256 KiB. A 256 KiB window drained before credit is returned has an approximate 41.7 MiB/s timer ceiling before scheduling/network overhead; this is a mechanism estimate, not a measured throughput. The audit now A/Bs 256 KiB, 1 MiB and 4 MiB with repeated byte-exact 32 MiB transfers on all embedded/remote Docker/Podman paths. The old review also overstated health-check event frequency: Docker documents health_status emission on status changes, not every healthy check (https://docs.docker.com/reference/dockerfile/#healthcheck).

New source finding at d1eefa9: production ContainerLogPanel, exec-session and TerminalPane explicitly request recv_window=256 KiB. StreamClient returns one coalesced CREDIT globally every 6 ms (packages/stream/src/index.ts:810). The stream benchmark gives logs 4 MiB (packages/stream/src/benchmark_workload.ts:31) while exec gets 256 KiB. A 256 KiB window drained before credit is returned has an approximate 41.7 MiB/s timer ceiling before scheduling/network overhead; this is a mechanism estimate, not a measured throughput. The audit now A/Bs 256 KiB, 1 MiB and 4 MiB with repeated byte-exact 32 MiB transfers on all embedded/remote Docker/Podman paths. The old review also overstated health-check event frequency: Docker documents health_status emission on status changes, not every healthy check (https://docs.docker.com/reference/dockerfile/#healthcheck).

kayg commented

2026-10-04 16:55:03 +00:00

Author

Owner

Release build completed: Finished release profile [optimized] target(s) in 24m 06s (cold target, two workers, shared VM; not a runtime budget comparison). Stripped artifact /mnt/hdd/infraternal/jobs/perf-audit/product-stripped is 33,465,288 decimal bytes vs 25,000,000 budget, an 8,465,288-byte overage. At 2026-10-04T16:52:35Z, load1/5/15 = 23.46/16.56/13.59, embedded server RSS 25,904 KiB; standalone infraternal agent RSS 15,748 KiB (15.379 MiB), with file-backed 13,532 KiB and anonymous 2,216 KiB. /proc smaps later attributes 12,352 KiB to executable pages in the standalone infraternal agent. Inventories were actively growing during that later sample; this is not an idle certification. The full static bundle is only 2,291,200 bytes, so removing all web assets alone cannot close the binary overage. More provider/tunnel, throughput, allocation and browser measurements follow.

Release build completed: `Finished `release` profile [optimized] target(s) in 24m 06s` (cold target, two workers, shared VM; not a runtime budget comparison). Stripped artifact `/mnt/hdd/infraternal/jobs/perf-audit/product-stripped` is 33,465,288 decimal bytes vs 25,000,000 budget, an 8,465,288-byte overage. At 2026-10-04T16:52:35Z, load1/5/15 = 23.46/16.56/13.59, embedded server RSS 25,904 KiB; standalone infraternal agent RSS 15,748 KiB (15.379 MiB), with file-backed 13,532 KiB and anonymous 2,216 KiB. `/proc` smaps later attributes 12,352 KiB to executable pages in the standalone infraternal agent. Inventories were actively growing during that later sample; this is not an idle certification. The full static bundle is only 2,291,200 bytes, so removing all web assets alone cannot close the binary overage. More provider/tunnel, throughput, allocation and browser measurements follow.

kayg commented

2026-10-04 17:01:46 +00:00

Author

Owner

Measured availability delay at d1eefa9 (second run): native runtime had 306 labeled Podman fixtures (305 created/stopped + one running), while GET /api/v1/containers?host=&engine=podman&text= returned HTTP 200, total 289, unchanged from 16:58:55Z through the 16:59:55Z 60-second deadline. Load1 ranged approximately 19.8–23.2. Embedded Docker initially returned 502 engine_unavailable but recovered before its deadline. Both runs cleaned all owned fixtures; no demo containers touched. Raw responses/statuses and timestamps are in audit-second-failure.json. Cause is not established; source has independent native refresh and runtime reconciliation timers, with events published before refresh completion, which is a race candidate. To avoid treating failure as latency, the next capacity audit creates fixtures before server/runtime startup, so the initial complete snapshot is read from real engines. The churn failure remains a finding.

Measured availability delay at d1eefa9 (second run): native runtime had 306 labeled Podman fixtures (305 created/stopped + one running), while GET /api/v1/containers?host=<embedded>&engine=podman&text=<unique-run> returned HTTP 200, total 289, unchanged from 16:58:55Z through the 16:59:55Z 60-second deadline. Load1 ranged approximately 19.8–23.2. Embedded Docker initially returned 502 engine_unavailable but recovered before its deadline. Both runs cleaned all owned fixtures; no demo containers touched. Raw responses/statuses and timestamps are in audit-second-failure.json. Cause is not established; source has independent native refresh and runtime reconciliation timers, with events published before refresh completion, which is a race candidate. To avoid treating failure as latency, the next capacity audit creates fixtures before server/runtime startup, so the initial complete snapshot is read from real engines. The churn failure remains a finding.

kayg commented

2026-10-04 17:52:47 +00:00

Author

Owner

Architecture-review follow-up to the performance audit (2026-10-04):

The report measured an empty-target cargo build -p infraternal-provider-api at 51.221 s and a warm build at 0.327 s. A timestamp-invalidated server library rebuild took 3.208 s under load1 19–22, but that is not a semantic-edit or final-binary build measurement. The scripts/verify fast result was 864.038 s and is gate/build diagnostics, not a product-performance result.

Recommendation: after the higher-priority provider/session work, measure a separate local feedback profile and cache stability under controlled build conditions. Keep the shipping release profile, real production-browser checks and perf-test-VM A/B unchanged. No build-profile speedup is claimed by the architecture review.

Payoff: a measured local edit loop without changing shipped behavior. Estimate: 1 engineering day for a controlled profile/cache comparison. Priority: Later.

2026-10-04 owner decisions: matches performance-first work and the requirement to keep release verification unchanged.

Report: /mnt/hdd/infraternal/jobs/arch-review/report.md
Source: arch-review 2026-10-04.

Architecture-review follow-up to the performance audit (2026-10-04): The report measured an empty-target `cargo build -p infraternal-provider-api` at 51.221 s and a warm build at 0.327 s. A timestamp-invalidated server library rebuild took 3.208 s under load1 19–22, but that is not a semantic-edit or final-binary build measurement. The `scripts/verify fast` result was 864.038 s and is gate/build diagnostics, not a product-performance result. Recommendation: after the higher-priority provider/session work, measure a separate local feedback profile and cache stability under controlled build conditions. Keep the shipping release profile, real production-browser checks and perf-test-VM A/B unchanged. No build-profile speedup is claimed by the architecture review. Payoff: a measured local edit loop without changing shipped behavior. Estimate: 1 engineering day for a controlled profile/cache comparison. **Priority: Later.** 2026-10-04 owner decisions: matches performance-first work and the requirement to keep release verification unchanged. Report: /mnt/hdd/infraternal/jobs/arch-review/report.md Source: arch-review 2026-10-04.

kayg commented

2026-10-04 18:01:11 +00:00

Author

Owner

Measured findings at d1eefa9 (shared VM diagnostics): native allocator maxima for 500 Docker fixtures are 49,740 / 46,716 / 69,443 bytes across three runs, all below 128 KiB. Named CPU profile puts 26.09% of full-list CPU in summary filtering; finalized heaptrack contains 36,054,590 allocation calls, 20,229,060 through filtering. Remote 32 MiB stdin uploads closed the tunnel after 9,158,656 bytes (Docker) and 8,159,232 bytes (Podman). UI default 256 KiB versus 1 MiB recv windows measured remote Docker output medians 16.381 / 29.793 MiB/s. Real browser cold JS: Containers 117,684 gzip bytes, Host detail 123,055 (175 over 120 KiB), all identity transfer. 16,324 separate echoed input frames produced zero WAL growth; SQLx is 32.96% of their CPU profile. Idle trace counts 7,136 server and 921 standalone wakeups over 59.989 seconds. Formal perf gate failed on stale Search textbox selector, after real builds and evaluator tests; partial.json is incomplete and is not a baseline. Detailed report follows.

Measured findings at d1eefa9 (shared VM diagnostics): native allocator maxima for 500 Docker fixtures are 49,740 / 46,716 / 69,443 bytes across three runs, all below 128 KiB. Named CPU profile puts 26.09% of full-list CPU in summary filtering; finalized heaptrack contains 36,054,590 allocation calls, 20,229,060 through filtering. Remote 32 MiB stdin uploads closed the tunnel after 9,158,656 bytes (Docker) and 8,159,232 bytes (Podman). UI default 256 KiB versus 1 MiB recv windows measured remote Docker output medians 16.381 / 29.793 MiB/s. Real browser cold JS: Containers 117,684 gzip bytes, Host detail 123,055 (175 over 120 KiB), all identity transfer. 16,324 separate echoed input frames produced zero WAL growth; SQLx is 32.96% of their CPU profile. Idle trace counts 7,136 server and 921 standalone wakeups over 59.989 seconds. Formal perf gate failed on stale Search textbox selector, after real builds and evaluator tests; partial.json is incomplete and is not a baseline. Detailed report follows.

kayg referenced this issue

2026-10-04 18:03:21 +00:00

arch-review: provider seams and feature delivery cost #52

kayg commented

2026-10-04 18:18:27 +00:00

Author

Owner

Finished performance audit. HEAD/base remains d1eefa9a1bbd68b1ff75fdfb394aebb094d64cd7; tracked tree clean; no commits or pushes. Final report: /mnt/hdd/infraternal/jobs/perf-audit/report.md. CPU and allocation flamegraphs, Chromium traces, real fixture samples and cleanup evidence are alongside it.

Final additional measurement: real raw PTY 32 MiB transfers, three repeats per path/window. Remote Docker 256 KiB→1 MiB window medians 18.141→43.451 MiB/s (2.40×); remote Podman 23.595→67.354 (2.85×). All byte counts and exit codes asserted. An all-NUL Docker PTY output separately exited 0 with zero DATA bytes; record preserves this completion defect. Tiny-write yes throughput was source-bound and is excluded as capacity evidence. Finalized heaptrack: 36,054,590 allocations; filtering 56.11%, search 6,622,967 calls. Quiet 304 maximum 69,443 bytes (128 KiB cap passes); Containers JS 117,684 gzip bytes passes the 120 KiB cap, Host detail exceeds by 175 bytes. Idle actual wakeups about 119/s server and 15.4/s standalone. Ranked fixes separate ≤½-day changes from deeper work and state expected gain/risk.

Formal gate output, verbatim:

PASS perf: JavaScript dependencies (0.019s)
PASS perf: build web app (39.244s)
PASS perf: build release binary (533.815s)
PASS perf: install Chromium and system dependencies (7.761s)
PASS perf: evaluator tests (1.532s)
PASS perf: build process CPU clock sampler (0.236s)
PASS perf: build allocation measurement harness (1152.865s)
FAIL perf: measured performance budgets (724.342s)

The formal gate failed on the stale Search textbox selector; current UI has Filter Containers searchbox and explicit Open logs. partial.json is incomplete; no valid calibration or release certification was produced. Measurements are shared-dev-VM diagnostics, not perf-test-VM A/B evidence.

Cleanup verified zero fixtures with infraternal.test=perf-audit in Docker and rootless Podman, no owned product processes, stopped load monitor and removed known private test-server directories. The interrupted continuation bulk cleanup left 258 exact-run Podman fixtures; final exact-ID batches removed them with demo labels excluded. cleanup.json and cleanup-orphan-ids.json retain evidence. Issue stays open for the orchestrator.

Finished performance audit. HEAD/base remains `d1eefa9a1bbd68b1ff75fdfb394aebb094d64cd7`; tracked tree clean; no commits or pushes. Final report: `/mnt/hdd/infraternal/jobs/perf-audit/report.md`. CPU and allocation flamegraphs, Chromium traces, real fixture samples and cleanup evidence are alongside it. Final additional measurement: real raw PTY 32 MiB transfers, three repeats per path/window. Remote Docker 256 KiB→1 MiB window medians 18.141→43.451 MiB/s (2.40×); remote Podman 23.595→67.354 (2.85×). All byte counts and exit codes asserted. An all-NUL Docker PTY output separately exited 0 with zero DATA bytes; record preserves this completion defect. Tiny-write `yes` throughput was source-bound and is excluded as capacity evidence. Finalized heaptrack: 36,054,590 allocations; filtering 56.11%, search 6,622,967 calls. Quiet 304 maximum 69,443 bytes (128 KiB cap passes); Containers JS 117,684 gzip bytes passes the 120 KiB cap, Host detail exceeds by 175 bytes. Idle actual wakeups about 119/s server and 15.4/s standalone. Ranked fixes separate ≤½-day changes from deeper work and state expected gain/risk. Formal gate output, verbatim: ```text PASS perf: JavaScript dependencies (0.019s) PASS perf: build web app (39.244s) PASS perf: build release binary (533.815s) PASS perf: install Chromium and system dependencies (7.761s) PASS perf: evaluator tests (1.532s) PASS perf: build process CPU clock sampler (0.236s) PASS perf: build allocation measurement harness (1152.865s) FAIL perf: measured performance budgets (724.342s) ``` The formal gate failed on the stale `Search` textbox selector; current UI has `Filter Containers` searchbox and explicit `Open logs`. `partial.json` is incomplete; no valid calibration or release certification was produced. Measurements are shared-dev-VM diagnostics, not perf-test-VM A/B evidence. Cleanup verified zero fixtures with `infraternal.test=perf-audit` in Docker and rootless Podman, no owned product processes, stopped load monitor and removed known private test-server directories. The interrupted continuation bulk cleanup left 258 exact-run Podman fixtures; final exact-ID batches removed them with demo labels excluded. `cleanup.json` and `cleanup-orphan-ids.json` retain evidence. Issue stays open for the orchestrator.

kayg commented

2026-10-04 18:24:37 +00:00

Author

Owner

Starting the owner-requested issue-filing pass for the 2026-10-04 performance audit.

Worktree: /mnt/hdd/infraternal/wt-lp/issues-perf (detached at dev); base HEAD: 708580114d325690d5f8b407168c4e5d0b7bbd07. I am checking each recommendation against the current source and searching issues #53–#75 for overlap before filing. No repository files are being changed.

Starting the owner-requested issue-filing pass for the 2026-10-04 performance audit. Worktree: `/mnt/hdd/infraternal/wt-lp/issues-perf` (detached at dev); base HEAD: `708580114d325690d5f8b407168c4e5d0b7bbd07`. I am checking each recommendation against the current source and searching issues #53–#75 for overlap before filing. No repository files are being changed.

kayg commented

2026-10-04 19:04:03 +00:00

Author

Owner

Finished the audit issue-filing pass.

Worktree: /mnt/hdd/infraternal/wt-lp/issues-perf, detached at the dev base. HEAD: 708580114d. No repository files changed and no commit was made.

Created issues:

#77 remote 32 MiB stdin stalls near 8 MiB
#78 raw PTY output loses NUL-prefixed bytes
#79 container search-text/list allocation
#80 bounded receive-window experiment
#81 live log serialization throughput
#82 embedded asset compression
#83 browser log decoder allocation
#84 Host detail cold JavaScript over budget
#85 duplicate crypto backend/binary-size audit
#86 idle authorization polling and wakeups
#87 Podman inventory stuck at 289/306
#88 standalone infraternal agent RSS
#89 stale perf-gate selectors
#90 performance budget definitions

Added the sparse-input authorization profile and performance target to existing issue #67 rather than duplicating its authority/session work. Related architecture issues are cross-linked from the new bodies.

Gate output: “Not run: read-only issue-filing task; no product code changed.”

Finished the audit issue-filing pass. Worktree: /mnt/hdd/infraternal/wt-lp/issues-perf, detached at the dev base. HEAD: 708580114d325690d5f8b407168c4e5d0b7bbd07. No repository files changed and no commit was made. Created issues: - #77 remote 32 MiB stdin stalls near 8 MiB - #78 raw PTY output loses NUL-prefixed bytes - #79 container search-text/list allocation - #80 bounded receive-window experiment - #81 live log serialization throughput - #82 embedded asset compression - #83 browser log decoder allocation - #84 Host detail cold JavaScript over budget - #85 duplicate crypto backend/binary-size audit - #86 idle authorization polling and wakeups - #87 Podman inventory stuck at 289/306 - #88 standalone infraternal agent RSS - #89 stale perf-gate selectors - #90 performance budget definitions Added the sparse-input authorization profile and performance target to existing issue #67 rather than duplicating its authority/session work. Related architecture issues are cross-linked from the new bodies. Gate output: “Not run: read-only issue-filing task; no product code changed.”

kayg referenced this issue

2026-10-05 05:22:28 +00:00

Foundations design: one Host model, runtime and event route #92

Rows
Columns

Performance audit: release end-to-end hot spots at d1eefa9 #51

Performance audit: release end-to-end hot spots at `d1eefa9` #51