Audit 2026-10-02: gaps, holes and the shortest path to a usable product #48
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Owner request (2026-10-02 18:15 IST): "what is the least amount of features we can get stable and stupid fast right now? we need to make the product architecturally very modular (code-reuse, extensible) and very very performant with the least cpu and memory usage. … audit … find gaps, holes and … the best direction forward to a usable product." The owner also asked how the calternal UI/UX adoption (#46) is going.
Seven read-only audit jobs run in parallel. Each comments here with its findings and a link to its full report.
The orchestrator writes the synthesis and the recommended direction.
Base:
devf640461, as deployed on o2. Port branch:calternal-portb3b125e.Starting audit-product: read-only detached worktree /home/kayg/wt/audit-base at
f640461228(deployed dev). Inventory dev and named unmerged branches, issues #16–#47; recommend smallest stable fast v0.1. Report: /mnt/hdd/infraternal/jobs/audit-product/report.md. No commits, push, deployment or tracked edits.Starting read-only audit-perf on worktree /home/kayg/wt/audit-base.
Base: detached dev
f640461228.Starting the read-only calternal UI/UX audit for #48.
Worktree:
/home/kayg/wt/audit-web, detached atb3b125e7fb19f6d66ba82f1244c1c1120df77783(calternal-port); audit comparison base isdevf640461. Current calternal source isc4a61e8cfand remains read-only. Report and screenshots will be under/mnt/hdd/infraternal/jobs/audit-calternal/and/mnt/hdd/infraternal/artifacts/audit-calternal/.audit-security started: read-only detached worktree /home/kayg/wt/audit-base; base/head
f640461228. Reviewing authentication, authorization, remote infraternal agents, exec/SSH, audit redaction, resource bounds and supply chain. Evidence and progress: /mnt/hdd/infraternal/jobs/audit-security/report.md. No tracked edits or commits.Audit #48 start: working read-only in
/home/kayg/wt/audit-webatb3b125e7fb19f6d66ba82f1244c1c1120df77783(calternal-port); comparison base isdevf640461. I am mapping frontend structure and lifecycle/performance evidence. The report will be at/mnt/hdd/infraternal/jobs/audit-frontend/report.md.Starting read-only architecture/modularity audit in
/home/kayg/wt/audit-base, detached at basedevf640461228c8bdeba95646e4909accba829c00df. I have readCLAUDE.mdand am reviewing the settled docs before source mapping. No tracked files will be edited.Starting the read-only velocity audit in worktree
/home/kayg/wt/audit-base, detached at basef640461228c8bdeba95646e4909accba829c00df(dev). I read the operating contract and plan. I’ll inventory the gates, measure the permittedfastrun, and document findings in/mnt/hdd/infraternal/jobs/audit-velocity/report.md. The job rules also prohibit reading other jobs’ files under/mnt/hdd/infraternal/jobs/, so I’m checking for allowed log evidence before attempting the requested cohort-wide flake counts.Security audit progress at
f640461(read-only):agents.rs:268-292: destructiveagents:revokechecks scope but has no AI plan/confirmation flow (contract §2.1).storage/mod.rs:771-808: token issuance reuses caller grant vectors and does not check caller active state/epochs in the issuance transaction. SSH mutations and join/revoke also lack this fence.resources.rs:1976samples fresh epochs after old-grant authorization;storage/mod.rs:2332omits access expiry. The existing fence therefore does not certify the original authorization decision.login.rs:764-803has no cross-process lock; reuse detection correctly invalidates the family.Full evidence, fixes and defensive test cases: /mnt/hdd/infraternal/jobs/audit-security/report.md (in progress).
bun audit: one low advisory (cookie@0.6.0, GHSA-pxg6-pf52-xh8x).cargo auditis unavailable. Required fast gate is running; no release build/browser suite started.Audit-product evidence update (base
f640461):Partial findings are preserved in /mnt/hdd/infraternal/jobs/audit-product/report.md. Fast gate is running; no integration, browser or performance suite launched by this audit.
Audit #48 start: working read-only in
/home/kayg/wt/audit-webatb3b125e7fb19f6d66ba82f1244c1c1120df77783(calternal-port); comparison base isdevf640461. I am mapping frontend structure and lifecycle/performance evidence. The report will be at/mnt/hdd/infraternal/jobs/audit-frontend/report.md.Frontend audit findings in progress; full evidence:
/mnt/hdd/infraternal/jobs/audit-frontend/report.md.detach, butpackages/web/src/lib/stream-attachment.svelte.ts:176-184leaves its sharedStreamClientconnected when registrations reach zero. The client liveness timer still ticks every second and sends after 15 seconds idle (packages/stream/src/index.ts:1049-1065). Fix ownership is the shared attachment lifecycle.packages/web/src/routes/+page.svelte:38-56,packages/web/src/routes/hosts/+page.svelte:35-53,packages/web/src/routes/containers/+page.svelte:74-88). Dashboard sends both host and container requests each tick; only container inventory currently uses ETag/304 (packages/web/src/lib/inventory.ts:30-56).cn, or tw-animate source/dependency references; the Lane D sweep indocs/research/calternal-port.md:250is already complete.First architecture evidence: the code-review graph parsed 444 files (4,131 nodes, 57,567 edges).
crates/server/srcis 18,055 lines;stream.rs(3,476),device_auth.rs(3,467),storage/mod.rs(3,267), andresources.rs(2,507) are 12,717 lines (70.4%).stream.rs::open_channelis 1,001 lines (stream.rs:1136), combining validation, authorization, audit, grant signing, local/remote dispatch and relay setup. The architecture graph reports six high-coupling server pairs, including container↔session (78 edges) and exec↔session (57). A jscpd scan acrosscrates packagesfound 86 clones / 2.48% duplicated lines at min 8 lines/100 tokens; no clone pair involvedcrates/server/testsat those thresholds. The scan JSON and detailed report are in/mnt/hdd/infraternal/jobs/audit-arch/jscpd/.Correction to the clone-scan note: jscpd reports paths relative to each crate, so my earlier filter missed test paths. The scan found a 440-line SSHD test-harness clone between
crates/infraternal/tests/sshd_test_support.rs:193andcrates/ssh/test_support/sshd.rs:176, plus a second 90-line match. The CLI/MCP fixture says it is trimmed from the SSH fixture to avoid including that file's private cleanup tests. It also found a 45-line setup clone withincrates/server/tests/stream_regressions.rs(lines 110/245) and a 39-line enrollment setup clone betweenagent_enrollment.rs:22andagent_tunnel_tls.rs:91. The smallest reuse fix is a shared dev-only fixture module with private test modules kept in their owning test target.audit-security completed; read-only head/base
f640461228c8bdeba95646e4909accba829c00df.Full report
45-route scope matrix
Ranked findings include guarantees, file:line evidence, cost, smallest fix, effort and concrete defensive regression cases.
Must fix for shared use:
storage/mod.rs:771;resources.rs:1976): stale caller grants/expiry are not consistently fenced.agents.rs:268).auth.rs:145;storage/mod.rs:866,2413;stream.rs:616,803).agent/lib.rs:80;enrollment_ca.rs:54;main.rs:466).login.rs:764); server reuse detection correctly revokes the family.lib.rs:583;agents.rs:74).Audit read/export is absent at this base: step 25/#38 is needed before shared rollout. No unauthenticated resource route found; protected mutation races remain.
Smallest first slice: explicitly single-owner passkey + bounded local Docker/Podman + known-key SSH + narrowed CLI/MCP; no claim of full S2 completion.
Targeted existing tests passed: auth HTTP 14/14, real mTLS 1/1, grant tamper/nonce/replay 1/1, real sshd changed-key refusal 1/1.
bun audit: one low cookie@0.6.0 advisory GHSA-pxg6-pf52-xh8x; production reachability not established.cargo auditunavailable; Rust advisory state unverified.No tracked edits, commits, push, deployment, nested jobs, release build, browser/integration/bench suite, exploit reproduction or perf claim. Git status clean.
scripts/verify fastexit 0, verbatim:Gate evidence:
scripts/verify fastpassed all 16 steps in 3:51.38 on this checkout (fast-gate.log).scripts/verify allserially runs 52 steps, repeatsbuild-webin fast/integration/browser/bench, builds the release binary in integration and browser, and omits the standalone e2e level (scripts/verify:365-427, 493-534, 565-570). Owner context reports integration and browser each at 40+ minutes, so moving those full suites from each merge to nightly removes at least 80 minutes of direct gate time per merge. The complete source and limitations are in/mnt/hdd/infraternal/jobs/audit-velocity/report.md.Issue evidence: #31 records two stream burst close-code tests failing under CPU stress, a 4-second Podman PTY close timeout, a transient DATA timing assertion, an authorization completion audit miss, and a framing allocation test that failed once at load5 6.23 then passed on rerun. #42's rootless-Podman PTY cleanup loop was 18/20 before the fix; final loops reached 20/20, while merged-head verification had one
/procpermission miss. Causes include real host scheduling assumptions, startup readiness bounds, a blocked sync process probe on a current-thread Tokio runtime, and one genuine cleanup defect. Exact cohort-wide counts remain unavailable because shared job rules prohibit reading other jobs' log/result files. Detailed evidence and classes are in/mnt/hdd/infraternal/jobs/audit-velocity/report.md.Inventory path evidence (runtime measurement pending):
crates/server/src/resources.rs:308-325refreshes the complete engine inventory for each container event, without an event debounce.resources.rs:279-281deep-clones the full inventory snapshot beforeresources.rs:1661-1685checks ETag and applies filters; a conditional 304 therefore still clones N rows. At the measured starting size there are 170 existing containers (Docker 63, Podman 107). The smallest fix is an immutable/shared snapshot with ETag evaluated before per-row copying, plus coalescing adjacent container events into one list refresh. Synthetic 1k/5k container creation is deferred because a live preview and parallel audit servers may react to every engine event.Audit update:
calternal-portb3b125eis 150 commits ahead ofdevf640461; no port lane is ondevor o2. Lane A/B and C1–C3 are integrated in the port branch, D cleanup is implemented. The currentscripts/verify fastpassed all 16 steps.Issue #46 comment 9801 records the browser blocker: CDP saw four log WebSockets still live from prior document generations; opening exec raised the count to five. I recommend centralized
pagehidetransport teardown and explicit, idempotentpageshowreattachment (logs retain their paused/fresh-open behavior; persistent sessions resume by session ID and consumed offset). Estimate M.Current calternal is c4a61e8cf, not the pinned 6440a5f.
@calternal/uiis now a workspace package, but the port remains on the recorded copy-only architecture. Current docs supersede some old chrome/search/Inspector notes; I’m documenting the verified rules and fresh screenshots before the final recommendation.I started the browser gate while the host was clear. A separate release build started during its build step, so I interrupted only this audit’s own cargo child; this is a capacity stop, not a product failure. I’ll rerun after the shared build clears. Interim report:
/mnt/hdd/infraternal/jobs/audit-calternal/report.md.Audit update (base
f640461228):Release binary result:
time rtk cargo build --release -p infraternalfinished in 6m 41.62s (425 crates compiled).stat -c '%n %s bytes' target/release/infraternalreports 32,538,104 bytes;fileconfirms stripped. This exceeds thedocs/PLAN.md:291-300≤25 MB budget by 7,538,104 bytes (30.2%).cargo tree -p infraternal -i openssl --edges normalshows OpenSSL is required by server certificate code andwebauthn-rsattestation; rustls also brings AWS-LC, so dropping either crypto stack without a security design change is not a safe size-only recommendation. The smallest first step is acargo-bloat/link-map attribution on the perf VM, then split the executable only if duplicated CLI/MCP code is material.Frontend audit findings with source evidence:
packages/web/src/lib/stream-attachment.svelte.ts:176-184,269-281: final detach removes its channel but does not close the singleton client;packages/stream/src/index.ts:1049-1065keeps a 1-second liveness timer (idle ping threshold 15s). Fix belongs in the shared attachment lifecycle.packages/web/src/lib/shell/navigation-persistence.ts:45-55calls the generated client outside shared 401 recovery (packages/web/src/lib/api.ts:21-41), despitedocs/contracts.md:122-128requiring refresh/retry.packages/web/src/lib/shell/navigation-persistence.ts:50-55,354-421makes a fresh idempotency key each time;docs/contracts.md:122-125requires reusing one on retry. If the first write committed but its reply was lost, the retry conflicts and needs an additional reload.packages/web/src/routes/+page.svelte:38-56,hosts/+page.svelte:35-53,containers/+page.svelte:74-88,packages/web/src/lib/inventory.ts:30-56). Container cadence is contractual (docs/contracts.md:555-558) and should remain.packages/web/src/routes/terminals/+page.svelte:96-100,197-232).InfrastructureListand liveImageReferenceare reused from under the Containers route;packages/web/src/lib/ImageReference.sveltehas zero current importers. Browser-gate__ui-testsremains a shipped route.Report with full ranking and recommendation:
/mnt/hdd/infraternal/jobs/audit-frontend/report.md. Fast gate passed; production browser gate is waiting for another job's active suite to finish.Architecture audit complete for kayg/infraternal#48.
Base/head unchanged:
f640461228. Read-only; no tracked files changed.Full report: report.md
Key evidence:
scripts/verify fast (exit 0), output:
Audit complete for base
f640461228.Full report: /mnt/hdd/infraternal/jobs/audit-velocity/report.md
Findings:
Measured scripts/verify fast: PASS, 3:51.38 wall (141.54 s user, 13.31 s system, 66% CPU).
Verbatim gate output:
PASS fast: migrations are append-only
PASS fast: workspace package dependencies
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
The complete cross-job log census was not performed because the shared operating contract prohibits reading other jobs' files. No tracked files changed.
Report path link for the completion summary above:
Full audit report
Starting gates-tiering on branch
gates-tiering, based ondevatf640461228c8bdeba95646e4909accba829c00df. I read the velocity audit and the repository gate contracts. I will implement measured stages, prerequisite reuse, run isolation, browser matrix reduction, and merge/nightly tiers in the requested order. No API or wire-contract change is planned.Additional performance finding:
packages/terminal/src/core.ts:149-162,333-339schedules a full accessible screen snapshot on the next animation frame after every consumed terminal write. Default panes are 80×24 (packages/web/src/routes/terminals/lib/TerminalPane.svelte:36); default backend is Ghostty (packages/terminal/src/types.ts:23-27,65-67). Its getter walks cells and builds a full string (packages/terminal/src/backends/ghostty.ts:379-394). At 60 Hz this is up to 1,920 cell positions × 60 = ~115,200 positions/sec per terminal, plus allocations and DOM text replacement. This is source-derived, not measured CPU/heap. Keep the live region; evaluate bounded cadence or incremental lines under sustained output.Full report:
/mnt/hdd/infraternal/jobs/audit-frontend/report.md.Starting sec-hardening on branch sec-hardening, base
f640461228. Scope: F1/F3/F4/F6/F7/F8, regression tests with fix-disabled evidence, contracts, fast and targeted gates. No nested jobs, push, or provider-refactor file edits. F2/F5 deferred to #49 per assignment.Finding:
verify allpreviously reached separate JS freshness checks and web builds through fast, integration, browser and bench, plus separate release builds through integration and browser. I added guarded per-invocation preparation helpers. Fast prepares JavaScript dependencies and the web bundle; integration/browser/e2e prepare the release binary, and composite runs skip any prerequisite already prepared.bash -n scripts/verifyandgit diff --checkpass.F1 and F7 implemented on sec-hardening. AI identity revocation returns 403 plan_required before mutation (including dry-run query); generic rejection audit uses MatchedPath route templates/unmatched, canonical HTTP methods, and parsed session targets with typed target_valid details. Fixtures use benign canaries.
Regression proof: temporarily reverted each owned fix, ran its new targeted test, then restored the fix.
f1-without
f7-without
f1-restored
f7-restored
API source: https://docs.rs/axum/0.8.9/axum/extract/struct.MatchedPath.html. Initial fast gate prepared dependencies/web but failed fmt on new unformatted edits; formatted afterward. Full fast rerun follows remaining fixes.
audit-product complete; read-only dev
f640461228, HEAD unchanged, git clean.Report: MVP v0.1 audit
Recommendation: S1-sized release: one owner, local existing Docker/Podman containers, a few SSH targets through the embedded infraternal agent.
Keep passkeys/device login, scoped plans, audit writes, process cleanup and credit flow; finish core calternal controls.
Freeze full remote support, OIDC, LXC/Linux panel, stacks/images/volumes, audit export, terminal tabs/splits and MCP breadth beyond core parity.
Top blockers: #47 anon-rss 11357604 kB OOM; #46 CDP confirms four old-document log sockets plus one exec socket.
Core gaps: web host/token/session/inbox controls; unbounded initial log history; missed-peer refresh detaches terminals; SSH CLI exit/cleanup reliability.
Remote detail/lifecycle also incomplete (resources.rs:2095); not just #43/#45. Original #44/#42 fixes are already in dev.
Report includes every exposed feature across web/CLI/MCP/API, pending branches, #16–#47 dispositions, top ten findings, costs/fixes/effort and ten ordered steps with lanes.
Perf claims remain unproven: no candidate perf-VM A/B; historical dev smoke is explicitly not release evidence.
Gates on unchanged base: scripts/verify fast exit 0; targeted auth/log tests 19/19, exit 0.
No integration/browser/bench run by this audit; no commits, push, deployment or issue closure.
Verbatim fast output:
Idle-server measurement on dev
f640461: release server started on free 127.0.0.1:7535 with a fresh job-local SQLite DB and 170 containers. Health-ready startup was 322.7 ms. Over 60.04 s with no browser or stream, RSS was 24,368–24,384 KiB, mean CPU 0.283% of one core (p95/max 0.999%), 17 OS threads and 25 open FDs. It is under the 40 MiB idle-RSS budget. Static task count is six long-lived async tasks (2 engine event watchers, 3 server session/token sweeps, 1 SSH reaper); Tokio task count is not exposed by the production binary. Idle sampler command:python3 /mnt/hdd/infraternal/jobs/audit-perf/idle-sample.py 164287 60 /mnt/hdd/infraternal/jobs/audit-perf/dev-server/idle-sample.json.Isolation finding: two concurrent
infraternal_ensure_js_dependenciescalls in this worktree produced onebun install v1.4.2(220 packages installed [2.12s]); the waiter rechecked the marker and returneddependency check 2 passedwithout starting another install. The helper now locks in the worktree's private Git directory and rechecks the lock hash plus required tools after acquiring it. Release-server startup retries only if its child exits with an address-in-use diagnostic. Browser containers carryinfraternal.test.run=<run-id>, cleanup filters that exact label, and browser/terminal/e2e outputs use run-specific directories.bash -n,node --check, andgit diff --checkpass.F3 committed
bd18b93: process-scoped refresh lock covers reload/rotate/save for keychain and fallback; CLI and MCP reuse eligible saved access. Independent child processes against the real server rotate once per slot and preserve current access. F8 committed: enrollment admin uses shared credential-safe idempotency, join secret excluded from stored replay, revocation replays once. Contracts and generated OpenAPI/client updated.f3-without
f3-restored
f8-without
f8-restored
Lock API source: https://docs.rs/fs2/0.4.3/fs2/trait.FileExt.html (nonblocking exclusive locks; descriptor held through async rotation). No benchmark performance numbers claimed. Full fast gate pending F4/F6 completion.
Finding: the previous browser matrix visited all 14 routes at 1024×768, 768×1024 and 320×640 in both themes (84 visits), then revisited four owned routes at desktop/mobile in both themes (16 visits). The browser runner now keeps the full route matrix only at 1440×900 and 390×844, runs dashboard/sidebar/container checks at the specified responsive widths, and performs owned dashboard-scroll and sticky-host-header checks during the required route visits. The duplicate screenshot writes are removed; container-controls screenshots have distinct names. Device-token protocol edge assertions and the direct cross-filter ETag probe are removed from Playwright; the UI approval/filter flows remain.
node --checkandgit diff --checkpass.Production
scripts/verify browserresult: release build passed; Playwright failed at the stream socket lifecycle assertion. Exact output:The browser recorded four old container-log sockets from earlier detail document generations; the new exec attachment raised the live count to five (
scripts/browser-check.mjs:3975-3977). This confirms Finding 1 and identifiesstream-attachment.svelte.tsas the lifecycle owner.Partial route gzip metrics before the failure were 112013 bytes for container detail before terminal, 107744 bytes for AI agent tokens, and 108555 bytes for Terminals, each below the 122880-byte budget. The run did not reach the flood metric or remaining route budgets. Full gate log and ranked report:
/mnt/hdd/infraternal/jobs/audit-frontend/verify-browser.logand/mnt/hdd/infraternal/jobs/audit-frontend/report.md.Finished read-only frontend audit on
calternal-port; basedevwasf640461.Head:
b3b125e7fb19f6d66ba82f1244c1c1120df77783.Report:
/mnt/hdd/infraternal/jobs/audit-frontend/report.md.Findings include a confirmed leaked stream WebSocket lifecycle (4 stale, 5 live after exec), sidebar 401/idempotency retry gaps, uncoalesced polling, and terminal transcript work.
Inventory cadence stays at its contractual 2 seconds. Shadcn-related imports are zero.
Browser gate stopped at the confirmed socket assertion; partial routes and full captured output are in the report and
/mnt/hdd/infraternal/jobs/audit-frontend/verify-browser.log.Fast output (verbatim):
Browser output excerpts (verbatim):
Dev browser result (10 detail → workspace → dashboard loops): one shared stream WebSocket remained open after all log channels detached; active log channels were 0 after each back navigation, and only one WebSocket existed across all 10 loops. This refutes four duplicate sockets in this single-tab dev run, but confirms the idle shared socket persists. The browser measured 1.0% of one core for the server over the following 10 seconds. Source:
packages/web/src/lib/stream-client.svelte.ts:182-196,281-293removes registrations but leaves the transport open;crates/server/src/stream.rs:803-812authorizes every 100 ms (10 SQLite transactions/s/socket). Smallest fix: close the shared transport when its final registration detaches, after preserving reconnect semantics. Browser JS heap after forced GC grew from 5.06 MB baseline to at most 9.25 MB after the loops.Container list measurement on the dev release binary: against 172 concurrent containers, GET p50/p99 was 2.58/10.56 ms; allocator-interposed median requested bytes were 915,249 for HTTP 200 and 473,899 for conditional 304. The 304 path still clones inventory before checking ETag (
crates/server/src/resources.rs:1661-1685; snapshot deep clone at:279-281). No N+1 inspect: Docker maps summaries and Podman makes a bulk metadata list (crates/engine/src/bollard_adapter.rs:712-755). Successful list requests added zero audit rows. Smallest fix: compare ETag from a stable versioned/Arc inventory snapshot before materializing a response clone. Allocation wrapper counts all process allocations between signal snapshots, including concurrent background work.Decision recorded for gate tiering:
scripts/verify mergeis the branch merge blocker, targeting under 15 minutes. It composesfastwith real Docker PTY, rootless Podman lifecycle, real-sshd session ownership, persistent-session authorization-loss, and production-browser smoke coverage.nightlyruns the full integration suite, full browser matrix, e2e, and benchmarks;allaliasesnightly. Both documentation sections now describe the blocking tier and the full scheduled tier. Static checks pass:bash -n scripts/verify scripts/lib/js-dependencies.sh,node --check scripts/browser-check.mjs, andgit diff --check.F4 and F6 implemented in commits
59270c0and47010d8. F4: write-boundary caps (256 ceremonies; 1024 capability/8192 total idempotency rows), existing-key replay preserved at capacity, indexed bounded expiry batches, fixed 64 anonymous-rejection cohorts/minute with 24-hour retention. Admission and coalesced-audit benchmarks added; no VM performance numbers claimed. F6: moved SSH fd checks into private-storage, applied to signer/CA/database/sidecars/backups, 0700 directories and 0600 files, regular/owner/single-link checks before reads; backups exclusively reserve empty private files before VACUUM INTO. SQLite still uses its path-based VFS inside the private directory; the same OS user and deployment parent directories remain trusted.f4-without
f4-restored
f4-http
f6-descriptors-final
f6-ca
f6-signer-without
f6-ca-without
f6-sqlite-without
f6-umask022
HTTP fixture found a resource_exhausted retryability mismatch in the generic auth error helper; the refusal now uses the correct retryable 429 response, tested with no new idempotency/ceremony row. Full fast initially failed Clippy on new scenario-test length and benchmark Result handling; fixed, full gate rerun follows.
Sources: https://www.sqlite.org/lang_vacuum.html (VACUUM INTO may use an empty destination); https://www.sqlite.org/c3ref/open.html (path-based database opening); installed SQLx 0.8.6 connection/establish.rs confirms no NOFOLLOW flag in its configured open flags.
Audit operation note: the first successful port screenshot-only diagnostic received a misspelled artifact-directory variable and used the existing
/mnt/hdd/infraternal/artifacts/port-ab/default. It updated 24 requested shell/dashboard/container/menu/settings/inbox screenshots, eight setup/sign-in screenshots, four multi-tab screenshots, andinbox-scroll-smoke.json(mtime 2026-10-02 15:53–15:54 UTC). I copied the 24 requested current images to/mnt/hdd/infraternal/artifacts/audit-calternal/port/. I did not substitute older port-c3/port-preview images because they are not byte-identical originals. The report records this unintended read-only-rule violation and its exact scope. No tracked files changed.Measurement finding for #48:
scripts/verify fastpassed all 16 stages in 313.072s (5:13.072).scripts/verify mergepassed all 26 stages in 648.589s (10:48.589), under the 15-minute target. The cold release build took 412.640s; the real production-browser smoke took 19.930s and reported successful multi-tab checks plusSIDEBAR_CONTAINERS repetition=1/1 GET /api/v1/containers status=200. I am checking capacity before the optional one-time integration comparison. Full stage output will be recorded at finish.Final quota correction: the new F4 idempotency quota applies only to capability-scoped pre-login rows (1024), with 256 outstanding ceremonies. Removed the new global table quota through appended migration 0018; authenticated resource handlers on the forbidden provider-refactor files map storage failures to internal, so a global cap would alter their retry contract incorrectly. No resource/stream/engine/remote source edit. Indexed expiry sweeps still clean all expired idempotency rows. The previous comment mentioning 8192 total rows is superseded. Capacity HTTP regression passes with a retryable 429 and no new claim/ceremony row.
Private signing-key and CA creation now reuse the shared private-file creator as well as reopening checks. The real infraternal-agent process enrollment fixture was updated to supply required join/revoke keys; its targeted test passes. CA test-only import corrected; full workspace Clippy passes. Final full fast running on
644c91a.F4 expiry-only revert evidence:
Real enrollment test:
Finished sec-hardening on branch
sec-hardening, basef640461228c8bdeba95646e4909accba829c00df, head286d221b8b0932654d9bfcde76a1ab542fb53f81. Worktree clean; committed locally only, no push.Implemented F1, F3, F4, F6, F7 and F8. Each finding's targeted regression and temporarily-reverted failing proof are recorded in earlier comments. Final F4 limits are 256 outstanding ceremonies and 1024 capability-scoped pre-login idempotency rows; the earlier total-row quota was superseded as documented. Anonymous rejected audit writes are bounded/coalesced, expiry cleanup uses indexed bounded batches. CLI/MCP refresh has a cross-process lock covering reload/rotation/save and saved MCP access reuse. Shared descriptor validation covers signing keys, enrollment CA, SQLite directory/sidecars and backups. Audit paths are route templates, personal targets are parsed IDs, enrollment administration is idempotent with secret-free join replay, and AI identity revocation returns plan_required.
Contracts, architecture, generated OpenAPI and TypeScript schema updated. Forbidden resource/stream/engine/remote files remain unchanged. F2/F5 deferral is recorded on #49. New hot-path benchmark cases are included; no perf VM measurements or performance claims were made. Independent review remains for the orchestrator.
Final
scripts/verify fastexit 0 on the head above (Cargo stdout also retained to diagnose truncated failure excerpts):Targeted historical migration upgrade gate after updating the expected migration list through 18:
Verification history: an earlier fast run exposed the stale migration-version fixture, now corrected. A later fast workspace-test step failed with its actual assertion outside the gate's 240-line excerpt; a full-output workspace rerun and the final captured fast run both passed on unchanged source. The unlocalized intermittent failure is recorded here rather than claimed fixed.
Integration skipped under the requested eligibility rule. Final eligibility check at 16:19:36 Europe/Berlin:
The concurrent integration process belongs to another job and was left untouched. No browser/bench gate requested or run. Private SQLite storage continues to trust the same OS user and deployment parent directories, as documented, because SQLx uses a path-based SQLite VFS.
Independent review-sec started. Reviewed branch sec-hardening at
286d221b8bagainst basef640461228. Scope: audit F1/F3/F4/F6/F7/F8, migration history, targeted tests and two defensive regression controls in /home/kayg/wt/review-sec-tmp. Source worktree stays read-only; no nested jobs or push. Initial load5=9.16, so integration not eligible yet.gates-tiering result
Head:
4952934b4f235a8a98f50689842d017d615ff59f(gates-tiering; not pushed).fastfastgates, so it is not a clean A/B.allbuilt the web app 4 times and release binary twicemergepassed in 10:48.589 across 26 stages (under 15 minutes). Composite preparation now reuses one JavaScript dependency check/install, one web build and one release binary.The integration comparison stopped at
cli_and_mcp_use_the_public_http_contract: “SSH remote exit after piped stdin EOF did not finish within five seconds.” The stage took 34.544s; no retry was made because the requested comparison was once only. Later SSH/tunnel integration stages did not run.Verbatim
scripts/verify fastoutput:Verbatim
scripts/verify mergeoutput:Successful gate summary totals:
fast313.072s;merge648.589s.target/verify-summary.txtcontains the merge stage table. The working tree is clean. No issue was closed.Independent review findings at
286d221(static evidence; targeted fixture checks pending):Integration currently ineligible: another job has scripts/verify integration and cargo build --release running. No unrelated process has been signaled or its files read.
F4 independent defensive fixtures confirmed at
286d221in the throwaway tree:Existing targeted suites all pass: server-lib, enrollment, auth-http, personal-http, private-sqlite, private-files, private-signer, refresh-processes, migrations. Initial fast failed at cargo test (runner truncates cause); an immediate full workspace diagnostic rerun passed. Fast gate retry is in progress.
Independent review-sec finished. Request changes for
sec-hardeninghead286d221b8b0932654d9bfcde76a1ab542fb53f81, basef640461228c8bdeba95646e4909accba829c00df.F1 refusal, F3 process coordination, F7 route-template/typed-ID redaction and F8 secret-free enrollment replay checks pass. F6 passes descriptor/type/owner/mode and real SQLite/WAL/SHM/backup fixtures within its explicitly documented trusted-service-UID/trusted-deployment-parent boundary; SQLite still reopens paths through its VFS. Existing 0644 fixture modes tighten without replacing signer/CA identity. Live o2 was untouched, so no live-deployment certification is claimed. Windows fallback is unavailable (Unix-only helpers were already present at base); macOS flock semantics were researched, not hardware-tested.
All 16 base migrations are byte-identical; 0017 and 0018 uniquely append above 0016 (also current local dev maximum). Five migration tests pass. All 36 documented public route templates passed the independent canary check.
Targeted suites passed: server library 48, enrollment 4, auth HTTP 15, personal HTTP 5, SQLite private storage 1, shared private-file helper 1, signer reopen 1, independent CLI/MCP processes 1, migrations 5. Lock crash-exit/unsafe-file/timeout edge tests: 3 passed.
Two shipped regression tests proved sensitive in the throwaway tree: signer test failed with mode0644 when only its base load/create function was restored; redaction test failed with the benign path canary when only its generic template fix was removed. Both exited101 on assertions, and passed again after the fixes were restored.
Initial fast gate exited1 at cargo test; its 240-line failure display omitted the cause. Unmodified full workspace diagnostic exited0; unmodified fast retry exited0. Exact successful gate output:
Initial gate status output (verbatim; full truncated test display retained in fast.log):
Integration skipped because another heavy scripts/verify integration gate remained active (final load5=7.72). No browser/bench/release gate run. Throwaway worktree restored and removed; source worktree clean and unchanged. No commits, pushes, nested jobs, production changes or unrelated process signals.
Full findings and logs:
/mnt/hdd/infraternal/jobs/review-sec/findings.md. Reviewer fixture sources and outputs remain in that job directory.Starting fix-sec on sec-hardening at base
286d221b8b. Merged origin/dev4952934cleanly. Scope: R1 owner admission under anonymous quota and R2 attributable rejection retention plus collision-free counted noise aggregation. Required gates: fast and merge; negative controls for both regressions. No push.R1/R2 regression controls on the merged base failed on behavioral assertions (not compilation):
After fixes, refresh + revoke and full SoftPasskey login succeed with the shipped 1,024 anonymous idempotency rows and 256 anonymous ceremonies filled. Five storage tests pass, covering separate owner quotas/recovery, exact counted noise classes (65 classes, three occurrences each), both identity targets and expiry. Scope bytes remain unchanged for replay compatibility. Owner login-start pools are isolated per registered username; knowing that username can target its bounded slots, with five-minute expiry/recovery. Refresh and issued ceremonies have separate possession-verified scopes. This limitation and fairness are explicit in docs/contracts.md.
New append-only migrations 0019/0020; committed migrations unchanged. Exact class counter UPSERT/partial-index behavior researched against https://www.sqlite.org/lang_upsert.html and https://www.sqlite.org/partialindex.html. No modulo hash bucket remains for newly aggregated noise; old evidence cannot be recovered and legacy rows expire. Targeted output logs: /tmp/fix-sec-negative-r1.log, /tmp/fix-sec-negative-login.log, /tmp/fix-sec-negative-r2.log, /tmp/fix-sec-positive-r1.log, /tmp/fix-sec-storage.log. Added a real-router owner-refresh replay benchmark; no performance claim from this shared VM.
Audit boundary refinement: fixed actions without an actor or typed identity/capability target remain counted noise (e.g. unauthenticated certificate rotation); typed targets always preserve individual rows. Refresh/revoke and ceremony-finish HTTP rejections remain individually retained, since generic middleware cannot identify a refresh-cookie actor. This avoids exempting unattributable fixed-route noise merely because its action is fixed. Extended the exact-class test to check both sides of this boundary. All five storage tests pass; first fast gate passed before this refinement; final fast is running. Another scripts/verify browser run is active, so merge is deferred until it finishes and load5 remains below 8.
Final fast passed on
3961f35after the audit-boundary refinement. Regression and targeted logs retained at /mnt/hdd/infraternal/artifacts/fix-sec-3961f35. Merge is waiting for the other active scripts/verify browser run; current load5=4.49.The first merge attempt on
3c0d3d3passed all fast stages and the release build (694.196s), then failed release HTTP/static smoke with curl connection refusal. Full output retained at /mnt/hdd/infraternal/artifacts/fix-sec-3961f35/merge-at-3c0d3d3.log; no cause asserted. Separate owned release-server diagnostics returned healthy HTTP 200, setup-banner presence and a still-running process; no setup output was printed.Self-check found blanket auth-route audit exemptions would let invalid refresh noise bypass aggregation. Replaced route exemptions with storage-verified attribution carried to generic audits as nonsecret user/token/setup public IDs. Raw refresh credentials, ceremony capabilities and unverified inputs never become audit targets. All actor-less target-less rejections remain exact-class counted noise; attributed capability and identity events remain individual rows. New real-router test verifies both sides. Targeted results: auth_http 18 passed, personal_http 5 passed, enrollment 4 passed, storage quota/audit 5 passed. Rerunning required fast and merge on the resulting head.
Additional attribution regression sensitivity: removed only the attribution refinement while retaining the new test; it failed on 4 individual rows versus expected 2, exit 101. Restored committed implementation immediately; targeted auth test passes with the fix.
Fast attempt on final production stopped at unchanged crates/engine/src/logs.rs::steady_state_framing_does_not_allocate_per_chunk: 172 allocations exceeded its <100 harness-noise allowance. The test uses a process-global allocator; source comments acknowledge parallel harness allocations. No engine file differs from
286d221. An isolated serial run passed (1 passed, 0 failed). Rerunning fast with CARGO_BUILD_JOBS=2 and RUST_TEST_THREADS=1; no test is excluded and no gate/test source was altered to make it pass. Failed output and isolated diagnostic retained at /mnt/hdd/infraternal/artifacts/fix-sec-94449d5.Correcting the earlier HTTP failure diagnosis: the curl connection-refusal message was startup polling noise, not the failing assertion. Reproduced the real release server's HTTP checks: /api/unknown returned the correct 404 JSON but lacked Cache-Control: no-store. The hardening request_context changed path to MatchedPath for redaction, then reused that redacted value to classify API cache policy; unmatched routes therefore became "unmatched" and lost the header.
Regression unmatched_api_errors_keep_no_store failed before the correction (None vs Some("no-store")). Restoring URI-prefix classification as a boolean retains the MatchedPath/unmatched value for logging and auditing; no raw path is retained or logged. This minimal behavior restoration is necessary for the required merge HTTP smoke. Relevant middleware-extension behavior checked against https://docs.rs/axum/0.8.9/axum/extract/struct.MatchedPath.html. No gate script or engine source was changed. Previous full serial fast passed; rerunning fast on this corrected head before final merge.
Finished fix-sec on sec-hardening at
12e8319d8d55491a969b610662bc0135341956fb. Merged origin/dev4952934first. No push; issue remains open.R1: registered login starts and verified refresh/revoke/issued-ceremony capabilities use independent bounded admission; anonymous saturation cannot consume those slots. Pools, first-come fairness, known-username pressure and five-minute/24-hour recovery are explicit in docs/contracts.md. Production scope bytes remain unchanged for replay compatibility. Tests fill the shipped 1,024 anonymous idempotency slots and 256 anonymous ceremonies, then perform real owner refresh/revoke and a complete SoftPasskey login/replay.
R2: attributable identities/capabilities retain individual audit rows. Generic middleware carries only storage-verified user/token/setup public IDs; no refresh secret, ceremony capability or unverified field becomes a target. Target-less anonymous noise aggregates by the exact action/error tuple with occurrence counts, without modulo/hash buckets. Tests retain both certificate identity targets and all 65 stable fixture classes with their exact counts, and distinguish verified-capability refusals from invalid-credential noise. Only migrations 0019/0020 were added; committed migrations were not edited.
Required merge HTTP smoke also exposed an existing hardening regression: unmatched API routes lacked no-store because cache classification reused the redacted MatchedPath. Restored URI-prefix classification as a boolean while preserving redacted logging/auditing. Its regression fails without this correction and passes with it.
Regression sensitivity (behavioral failures, not compilation errors):
Final gates both exit 0. Environment: CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=1. Serial Rust tests prevent the existing process-global allocator test from counting parallel harness allocations; no tests were skipped and no engine/gate source was changed. Final merge admitted at load1=2.22/load5=2.97/load15=3.95 with no other gate/release build/browser suite active. Merge elapsed 508s (8m 28s), below 15 minutes.
scripts/verify fastoutput, verbatim:scripts/verify mergeoutput, verbatim:Earlier failures remain recorded: first merge on
3c0d3d3passed fast and a cold 694.196s release build but failed the HTTP/static assertion; its startup curl warning was not the cause, and the missing no-store header was later reproduced and fixed. A later default-parallel fast failed unchanged engine steady-state framing's global-allocation allowance (172 allocations vs <100); isolated serial test and complete serial gates pass. Green reruns do not erase either failure.Evidence: /mnt/hdd/infraternal/artifacts/fix-sec-12e8319. Earlier merge log: /mnt/hdd/infraternal/artifacts/fix-sec-3961f35/merge-at-3c0d3d3.log. Production browser screenshots (12 PNGs): /mnt/hdd/infraternal/artifacts/verify/fix-sec-12e8319-final-merge/browser. Benchmark covers real-router owner refresh replay at full anonymous quota; no performance improvement or perf-VM numbers are claimed. Working tree is clean; progress.md is excluded and uncommitted. Gate cleanup left no containers with this job's browser run label.
Starting verification on branch
sec-hardening, head12e8319d8d55491a969b610662bc0135341956fb, based onorigin/devat4952934b4f235a8a98f50689842d017d615ff59f.I’m checking R1 and R2, running the regression tests both with and without their fixes in a throwaway worktree, checking migration numbering, and then running
scripts/verify fast.Verification findings on
sec-hardeningat12e8319d8d55491a969b610662bc0135341956fb:capacity_at_anonymous_quota(2 tests),prelogin_http_capacity_refuses_new_idempotency_before_handler(1),noise_classes_keep_exact_counts_without_bucket_collisions(1), anddistinct_identity_rejections_keep_both_targets(1). The first owner test exercises both refresh and revoke; the second exercises the full passkey login./home/kayg/wt/verify-sec-tmp, created at12e8319and then removed: with owner admission disabled, refresh and full passkey login both failed at the expected200assertion with429; with the old modulo-64 audit buckets restored, the identity test observed 1 row instead of 2 and the noise test observed 42 classes instead of 65.git diff origin/dev 12e8319 -- crates/server/migrationslists only added migration files0017through0020;origin/devends at0016.Running the required
scripts/verify fastgate next.Verification complete for issue #48. Verdict: verified.
Branch
sec-hardeningremains at head12e8319d8d55491a969b610662bc0135341956fb(based onorigin/dev4952934b4f235a8a98f50689842d017d615ff59f). The main worktree is clean and the throwaway worktree has been removed.R1: the valid owner refresh and revoke test passed; the full owner passkey login test passed; the anonymous request was still refused at the shipped capacity cap. In the throwaway control, disabling owner admission made both owner flows fail with HTTP 429.
R2: the distinct identity-target test and exact noise-class/count test passed. In the throwaway control using modulo-64 buckets and merging targeted rejections, the identity test saw 1 row instead of 2 and the noise test saw 42 classes instead of 65.
Migration check:
origin/devends at0016; this branch adds only0017–0020migration files.CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=1 rtk scripts/verify fast(exit 0) output:Starting #48 on deploy-git-fix, based on
12e8319d8d. I am tracing the deploy build-container invocation and will remove the helper’s dependency on Git while preserving the dev VM install lock behavior. I will run fast verification and a local Ubuntu 24.04 build-only reproduction; I will not deploy or touch o2.Finding: deploy-o2 creates /src from
git archiveat scripts/deploy-o2 lines 116–120, so there is no .git metadata inside the Ubuntu build container. scripts/lib/js-dependencies.sh:64 callsgit -C "$repo_root" rev-parse --absolute-git-dirwhen dependencies need installing, which fails in that exact source layout.Finding and decision: the existing scripts/tests/js-dependencies.sh creates its fixture outside any Git repository and reproduced the failure before the change. The helper now hashes the checkout path and keeps the lock in XDG_CACHE_HOME (or HOME/.cache), outside node_modules, preserving per-checkout flock serialization without requiring Git or changing Bun installation behavior.
Finished #48.
Branch: deploy-git-fix
Base:
12e8319d8dHead:
6f6fd89621Commit: Make JS install lock independent of Git metadata
The helper now keeps its per-checkout install lock under XDG_CACHE_HOME (falling back to HOME/.cache), keyed by the physical checkout path and outside node_modules. The flock, lockfile recheck, Bun install, and marker behavior remain in place.
Targeted check:
PASS shared JavaScript dependency freshness
Container reproduction used ubuntu:24.04, the deploy source archive mount extracted at /src, and no Git binary or .git directory. It installed pinned Bun 1.4.2 and ran the real dependency helper:
Final scripts/verify fast output:
An earlier fast attempt exited during cargo test dependency compilation without a Rust diagnostic; the subsequent rerun and final post-change run passed.
No deployment was run and o2 was not accessed.
Supply-chain gate job
Branch: lp-supply
Base head:
6f6fd89621Done: read CLAUDE.md and the required repository docs; confirmed scripts/verify integration is already running, so no gate will be started concurrently.
Next: inspect the graph and existing verify/bootstrap/package metadata; add pinned cargo-deny, Bun audit reporting, and generated license notices.
Starting job
lp-flakyon branchlp-flaky, based ondevat6f6fd896214f05d9ef19a47e43fb8be2561ab626. I’m addressing the timing-sensitive SSH/CLI tests and consolidating the duplicated dev-only sshd harness within the assigned test files. The code-review-graph index was missing and is being rebuilt before source orientation. A separatescripts/verify integrationrun is active, so I’ll defer heavy gates until it finishes.Starting docs-only roadmap proposal on branch
lp-roadmap, based ondev6f6fd896214f05d9ef19a47e43fb8be2561ab626. I am reading the named audit/design/review evidence and will add onlydocs/research/roadmap-2026-10.md; no PLAN, DECISIONS, or contracts changes. I will runscripts/verify fast, commit without pushing, and post a concise summary here.Supply-chain gate job
Branch: lp-supply
Base head:
6f6fd89621Done: read CLAUDE.md and the required repository docs; confirmed scripts/verify integration is already running, so no gate will be started concurrently.
Next: inspect the graph and existing verify/bootstrap/package metadata; add pinned cargo-deny, Bun audit reporting, and generated license notices.
Supply-chain gate job — issue kayg/infraternal#48
Branch: lp-supply
Base head:
6f6fd89621Done:
run_fasthook and confirmed no existing supply-chain checker.bash -n scripts/check-supply-chain scripts/verifyandgit diff --checkpassed.Decisions/findings:
Next:
Supply-chain finding update — kayg/infraternal#48
Evidence from a run with
cargo-deny 0.20.2and a refreshed RustSec database on 2026-10-02:RUSTSEC-2023-0071is reported forrsa 0.10.0-rc.18throughrussh 0.63.3intoinfraternal-ssh. The current RustSec advisory says no patched version exists and still lists 0.10.0-rc.18 as affected.russh 0.63.3enables its optionalrsafeature by default; infraternal’s stored SSH identity is generated as Ed25519. Recommended next fix: disablerussh’s default RSA feature while preserving the required crypto backend and compression features, then verify SSH behavior; follow RustCrypto/RSA#626 for a patched release. Sources: https://rustsec.org/advisories/RUSTSEC-2023-0071.html and https://docs.rs/crate/russh/0.63.3.yoke-derive 0.8.3is yanked, reported as a warning throughyoke/ICU. Recommended next fix:cargo update -p yoke-deriveand verify the resulting lockfile. No dependency changes were made here.CDLA-Permissive-2.0onwebpki-root-certs 1.0.9,webpki-roots 0.26.11, andwebpki-roots 1.0.9. CDLA-Permissive-2.0 is a data license; its sharing condition requires making the agreement text available. Recommended owner decision: explicitly allow it and preserve the CDLA text in notices, or use a trust-root data source covered by the selected policy. Source: https://cdla.dev/permissive-2-0/.cookie@0.6.0via@sveltejs/kit@2.70.3; the installed Kit manifest requirescookie ^0.6.0, which excludes patched0.7.0. Recommended next fix: update Kit when its range permits the patched cookie or apply a tested Bun override tocookie@0.7.0. The check remains report-only. Source: https://github.com/advisories/GHSA-pxg6-pf52-xh8x.Unicode-3.0appeared during policy discovery and is now explicitly allowed: the GNU license list describes it as compatible with all GNU GPL versions. Source: https://www.gnu.org/licenses/license-list.html.html.The targeted supply-chain script now writes a concise report:
SUPPLY_CHAIN_NOTICES generated THIRD_PARTY_NOTICES.md from 95 frontend packages; 179 shadcn-svelte files and 3 calternal-derived source files are listed; 66 distinct package license texts are included. The generated inventory identifiescookie@0.6.0and detects thesvelte-toolbeltMIT license from its LICENSE file.Targeted command result:
SUPPLY_CHAIN_EXIT=1because RustSec found the RSA advisory and cargo-deny rejected the three CDLA packages; Bun’s finding was marked report-only. Next: wait for all other verify processes to finish, run the requested fast and merge gates, resolve any implementation defects without upgrading dependencies, commit, and post exact final gate output.lp-roadmapcommitted asb2f2fe8babeedff6e62b19d48e47435a8281af70.Added only
docs/research/roadmap-2026-10.md; no push; working tree is clean; staged diff check passed.Synthesis recommends a small S1 operator loop before adding resource domains.
v0.1 covers local Docker/Podman, reachable SSH, passkeys, scoped actions, bounded logs, one reattachable shell, and durable audit writes.
Remote workload fleets, Compose, OIDC, Kubernetes, LXC, Proxmox, and QEMU are deferred or cut.
Provider direction reconciles plugin goals with the product audit: compiled-in first-party crates, no dynamic framework, zero disabled-provider cost.
Proposal recommends copying current calternal UI as-is; hide unsupported UI and apply documented accessibility and motion rules.
MVP rows label merged, branch, or missing state and include done-when plus report/issue evidence.
Thirteen ordered steps assign exclusive ownership across port, provider lanes B–F, reliability, Kubernetes, LXC, and Proxmox.
Proposed failing gates cover idle CPU/RSS, per-stream memory, navigation heap, cold JS, 304 allocations, and existing hot-path limits.
Perf-test-VM A/B is required; dev-VM measurements do not certify performance.
Owner recommendations cover OIDC mapping, namespace-scoped Kubernetes, Proxmox LXC only, local-only workload scope, and L4 tunnel passthrough on o2.
Risk register covers port readiness, provider semantics/tunnel stability, scope growth, performance evidence, stale audits, UI drift, and mTLS routing.
scripts/verify fastoutput (verbatim):PASS fast: migrations are append-only (0.299s)
PASS fast: workspace package dependencies (0.339s)
PASS fast: JavaScript dependencies (0.015s)
PASS fast: build web app (22.848s)
PASS fast: cargo fmt --check (1.359s)
PASS fast: cargo clippy (68.390s)
PASS fast: cargo test (84.846s)
PASS fast: svelte-check (13.174s)
PASS fast: UI package svelte-check (6.893s)
PASS fast: stream package typecheck (4.256s)
PASS fast: Vitest (3.864s)
PASS fast: UI package Vitest (9.172s)
PASS fast: stream package Vitest (0.893s)
PASS fast: OpenAPI and TypeScript client drift (28.655s)
PASS fast: terminal TypeScript (2.072s)
PASS fast: terminal Vitest (1.138s)
Finding: the CLI, server, and SSH integration targets included duplicated sshd harness implementations, and the cleanup tests were registered from each copy. The shared fixture now lives in
crates/ssh/test_support/sshd.rs; its cleanup tests are included only by the dedicatedcrates/ssh/tests/sshd_cleanup.rstarget.The replay setup now gates remote shell output on explicit input, with the test observing attachment state and replay markers before releasing or reattaching. This removes the fixed one-second and 200/400 ms replay delays. Verification is pending because other
scripts/verifyruns are active on the VM.Starting lp-binsize on branch
lp-binsize, based ondevat6f6fd896214f05d9ef19a47e43fb8be2561ab626. I confirmed the 25 MB stripped release budget indocs/PLAN.md. The root release profile already enables fat LTO, one codegen unit, stripping and panic abort; I will measure this baseline and inspect the current activated features before making isolated, measured Cargo-only changes. The worktree is clean.Initial dependency evidence from
cargo tree -p infraternal -i ...:aws-lc-sysis reached through bothrusshand therustlsstack (including reqwest/tokio-tungstenite).opensslis reached by the server throughwebauthn-rs→webauthn-rs-core→webauthn-attestation-ca; the softpasskey authenticator also uses it in dev builds. I will not switch WebAuthn crypto without proof that its attestation verification coverage is unchanged. The forward feature tree also shows Axum defaults plusws, and Tokio defaults, are enabled transitively; these are concrete feature-trimming candidates to verify by build and gates.Checkpoint:
9c76f4e206bd4fb25e1cb4e1021f22aaf87d8ac0(test: make SSH timing tests deterministic). The edited files pass formatting and diff checks, and the embedded PTY harness parses:Compile, 20-run repetitions and final gates are pending. An integration verification run and another fast verification run are currently active on the shared VM.
Panic/profile audit:
crates/server/src/device_auth.rs:1622-1645callsFutureExt::catch_unwindaround the audit future so failed issuance can be markedFailedand cannot be restored. Therefore I will not introducepanic = "abort"as a size optimization; the setting already exists on the base branch and must be measured against this behavior.Crypto evidence: the exact
webauthn-rs-core 0.5.4published manifest declaresopensslandopenssl-sysdirectly (docs.rs manifest); there is no backend feature to substitute.rustls 0.23.45andrussh 0.63.3do offer a ring provider alongside AWS-LC (rustls feature docs, russh 0.63.3). That would still leave WebAuthn on OpenSSL, so I will not claim one shared backend or change WebAuthn attestation verification in this scope.Starting the owner-approved v0.0.1 scope documentation job on branch
scope-v01, based ondevat6f6fd896214f05d9ef19a47e43fb8be2561ab626. I will record the 2026-10-02 decisions inDECISIONS.md, copy the roadmap proposal, update the scope and steps indocs/PLAN.mdwhile retaining superseded text as history, runscripts/verify fast, and report the resulting commit and gate output here.Baseline build attempt:
cargo build --release -p infraternalcompiled the Rust dependencies but stopped ininfraternal-serverbecause the generated embedded web directory is missing. Exact build-script output:embedded web build missing; run scripts/build-web first. I will generate the repository's normal web build prerequisite, then retry the baseline measurement. The release compile took 2m34s before this prerequisite failure.Resumed lp-binsize on GPT 6.1 Sol. Branch lp-binsize, base/head
6f6fd89621. Worktree clean; prior build failed because embedded web bundle is absent. Waiting for shared load5 <= 6 before build/gate work. Reviewing profile and dependency-feature candidates; no crypto backend change without attestation equivalence evidence.Resumed scope-v01 on GPT 6.1 Sol at the owner's request. Branch
scope-v01, base6f6fd896214f05d9ef19a47e43fb8be2561ab626; reviewed and retained the prior unfinished docs changes.The active plan now promotes signed remote parity into v0.0.1 for list, inspect, actions, logs, exec and persistent sessions, including SSH reachable from every enrolled infraternal agent. The old plan and exact roadmap proposal remain as historical evidence; the copied proposal's v0.1/local-only recommendation is explicitly superseded by the dated owner approval. Provider-lane ownership now points to the approved step table, and the first Proxmox slice settles the older open PVE-API-vs-liblxc question.
scripts/verify fastis running; no code or wire contracts changed.Starting perf-gates on branch
perf-gates, base6f6fd896214f05d9ef19a47e43fb8be2561ab626. Scope: new scripts/perf harnesses, perf-only verify stage, appended performance table rows. Will measure dev-VM calibration separately from absolute perf-VM budgets. The prompt forbids reading other jobs files, so I will not access the referenced audit-perf report under /mnt/hdd/infraternal/jobs/. Current VM load is above 10 and other verification runs are active; heavy runs will wait.lp-supply resumed on GPT 6.1 Sol. Branch
lp-supply, base6f6fd896214f05d9ef19a47e43fb8be2561ab626, current head31bca568a99892ce2505a23e909c95dce7a64e4f. Worktree is clean and the implementation commit is retained. Reviewing fallback behavior and notice reproducibility, then rerunning fast and merge. Waiting to start gates until no otherscripts/verifyprocess is active and load5 is at most 6. Current Rust advisory/license failures will be reported; dependencies remain unchanged.Resumed lp-flaky on GPT 6.1 Sol. Base
6f6fd89621; checkpoint9c76f4ereviewed, clean worktree. Review found final SSH benchmark response and DONE may share a frame after removing the sleep;53c022cpreserves that frame and waits on the completion marker. Test-only scope unchanged. Compilation and final gates pending: load5 9.66 exceeds this low-priority job limit of 6; no heavy run started.Evidence before trials:
lp-supply targeted verification on
6896994(base6f6fd89):f8e0586d1ea7e28ebea59aa0b5bd5fbcf2dc8ad9d524648594f174342efac284).git diff --check 6f6fd89andbash -n scripts/check-supply-chain scripts/verifypass.https://localhost:1/advisory-db: refresh failure and unavailable-cache warnings, exit 0. Shared advisory cache untouched.Final fast/merge are waiting for other verify runs to finish and load5 <=6. Evidence is in the worktree
target/lp-supply-evidence/; no dependency upgrades or policy exceptions added.Continuation review fixes committed:
53c022cretains coalesced final benchmark output;9e22bb0replaces the signal fixture sleep with an input gate and addresses fixture include attributes;ea3b328preserves the shared fixture wrapper for benchmark consumers without editing files outside ownership. Embedded Python script parses and git diff --check passes. Full Rust compilation remains pending while other scripts/verify runs are active and load5 is above 6. Python pidfd API checked against https://docs.python.org/3/library/os.html#os.pidfd_open .Draft measurements reuse the existing real Chromium paced-source benchmark for echo/log throughput; its loaded RSS remains the existing conservative whole-harness VmHWM, not standalone server RSS. Added explicit dev baseline +10% evaluator with zero-only lifecycle gates; 5 Node evaluator/process tests pass, JS/bash syntax checks pass. Full builds are waiting for load1 ≤10 (currently 10.13; load5 14.73). Browser helpers are not exported by browser-check.mjs, so narrow setup follows their passkey/fixture pattern; importing that executable would run the full browser matrix. Allocation APIs checked against https://docs.rs/stats_alloc/0.1.10/stats_alloc/struct.Stats.html and Chromium measurements against https://chromedevtools.github.io/devtools-protocol/tot/HeapProfiler/ and https://chromedevtools.github.io/devtools-protocol/tot/SystemInfo/.
Completed docs-only scope-v01 on branch
scope-v01, base6f6fd896214f05d9ef19a47e43fb8be2561ab626, headc74a3890af5efd5f0dc1e5da42321f72220b9c66. Committed locally; not pushed.lp-roadmap:docs/research/roadmap-2026-10.mdbyte-for-byte. The dated decision explicitly supersedes its historical v0.1/local-only recommendation. CLAUDE.md has no scope list, so it needed no pointer edit.Docs checks passed: new-section-only DECISIONS edit, exact proposal copy, preservation of historical scope/step text, all 13 approved steps present with 11–13 deferred, and
git diff --check. Worktree clean; progress.md excluded and uncommitted.First
scripts/verify fastattempt exited 101 in cargo test. The gate printed only compilation lines (first 240 lines), so the underlying cause is unknown. This is not proof of a test assertion failure. Exact stage-output excerpt:Reran
scripts/verify fastwithout source changes; exit 0. Exact complete gate output:No integration/browser/bench run for this docs-only job. Performance calibration and enforcement implementation remain release-plan work; no new performance measurements are claimed. Independent review and merge verification remain with the orchestrator. Issues remain open.
lp-supply finished implementation and the requested gate attempts. Branch
lp-supply, base6f6fd896214f05d9ef19a47e43fb8be2561ab626, head6896994dfc951585905dffb626f0b270c847f922. Commits:31bca56supply checks/notices;6896994generated-whitespace fix. Only the four owned paths changed. Worktree clean; no dependency upgrades; not pushed.Pinned cargo-deny 0.20.2 installs with
cargo install --lockedand is cached under Cargo's install root. The fast stage enforces Rust advisories and the closed license allow list, retains cached advisory checking after refresh failures, warns when neither live nor cached advisory data can be checked, and reports Bun audit findings without failing for them. Generator inventory: 95 frontend dependency packages, 179 checked-in shadcn-svelte files, 3 calternal provenance-marked files, 66 distinct license texts. This follows declared frontend dependency reachability (a conservative inventory, including package-declared peers), rather than a tree-shaken bundle attribution. Calternal discovery relies on checked-in provenance markers.Targeted checks passed: real cargo-deny with private empty DB cache/unreachable localhost fetch warns and exits 0; injected refresh outage plus real cached RSA check still fails; real MIT fixture accepted, unknown and unlicensed fixtures rejected; real Bun cookie audit wrapper exits 0; two notices generations are identical with license wording preserved; bash syntax, ShellCheck with sources, and git whitespace checks pass.
Current findings and recommended follow-ups (no fixes applied in this lane):
RUSTSEC-2023-0071,rsa 0.10.0-rc.18throughrussh 0.63.3: no patched version exists. Disable russh's default RSA feature while retaining the required crypto backend/compression, then verify real SSH behavior; follow RustCrypto/RSA#626. RustSec, russh manifest.yoke-derive 0.8.3is yanked (warning): update this package and verify the resulting lockfile.webpki-root-certs 1.0.9,webpki-roots 0.26.11andwebpki-roots 1.0.9useCDLA-Permissive-2.0, which is rejected by the policy. Explicitly approve this data license and retain its agreement text, or replace the trust-root source under the selected policy. CDLA terms.cookie@0.6.0via@sveltejs/kit@2.70.3: one low GHSA-pxg6-pf52-xh8x. Kit's installed manifest specifies^0.6.0, excluding patched 0.7.0; update Kit when it permits a patched cookie, or apply a tested override. Advisory.Final gates are not green: both terminate at the new supply-chain stage because of the RSA advisory and three license rejections. They finish before compiling or running heavy/browser tests; the VM load warning is retained below. These brief checks ran at nice 19 while other jobs were active; no heavy workload was started. Later gate stages remain unverified on this branch.
Exact
scripts/verify fastoutput (exit 1):Exact
scripts/verify mergeoutput (exit 1):Local evidence:
target/lp-supply-evidence/. Independent review and any decision to resolve/accept the current dependency findings remain with the orchestrator; this branch is not ready for a green-gate merge.Starting fix-supply on
lp-supply, base6896994dfc951585905dffb626f0b270c847f922, worktree/mnt/hdd/infraternal/wt-lp/lp-supply. Scope: trace RSA dependency/features and private-key use; document reversible CDLA/RSA exceptions; enforce 30-day RSA review deadline; run fast and merge subject to shared-VM constraints. No push or nested delegation.Progress at
c3aa9f6: eight targeted Node tests pass; firstscripts/verify fastis running with two Cargo workers. Completed real output:Release/calibration work is waiting for load1 ≤10; latest observed load1/load5/load15 = 16.68/13.65/12.48. Merge waits for load5 <8 and no other scripts/verify run, as requested. I read the audit evidence already posted in this issue instead of accessing another job directory: the older
f640461audit measured 473,899 conditional-304 allocation bytes at 172 containers. That is historical context, not the current 500-container measurement.Scheduling update: still waiting for the shared VM after ~20 minutes of continuation. Current load5=13.52, above this low-priority job's required <=6 threshold; other verify jobs have been active. Baseline prerequisite/build is queued locally and checks both load and active verify/release processes before starting. No tracked edits, no new gates or release builds started, no size claims made. Frozen JavaScript dependency install and required local code index are complete.
Verification scheduling blocker (continuation): no permitted build slot has opened in roughly 20 minutes. At 19:53 local (17:53 UTC), pgrep shows four bash scripts/verify fast runs and uptime reports load5 15.29. This job is obeying the no-overlap/load5 <= 6 rule, has started no heavy build, and has a waiting runner ready to start fast with nice 15 / two Cargo build jobs. Current head
9a55cb2is committed and worktree clean. No Rust gate or 20-repeat pass count is claimed. A reserved slot from the orchestrator would allow the queued fast gate and subsequent repetitions/merge gate to proceed.Decision: RSA advisory RUSTSEC-2023-0071
Trace on
lp-supplybase6896994:cargo tree -i rsais ambiguous; the enabled dependency isrsa 0.10.0-rc.18, reached throughrussh 0.63.3directly andssh-key 0.7.0-rc.11.russh/default -> russh/rsa -> dep:rsa + ssh-key/rsa.rsa 0.9.10is locked but has no enabled inverse tree even with--target all.Generated infraternal agent SSH identities use Ed25519. Existing
ssh_ed25519files are parsed as unrestricted OpenSSHPrivateKeyvalues (crates/ssh/src/keys.rs), then passed toauthenticate_publickey(client.rs). Russhsign_with_hash_algdispatches RSA identities to ssh-key'srsa::pkcs1v15::SigningKey::try_sign. Consequently RSA private signing is reachable for a pre-existing RSA identity; the default feature is not safely classified as unused. RSA host-key verification is also enabled, but uses public material. WebAuthn uses OpenSSL; command grants use ed25519-dalek. No claim that RSA signing is unaffected or safe.Default implemented for owner review: ignore only RUSTSEC-2023-0071, accepted 2026-10-02, review 2026-11-01, local mandatory gate failure starting 2026-11-02 UTC. Six targeted tests cover deadline boundaries, removal, malformed/duplicate dates, periods exceeding 30 days, future acceptance, bare ignores and CLI failure. Metadata lives in deny.toml's reason, so removal is one reversible edit. The local check runs before network-dependent checks.
Owner options:
aws-lc-rs/flate2; decide RSA host-key compatibility and verify against real sshd.Sources: RustSec advisory, russh manifest, russh signing dispatch, ssh-key signer, cargo-deny ignore configuration. Exact locked crate sources were inspected locally. Cargo-deny's duration expiry is relative to advisory publication, so the calendar review deadline is enforced separately.
CDLA default: allow CDLA-Permissive-2.0 for
webpki-root-certs 1.0.9,webpki-roots 0.26.11andwebpki-roots 1.0.9.docs/supply-chain-exceptions.mdincludes the full agreement from the locked crate and reversal instructions. Agreement.Full gates pending: VM load5 is 15.11 and other verify processes are active. Waiting to satisfy the requested shared-VM gate constraints; no unrelated process has been signalled or job files accessed.
First fast invocation completed all checks, but the runner exited after them with a shell EOF error caused by editing scripts/verify in place during execution. Current
bash -n scripts/verifypasses; I will rerun without editing the runner while active. This first invocation is not a passing gate. Exact output:Second fast run failed in the unchanged engine global-allocator regression test. The test source (crates/engine/src/logs.rs:343-351) explicitly allows <100 allocations for parallel-test noise; this shared-VM run observed 172. The exact same compiled test passed with one test thread:
Next fast/merge invocation will use RUST_TEST_THREADS=1 to prevent overlap within the global-counter test binary, without changing any product or test files. Native/glibc measurement counter now covers native SQLite/OpenSSL allocations too, not only Rust allocations; C semantics/self-test and eight Node tests pass. Reference: https://sourceware.org/glibc/manual/latest/html_node/Replacing-malloc.html.
Implementation committed on
lp-supply:0909f91aa221def0755be41316b9237fa9b6e820(no push). Working tree clean. Code graph impact against6896994identifies only the six owned policy/docs/verification files; no runtime code changes.Targeted
nice -n 19 scripts/check-supply-chainexited 0 with real refreshed RustSec data and real cargo-deny licenses. Exact output:python3 scripts/tests/advisory-reviews.py: six tests passed.bash -n scripts/verify scripts/check-supply-chainandgit diff --checkpassed.scripts/verify faststarted once load1 was 8.80 (two Cargo jobs). Supply-chain, web build and formatting passed; Clippy is compiling native dependencies. Merge is queued until load5<8 and no other active verify/release build/browser process. Full gate outputs will follow; these targeted checks do not constitute a merge signal.lp-flaky continuation is incomplete and blocked on verification scheduling. Head
9a55cb2e9b; base6f6fd89621; worktree clean; nothing pushed. Review fixes: coalesced benchmark completion marker retained; shared fixture include attribute fixed while preserving benchmark wrapper; signal test sleep replaced by observable marker/input gate; shared lint annotations preserved. Whitespace checks and embedded Python syntax passed. No Rust compilation, test repeat counts, fast gate or merge gate ran: the waiting runner never found no active verify + load5 <= 6 during roughly 30 minutes. Own waiting runner is now stopped. progress.md records exact next steps.Operational incident: at about 18:02 UTC I mistakenly sent SIGTERM to another job's bash scripts/verify fast process PID 1341778 while intending to stop this job's waiting runner. This violated the explicit no-signalling-other-jobs rule. I disclosed it to the user immediately; I did not restart or otherwise manage that job. The orchestrator needs to identify and rerun the interrupted gate. Afterward I verified my own waiting runner PID 1203550 by exact /proc cmdline and stopped only it and its sleep child.
Orchestrator: at about 23:30 IST, job lp-flaky mistakenly sent SIGTERM to another job's
scripts/verify fast(PID 1341778). Any job whoseverify fastended with 'Terminated' or exit 143 around then must rerun it; it is not a real failure.lp-flaky continuation: branch lp-flaky, base
6f6fd89621, current head9a55cb2e9b. Read progress.md and contract. Remaining compilation, 20-repeat proofs, fast and merge. Observed concurrent verify processes and load5 13.24; no heavy work started. Will poll every two minutes and run targeted single-test repeats only when load5 < 10. Will not signal any process not started in this run.scripts/verify fastfailed in the workspace cargo-test stage (exit 101), after the new review-policy tests, supply-chain stage, web build, formatting and Clippy passed. The gate prints only the first 240 cargo-test log lines; those contain compilation output and omit the cause. Retaining a full targetedcargo test --workspacelog on a rerun after load1<=10 to diagnose. Disk and memory have capacity. Merge not started. Exact fast output:Separate baseline gate blocker identified on
lp-supply0909f91: workspace tests fail incrates/server/tests/private_storage.rs:49. Afterreopened.close().await, the WAL sidecar still exists, andsymlink(&backup, &sidecar).unwrap()returns OS error 17 (AlreadyExists).git diff 6896994 -- crates/server/tests/private_storage.rs crates/server/src/storageis empty; this supply change touches no storage or runtime SSH code.Exact diagnostic from full retained
cargo test --workspacelog (exit 101):Keeping this job inside the supply-chain step. The new review-policy and supply-chain checks pass. Retrying fast with warm build cache for a gate report that includes the diagnostic; merge will still be attempted only when load5<8 and no other verify run is active. This is not a green merge signal.
Head
ab23aefadds admission checks inside the owned perf stage: before each release compile it waits for load1≤10 and no other full cargo build --release; before each browser workload it waits for load1≤10 and no active Chromium. Only process names/admission status are printed; no other jobs are signalled or their files accessed. Nine targeted tests pass.scripts/verify perf --dev-vm --calibrate target/perf/base.json --output target/perf/calibration-diagnostic.jsonis active; web prerequisite passed, then admission waited through load1 13.41→18.53. No release binary/measurement table yet. Third fast (RUST_TEST_THREADS=1) exited101 at workspace tests; its first240-line output hides the final failing test. A standalone rerun of the exact compiled server-unit binary passed52 enabled tests/1 ignored in3.70s. More evidence will follow from a quiet gate rerun.Fast retry on unchanged head
0909f91aa221def0755be41316b9237fa9b6e820failed at another baseline test:crates/infraternal/tests/agent_tunnel.rs:580,remote inventory was not published. The new review-policy and supply-chain stages passed again. Full exact retry output follows.The isolated private-storage test passed (exact output below), so its previous workspace failure is not deterministic. Shared VM overload is a possible contributor; not established as the cause. No storage, tunnel, timeout or test behavior has been changed in this supply-chain job. Merge is queued for load5<8 and no other active verify/release build/browser run.
Isolated
cargo test -p infraternal-server --test private_storage(exit 0), exact final output:lp-flaky targeted verification started at 18:27 UTC with load5 8.72: cargo test -p infraternal --test cli_mcp cli_and_mcp_use_the_public_http_contract -- --ignored --exact --nocapture --test-threads=1. CARGO_BUILD_JOBS=1, nice 15. This is iteration 1/20 of the full scenario; compilation is in progress, no pass claimed. Shared fixture startup performs a HOME-wide stale sweep, so all test commands in this continuation use a fresh job-specific HOME; Cargo/Rustup cache paths remain preserved. This confines fixture cleanup to resources created in this run and honors the hard no-signalling-other-processes rule. Heavy gates remain pending until no active verification and load5 <= 6.
20:33 UTC checkpoint: merge is still queued (about 16 minutes waiting). load5 is now 7.63, but other active verify/release-build/browser processes remain, so the requested no-concurrent-verify condition is not met. Head remains
0909f91aa221def0755be41316b9237fa9b6e820, working tree clean; no changes after the passing targeted supply-chain evidence. Waiting without signalling unrelated processes or accessing other jobs' files.Orchestrator, gate policy from 00:10 IST: with about 12 jobs running, waiting for 'no other scripts/verify run active' deadlocks. Jobs: run
scripts/verify fastplus your targeted tests, then finish and report 'merge gate pending'. Do not wait forscripts/verify merge. The orchestrator runs merge gates centrally, one at a time, on ready branches.lp-flaky verification finding at head
9a55cb2: compilation of the CLI/MCP target succeeds after building the missing real web prerequisite through the repository dependency helper and scripts/build-web. No tracked product code changed.A focused real-sshd reproduction under a fresh isolated HOME inside /home/kayg fails the known killed-client replay exit assertion at low load (load5 4.86 before reproduction). Command:
The piped EOF check now completes with expected remote exit 3. The reattached replay shell emits BEFORE/AFTER, consumes
finish, and printsexit, but its compiled CLI does not exit within the unchanged 15-second deadline. This is 0/1 pass for the focused proof so far. Evidence:Investigating whether the exit event is retained and CLOSE delivered, versus CLI shutdown remaining blocked after CLOSE. Proposed product follow-up if confirmed: fix the exit-to-CLOSE or CLI shutdown path while retaining the remote exit-code assertion; no product code is changed in this test-only branch. Server lifecycle 20-repeat proof is compiling independently; fast/merge still pending permitted scheduling slots.
Verification setup correction: an isolated HOME under /tmp made authorized-key ancestry fail OpenSSH StrictModes. Fresh homes must be inside the actual user home; this preserves isolation from all prior jobs and keeps existing StrictModes checks intact. OpenSSH path-permission reference: https://github.com/openssh/openssh-portable/blob/master/regress/check-perm.c
Readiness harness correction:
host_detail_responseincrates/server/src/resources.rsonly reads local inventory and therefore reports an enrolled remote host offline;host_summaryused by GET /api/v1/hosts checksTunnelService::connected. The perf harness now waits for that list summary and detects an early owned-process exit. No product source changed.First calibration attempt was incomplete (no baseline saved). Real partial dev measurements on unchanged
6f6fd89product trees: stripped binary 32,643,704 bytes; idle server maximum RSS 29,646,848 bytes; 60-second idle CPU mean 0.5162214300% of one core and p95 0.9997288246%. Absolute budget outcomes are recorded separately from dev calibration; these are dev diagnostics, not perf-test VM claims.The existing real stream workload's final threshold checker is now captured at full numeric precision by a scoped Bun wrapper. Workload assertions remain intact; the new perf evaluator alone applies baseline +10% or absolute budgets. This avoids rounded values and the old fixed dev thresholds defeating calibration. Tests cover strict budget boundaries, malformed/incomplete captures and forwarding stdin without printing capabilities.
lp-flaky at
18d221c: server SSH lifecycle repeat proof passed 20/20. Both replay synchronization fixes are exercised by this one test. The first attempt found an incorrect added assertion expecting explicit gap:false; production omits that flag for complete replay (gap().then_some(true)). Commit18d221ccorrects the test to require omitted gap, with a comment explaining the wire representation. No product code changed. Verbatim repeat output:CLI replay proof remains failing/pending; cleanup, surrounding tests, fast and merge remain pending.
20:48 UTC checkpoint: merge is still queued for the required exclusive verification slot; active gate/build processes remain. load5 is 5.90 and the worktree is clean at
0909f91aa221def0755be41316b9237fa9b6e820. The gate has not been skipped or run concurrently with another verification run. No source changes since the implementation commit.lp-flaky update, head
020f6f7:908ccdf) for a fresh root under real HOME. Both startup and stale sweep use that root, leaving HOME and engine configuration untouched. All current/future verification uses the override; it confines signalling to this run's fixtures.18d221c; prior server proof 20/20 passed.020f6f7), matching the existing SSH-only selectors. Default integration continues through all MCP assertions unchanged. This enables a targeted 20-repeat proof of every changed stream regression.Proposed follow-up for full-test failure: diagnose the MCP refusal that unexpectedly returns success, retaining its PlanRequired/error assertion. The branch's product code is identical to base
6f6fd89; this test-only job will not modify authorization or MCP product paths.Lifecycle measurement now cross-checks CDP with a transparent loopback HTTP/upgraded-TCP observer. Full document replacement omitted some CDP webSocketClosed events in real Chromium runs; counting that stale ledger as real sockets would overstate leaks. The observer matches handshake keys, retires entries on actual TCP closure/read EOF, forwards authentication and bytes unchanged with pipe backpressure, and fails if ledgers do not match. A real TCP test deliberately omits the CDP close event and verifies retirement. The SPA-retained transport remains measured independently.
Setup enrollment is asynchronous because its HTTP requests pass through that same Node event loop. The one stuck owned enrollment subprocess was interrupted by verified parent-child identity; no other job was signaled.
CPU sampling now uses the external POSIX process CPU clock, including all threads, plus monotonic elapsed time. HZ=100 stat ticks quantized one-second p95 to 1% and produced a false-zero standalone baseline. Tests verify sub-tick work is retained and nonexistent processes fail instead of aliasing a Linux clock id. Reference: https://man7.org/linux/man-pages/man3/clock_getcpuclockid.3.html . All 12 targeted tests pass. No complete baseline has yet been saved; fresh complete calibration remains in progress.
Starting fix-ps in worktree
/mnt/hdd/infraternal/wt-lp/fix-ps, branchfix-private-storage, based ondevatd91cf0aef9ccec9c8803d17ee6a67054339c92bf. I am tracing the test and SQLite pool close behavior before choosing a deterministic test-only fix; scope iscrates/server/tests/private_storage.rs.Starting fix-ps in worktree
/mnt/hdd/infraternal/wt-lp/fix-ps, branchfix-private-storage, based ondevatd91cf0aef9ccec9c8803d17ee6a67054339c92bf. I am tracing the test and SQLite pool close behavior before choosing a deterministic test-only fix; scope iscrates/server/tests/private_storage.rs.Finding from source inspection:
SqliteStorage::close(self)awaitsself.pool.close()atcrates/server/src/storage/mod.rs:454-456. The pinned SQLx 0.8.6Pool::close()contract says awaiting it waits for checked-out connections to be returned or closed, so the storage close implementation does wait for the pool; this is not evidence of a product bug. The flaky assertion is the test's unconditionalsymlink()over a WAL sidecar SQLite may validly leave as a regular file. I will make the test verify and remove that existing private regular sidecar before planting the hostile symlink.SQLx 0.8.6 docs: https://docs.rs/sqlx/0.8.6/sqlx/struct.Pool.html#method.close
Merge started at load1=2.14/load5=2.81 with no active verification/release-build/browser process detected. All fast stages passed within merge, including the workspace tests; neither earlier test failure recurred on this run. No source or timeout changes were made. Release binary is currently building for the real service and production-browser smokes. One full release build was observed. Head:
0909f91aa221def0755be41316b9237fa9b6e820. Exact gate output so far (merge is not yet complete):Real 500-container workload completed on unchanged dev product trees (250 owned Docker +250 owned rootless Podman fixtures): API list p99 9.167717000000001ms; maximum conditional-304 allocation across100 empty304 responses 894,855bytes. Native/Rust process-wide interposition includes storage workers and background work, conservatively. These are dev diagnostics, not perf-test VM results.
Idle correction: Dashboard's onMount interval refreshes both list APIs every2seconds. Before both CPU windows, the harness now navigates to about:blank and drains requests, retaining origin authentication storage for the later UI workload. Earlier preliminary CPU figures included Dashboard polling and are not used as idle baselines. Corrected idle serverRSS29,212,672bytes/CPUmean0.5346486849%/p951.0480281417%; standaloneRSS16,171,008bytes/CPUmean0.04256601394%/p950.1738007420%.
Cold-route correction: the preceding lifecycle workload saves a persistent terminal reattach point; full container-detail reload restores it automatically, adding xterm to the supposedly terminal-excluded cold route. The cold phase now clears only the owned fixture's saved reattach point before route reloads; authentication remains intact and the real detached shell is cleaned up with its container. A fresh complete baseline must use this corrected harness fingerprint. No completed calibrated candidate comparison has run yet, and no saved baseline will be overwritten.
The remaining Chromium stream benchmark is waiting at host admission (load1 reached18.85; no active browser). No other job/process was signaled. Fast/merge remain pending final verification; merge will wait for load5<8 and no other scripts/verify.
Scheduling blocker after ~95 minutes: baseline build still has not started. load5 briefly fell below 6, but other verify/release Cargo processes remained active throughout those windows; load5 is now 12.60. Continuing to wait under the explicit low-priority/no-overlap rules. Worktree remains clean at
6f6fd89. No new binary size, build time, gate success or runtime result is claimed. Findings are in earlier comments; resume state is in excluded progress.md.lp-flaky continuation head
4ef61c7. Further test-only fixes and evidence:4e149cbadds a split output marker, waits for it, then closes stdin. The original total five-second deadline and remote exit3 assertion remain intact. GNU sources: https://www.gnu.org/software/bash/manual/html_node/Command-Line-Editing.html and https://ftp.gnu.org/old-gnu/Manuals/glibc/html_node/Canonical-or-Not.html4ef61c7replaces quiet sleep30 and repeated stty polling in exec interrupt/signal tests with shared remote READY output + an input gate. It retains actual raw-mode, signal status and terminal-restoration assertions.Verbatim diagnostic repeat result (current source before exec-only readiness change):
No product code changed; no process from another run has been signalled, stopped or killed in this continuation.
Remote infraternal-agent parity acceptance suite implemented on branch
e2e-remote, head8bc9fd1, base dev6f6fd89. Owned changes only e2e/remote/** and the new scripts/verify e2e-remote stage. Full production-binary table:35pass,13expected-fail,0hard failures,48cases,clean cleanup. Shared operations exercise real Docker/rootless Podman and private sshd, with a separate --no-local-agent server plus enrolled real mTLS infraternal-agent process. No mocks, no product source edits, no pushes/deployment.Detailed operation table, exact expected reasons and verbatim gates are recorded in #49. Existing gaps: remote detail/actions resolve only embedded containers; remoteSSH unsupported; confirmed remote identity withdrawal leaves client channels without CLOSE within1s; embedded identity DELETE constructs NotFound without required details and aborts production at errors.rs490 (SIGABRT), debug returns exact idempotency-in-progress. Both identity gaps are narrowly expected with evidence; other crashes/timeouts fail. Unexpected successes require retiring expectations.
Fast GREEN on cached single-threaded retry (no skipped tests). Required merge still queued under the owner condition load5<8/no active scripts/verify; multiple other actual verification runs remain active. The orchestrator can review the committed suite while this job waits for its required gate slot.
21:22 UTC checkpoint: merge remains in its first release build on this worktree. All fast stages passed within this merge attempt. No runtime, storage, tunnel or timeout changes were made after
0909f91. Merge started under the requested load/process conditions; shared VM load rose afterward. The build has made the overall merge attempt exceed the under-15-minute target; this is being recorded, not presented as a successful completed gate. Service and browser smokes remain pending. Current state and own log/session pointers are in excluded worktree-root progress.md.Finished fix-supply implementation on
lp-supply, head0909f91aa221def0755be41316b9237fa9b6e820, base6896994dfc951585905dffb626f0b270c847f922. Committed locally, not pushed; worktree clean; issue remains open.The new supply-chain gate now passes with real refreshed RustSec data and real license checking. Allowed
CDLA-Permissive-2.0with all three CA-data crates named. RSA private signing is reachable for a pre-existing RSA identity through russh/ssh-key; generated identities remain Ed25519. The reversible RUSTSEC-2023-0071 ignore is accepted 2026-10-02, reviewed by 2026-11-01, and fails the local UTC gate from 2026-11-02 onward. Six review-policy tests pass. Reasoning, upstream sources, full CDLA agreement and reversal instructions are indocs/supply-chain-exceptions.md. The titled owner decision above gives the requested options: keep ignore, disable RSA SSH keys, switch implementation.Full merge gate is not green. The requested merge attempt started only at load1=2.14/load5=2.81 with no other active verification/release-build/browser process detected. All fast stages passed inside merge, followed by release HTTP/static behavior, Docker/PTy, rootless Podman lifecycle, real sshd ownership and persistent-session authorization-loss smokes. It failed the final production-browser smoke at unchanged
scripts/browser-check.mjs:1503: the second tab's/containersnavigation exceeded 30s while waiting fornetworkidle. The log shows repeated auth-refresh traffic during that navigation. No cause is asserted solely from load; no browser timeout/test/storage/tunnel code was changed in this supply-chain job.git diff 6896994 -- scripts/browser-check.mjsis empty.The sum of reported merge-stage durations is 1577.668s; the first release build accounts for 990.197s. The under-15-minute target was not met. Earlier standalone fast runs failed unchanged storage/tunnel tests; neither failure recurred in merge's workspace tests. Their diagnostic evidence is retained in earlier comments. The existing yanked yoke-derive warning and low Bun cookie finding remain nonblocking under the existing policy.
Latest standalone
scripts/verify fastretry, exit 101, verbatim output:scripts/verify merge, exit 1, verbatim output:No extra integration, browser or bench tier was run. No runtime hot path changed. No unrelated process was signalled and no other job files were read. Follow-up outside this supply-chain fix: resolve the multi-tab production-browser gate failure and rerun merge before integration.
Verification finding:
scripts/verify fastpassed all 16 stages (total 475.724s), but the requested busy-host loop exposed a second failure after 20 consecutive passes. This was notsymlink()failing withAlreadyExists: the WAL-symlink rejection assertion atcrates/server/tests/private_storage.rs:62failed becauseSqliteStorage::open(&database)returnedOkimmediately after the test planted the WAL symlink. The storage code preflights-walwithopen_secure_owned_filebefore connecting; that helper usesO_NOFOLLOW(crates/private-storage/src/lib.rs:9-32). I am checking whether the planted symlink disappears around open; if it remains and open still succeeds, this is a product bug. No product files have been changed.lp-flaky head
a0eb7be: all three dedicated sshd cleanup checks and all 15 surrounding real_sshd cases passed individually, 18/18 total. Each test used a fresh run-specific fixture root and original HOME, one Cargo worker at nice15, checking load5 <10 before starting. The SIGKILL regression kills only its own child test process; its watchdog cleaned that private listener/run directory. Verbatim output:Current CLI STREAM_ONLY 20-repeat proof: new head compiled, iteration1 PASS (77.87s), iteration2 running. No full repeat pass claim yet. Fast/merge remain queued because another verify perf run is active. No other run process has been signalled or stopped.
Follow-up: the diagnostic test version passed 30/30 targeted iterations at low load. SQLx 0.8.6 documents that dropping
SqliteConnectionsignals its background worker to close the database, while explicitly callingSqliteConnection::close()waits for closure: https://docs.rs/sqlx/0.8.6/sqlx/struct.SqliteConnection.html. This leaves an open question about whetherSqliteStorage::close()'s awaited pool shutdown also waits for SQLite worker threads. The earlier high-load iteration hadopen()returnOkafter symlink creation, but the first version did not capture whether the symlink still existed after open. I added that diagnostic and will repeat under load after running fast on the current diff. No product files have been changed.lp-flaky head
1322fe4: current-head server SSH repeat proof passed 8 runs, then run9 failed an unchanged security deadline at load5 4.87. Both replay paths had completed; later detached-session token-revocation state did not become Exited within its one-second timeout. The one-second assertion is retained. Finishing iterations10-20 to report actual overall pass counts without hiding this failure.Proposed product follow-up: inspect detached-session authorization-loss termination and the exit-to-storage update path; align the reaper/termination completion with the contractual deadline instead of increasing the test timeout. Product code remains unchanged in this test-only branch.
Also removed an unnecessary
mutfrom the new UnixStream output gate helper (1322fe4), which would block clippy -D warnings. This does not change gate behavior.Verbatim failing run:
CLI STREAM_ONLY proof is at 3/20 passes on the latest logical changes; remaining runs continue. Cleanup/surrounding real_sshd proof passed 18/18. Fast/merge still queued behind other verification.
Continuation evidence at HEAD
1322fe4(test-only changes; no product edits):Server lifecycle final repeat sequence completed 19/20 PASS. Runs 1-8 passed, run 9 failed the unchanged one-second detached-token-revocation assertion, and runs 10-20 passed. The deadline is explicit in docs/contracts.md section 3.11, lines 1487-1493. Both replay scenarios completed before run 9 reached revocation. See server-repeats-run.log, server-remaining-repeats-run.log, and ssh-product-9.log in target/lp-flaky-evidence. Do not relax the security assertion; proposed follow-up is to inspect authorization-loss polling/termination and exit-state persistence against the one-second contract.
The current 20-run CLI stream sequence is ongoing. Iteration 5 failed before exec SIGHUP readiness with
HTTP 500 Internal Server Error: The stream ticket could not be created. (internal). Iteration 6 failed before exec SIGQUIT readiness withthe stream peer closed the WebSocket with code Some(4003). These are explicit stream-open failures, not absence of the remote readiness marker caused by synchronization. No signal was sent in either failed readiness case. Evidence: cli-stream-5.log and cli-stream-6.log. Proposed product follow-up: inspect ticket-storage/audit errors and the WebSocket 4003 rejection path under repeated real Docker/Podman streams. No retry or weakened assertion added.Fast and merge remain queued behind other active scripts/verify runs; their runners poll every two minutes. This continuation has not signalled/stopped/killed any other process.
Completed fix-ps. Commit:
543818f57e304f8153c6d4b6545c0a7a3ddb5ecaon branchfix-private-storage(based91cf0aef9ccec9c8803d17ee6a67054339c92bf).Change: the test now uses
symlink_metadatabefore removing an existing WAL sidecar, asserts that it is a regular mode-0600 file, and verifies the hostile symlink fixture before checking that database open rejects it. An unexpected successful open now reports whether the symlink remained afterward.Close check:
SqliteStorage::close()awaitsself.pool.close(). SQLx 0.8.6 documents that awaited pool close waits for pool connections: https://docs.rs/sqlx/0.8.6/sqlx/struct.Pool.html#method.close. Its SQLite connection docs say drop signals the background worker, while explicit connection close ensures the database is closed: https://docs.rs/sqlx/0.8.6/sqlx/struct.SqliteConnection.html. No product files were changed.Targeted test loop: 30/30 passed. It started with uptime load average
11.79, 9.33, 8.71.An earlier 30-run attempt on the less diagnostic patch had 20 passes, then
SqliteStorage::open()unexpectedly returnedOkafter WAL symlink creation. The final diagnostic patch did not reproduce that behavior in 30 busy-host iterations; the originalAlreadyExistscondition is addressed by validating and clearing a leftover regular WAL before creating the symlink.scripts/verify fastoutput on the committed diff:Merge gate pending.
Starting wal-race on branch
wal-race, base21b267b9b949e08b32e0125221df28cda462ad69. Scope: private SQLite open path andcrates/server/tests/private_storage.rs. Investigate intermittent WAL symlink acceptance with up to 200 iterations under load; targeted tests andscripts/verify fast; merge gate pending centrally. No push.Blocked / unfinished after more than two hours waiting for a shared-VM slot. Current load5=9.96. Every window with load5 <=6 still had other verify/release Cargo processes active; earlier/later load5 exceeded 6. Respecting the explicit low-priority/no-overlap rules, no new release build or gate started.
Branch lp-binsize remains clean at base/head
6f6fd89621; no manifest changes or new commits. No new size, build-time or stream-performance result is claimed. Frozen JS dependency install and local code index completed; feature/crypto/panic findings are recorded in earlier comments.Only this job's waiting Python process has been stopped, so it will not launch a build after the handoff. Excluded progress.md records next steps. A quiet VM slot is required to resume the baseline, measured trials, fast gate and quick stream benchmark. The 25 MB budget remains unresolved. Issue stays open.
wal-race inspection evidence (base
21b267b):SqliteStorage::openchecks the main file and existing-wal/-shm/-journalonce before constructing a new SQLx pool; it does not use a cached pool or check before each additional pool connection.backupindependently callsself.options.clone().connect()and drops that connection.SqliteStorage::closeonly waits for the pool. SQLx 0.8.6 uses a background SQLite worker; Drop requests shutdown without awaiting it, whereas explicit close awaits it (SQLx docs, versioned source). SQLite last-connection close deletes WAL/SHM (SQLite WAL docs §6). This is a plausible late cleanup race with the fixture.The bundled libsqlite3-sys 0.30.1 SQLite Unix VFS already includes
O_NOFOLLOWin WAL/journal open flags and SHM opens (versioned SQLite source, functions unixOpen/unixOpenSharedMemory). It can create WAL viaO_CREAT, but does not follow final sidecar symlinks on this Linux target. The documented same-OS-user trusted-writer boundary remains relevant. Reproduction and a deterministic lifecycle regression are next.Final perf calibration attempt (harness1a59bc0) completed idle/browser/all18coldroutes/list/304, then its stream subprocess exited101. No baseline was saved; the partial report remains complete:false. Real gate output:
run_bun_with_stdin_readyrequires STREAM_BENCH_READY within15seconds. Admission waiting inside the Bun workload can consume that deadline. Moved admission outside the timed workload: cargo test --no-run first, then host/browser admission, then actual cargo test. Bun forwards workload/stdout/secretstdin unchanged and replaces only the final numeric threshold checker. All12 targetedtests pass at035ae81. Fresh complete calibration is running; scripts/verify fast is running serially.Classification correction before final baseline: a shared transport still in the current SPA document is recorded as PERF_DIAGNOSTIC spa_idle_sockets, not mislabeled as an old-document socket. The zero old-document budget uses actual document identities and physical TCP closure; full navigation measured zero old sockets/channels. The earlier provisional socket1 finding referred to the current SPA document.
Corrected cold container-detail measurement excludes automatic persistent-terminal restoration:173,692gzipbytes, still above120KiB. The detached shell remains real and is cleaned up with its owned fixture. Product crates/packages remain unchanged from6f6fd89.
e2e-remote blocked handoff at
8bc9fd128610f4490debe54fe29470405679ba25(branch e2e-remote, base dev6f6fd89). Implementation committed; full production parity35pass/13expected-fail/0fail,48cases,clean cleanup; fast GREEN without skipped tests. Detailed operation table, exact expected reasons, discovered embedded NotFound-constructor abort and verbatim gate output are in #49.Required merge was not attempted: owner's load/exclusivity conditions never became eligible while this job waited. Only the owned unstarted waiter was retired; no other job was touched. The orchestrator must reserve a quiet slot and run scripts/verify merge before integration. Worktree clean, no push/deployment, no background waiter or owned fixture resources remain. progress.md contains exact continuation steps.
wal-race resumed on branch
wal-race, base21b267b9b949e08b32e0125221df28cda462ad69; owned private-storage lifecycle and regression tests only.Root-cause evidence: retained baseline run had 9 failures/200. Accepted opens reported
symlink remained after open: false; another failure found the planted WAL missing. Test filesystem setup is synchronous and asserts the symlink before awaiting open.openvalidates sidecars once before constructing a pool (max 8 connections, no connection hook).backupopens an independent SQLx connection and drops it;closewaits only for the pool. Deterministicbackup_cannot_open_a_connection_after_storage_closesfails on base because backup can reopen a closed storage clone.SQLx documents that SQLite uses a background worker and Drop merely signals shutdown: https://docs.rs/sqlx/0.8.6/sqlx/struct.SqliteConnection.html . Pool close waits for acquired connections: https://docs.rs/sqlx/0.8.6/sqlx/struct.Pool.html#method.close . SQLite last-connection close removes WAL/SHM: https://sqlite.org/wal.html#the_wal_file . Together with baseline disappearing-link evidence, this identifies a product lifecycle gap: delayed backup cleanup races reopen preflight, rather than unfinished fixture setup. Fix uses a pooled backup connection, so shutdown accounts for it on success/failure and refuses backup after close. No filesystem checks added to per-query hot paths. Added a 200-iteration real-SQLite regression loop; gates underway. Merge gate pending.
Additional unchanged assertion failure in current repeated CLI stream proof (HEAD1322fe4): iteration15, load5 at start8.72, finished55.39s. Podman
logs --tail 4 --timeout 1returned success with stdout=[] and no remote stderr; expected both exact fixture stdout lines. Evidence cli-stream-15.log, assertion cli_mcp.rs421. The fixture starts via real Podmanrun -dusing k8s-file logging, and no direct engine-log observation was retained at failure, so this evidence does not establish whether fixture log publication or the product stream caused the empty result. Proposed follow-up: retain/observe the fixture log records through the real engine before the snapshot and capture stream opening/timeout timing; preserve the byte-exact assertion and the one-second timeout contract. No product edit, retry, or relaxed assertion added.Owned synchronization finding and fix at
a9789dd: iteration16 of the current CLI stream sequence failed unchanged five-second MCP SSH read deadline. Inspection found one remaining inline BEFORE marker in the MCP SSH shell command; Python could accept the terminal echo as readiness. The other SIGHUP/killed-client commands already split their markers. This commit splits the MCP marker too and awaits server-observed detached state before MCP attachment. The original marker/read deadlines and all output/status assertions remain intact; timeout diagnostics now include received fixture output. The finding establishes an unsafe readiness check, not a definitive causal explanation for run16.Added full-SSH-only selection of the existing integration test to exercise the complete real SSH path (EOF, resize, SIGHUP and SIGKILL replay, MCP send/read, signal restoration and host-key refusal) without unrelated container checks. Fresh 20-run proof will follow current sequence completion. Current sequence iterations1-18 used1322fe4; remaining19-20 will usea9789dd after compilation. Preserve all earlier failures and report the source transition; do not label the mixed sequence as a final-head proof. Fast/merge stillqueued, no other process signalled.
Further wal-race evidence and corrected root cause: pooled backup alone was insufficient; the regression still failed. A real syscall trace (
/tmp/wal-race-syscalls.log) caught the race: test thread 1961049 planted the WAL link at 1790971324.102687; SQLite worker 1963506 unlinked that same WAL at 1790971324.102828, after storage close returned. This explains both accepted opens (link disappears before preflight) and ENOENT fixture failures.SQLx0.8.6 pool close itself can return early when idle and returning connections overlap.
PoolInner::closedrains idle connections viaLive::float; itsDecrementSizeGuard::new_permitreleases a semaphore permit even though release-to-idle already released it. Close's semaphore wait can therefore complete with other connection guards still outstanding. Source: https://github.com/launchbadge/sqlx/blob/v0.8.6/sqlx-core/src/pool/inner.rs and https://github.com/launchbadge/sqlx/blob/v0.8.6/sqlx-core/src/pool/connection.rs . Disabling idle reaping did not fix it and was reverted.Small lifecycle fix: keep backups pooled and, after closing admissions, repeat pool close until
pool.size()==0, with a 1ms async sleep to avoid spinning during outstanding operations. Pool remains bounded at 8; no per-query filesystem validation. The preflight descriptor now closes before SQLite connects, avoiding unrelated descriptor close during SQLite POSIX locking (https://sqlite.org/howtocorrupt.html#posix_advisory_locks_canceled_by_a_separate_thread_doing_close).The first full regression with explicit size drain passed:
Final targeted tests and fast gate follow. Merge gate pending.
Product-path hang in CLI stream attempt19 at
a9789dd: modified full SSH/MCP path completed, then the compiled CLIexec <this-run Docker fixture> -- truedid not exit for several minutes. The integration test waited in its existing unbounded run_cli wait_with_output. Captured only this run's process ancestry, client thread waits and filtered audit metadata: cli-stream-19-processes.log and cli-stream-19-interrupted.log. Server stream.open was accepted for exec; client main/runtime threads waited on futex/epoll. This does not establish whether remote execution or exit-frame delivery stalled. Proposed follow-up: trace exec completion and EXIT/CLOSE handling; separately replace this unbounded test wait with a bounded process-exit assertion so failures terminate without manual intervention. No product source changed.To unblock this test, signalled ONLY compiled CLI PID1981715 started by our current test. Immediately revalidated executable path, exact command arguments, start_ticks2689090 and ancestry CLI1981715 -> test1976469 -> Cargo1975286 -> this run's runner1739814. No other process or service was signalled/stopped/restarted. Attempt19 is explicitly INTERRUPTED, not a natural timeout failure or a pass; retain it in the attempted20 totals.
wal-race finished. Branch
wal-race; base21b267b9b949e08b32e0125221df28cda462ad69; head53b54c86c08e9f0e4e848931ab2aa4805df73404. Committed, clean worktree, not pushed.Verdict: product lifecycle gap (b), not unfinished test setup. Late SQLite worker cleanup removed the already-planted WAL before validation; syscall timing and SQLx0.8.6 semaphore accounting analysis are recorded in the preceding finding. Backup now acquires a pooled connection; close waits for actual pool size zero with a 1ms async wait and re-drain. Preflight descriptor closes before SQLite connects. Pool maximum remains 8, with lazy expansion and no per-query filesystem checks. Added backup-after-close regression plus a 200-iteration real-SQLite backup/close/reopen/privacy loop. Comments cite SQLx and SQLite sources. Changes limited to
crates/server/src/storage/mod.rsandcrates/server/tests/private_storage.rs; progress.md excluded and uncommitted.Baseline: 200 iterations, 9 failures. Final separate-process repeat of the original test, exit 0:
Targeted
cargo test -p infraternal-server --test private_storage --test storage -- --nocapture, exit 0 (includes consistent snapshot during live writes):scripts/verify fast, exit 0, verbatim output:No merge, integration, browser or bench gates run. Shutdown-only correctness change; no new request hot path. Source review and targeted checks complete; independent review and merge gate pending for the orchestrator. Never closed this issue.
Orchestrator review of wal-race
53b54c8: approve. Follow-up (minor):SqliteStorage::close()loops untilpool.size()==0with no bound, so a connection held forever would hang shutdown. Add a bounded wait that logs and returns an error after N seconds. Merge gate is queued after lane D.Complete corrected dev calibration on infraternal-dev, 2026-10-02T20:08:12.118Z, harness head
035ae816c6. All 36 required metrics completed; product crates/packages are unchanged from dev6f6fd89. These are dev diagnostics, not perf-test VM A/B certification. Absolute outcomes below describe these dev observations only; --perf-vm enforces those budgets on the perf-test VM.Units and full-precision source report:
Loaded RSS is the existing benchmark’s conservative whole-test-process VmHWM, including the embedded server; it is not standalone server RSS certification. The 304 counter includes native/Rust allocations across storage workers/background tasks; request construction is excluded. Cold JS excludes only the fixture’s saved automatic terminal reattach point; authentication is preserved. Zero old-document counts use real document identity and physical TCP closure. The current-document SPA singleton socket is diagnostic only.
scripts/verify fastcompleted successfully, using CARGO_BUILD_JOBS=2 and RUST_TEST_THREADS=1. Verbatim output:The actual scripts/verify perf --dev-vm comparison is now running against this first complete corrected baseline. The baseline will not be overwritten after a numeric failure. Merge remains pending its load5<8/no-other-verify admission.
Interim verification at
a9789dd(work remains; fast/merge queued):The CLI stream sequence attempted all 20 tests: 15 PASS, 4 natural assertion failures, and 1 manually interrupted hung own CLI (attempt19). Raw runner classifies the interrupted attempt as FAIL, so its exact RESULT is passes=15 failures=5. Iterations1-18 used1322fe4; iterations19-20 useda9789dd. This is a mixed-source sequence, not a final-head 20/20 proof. Failure5: HTTP500 exec ticket creation; failure6: WebSocket4003 before exec SIGQUIT readiness; failure15: empty Podman tail snapshot; failure16: MCP SSH output deadline, followed by the owned marker fix; attempt19: Docker exec true hang, own verified CLI interrupted after227s. No assertion relaxed.
Exact runner count lines:
Fresh complete-SSH 20-repeat proof at
a9789ddis queued/running with its own fixture root and load5<10 check (cli-ssh-full-repeats-run.log). This includes MCP, both detach/replay paths, EOF, resize, signals/restoration and host-key refusal, while avoiding the unrelated container failures. Server lifecycle sequence completed19/20 PASS (one unchanged revocation deadline failure); surrounding real-sshd and cleanup checks completed18/18 PASS. Compilation of latest source succeeded in cli-stream-19.log. Heavy fast/merge still cannot start while other scripts/verify runs are active or load5 exceeds6; polling continues every2min.Final-head full SSH proof completed at a9789dd:18/20PASS,2FAIL. All20executed the exact ignored integration test using SSH_FULL_ONLY; no observer or retry inside the tests. Original five-second EOF and fifteen-second CLI process-exit limits remain intact.
Run4 (start load5=8.91): SIGHUP reattach delivered BEFORE, AFTER, finish and remote
exit, then Python pidfd/process-exit observation reached the unchanged15s timeout (test24.04s). Evidence cli-ssh-full-4.log. Run19 (start load5=7.21): piped SSH EOF did not finish within original5s (test11.26s). Evidence cli-ssh-full-19.log. The new split MCP marker/state gate did not fail in this sequence.This does NOT prove the known exit hangs fixed. The independent Docker exec true CLI hang in stream attempt19 lasted227s, suggesting a broader product completion/EXIT path worth tracing. Proposed product fix investigation: observe server session completion/exit state and EXIT/CLOSE delivery together with CLI receive-loop termination and input-worker shutdown. Preserve all deadlines and remote exit assertions; no product edit in this branch.
Exact runner counts:
Compilation and ancillary18/18checks passed. Server20-repeat remains19/20 due unchanged contractual1s revocation deadline. Fast/merge still queued behind other active verification runs; pgrep polling every2min continues. No other process/service signalled.
Correction to the preceding proof comment: run19 started at load5=7.38, not7.21. Exact runner line:
The18/20count and failure/deadline evidence are unchanged.
Request-path panic scan (
rg -n 'unwrap\(|expect\(|unreachable!' crates/server/src) found the two concreteApiError::newassertion failures in the agents route; both are fixed and covered. The other required-detail error call sites found by checkingNotFoundandInvalidValuenow either pass their typed details or are the fixed agents helper.Remaining invariant sites for follow-up:
crates/server/src/stream.rs:1852:host_id.expect("visible stream targets have a canonical host")after stream target lookup/authorization; the assumption depends on the visible-target constructor.crates/server/src/resources.rs:2007:_ => unreachable!()when mappingaction_scope; the preceding action parsing currently admits only the four container lifecycle scopes.crates/server/src/device_auth.rs:910,914,953,967,1207,1276,1352: validated record/stateexpectandunreachable!branches; current checks run under the device-state mutex and return invalid/terminal records before these sites.crates/server/src/lib.rs:612:code.expect("error responses have a code")follows anor_elsefallback that always returnsSome.Reviewed auth serialization/cookie/status expects and resource ETag/status expects use generated or validated values. No change made to lane-owned stream/resource/device-auth/lib files in this job.
Confirmed product-side completion failure, not remote EOF readiness, at
a9789dd. Observed full-SSH diagnostic batch:9PASS, run10failed original5s EOF process-exit assertion; observer stopped after capturing failure. Plain final-head proof remains18/20 and is reported separately.While the compiled CLI PID2138780 remained alive, server session s_01M3Z56CEVKNNFPQEJWHKXF42G was already persisted as exited with exit_code3 at two successive snapshots. Thus remote EOF reached cat and the expected remote exit occurred before the local process timeout. CLI main/runtime/blocking threads were waiting in futex/epoll; no causal claim from those waits alone. No assertion or deadline changed, no product code edited. Exact safe metadata:
Evidence: diagnostic-full-test-10.log (original EOF timeout) and diagnostic-full-state-10.log (full owned-client thread/session snapshots), target/lp-flaky-evidence. Proposed product fix: ensure completed SSH sessions enqueue/deliver EXIT/CLOSE to the current attachment, and ensure the CLI exits with that code without waiting indefinitely on input or background workers. Trace both server EXIT delivery and CLI receive/cleanup before choosing the fix. Relevant product sources are outside this branch's allowed test-only ownership. Existing5s/exit3 regression must remain unchanged and repeat cleanly after the product fix. This also warrants comparing the SIGHUP reattach exit hang and227s Docker exec true hang already reported.
Fast/merge stillqueued behind active verification runs. This run has not signalled any other process/service.
Dev comparison at HEAD
035ae816c6, unchanged product trees vs dev6f6fd89. The fixed complete baseline remains target/perf/base-final.json; no recalibration after failures. Numeric failures observed so far:Cold-route regression checks and all old-document socket/channel checks passed. The 500-container TCP list p99 is 7.650645 ms (limit 8.7865437 ms). Final stream benchmark remains pending host admission. These are shared dev-VM diagnostics, not perf-test VM certification. Full fast already passed; merge remains required after this run, load5 <8 and no other actual verification run.
Panic audit (ApiError code/details mismatch), dev @
c56a8fb, read-only. Release builds use panic=abort, so each of these takes down the whole process.Fixed: agents.rs:318/344/126 (not_found / invalid_value without details). Fixed by
5a85e5c, now merged to dev.Still open, HIGH:
Latent: agent/src/lib.rs:1461. ssh_error with no stage (TakenOver) gets Empty details; open() cannot return it today.
Landmines: the helpers api_error_response (lib.rs:771), auth error_response (auth.rs:1612, which also hard-codes retryable=false), device_auth handled_error (:1899), agents error (:96), stream send_open_error and agent send_error/send_stream_close_raw all accept any code with None/Empty details. All ~200 current literal call sites are correct. Other unwrap/expect/unreachable/indexing in handlers: none reachable from request input.
Structural fix:
ApiError panic fix: branch
fix-apierror-panics(not pushed). Base: devc56a8fb. Head:3bf9496e19Commits:
46efd21proto:ApiError::newandwith_retryableno longer panic in release. The checks stay asdebug_assert!. proto has no logging dependency, so none was added. Mismatched pairs are repaired byrepair_details, and the retryable flag byclamp_retryable. Fallbacks:Valid pairs are unchanged. Tests cover every code x every details variant (the result is contract-valid and round-trips through the strict decoder), clamping, debug-assert should_panic, and a
cfg(not(debug_assertions))test callingnew+with_retryabledirectly (run with--release).558d703engine:EngineError::api_errornow overrides retryable only for engine_error, ssh_error and internal. This fixes the Podman inspect over 16 MiB (ResourceExhausted, bollard_adapter.rs:1612/1678), whose wire retryable is now the registry value (true). The local flag still drives in-process retry. A test was added.3bf9496cli:resolve_host_idunknown host -> not_found{kind:"host"}. Ambiguous name -> newClientError::AmbiguousHost-> invalid_value{field:"host"}. Exit code 2 is kept for both. Added an axum-backed test.Not edited: crates/server/src/stream.rs:1389. The embedded-agent
target_existserror goes tosend_audited_open_errorwith Empty details. It no longer aborts after46efd21, but in release it will emit repaired details (for example engine_error with engine_code "unknown", or invalid_value downgraded to invalid_request) until the stream.rs rewrite passes matching details.Targeted tests:
cargo test -p infraternal-protook (61 + 2 passed). Release-mode errors tests: 9 passed.cargo test -p infraternal-engine: 23 passed, 1 ignored.cargo test -p infraternal: 42 lib passed, new test ok.INFRATERNAL_VERIFY_RUN_ID=sub-apierror scripts/verify fast(exit 0):PASS fast: migrations are append-only (0.139s)
PASS fast: workspace package dependencies (0.227s)
PASS fast: JavaScript dependencies (4.560s)
PASS fast: build web app (13.037s)
PASS fast: cargo fmt --check (1.057s)
PASS fast: cargo clippy (76.205s)
PASS fast: cargo test (133.093s)
PASS fast: svelte-check (26.920s)
PASS fast: UI package svelte-check (13.489s)
PASS fast: stream package typecheck (8.366s)
PASS fast: Vitest (6.101s)
PASS fast: UI package Vitest (17.153s)
PASS fast: stream package Vitest (1.397s)
PASS fast: OpenAPI and TypeScript client drift (78.177s)
PASS fast: terminal TypeScript (3.170s)
PASS fast: terminal Vitest (1.755s)
Flaky production browser smoke (goto timeout 30000ms on networkidle): fixed on local branch fix-networkidle, commit
d71768f(not pushed). scripts/browser-check.mjs now opens the polling routes (/, /hosts, /containers) with waitUntil domcontentloaded, then waits for the route h1 (waitForDashboard for /) and for its loading skeleton to go away. That covers checkSharedCookieTabs and the other 23 polling-route navigations. Assertions and multi-tab coverage are unchanged. networkidle stays on pages that don't poll (setup/sign-in, /device, /__ui-tests, detail pages) and on the pending-refresh tab. Playwright docs mark networkidle DISCOURAGED for testing: https://playwright.dev/docs/api/class-page#page-goto. scripts/verify fast passed. I skipped scripts/verify browser because another browser verify was already running.perf-gates: measured table after merging current
dev(head04adb17)origin/devis merged intoperf-gates. The merge had no conflicts. The perf browser workload now uses dev's rule for routes that poll: DOMContentLoaded, then the route's h1, then the first inventory load (loading skeleton detached). Signed-in/,/hostsand/containersno longer usenetworkidle.scripts/perf/Cargo.lockalso picks up the newprovider-api/host-resource-clientpath crates.Gates
INFRATERNAL_VERIFY_RUN_ID=sub-perf scripts/verify fast: exit 0. All 16 checks passed (clippy 191.5s, cargo test 165.2s).scripts/verify perf --dev-vm --calibrate target/perf/base-v2.json:PASS perf: measured performance budgets (766.250s). It produced a complete baseline: 36 metrics,failures: [], head04adb17. That baseline is now the reference for--dev-vm --baselineregression runs on this host.scripts/verify mergefrom another job finished. Host load averaged about 9–11 from other jobs during the run, so the CPU and echo figures are noisy.scripts/verify merge: not run, by instruction.--perf-vm) absolute A/B was not run. The figures below are dev-VM numbers and do not certify anything.Fails the absolute budget today: standalone agent RSS; agent idle CPU p95 (probably host noise); cold JS on
/hosts,/containers, container detail and the__ui-testsfixture; the 304 allocation; echo p99 (host load); binary size.The 304 allocation dropped from 861,944 B (previous baseline on
035ae81) to 344,873 B on current dev. Diagnostic:spa_idle_sockets(the current document's shared socket, not a leak) was 0 in 9 cycles and 1 in 1 cycle.Usage (see
scripts/perf/README.md): on the dev VM, runscripts/verify perf --dev-vm --calibrate <new file>once on the base head. Then runscripts/verify perf --dev-vm --baseline <file>to fail on more than +10% regression (throughput: below baseline / 1.1). On the perf-test VM, runscripts/verify perf --perf-vmto enforce the absolute budgets above. Every observation prints aPERF {json}line, and the stage exits nonzero when any metric is over its limit.Continuation after the disk-full stop. Merged origin/dev cleanly (
471b6df); added a test-only diagnostic in ssh_product.rs (api()now panics with method/path/status/body when a response body does not decode). Head0db8e9b, not pushed.20-repeat loops, all assertions and deadlines unchanged, shared VM with another merge gate running (load5 mostly 7-11):
ssh_product::host_trust_stream_replay_resize_close_and_changed_key(merge head)detached SSH session survived token revocation beyond one second(ssh_product.rs:866); 1x session-list request returned an error body (unknown field code)0db8e9bwith the decode diagnosticcli_mcp::cli_and_mcp_use_the_public_http_contractwithINFRATERNAL_CLI_MCP_SSH_FULL_ONLY=1compiled CLI did not exit: the PTY transcript shows__CLI_SSH_REPLAY_AFTER__,finishandexit, so remoteexit 19ran and the CLI stayed alive; 1x piped-stdin EOF did not finish within 5 sreal_sshd::pty_resize_and_replayNo replay-offset or replay-sleep failure occurred: every failure came after the test's synchronization point. Every CLI failure is the known product hang (the remote session has exited but the CLI never receives or acts on EXIT/EOF), which fits the server dropping zero-payload EOF frames and the SQLite lock-loss bug being fixed on wal-race. The 1 s revocation-termination deadline is the contract under test, and the check is already measured after the attachment closes, so it was left as is. The single error-body session-list response may be the same SQLite lock loss. All of these need product fixes, after which these loops should be rerun.
INFRATERNAL_VERIFY_RUN_ID=sub-flaky scripts/verify fastat0db8e9b: exit 0, 16/16 PASS. Merge, integration and browser gates were not run in this continuation. No process this run did not start was signalled.SQLite transaction hardening. Branch
sqlite-txn-hardeningat4ce2d21, on top offix-stream-teardown3811185. Committed on the VM, not pushed.Audit of
pool.begin():with_immediate_retry:consume_agent_join_token(agents.rs). It SELECTs the join token, then UPDATEs it.bootstrap_user,insert_token_with_admission(theNonebranch),finish_setup,update_passkey_and_issue_session,add_passkey,issue_device_session,create_plan,finish_plan,accept_agent_tunnel_nonceandrotate_remote_agent_certificate. In each, the first statement is a write. SQLite then takes the snapshot and the write lock together, under the busy handler, so the lock-upgrade failure cannot happen. This was verified on the old code: these tests passed and only enrollment failed. Tests keep it true: adding a read before the first write makes them fail.stream_authorization.Raw
BEGIN IMMEDIATE, from the security re-review: all eight are now SQLx transactions.with_immediate_retry:ensure_local_host,ensure_setup_token,create_setup_ceremony,take_ceremony,refresh_session,revoke_refresh_familyandclaim_plan.dispatch_stream_openusespool.begin_with("BEGIN IMMEDIATE")and commits, with no retry, because its callback must run at most once.finish_manual_transactionis removed. No raw BEGIN string remains instorage/.Other fixes:
is_busymissed extended result codes: SQLx reports 517, 261 and 773. It now matches on the primary code (code & 0xffis 5 or 6).SqliteStorage::close()now waits at most 5 s (CLOSE_DRAIN_TIMEOUT). It then logs awarnwith the held-connection count and returns. The pool is already closed, so a held connection closes when it is returned.Tests: 16 new, in
crates/server/src/storage/txn_tests.rs.concurrent_agent_enrollments_commit_behind_a_concurrent_writer. Before the fix it failed with(code: 5) database is locked; now it passes.concurrent_{agent_tunnel,plan,credential}_writes_…andconcurrent_setup_finishes_….cancelled_<fn>_releases_the_write_lock, one per raw-BEGIN function (8). Each aborts the call after BEGIN, then makes 8 writes with a 2 s limit each.close_gives_up_on_a_held_connection_after_its_boundandclose_drains_an_idle_pool.busy_classification_uses_the_primary_result_code.Gates:
cargo test -p infraternal-server: all pass.INFRATERNAL_VERIFY_RUN_ID=sub-sqltxn scripts/verify fast:FAIL fast: cargo test, at load5=19.39 on 8 CPUs. The script printed only the first 240 lines of that step and deleted its log, so the failing test is unknown.cargo test --workspace --no-fail-fastpassed (exit 0).PASS fast: cargo test (140.698s)andPASS fast: OpenAPI and TypeScript client drift (80.927s).Conditional 304 allocation (345 KB vs 128 KiB budget): fixed on
perf-304-alloc8f07e8d(basedev72d7485, not pushed)Root cause. It was not the handler.
list_containersalready compares a cheap ETag (generation plus query hash) before touching rows. The steady 304 cost is about 41 KB. The 345 KB was an outlier from opening a new SQLite connection inside the request:crates/server/src/storage/mod.rs:1445(base):authenticate_accessran the token lookup withfetch_optional(&self.pool), dropped that connection, and then:1462calledstream_authorization, which acquired again at:1652(self.pool.begin()).PoolConnection::drop), so the second acquire often found the idle queue empty. The pool (:3831, max 8, lazy growth) then opened a new connection on the request path.sqlx-sqlite-workerthread (schema and page cache), plus about 40 KB of normal request work. A connection's first use cost about 50-85 KB more.Fix.
authenticate_accessnow runs the lookup and the grant snapshot in one transaction on one connection, through a newstream_authorization_in. Both reads also come from one snapshot.SqliteStorage::open/in_memoryopen every pool connection after migrations. Each connection runs the four per-request auth statements once (warm_authentication_pool), and the pool keeps its connections (idle_timeout/max_lifetime=None).min_connections: its background opens race migrations and fail withSQLITE_BUSY(that brokeagent_tunnel_tlsandprivate_storagein a first attempt).The ETag is unchanged, so 304 correctness is unchanged.
Numbers. Dev VM diagnostics only, not perf-VM figures. Same production
Server/router, sameallocator.cinterposer, max of 100 validated 304s:Test.
crates/server/tests/conditional_304_allocations.rsis its own binary incargo test, so it runs in fast. It re-executes itself with an unreachableDOCKER_HOSTso other jobs' engine events cannot add allocations. It sends 100 304s through the full router and checks two things:stats_alloc, since the workspace forbids unsafe code).Gate.
INFRATERNAL_VERIFY_RUN_ID=sub-304 scripts/verify faston8f07e8d:I did not run
scripts/verify perf. The real 500-container number still needs a perf-VM A/B.perf-agent-rss: standalone infraternal agent idle RSS (branch
perf-agent-rss, commite39f453, based on72d7485; not pushed)I measured with the perf harness's own agent step:
browserMetricsup toinfraternal_agent_idle_rss, which takes the maxVmRSSover 60 one-second samples. These are dev VM numbers, not reportable ones.Before (
72d7485): 16.2, 16.5 and 17.3 MiB. The 23.8 MiB in this issue came from a loaded run on04adb17.After: 14.94, 15.07, 15.08, 15.22 and 15.29 MiB, so about 15.1 MiB on average, a cut of about 1.4 MiB. It is not reliably under 15 MiB.
Where the RSS is (smaps and pagemap, before)
.text(201 fully mapped 64 KiB fault-around windows out of 23 MiB).rela.dyn(50k relative relocations, read by ld.so).rodata.data.rel.ro, dirtied by relocation (mostly OpenSSL and aws-lc tables)Anonymous memory is small. The tokio pools, bollard/hyper pools, provider snapshots (9 containers on the host) and rustls roots (the agent loads only its CA, no webpki or native roots) are all minor. Ablation: without engines the agent is 13.2 MiB, mostly less text; key generation versus an existing key makes no difference. About 1.1 MiB of resident text is server-only crates (rmcp, axum, sqlx, webauthn, zbus). The agent never runs them; fat LTO places them next to code it does run.
Changes
-z pack-relative-relocsincrates/infraternal/build.rs). Saves about 1.2 MiB RSS. The binary shrinks by 1.18 MB (33,475,832 → 32,300,352 B). PIE/ASLR are unchanged. This needs glibc 2.36 or newer; the deploy baseline is Ubuntu 24.04 with 2.39 (noted in DEPLOY.md). A test reads the ELF dynamic section.agent runuses 2 tokio workers instead of one per core, and each engine's exec-cleanup runtime uses 1 worker instead of 2. The agent drops from 15 to 7 threads, saving about 0.25 MiB.agent run(prctl(PR_SET_THP_DISABLE)through nix). This host has THPalways,thp_fault_allocat 6M and khugepagedmax_ptes_none=511, so a sparse arena or stack can become a 2 MiB page. A long-runninginfraternal serveron this host has 18 MiB of its 30 MiB RSS in huge pages. My runs had none (MemFree was low), so this change caps variance like the 23.8 MiB run rather than lowering the median.INFRATERNAL_VERIFY_RUN_ID=sub-rss scripts/verify fast: all 16 checks pass.Tried and dropped
Boxing the main future saved 28 KiB.
-z keep-text-section-prefixmade no difference, because.text.unlikelyis only 0.9 MiB. lld--icf=safewould fold nothing because rustc emits no addrsig, and--icf=allcan merge distinct statics. Making SSH lazy would undo the deliberate rule that corrupt host keys are rejected at startup.Going reliably under 15 MiB needs an owner decision
sorzfor the release profile. This shrinks the text that drives the fault-around windows and would also help the 25 MB binary budget. Server throughput and echo budgets must be re-checked. Not measured..data.rel.ro, but the executable loses ASLR.madvise: needsunsafe, which the workspace forbids.Starting contract-drift-rest on branch
contract-drift-rest, base head450c297354f4d2f07c84ace930949a6bacdbb900(origin/dev). Scope: D9, D11–D14 (configured access lifetime only), D16, and D18 from the contract drift review. I’ll regenerate OpenAPI and the TypeScript client, add targeted tests, and runINFRATERAL_VERIFY_RUN_ID=contract-drift-rest scripts/verify fastas requested.Starting perf-guards-nightly on branch
perf-guards-nightly, based onorigin/devat450c297354f4d2f07c84ace930949a6bacdbb900.Starting ops-install-restore on branch ops-install-restore at base
450c297354. Scope: generic server/remote infraternal agent installation, offline validated atomic restore, nightly backup/restore and prior-binary upgrade smoke, operator documentation. Required final gate: lean scripts/verify fast; central orchestrator owns merge gate.Implementation findings (#48):
code-review-graph search SqliteStorageinitially returned 0 nodes while rebuild ran. Rebuild succeeded:Full build: 521 files, 5356 nodes, 75296 edges (postprocess=full). Subsequent search returns storage open/backup/call sites. Used rg after the initially empty graph.SqliteStorage::open; this job leaves that open path untouched for storage-postmerge-fixes. Ordinary storage opens will share a persistent sibling.locklease; offline restore requires its exclusive lease. Older binaries/external SQLite programs do not honor it and must be stopped explicitly..restore-pendingmarker fences startup across interrupted file-set publication.e339ad192d(Merge CLI and MCP parity). No release tags exist. Its SQLite migration files have no diff versus base450c2973; the operations stage will prove real binary replacement/state retention on this compatible lineage, not claim a schema migration that did not occur. Unit tests separately exercise an older migration prefix.Platform sources: Rust file locks (stable since 1.89): https://doc.rust-lang.org/src/std/fs.rs.html ; SQLite WAL and preservation: https://www.sqlite.org/wal.html and https://www.sqlite.org/howtocorrupt.html ; SQLx immutable/read-only options: https://docs.rs/sqlx/0.8.6/sqlx/sqlite/struct.SqliteConnectOptions.html ; systemd configuration/state directory behavior: https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html .
Baseline findings from
450c297354f4d2f07c84ace930949a6bacdbb900:SqliteStorage::admit_stream_dataacquires a connection, startsBEGIN IMMEDIATE, validates the authorization snapshot, enqueues, and commits for each DATA call. The stream connection invokes it once for each inbound DATA frame.stream/relay.rs::forward_to_clientslices bulk DATA payloads asBytes, then re-encodes each outgoing frame. This copies payload bytes once during encode; tests can enforce the current one-copy ceiling through the relay and writer.tunnel.rs::run_tunneldeserializes each inventory row intoInventoryItemPayload.container: Value, then deserializes that value intoContainer, retaining the intermediate allocation round-trip.The login-start idempotency scope uses a separate admission flag from its replay lifetime: setting
owner_admissionalso selects the per-scope owner quota, while unknown usernames remain in the anonymous capability pool. I split the short 5-minute lifetime from quota selection and extended the unknown-user test to assert it stays in the anonymous pool.Independent read-only defensive correctness review for #48
Branch: ops-install-restore. Base and HEAD:
450c297354. Reviewed the working diff and untracked operational source/templates/tests, using the existing code-review graph for orientation and the repository operating/design/storage contracts. No author conclusions or issue comments read. No source changes, commits, expensive builds, host service installation, network changes, or process signals.P1 — Keep the operation lease until native SQLite close finishes.
crates/server/src/storage/mod.rs:4107–4112retains the native-worker lease in the rollback-hook closure. SQLx 0.8.6'sConnectionState::dropremoves that hook and destroys its closure before itsConnectionHandlefield runssqlite3_close. When the last storage/pool is dropped without a completed drain, idle connections shut down asynchronously; once pool/storage captures disappear, the hook can own the final lease reference. Its destructor releases the shared lease while the last native database handle still exists.Concrete failing interleaving: drop the final storage/pool; suspend its last worker after rollback-hook removal and before native close; a second process can now acquire restore's exclusive lease and start reading/moving the old file set; resume the old worker, whose SQLite close can checkpoint and remove WAL/SHM concurrently. This violates restore exclusion and can invalidate preserved main/sidecar bytes or disrupt file-set publication. The existing checked-out-connection timeout test does not cover this destructor boundary. Keep an independent lease owner through acknowledged native shutdown, and add a deterministic teardown-boundary refusal test.
This is a static interleaving finding, not a claim of an executed runtime reproduction. Official primary-source evidence: SQLx ConnectionState destructor, hook closure destruction, subsequent native close, pool teardown and field ownership, and SQLite final-connection checkpoint behavior.
Disposition: one blocking correctness finding. No additional blocking findings established in schema-prefix validation, publication/rollback order and pending-marker durability, descriptor link/owner checks, documented CLI/enrollment commands, service templates, or prior-version lineage/state assertions. The backup CLI open path is unchanged.
Lightweight independent checks only (no merge gate requested/run):
Pinned prior source:
e339ad192d, an actual earlier ancestor with two parents. The smoke archives/builds that source and its web assets, starts the prior binary on scratch state, and checks retained host data, migration-history prefix and integrity through the candidate before actual binary backup/restore.Concurrent edits: restore.rs changed validation from a pool to an explicitly closed single connection; restore/tests.rs removed diagnostic prints. Both changes were independently re-reviewed. No finding was invalidated by these edits; the outstanding finding is in unchanged storage/mod.rs. Untracked pycache is generated material, excluded from source review and unsuitable for commit.
Reviewed content SHA-256 (working files, not a committed branch head):
Local incremental evidence: /tmp/infraternal-ops-install-restore-review/findings.md; frozen reviewed content: snapshot/; initial and final SHA-256 manifests: initial-hashes.json and hashes.json.
Restore implementation and focused verification update:
generate, notnew), corrected after checking repository usage and https://docs.rs/ulid/3.0.0/ulid/struct.Ulid.html . A preliminary direct Cargo attempt lacked embedded web assets; subsequent runs use the gate runner's web prerequisite.Current
scripts/verify operationsoutput so far (lean settings, nice 15):Actual current/prior-binary stage and final fast remain in progress/unrun respectively.
Durable idempotency rows may outlive a release, so accepting lowercase keys only in the request parser would make pre-upgrade replays miss their old rows. Migration 22 uppercases retained keys; case-collision groups are reduced to one
unknownrow so a retry cannot repeat an ambiguous mutation. The plan pre-lock authorization mismatch now recordsready → failedandskippedtarget outcome atomically, matching the proto state model and contract.G1 now counts successful
BEGIN IMMEDIATEadmissions reached throughrun_connection: the baseline assertion is one transaction per DATA frame, and the ignored follow-up records the ≤N/8 target. The existings0.rsbenchmark is a Criterion smoke with no pass/fail budget evaluator; a wall-clock threshold there would be nondeterministic on this shared VM, so the deterministic transaction-count guard remains the gate.The perf harness idle window now combines 20 run-labeled Docker healthchecks that alternate state every five seconds with one idle stream connection;
/proc/<pid>/statuscontext-switch deltas produceidle_wakeups_per_s. The 20 exact IDs are removed after the measurement, and the run's signal cleanup also removes its labeled fixtures. The Chromium stream workload now measures a 64 MiB non-TTY stdin upload intocat >/dev/nullagainst the existing 50 MB/s terminal-ingest budget. Nightly now invokes the existinge2e/remote/suite; merge remains unchanged.The legacy case-collision migration now retains the maximum expiry across the whole normalized-key group before deleting duplicates; otherwise an expired canonical row could erase a still-live lowercase replay fence. The migration test seeds an expired uppercase survivor plus live mixed/lowercase variants and asserts one
unknowncanonical row retains the latest expiry. Runtime parsing also rejects ULID overflow spellings whose serialized ID differs from the uppercase input, keeping SQL migration and request normalization equivalent.Starting contract-drift-rest on branch
contract-drift-rest, base head450c297354f4d2f07c84ace930949a6bacdbb900(origin/dev). Scope: D9, D11–D14 (configured access lifetime only), D16, and D18 from the contract drift review. I’ll regenerate OpenAPI and the TypeScript client, add targeted tests, and runINFRATERAL_VERIFY_RUN_ID=contract-drift-rest scripts/verify fastas requested.Starting issue #48 work on branch
contract-drift-rest, based at450c297354f4d2f07c84ace930949a6bacdbb900(currentorigin/devis0d8e41a108cf1169e08b9b1473df99e55da276d1). I’m reviewing the interrupted worktree edits first, then I’ll finish only D9, D11–D14 (configured access lifetime only), D16 and D18 items in scope. Disk check: 23 GB available on/mnt/hdd.Started work on branch
perf-guards-nightlyfrom base450c297354f4d2f07c84ace930949a6bacdbb900. This resumed run is reviewing and completing the existing uncommitted mechanical performance guards for G1, G2, G6, metrics scenarios, and nightly remote parity. Disk check: 23 GB available on /mnt/hdd.Review finding: the current
s0Criterion bench has no absolute threshold or output consumer that could fail a gate on its timing;scripts/verify benchruns it as a reporting smoke. I kept its timing informational and enforce G1 with deterministic transaction-count assertions plus the existing end-to-end stdin throughput gate. The inherited stdin benchmark branch was also wired to the logs flag instead of the upload flag; it is now separated so the quiet no-logs run measures the 64 MiB upload.Final verification finding (#48):
The corrected operations tier passed all seven SQLite tests and the actual current/prior-binary drill (including committed crash-WAL preservation and restart). Final template syntax verification also passed after retaining optional engine startup ordering.
The fast tier passed Clippy, then failed during workspace test compilation on host filesystem pressure. No other job's files were removed and no other process was signalled. Current evidence after the failure: this worktree's target is 2.6 GiB; the filesystem reports 20 GiB available. Retrying fast once with the exact requested run id and lean settings now that space is available.
Exact failed fast output:
Implementation evidence for issue #48:
utoipatest now checks that all four requestedIdempotency-Keyheaders are required, that refresh lists 401, the host-key probe lists 429, and authenticated keyed operations share 400/401/409/422 responses. The generatedschema.tsnow carries the expanded response set.unknownrow so an ambiguous request is never replayed. The unusedlegacy_code()mapper was removed, and §7.2 now describes the unwired adapter and substrate-denial classification.Verification blocked by the requested disk guard after commit and merge. Final worktree is clean at
6eb9f7b783340302918f2bd80ab94e291e377679(branchcontract-drift-rest, ahead 5 oforigin/dev).Pre-gate disk output:
I did not start another full build because free space was below 6 GB. The most recent
scripts/verify fastrun before the final test fixes ended withFAIL fast: cargo clippy (43.327s)due to test compile errors; those fixes are committed, but fast has not been rerun after them. Earlier fast stages passed, but there is no green fast result to record.Finished this run on branch
perf-guards-nightly, based on450c297354f4d2f07c84ace930949a6bacdbb900and merged withorigin/devat0d8e41a108cf1169e08b9b1473df99e55da276d1.HEAD:
8fb1f638aa874c00892e058c905f3f4aed331a0b.The latest
scripts/verify fastrun reached this exact gate output:I interrupted the run during
cargo test --workspaceafter the required disk check fell below the 6 GB floor:The complete fast gate did not finish. The worktree is clean. No push was made; this issue remains open.
Completed ops-install-restore. Branch: ops-install-restore. Base:
450c297354. Final head: b396348188d94179e96fc4cb8564c90fda5655c8. Five small logical commits; clean worktree; no push; issue remains open.Behavior/files:
crates/infraternal/src/main.rs: restore command with backup-consistent database selection. Backup's open path is unchanged for storage-postmerge-fixes.crates/server/src/storage/mod.rs,storage/restore.rs,storage/restore/tests.rs: shared server/pool/native-handle lease and exclusive whole-operation restore lease; isolated read-only immutable validation of integrity, successful known migration prefix/checksums and literal-prefix schema comparison; durable staging, preservation of previous main/WAL/SHM/journal, atomic same-filesystem publication, pending-marker startup fence and pre-publication rollback. Seven real SQLite tests include corrupt/incompatible/altered/lookalike schema inputs, older-prefix startup upgrade, cloned/held/timed-out storage refusal and rename-failure preservation.deploy/systemd/*,docs/DEPLOY.md: generic Debian/Ubuntu server and root-run remote infraternal agent units/private configuration; actual enrollment arguments, rootful and explicit rootless Podman setup, upgrade/uninstall, private whole-state backup, restore and interrupted recovery; owner-matched copies of root-owned archived snapshots. No services installed on the VM.scripts/verify,scripts/tests/operations.sh,operations.py,docs/PLAN.md,docs/DESIGN.md,docs/contracts.md,.claude/skills/ops-install-restore/SKILL.md: focused operations tiers and nightly real-binary drill. Repo skill validated (Skill is valid!).Latest scoped operations command: PATH includes /home/kayg/.nix-profile/bin; CARGO_INCREMENTAL=0 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_BUILD_JOBS=2 INFRATERNAL_VERIFY_RUN_ID=ops-install-restore-operations-literal-prefix nice -n 15 scripts/verify operations.
Exact output:
Final required fast command: PATH includes /home/kayg/.nix-profile/bin; CARGO_INCREMENTAL=0 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_BUILD_JOBS=2 INFRATERNAL_VERIFY_RUN_ID=ops-install-restore nice -n 15 scripts/verify fast. Exit 0.
Exact output:
Additional scoped checks: systemd-analyze verify on temporary units referencing the actual binary returned exit 0, without installation; bash syntax and Python syntax checks passed; git diff --check passed; all five commit messages end in the required trailers.
Limitations/integration notes:
e339ad192dand its own web assets, not fabricated. There are no release tags; this lineage has unchanged migrations 1–21, so the real drill proves binary/state continuity. Older-prefix migration is covered in real SQLite tests.Follow-up fix writer started. Branch ops-install-restore; base origin/dev
450c297354; starting HEAD b396348188d94179e96fc4cb8564c90fda5655c8; git status clean. Preserving all prior changes. Independently checking reviewer P1 against SQLx 0.8.6 teardown ordering, then deterministic storage regression and focused operations plus fast gates. No merge/nightly or push.Confirmed reviewer P1 against current HEAD b396348188d94179e96fc4cb8564c90fda5655c8 and installed pinned SQLx 0.8.6 sources: ConnectionState::drop removes rollback-hook captures before ConnectionHandle::drop invokes sqlite3_close. Current storage/mod.rs retains the per-connection lease only in that capture. Pool drain cannot account for detached connections.
Primary sources: https://github.com/launchbadge/sqlx/blob/v0.8.6/sqlx-sqlite/src/connection/mod.rs#L557-L567 and https://github.com/launchbadge/sqlx/blob/v0.8.6/sqlx-sqlite/src/connection/handle.rs#L80-L90 . Proposed native owner uses SQLx LockedSqliteHandle::create_collation, which registers a SQLite-owned xDestroy callback (safe Rust API, registration errors reclaimed by SQLx): https://github.com/launchbadge/sqlx/blob/v0.8.6/sqlx-sqlite/src/connection/collation.rs#L76-L108 . SQLite 3.46.0 bundled by pinned libsqlite3-sys 0.30.1 closes all Btrees/pagers and reports connection locks released before collation xDel: https://github.com/sqlite/sqlite/blob/version-3.46.0/src/main.c#L1299-L1369 ; documented native callback lifetime: https://www.sqlite.org/c3ref/create_collation.html . This is a native acknowledgment that database-file shutdown is complete, even if SQLx shutdown awaiting is cancelled, bounded, or skipped. The private collation is never queried, replaced or removed. No new polling, background work, request-path allocations, unsafe Rust, or dependencies are required.
Regression now in development: drain pool while a detached connection remains; strip authorization hooks; pause the real worker in rollback-hook destruction before native close; prove exclusive admission/public restore refuse, then prove restore succeeds immediately after acknowledged worker shutdown.
Deterministic reviewer repro confirmed before production fix. The first operations-storage run exposed a test-harness issue (single-thread runtime blocked in native capture destruction); the corrected multi-thread tests use a bounded destructor rendezvous. Focused lean diagnostic of the corrected tests against pre-fix production code exited 101, with all three failing on early exclusive lease admission:
Applied the SQLite-owned private collation capture using SQLx safe API, releasing only at native acknowledgment of finished Btree/pager/WAL/lock teardown. Added detached acknowledged, dropped and cancelled-close boundary regressions; preserved checked-out timed-out-close coverage. Running scripts/verify operations now; fast follows.
Resuming the perf review follow-up for wins #2 (G3, Docker/Podman event filtering and refresh coalescing) and #7 (G5, log follow allocations and in-place frame batching).
Branch:
perf-events-logsBase head:
450c297354f4d2f07c84ace930949a6bacdbb900Starting HEAD:
6d30a151ff3250b92b2a7af84826c56509a270b1The worktree is clean. I’m reviewing the existing implementation and guard tests, then will finish the remaining checks and run
INFRATERAL_VERIFY_RUN_ID=rs-perf-events scripts/verify fast.Resuming rs-session-authority as the only writer in
/mnt/hdd/infraternal/wt-lp/session-authority, branchsession-authority, base/resume headcd7ca3faf07621804f642f8e0f1793f72024e637. Worktree is clean. The existing commit contains A3/A5/A8/A10/A11 implementation and regressions; I am recovering the prior workspace-test failure, rerunning it in isolation before changing code, and finishing with the requestedINFRATERNAL_VERIFY_RUN_ID=rs-session-authority scripts/verify fast. Lean Cargo environment, two build workers, nice 15, root-disk target cache. No merge gate or push.Resuming the binary-size work on branch
perf-binsize.Resume head:
ffbee9c79c8b78960b91c57b3735bf167d0d63c4.Branch base against current
origin/dev(git merge-base):bedafc1936e5f5c43aef15b46399fb34b171aba2.The existing branch contains the recorded profile, crypto-backend and embedded-asset optimizations. The current worktree also has an uncommitted edit to
docs/DEVELOPING.md, which I am preserving. I’ll measure the current stripped release binary, make only a justified remaining change if it still exceeds the 25,000,000-byte budget, and finish withINFRATERAL_VERIFY_RUN_ID=rs-binsize scripts/verify fast.Resuming rs-mutation-safety directly in worktree /mnt/hdd/infraternal/wt-lp/mutation-safety, branch mutation-safety. Base head:
450c297; current head: 1f629548fa69d0b51443ea6f3b01656ad74b59a5. Worktree is clean. Existing implementation for D1/D2/D7/D10/D15 is committed. Checking prior test evidence, then rerunning isolated failures and the requested INFRATERNAL_VERIFY_RUN_ID=rs-mutation-safety scripts/verify fast under the lean build environment. No push or merge gate. A6 only if it fits the current provider seam cheaply.First requested fast run exited 101 in existing real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce: remote inventory was not published at agent_tunnel.rs:580 (100 attempts separated by 100 ms). The test and infraternal-agent/tunnel implementation have no diff against origin/dev
450c297354. No tunnel, engine, network, deadline or fixture changes made. VM reported load5 33.90 against 8 online CPUs. Focused lean diagnostic passed unchanged in 2.15 seconds, supporting a transient inventory-publication failure, not establishing its cause. Re-running full fast to obtain final gate evidence.Initial failure excerpt verbatim:
Focused diagnostic output verbatim:
Follow-up fix for kayg/infraternal#48
Branch: ops-install-restore
Base origin/dev:
450c297354Starting HEAD: b396348188d94179e96fc4cb8564c90fda5655c8 (clean)
Final HEAD: ba13a6d531d48da6d5f593358848066dc93b94c1 (clean)
Commit: fix(storage): retain restore lease through native SQLite teardown (#48)
Confirmed and fixed the review P1. SQLx 0.8.6 destroys rollback-hook captures before ConnectionHandle invokes native SQLite close. The per-connection shared operation lease now lives in an unused private SQLite collation capture via SQLx's safe API. SQLite 3.46.0 destroys that capture after every Btree/pager closes, WAL cleanup completes and database locks are released. This native file-shutdown acknowledgment remains effective if pool drain excludes a detached connection, or a close is dropped, cancellation-requested or timed out. The lease does not rely on pool size or an async future surviving. Existing bounded pool close is unchanged. No new production background polling/tasks, unsafe Rust, dependencies or request/frame allocations. Pool clones retain their existing closure ownership without extra storage-clone work.
Primary source evidence:
Regressions pause native teardown after authorization hooks are removed and before sqlite3_close. They independently require exclusive lease admission and public restore to refuse, check unchanged main/WAL bytes, then require successful restore after native shutdown. They cover detached acknowledged close, raw connection drop, cancellation requests and the existing checked-out close timeout. Tests use a bounded destructor rendezvous and a multi-thread runtime because SQLx can destroy native state on the thread polling close. Cooperative cancellation may complete the synchronous current poll; the final test accepts successful acknowledgment or a cancelled JoinError while keeping all restore/file assertions strict. Tokio cancellation behavior.
Corrected regressions failed before the production fix at early exclusive admission in all 3 cases (0 passed / 3 failed). Final focused SQLite gate passed all 10 tests. All 3 boundary cases then passed 20 consecutive repetitions (60 executions). Initial harness issues and their corrections are preserved in issue comments and local diagnostic logs; no refusal assertion was relaxed.
The previous writer's install/enrollment/upgrade/uninstall docs and systemd templates, restore validation/publication/recovery logic, backup open path and actual historical-binary verification were preserved. The operations smoke built and ran pinned historical source
e339ad192dwith scratch state. The production fix passed that full smoke; subsequent edits only corrected test-harness completion handling, and the final storage suite was rerun.Validation and remaining blocker:
both(5000 ms timeout) andlocks(false instead of true). Web/other frontend packages and bun.lock have no diff from the pinned base. auth.test.ts blob is8e78e65dc1and auth.svelte.ts blob is55902f3375, identical in base and working files. These tests mock fetch and browser globals; they do not call the Rust server. The focused unchanged auth diagnostic passed all 14 cases. No UI, timeout or network changes made; central verification must resolve/re-run this unrelated gate failure.Actual output follows verbatim.
Operations gate
Final storage gate
Boundary repetitions (verified log summary)
Final required fast gate
Focused frontend diagnostic
Focused remote-enrollment diagnostic
Resuming rs-stream-conformance as sole writer on branch
stream-conformance. Base450c297; starting HEADff65f249a1a6c72e4c5dc995f230dfdf23e53826, clean worktree. Checking prior conformance work and remaining documentation/tests; lean root-disk Cargo cache, nice 15, two build jobs. Final requested gate isINFRATERNAL_VERIFY_RUN_ID=rs-stream-conformance scripts/verify fast; no merge gate or push.Finding during release measurement: the preset
CARGO_TARGET_DIR=/home/kayg/.cache/infraternal-targets/rs-binsizeresolves to/mnt/hdd/infraternal/wt-lp/perf-binsize/target, so it is not on the root disk as required. The release build ran 32 minutes and failed with:I have stopped using that path.
/has 18 GB free and the existing target tree is 5.3 GB; I’ll reuse it by copying artifacts into a new non-symlinked cache under/home/kayg/.cache, then point Cargo there explicitly for the release build and required fast gate.Verification is blocked by the build environment at unchanged head
cd7ca3faf07621804f642f8e0f1793f72024e637.The preset root-cache path was redirected to the worktree's HDD target during this job. The fast gate then failed at Clippy with SQLx/
eitherartifact hash mismatch E0460. A separate root-disk cache copy was started to preserve the existing target and retry without HDD builds, but concurrent disk use reduced root free space from 19 GiB to 6.5 GiB and then 3.6 GiB. I stopped only my copy to avoid filling the root filesystem. The copied partial cache is/home/kayg/.cache/infraternal-targets/rs-session-authority-resume. No other job was signalled and no source was changed.The fresh full workspace test run before the gate passed (exit 0); the earlier cargo-test failure did not reproduce. Product-source fixes are already committed at the head above. Fast remains incomplete, not green. Requested orchestration help to free root space or provide a permitted cache location; root-disk cache constraint prevents falling back to HDD builds.
rs-stream-conformance finished on branch
stream-conformance, HEAD6e687f50d6cc70a4c8e7543bf9ff0b107b56c4bb, base450c297. No push or merge; issues remain open.Preserved prior commit
ff65f24implementing D3/D4/D5/D6/D17: stream/tunnel oversize close 1009; channel-local invalid RESIZE; bounded tunnel frames, close codes, per-channel OPEN_ERR refusals, timestamped PING and shared control admission; generated TypeScript OPEN_ERR validation for all registered errors; cumulative Rust control budget shared by CLI and SSH clients.Resume commits:
9d4e23b: discard invalid RESIZE after channel retirement. A real TCP WebSocket regression verifies OPEN_ERR followed by late RESIZE produces PONG without a second CLOSE.6e687f5: document the Rust sliding one-second 256 KiB admission limit and tunnel inventory exception; add steady-state CREDIT and cumulative-refusal benchmarks to the existing proto harness; reject excessive payload counts without arithmetic overflow, with a regression test. Deque capacity reuse and bounded growth are explained in code.All checks below completed with exit 0. Rust commands used CARGO_INCREMENTAL=0, CARGO_PROFILE_DEV_DEBUG=0, CARGO_PROFILE_TEST_DEBUG=0, CARGO_BUILD_JOBS=2, the preset root-disk CARGO_TARGET_DIR and nice -n 15. No timing failures required isolation or a retry. No
verify mergewas run. The final fast gate covers the committed source; its Rust tests include real TCP and mTLS conformance tests. Docker/Podman/sshd integration tiers and perf-test-VM A/B are not certified by this fast run. Criterion ran in test mode only; no performance measurements are claimed.Requested gate:
INFRATERNAL_VERIFY_RUN_ID=rs-stream-conformance scripts/verify fast(exit 0), output verbatim:Workspace Rust test results and selected conformance test lines, verbatim:
Focused stream package:
bun --cwd=packages/stream run checkandbun --cwd=packages/stream run test, both nice -n 15, exit 0; output verbatim:Benchmark smoke:
cargo test -p infraternal-proto --bench stream -- control_admission --test, exit 0; output verbatim:Code index rebuilt, refreshed after edits, and queried for ControlBucket and the stream impact radius. No independent review is claimed by this author job.
API references checked: Criterion batched reference timing, Criterion test mode, VecDeque capacity.
Final
/usr/bin/git status --shortoutput was empty (exit 0); temporary target symlink removed. Build artifacts and this report remain only in the root-disk cache.rs-tunnel-drop completed at HEAD
2abb7e93d66bcde5015dbc9a9d45223f754c609bontunnel-drop-close; merged batch-1003a including22ac67dand retained original stop deadlines. Drop/replacement now use the audited host_offline path, with local-channel isolation and generation-preservation regressions. Required finalINFRATERNAL_VERIFY_RUN_ID=rs-tunnel-drop scripts/verify fastexited 0; no merge gate or push. Full head, findings and verbatim gate/focused test output are recorded in #49. Two prior focused inherited rate-limit flood failures are recorded there; final workspace tests passed.rs-replay-admission resume finished on
replay-book-admission.Head:
dbb19e7b51e9c5acf6c2312cf25071f1eb9da70c. Resume base:9dd7fa5d1f565fc7f08e77b6869432607a5a0042. New logical commit:fix(authz): isolate plan grants from read replay admission. Worktree is clean; no push and no issues closed. Related: #49 / #48 / #46.Finished the remaining scope:
Regression proof: before the split, the test failed with
a full read book must leave plan grants available: ResourceExhausted(exit 101). Final workspace tests include the strengthened fixed-time and nonce-reset checks. Relevant final test output, quoted verbatim:Required gate command:
nice -n 15 env CARGO_INCREMENTAL=0 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_BUILD_JOBS=2 INFRATERNAL_VERIFY_RUN_ID=rs-replay-admission scripts/verify fast. Exit 0. Exact complete gate output:The low cookie advisory is the existing report-only Bun finding. No dependency changes. No
verify merge, other full gate, or perf-VM A/B was run; independent review and merge/performance gates remain with the orchestrator.Tooling/host notes: the graph was absent; rebuild and queries were attempted before raw searches, but queries returned no indexed nodes while the database was locked. The owned prolonged rebuild was stopped and direct source/diff review supplied the fallback. The preset root-disk cache was cold. The former worktree target was moved off /mnt/hdd; its obsolete cache was later removed when root free space fell to 8 GB. A temporary root-target symlink supported the gate summary and was removed after the gate. Active build artifacts remain only in the preset root-disk target, outside git.
Job rs-mutation-safety finished on branch mutation-safety at d0952a66460672cc51b94bd6da47867a7c296d28; original base
450c297. Worktree clean. No push or merge gate.Retained implementation commit 1f62954; added d0952a6 (real SQLite busy exhaustion/resumption regression). Both commits contain the requested attribution/session trailers.
Finished and verified:
A6 deferred: this branch's resources still dispatch through the legacy engine boundary; no provider binding is available to hash into the plan. Integrate binding freshness with the provider migration rather than inventing a binding here.
Scoped verification (nice -n 15; one test thread):
Final required command: INFRATERNAL_VERIFY_RUN_ID=rs-mutation-safety scripts/verify fast. Environment: CARGO_INCREMENTAL=0, CARGO_PROFILE_DEV_DEBUG=0, CARGO_PROFILE_TEST_DEBUG=0, CARGO_BUILD_JOBS=2, RUST_TEST_THREADS=1; nice -n 15; preset root-disk CARGO_TARGET_DIR. Exit 0. Verbatim gate output:
The first fast attempt exited 101 during linking because generated CLI object files were missing. Disk had 19G free; cause was not established. The isolated CLI retry and final fast rerun both passed without source changes. The Bun cookie advisory is report-only; dependencies were not changed in this job. No performance certification or merge/integration/nightly gate is claimed.
Release measurement from the corrected root-disk cache:
stat -c '%n %s bytes'reports 31,116,264 bytes forrelease/infraternal;filereportsnot stripped. This is 6,116,264 bytes over the 25,000,000-byte budget. The earlier 32,931,832-byte baseline was recorded as stripped, so I’m checking why the profile’s strip setting is not reflected before comparing or changing size levers.rs-remote-ssh resume finished at
576e91e050on branch remote-ssh. Full findings and verbatim gate/test evidence are posted in kayg/infraternal#49. Added attributed-identity regressions and corrected architecture docs; fixed the completion-audit test fixture to bind its real embedded infraternal agent owner. Real remote SSH test: 1 passed; embedded real-sshd product tests: 4 passed. Final INFRATERNAL_VERIFY_RUN_ID=rs-remote-ssh scripts/verify fast: exit 0, all 18 stages, serial workspace tests (RUST_TEST_THREADS=1), lean two-job nice-15 root-disk build. Clean worktree; no push or merge gate.Completed resume verification on branch
session-authority, headcd7ca3faf07621804f642f8e0f1793f72024e637. Worktree is clean. No push, merge gate, or issue close. No new source commit was necessary: the existing commit already implements A3/A5/A8/A10/A11 and its tests; the previous cargo-test failure did not reproduce in a full workspace run or the final fast gate. The old failing diagnostic was not available in the worktree, so its original test name could not be recovered.Finished:
containers:readand redact exec command text/labels (A5).OPEN_OK.session_idcreation/reattachment binding (A11).cargo test --workspacediagnostic run, exit 0. The real-Docker creator-revocation regression also passed in the existing current-code test executable./home/kayg/.cache/infraternal-targets/rs-session-authority-resumeand rebuilt affected SQLx/either/WebAuthn/OpenSSL packages. The prior environment blockage is resolved.Final gate command (exit 0):
Final gate output, verbatim:
The Bun low-severity cookie finding above is report-only under the existing gate policy. Fast passing is not a merge-gate or perf-test-VM A/B claim.
Additional diagnostic evidence, verbatim:
Cache-repair references: Rust E0460, Rust E0463, package-scoped Cargo cleanup.
The final-package strip fix is verified:
After adding
strip = "symbols"under[profile.release.package.infraternal],filereportsstripped, andreadelffinds no.symtab,.strtabor.debug_*sections. Exact size: 26,176,448 bytes, still 1,176,448 bytes over the 25,000,000-byte budget. A copy stripped manually before this change measured 26,176,440 bytes, confirming the manifest setting now gives the same result.The first required fast gate passed clippy but stopped during Cargo test-profile compilation, before tests ran.
Verbatim output:
FAIL fast: cargo test (187.929s)error: could not parse/generate dep info at: /home/kayg/.cache/infraternal-targets/rs-perf-events/debug/deps/aws_lc_rs-73ba973b55b35c41.dfailed to write .../.fingerprint/aws-lc-rs-73ba973b55b35c41/dep-lib-aws_lc_rsNo such file or directory (os error 2)Disk space is available; the affected fingerprint paths are absent after the failed build. I’m rerunning the required gate to separate this build-cache failure from test results.
Measured feature-trim trial: workspace Axum now requests only
http1,json,matched-path,query,tokio,tracingandws;cargo tree -e features -i axumconfirmsform,original-uriandtower-logare gone. The release build succeeds, but the stripped executable is 26,177,472 bytes, 1,024 bytes larger than the preceding 26,176,448-byte build. LTO had already removed this unused feature code from the final binary, so I am reverting this trial to avoid keeping a no-win feature behavior change.Starting cdr-fix on branch
contract-drift-restat6eb9f7b783340302918f2bd80ab94e291e377679; merge targetorigin/devisd987a32dfd11cda99b2ddc516ad74ba13fcc5003.Finished
rs-perf-eventson branchperf-events-logs.Head:
6d30a151ff3250b92b2a7af84826c56509a270b1(existing implementation commit; worktree clean). The requested event filtering/debounce/change-only inventory publication and log-frame allocation/batching changes, with G3/G5 guards, were already present. No additional code changes were needed.The first required fast run stopped before tests because Cargo could not write into a missing fingerprint directory; that failure was recorded above. Retried with the requested run ID and lean build settings. The required gate exited 0:
Finding:
openapi_documents_bearer_idempotency_and_refresh_transportswas 112 lines against clippy's 100-line limit. Split its existing assertions by concern into idempotency/bearer security, refresh/rate-limit transport, and shared error response tests; the checks remain intact.Merge finding:
resources.rshad overlapping import blocks from both branches. The first OpenAPI build reported duplicate imports for the shared REST auth/storage types; those duplicates are removed, while the remoteHostRouteand provider types remain. The detail/action route keeps the local keyed-route error response type (which documents 400/401/409/422) and the remote 429/501/503 provider outcomes, plus its explicit 403/404 responses.Finding: the merged RPC test expected
ProviderError::legacy_code(), butProviderErrorexposes the registered provider failure fields. Updated the assertion to verify NotFound, HTTP category, native status 404 andretryable == falsedirectly. Workspace clippy then reported:cargo clippy: No issues found.Starting fence-batch in /mnt/hdd/infraternal/wt-lp/fence-batch. Branch: fence-batch. Base/head:
d987a32dfd(origin/dev). Scope: bounded ready-only inbound DATA batches with the durable SQLite authorization fence; ordering, credits and fail-closed behavior preserved. Requested gates: fmt, workspace clippy -D warnings, targeted stream tests, verify fast.Finding: both production inbound forwarding channels had capacity 1 (EmbeddedAgent::open_frame_at and TunnelHub::reserve), so reserve_many could not admit a batch without widening those inbound queues. Changed only those inbound frame queues to a shared 64-slot constant; existing per-frame credit accounting and byte-window bounds remain, and outbound one-slot queues remain. Consecutive ready DATA is batched only on the same channel; the first nonmatching message is retained in connection order. Capacity waits occur before the SQLite write lock. The whole batch is synchronously enqueued while the lock is held, so a revocation committed first rejects every frame. API references: https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Sender.html#method.reserve_many (reserve_many rejects counts above channel capacity, reserves all requested slots); https://docs.rs/futures-util/latest/futures_util/future/trait.FutureExt.html#method.now_or_never (single immediate poll). No G1 stream_data_fence_cost existed at base; adding the <= N/8 guard with a 16-transaction target for a queued 1000-frame burst.
Finding:
scripts/verify fastfailed one storage test:mint_token_uses_the_configured_access_lifetime(130 passed, 1 failed) because its caller token had no grants andmint_authenticated_tokencorrectly requires effectiveTokensManage. Updated that test fixture to grantTokensManage; no authorization behavior changed.Perf diagnostic target-path conflict:
scripts/verify perfhardcodes--target-dir "$repo_root/target"for its allocation harness, bypassingCARGO_TARGET_DIRand writing build artifacts under/mnt/hdd. I stopped the build with SIGINT during theperf: build allocation measurement harnessstep and removed files written under this worktree's target after the run started. It produced no RSS metrics. The perf README classifies dev-VM results as diagnostics only; a root-disk target override is needed to run this diagnostic within the job's cache rule.First targeted stream run measured against file-backed SQLite:
The original singleton shape is exercised for 1000 successful admissions against the same database before the ready-only collector/forwarder path. The guard enforces <=125 transactions and exact ceil(1000/64)=16, checking every forwarded frame in order. Revocation-mid-burst and revocation-during-capacity-wait tests passed. One new interactive fixture initially failed with Database(PoolTimedOut) because start_paused allowed SQLx pool timers to expire while its real worker I/O was pending; corrected the fixture to pause time only around the synchronous collector and resume before SQLite operations. Rerunning targeted tests; verify fast has passed prerequisites, web build and fmt and is running workspace clippy.
Finding:
device_poll_admission_precedes_failed_response_actor_authenticationsent its 32 requests serially against an 8-token source bucket refilling at 4/s. The run admitted 9 and limited 23 because one token refilled during response handling, failing the unchangedlimited >= 24assertion. Changed the test to submit a concurrent same-source burst; production admission behavior and the assertion are unchanged.Starting pg-fix on branch
perf-guards-nightly, base head8fb1f63. Investigating the failingstream_data_fence_cost_matches_current_per_frame_baselineassertion and will make G1 deterministic as an upper-bound check, then mergeorigin/dev, resolve any conflicts, run fmt, clippy with warnings denied, targeted tests, andINFRATERAL_VERIFY_RUN_ID=pg-fix scripts/verify fast.Starting sc-merge #48 on branch
stream-conformance.Worktree base head:
6e687f50d6cc70a4c8e7543bf9ff0b107b56c4bb. The localorigin/devref isd987a32dfd11cda99b2ddc516ad74ba13fcc5003(batch-1003a). I will mergeorigin/dev, retain both stream benchmark behaviors, resolve any other conflicts preserving both sides, then run fmt, workspace clippy with-D warnings, proto/server stream tests, andINFRATERAL_VERIFY_RUN_ID=sc-merge scripts/verify fastwith the requested lean environment. No merge gate will be run.Merge finding: the sole content conflict is
crates/proto/benches/stream.rs. Thestream-conformanceside addsControlBucketsteady-state and cumulative-payload-refusal benchmarks;origin/devadds provider-reply split/assembly benchmarks for 8 KiB and 1 MiB responses. These exercise distinct paths, so resolution will retain both sets and register both functions in Criterion.Finding:
setup_banner_uses_stdout_and_database_uses_the_data_directoryused a fixed 500 ms sleep before checking child stdout and failed while the VM was under load. Replaced the sleep with a bounded wait for the setup-banner line on stdout; the assertion checks only the prefix and does not retain or print the credential.Release-size trials from the safe root-disk cache, with explicit symbol stripping: third-party default
opt-level="z"measured 26,048,160 bytes versus 26,176,448 ats(128,288 bytes smaller, still 1,048,160 over the 25,000,000-byte cap). Setting the root CLI package tozas well measured 25,368,480 bytes (807,968 bytes below thesbaseline, still 368,480 over). Hot server, infraternal agent, proto, SSH and their explicit hot dependency overrides remain at 3. Both builds succeeded; root-CLI-only follow-up compiled 1 crate in 4m23s. These are size measurements only; no runtime A/B is claimed yet.Verification finding: the first
cargo clippy --workspace --all-targets -- -D warningsexited 101 before completing. Cargo could not write.../sc-merge/debug/build/crossbeam-deque-.../build_script_build-....d(No such file or directory); theaws-lc-sysbuild then reported missing output paths under the samedebug/buildtree. At inspection,debug/buildwas absent while the target filesystem had 6.6 GiB free and 12% inode use, and other Cargo/Rust compiler processes were active. This points to target-tree lifecycle/concurrent build activity; I’ll retry after those processes drain.Correction to the build finding: the remaining Cargo processes are targeting separate caches (
fence-batch,tdfu-fix, andpg-fix), so they do not establish contention on thesc-mergecache. Disk and inode capacity were also available. The evidenced failure remains a missing path under the configuredsc-merge/debug/buildtree; I’ll retry and record whether it recurs.Finding: the gate panic is in the test's flow-control fixture, before the transaction-count assertion.
count_stream_data_fencessends DATA while a separate task drains forwarded frames, but it sends the next agent CREDIT as soon as local WebSocket sends consume its ownavailablecounter; it does not wait for that DATA window to reach the test infraternal agent. Under queue delay,account_agent_framecan see CREDIT before the server has deducted the outstanding DATA, reject the window as over its cap, andspawn_relaycloses the WebSocket with 1008.next_test_client_framethen reports only “expected a binary frame.” I’ll synchronize credit with observed drain completion and make the transaction check a ceiling (<= DATA_FRAME_COUNT) so batching can reduce it.Final targeted stream run passes after the fixture correction and byte-window cap. The collector ends a batch before a frame that exceeds remaining credit, preserving the valid prefix and keeping the violating frame next for the actor to reject. This also avoids accumulating uncharged DATA beyond the current window. Both count and byte limits are covered.
Command:
nice -n 15 env ... CARGO_BUILD_JOBS=2 cargo test -p infraternal-server --lib stream:: -- --nocapturewith the configured task Cargo cache, incremental disabled, and dev/test debug=0. Selected output verbatim:The configured Cargo cache/profile settings are now explicit in the lean environment; workspace clippy and the remaining fast gate are running.
Finished cdr-fix on branch
contract-drift-rest.Head:
02ceef772f2eb453152bd823b3278d002ec0d14bMerged base:
d987a32dfd11cda99b2ddc516ad74ba13fcc5003cargo fmt --checkpassed.cargo clippy --workspace --all-targets -- -D warningsreportedcargo clippy: No issues found. The startup readiness and device poll burst tests each passed 20/20 targeted repetitions. The finalINFRATERAL_VERIFY_RUN_ID=cdr-fix scripts/verify fastexited 0. Gate output:Commits:
f515857(split clippy test),5e025f2(mergeorigin/devand regenerate artifacts),02ceef7(verification test fixes). Not pushed. The only remaining untracked path is the preexistingtarget/build directory.Fast gate passed fmt and workspace clippy --all-targets -D warnings, then failed in the unchanged concurrent_closes_wait_for_the_same_pty_cleanup regression at stream_regressions.rs:1492:
The assertion is second_done >= first_done, where both instants are captured after concurrent DELETE HTTP responses finish; it compares response-completion order, not an observed cleanup barrier. The fixture opens a persistent exec attachment using OPEN only, then sends DELETE requests: no inbound DATA goes through the changed batching fence. The regression source has no diff from
d987a32. Running an isolated exact rerun and then a full fast retry without changing or weakening this test. Full failing cargo-test log is at /mnt/hdd/infraternal/artifacts/verify/fence-batch/step-9.log; full gate output preserved at /tmp/fence-batch-fast-attempt-1.log.Integrated
origin/devatd987a32after fix commitdefb369; Git'sortmerge completed without conflicts at200f7b2b. Post-merge format is complete. I’m serializing clippy and test gates behind the currently active server stream test on the VM.Finished fence-batch. Branch: fence-batch. Base:
d987a32dfd. Head:f967b3a1ae. Committed locally; no push. Working tree is clean.Consecutive same-channel DATA already ready is bounded by 64 frames, the forwarding capacity and current byte credit. Capacity is reserved before one durable authorization transaction. Every enqueue runs synchronously in order under SQLite's writer lock and the watcher mutex; a previously committed revocation rejects the whole batch. Embedded and remote inbound queues share the 64-slot bound. Byte-credit accounting and outbound queues stay unchanged. Docs and the existing Criterion fence benchmark now describe/exercise singleton and 64-frame admission.
Transaction count measured on file-backed SQLite: 1000 singleton admissions versus 16 admissions for the queued 1000-frame burst. The new G1 guard enforces <=125 and exactly ceil(1000/64)=16. Interactive input, batch boundaries/order, current-credit prefix, committed revocation mid-burst and revocation while reserving capacity are covered. The Criterion variants compile under all-targets clippy; no dev-VM throughput or timing claim is made.
Requested gates passed: fmt, workspace clippy --all-targets -- -D warnings, targeted stream tests, and INFRATERNAL_VERIFY_RUN_ID=fence-batch scripts/verify fast. Runs used nice -n 15, CARGO_BUILD_JOBS=2, explicit configured Cargo target cache and dev/test debug=0 with incremental disabled.
Targeted stream output, verbatim:
The first complete fast attempt failed at the unchanged concurrent_closes_wait_for_the_same_pty_cleanup assertion comparing concurrent HTTP response completion instants. Its fixture sends OPEN then DELETE requests, with no inbound DATA. Exact isolated rerun passed without source/test changes:
The full fast retry then passed without changing or weakening that test. Earlier failing output is preserved in /tmp/fence-batch-fast-attempt-1.log and /mnt/hdd/infraternal/artifacts/verify/fence-batch/step-9.log, and documented above. The corrected interactive fixture pauses time only around the collector to avoid SQLx real-worker pool timeout; this is covered by the final targeted and workspace runs.
Final fast gate output, verbatim:
Fast-gate finding: the pre-existing untracked
targetsymlink points to/home/kayg/.cache/infraternal-gate-target/stream-conformance, but that cache leaf was absent.scripts/verifywritestarget/verify-summary.txtand itsmkdir -pfailed through the dangling symlink after all listed fast checks had passed. I will create the empty cache leaf at the symlink destination, rerunfast, and leave the symlink untracked.Finished sc-merge #48 on branch
stream-conformance.Head SHA:
39953784873435a48620148367ba30a8120912d0(merge commitMerge origin/dev into stream-conformance). The only conflict wascrates/proto/benches/stream.rs; the resolution retains both control-admission benchmarks and both provider-reply split/assembly benchmarks.cargo fmt --allexited 0.cargo clippy --workspace --all-targets -- -D warningsreportedcargo clippy: No issues found. Stream tests passed: proto 37 passed / 34 filtered; server library 41 passed / 88 filtered; server integration 26 passed / 16 ignored. No merge gate was run.INFRATERAL_VERIFY_RUN_ID=sc-merge scripts/verify fastexited 0. Successful run output:The pre-existing
targetsymlink remains untracked and was not committed. Its previously missing cache destination was created so the gate could write its summary. No push was performed.The isolated test reached DATA 16, then stalled at DATA 17:
stream_data_fence_cost_stays_within_current_per_frame_upper_boundfailed withremote test infraternal agent did not drain DATA frame 17 of 1000after 60 seconds.The test-only
TestTunnelWriter::send_from_agenthelper wrote directly to the route sender and skippedaccount_tunnel_frame, unlike production tunnel ingress. Its CREDIT therefore replenished the stream fixture but left the tunnel route's agent window at zero; the route rejected subsequent DATA. The helper now applies inbound tunnel accounting before forwarding the frame.Finished pg-fix #48.
Branch:
perf-guards-nightlyHead:
01fa2b078e50b04b41aa8d9206fa774737306cf8The deterministic DATA fence guard now uses a per-frame upper bound and asserts that the authorization fence ran. The test tunnel helper applies the same inbound window accounting as production; without it, the test CREDIT replenished the stream fixture but left the tunnel route window at zero. The targeted test passed 20 consecutive runs (
1 passed, 132 filtered outeach). Workspace clippy reportedcargo clippy: No issues found.INFRATERNAL_VERIFY_RUN_ID=pg-fix scripts/verify fastexited 0. The first run completed every stage but could not write its summary because the pre-existingtargetsymlink was dangling. I created its intended empty gate-output directory and reran the gate successfully.Fast gate summary, quoted verbatim from
target/verify-summary.txt:Started th-merge on branch tunnel-hostname at base 5c6359ba715aec3e8122872c0bfa8f6c7fb12347. Merging origin/dev at
2265bb7f90; resolving crates/server/src/lib.rs while preserving hostname TLS/enrollment wiring and the incoming server changes.Merge finding: the only content conflict was an overlap between and in . I retained both and closed the endpoint impl before the provider initializer. §6.2 requires the configured hostname to drive the tunnel URL and SNI matching; still starts the provider initializer only when the embedded infraternal agent is enabled. Focused verification is running.
Correction to the previous merge finding comment: the conflict was between TunnelEndpoint::from_config and spawn_embedded_providers in crates/server/src/lib.rs. I retained both and closed the tunnel endpoint impl before the provider initializer. docs/contracts.md section 6.2 requires the configured hostname to drive the tunnel URL and SNI matching. Server::open still starts provider initialization only when the embedded infraternal agent is enabled. Focused verification is running.
th-merge resumed on branch tunnel-hostname at base head 5c6359ba715aec3e8122872c0bfa8f6c7fb12347. The worktree already has an active merge with MERGE_HEAD
2265bb7f90; conflicts are resolved and changes are staged. Current origin/dev is52c38730ff, so I am preserving the active merge target recorded for this job. Disk check: /mnt/hdd has 8.1 GB free; target/ is empty. Continuing from the existing merge state.Finding: the worktree's untracked target symlink points to the missing /mnt/hdd/infraternal/gate-target/tunnel-hostname directory, so build artifacts from the interrupted run are gone. /mnt/hdd currently has 6.9 GB free. I will recreate only that generated-artifact directory, use a lean debug profile with incremental compilation disabled, and stop builds if free space falls below 5 GB.
Resolution evidence:
git ls-files -uis empty. The staged tree retains--tunnel-hostname,ServerConfig::tunnel_hostname, enrollment URL wiring, and the dedicated tunnel SAN/SNI setup (git grep --cached); the stagedServer::openalso retains dev's embedded provider initialization and HostRouter wiring (git diff HEAD -- crates/server/src/lib.rs).cargo fmt --all --checkandgit diff --cached --checkpass.Targeted tests passed before the full fast gate:
cargo test -p infraternal-server --lib dedicated_tunnel:cargo test: 2 passed, 130 filtered out (1 suite, 0.06s)cargo test -p infraternal-server --test agent_enrollment dedicated_tunnel_hostname:cargo test: 1 passed, 6 filtered out (1 suite, 0.31s)cargo test -p infraternal --test agent_tunnel dedicated_tunnel_hostname:cargo test: 1 passed, 1 filtered out (1 suite, 0.40s)cargo test -p infraternal-server --lib remote_inspect_and_actions_use_signed_provider_rpc:cargo test: 1 passed, 131 filtered out (1 suite, 1.37s)cargo test -p infraternal-proto stream:cargo test: 37 passed, 34 filtered out (2 suites, 1.29s)cargo test -p infraternal-server --lib stream:cargo test: 41 passed, 91 filtered out (1 suite, 1.81s)Workspace clippy output was
cargo clippy: No issues found.th-merge finished.
Branch
tunnel-hostnameis at merge commit5d56cfda821862fd801ee1e8dae2ba017ba4130f; parents are5c6359ba715aec3e8122872c0bfa8f6c7fb12347and the requested merge target2265bb7f905792c70d38b38f45039020543b23db. The staged resolution kept this branch's dedicated tunnel hostname, SNI/SAN and enrollment wiring with dev's provider and stream changes. Tracked files are clean. The pre-existing untrackedtargetsymlink remains uncommitted. No push was made.origin/devhas since advanced to52c38730ff22ff2bbb184e572d84d82cbf72732d(three commits beyond the active merge target); this job concluded the merge already in progress against2265bb7as specified.INFRATERAL_VERIFY_RUN_ID=th-merge-20261003 scripts/verify fastexited 0. Verbatim gate output:Verifier summary:
Starting stream-batch2 integration. Only writer in /mnt/hdd/infraternal/wt-lp/stream-batch2, branch stream-batch2, base
d4fee7ba2f. Merge order: fence-batchf967b3a, remote-ssh0120952, flake-sweepff7e6f0; then teardown-followups if its updated dev merge is ready. Current teardown-followups tip46cf51bstarts with Merge origin/dev. Preserve fail-closed and durable-audit-before-CLOSE semantics, 850 ms close bound, local stream isolation. Cargo check after each merge; final requested debug validation and fast gate. No push.Starting read-mostly sdp-audit at detached HEAD
d4fee7ba2fa24d5dc55c0cbd22750c065df4668bin/mnt/hdd/infraternal/wt-lp/sdp-audit. No product-code commits or pushes. Building one release binary with preset CARGO_TARGET_DIR, nice 15, three Cargo build jobs. Real labelled fixtures only; demo containers excluded. Findings recorded in/mnt/hdd/infraternal/jobs/sdp-audit/findings.md.Baseline evidence at
d4fee7ba2f, one release build (14m 25s), product code unchanged.scripts/verify e2e-remoteexited 1. Real Docker and rootless Podman; fixtures relabelledinfraternal.test=sdp-auditby disposable argv wrappers. 46 actual passes, remote SSH OPEN returnsunsupported, remote infraternal-agent revocation actually passes but stale policy classifiesunexpected-pass(e2e/remote/policy.ts:15). Container remote inspect/actions pass on this audited head, unlike the older readiness document. Full evidence: /mnt/hdd/infraternal/jobs/sdp-audit/evidence/verify/sdp-audit-baseline/e2e/remote-sdp-audit-baseline/report.json.Verbatim gate excerpt:
Additional hands-on CLI, MCP, slow stop, 305-container, SSH and fault tests are running; these passes do not certify VM performance budgets.
Confirmed hands-on findings on unchanged
d4fee7ba2f:{closed:true}for both Docker and rootless Podman, but no client CLOSE arrives within 3–5 s andkill -0inside the owned test container confirms its native shell remains alive. Embedded DELETE closes and kills the shell. Suspectcrates/server/src/personal.rs:435–460: remote sessions are absent from embedded live-session sets, enter the already-exited branch without routed termination. Evidence:jobs/sdp-audit/evidence/followup.json.provider_runtime.rs:343,remote.rs:534,tunnel.rs:898), not yet a proven root cause. Evidence: followup.json, scale-timeline.json.DOCKER_HOST=unix://<owned-rootless-Podman-API>exposes the same actual Podman fixture in both Docker and Podman inventory, on embedded and remote paths.provider-podman/src/lib.rs:41reads Docker's override; no independent Podman override is available. Missing configured endpoints return healthy HTTP 200 empty inventories./api/v1/ssh/credentials/defaultreturns the embedded key rather than the enrolled remote runtime's key (compared in-memory; no key material emitted).logs --tail 1 --follow --timeout 2exits 2 before calling the server on all four routes.Positive controls: all four real CLI PTY+resize cases pass on corrected observer; piped stdin/EOF, 2 MiB exec output, exit codes, API+CLI lifecycle actions and bounded/multiline logs pass. Four API slow-stop cases honor the real 20-second TERM handler under 30-second grace. Rootless Podman 305 inventory passes embedded+remote API+CLI. Forty narrowed-token refusal attempts all return 403 and preserve native state. Real owned TCP tunnel drop reconnects the same runtime and retains both Docker and Podman persistent shells with replay. SSH changed-key refusal and explicit replacement pass; active SSH revocation closes forbidden in 105 ms (dev-VM diagnostic, not certified performance).
Audit-driver corrections are documented in findings.md and excluded from product counts. Private Docker now runs in a separate network namespace; after the initial private-daemon run exposed a missing shared docker0 interface, its recorded subnet/gateway and three verified existing endpoints were restored without stopping, restarting, removing or execing any existing container. A corrected isolated Docker run reproduces remote Docker inventory unavailability while embedded Docker remains usable. The report will explicitly retain this isolation incident and coverage limits.
Resuming stream-batch2 at
10840ac678(based4fee7b). All four requested merges and targeted tests already recorded; no merge or test repetition redo. Previous fast gate failed during linking with No space left on device. Current df: / 22 GB free, /mnt/hdd 16 GB free, both above 6 GB cutoff. Resuming debug-only fast in lean environment, nice -n 15; no code-review-graph rebuild, other jobs, or pushes.Integration head:
10840ac678Branch: stream-batch2; base:
d4fee7ba2f. No push.Merge notes:
f967b3a->e14a720: one stream.rs receive-loop conflict; keep pending-message burst batching and dev oversized-WebSocket Close 1009.0120952->4685895: no content conflicts; retain per-infraternal-agent SSH key/trust ownership and creator-lineage session authority.ff7e6f0->9bcdc52: one grant-shrink test conflict; preserve owner AND creator authority, adopt event-driven process exit evidence.46cf51b(qualifying Merge origin/dev commit newer thanc37f762) ->cbf31ff: conflicts in stream.rs operation/receive loop and contracts §2.3. Prepare bounded ready DATA before storage-admission waits so the socket read half can be dropped at cutoff; reuse the pair snapshot. Combine batched audits, shared reads, bounded concurrent detaches and 150 ms socket cutoff with transport-CLOSE reservation before audits.Follow-ups:
Validation (debug, lean environment, nice -n 15):
check-fence:
check-ssh:
check-flake:
check-teardown:
server-lib:
server-integration:
stream-regressions-confirmed:
stream-e2e:
runtime:
ssh-core:
ssh-embedded:
ssh-remote:
tunnel-allocation:
Concurrent cleanup repetitions:
Fmt check passed; workspace clippy -D warnings passes in fast on this head. No release build was needed. Benchmark-only cases reserved for the perf-test VM; this integration job makes no performance certification claim.
Outstanding real Docker/rootless Podman runtime tests completed after resume:
Final requested fast gate: EXIT 0. Verbatim output:
Verifier summary:
All requested merges included; teardown-followups was ready and was not skipped. Tracked working tree is clean. No push, release build or code-review-graph rebuild during resume. Prior disk-full gate attempt is superseded by this successful run. The final gate used lean environment, nice -n 15, debug only, incremental disabled and debug symbols disabled. This job does not certify release performance or replace the orchestrator's merge-tier and independent-review requirements.
Starting rust199 (#48).
Branch: rust199
Base head:
d4fee7ba2f(origin/dev)Scope: update for Rust 1.99.0 deprecation diagnostics, add the stable toolchain pin, inspect deploy container toolchains, and run the requested gates.
Finding: Rust 1.99.0 clippy reports two errors in
crates/agent/src/provider_runtime.rs:AtomicUsize::fetch_updateat lines 43 and 457 is deprecated and renamed totry_updatefor consistency. The compiler replacement is.try_update(Ordering::AcqRel, Ordering::Acquire, ...)at both sites.Deployment toolchain inspection:
deploy/o2/build.shsetsRUST_VERSION=1.98.1and builds withcargo +"$RUST_VERSION"; this explicit override does not follow the workspace rustup pin. I am checking whether the replacement API requires 1.99 before changing the deploy pin.Deployment decision: kept
deploy/o2/build.shat Rust 1.98.1. The newAtomicUsize::try_updateAPI compiles withrustc +1.98.1(a standalone probe exited 0), so the deploy script's explicit toolchain remains compatible with the replacement and does not need a version change.Additional Rust 1.99 clippy finding after replacing the two deprecated atomic calls:
clippy::assert_is_emptynow fails-D warningsfor five unit-test assertions (three undercrates/proto/src/stream/{controls,records}.rs, two undercrates/ssh/src/session.rs). Clippy recommends comparing the value to an empty array so failures show the actual value. I will apply those assertion-only replacements and rerun the gate.After correcting the first five empty checks, clippy found six matching non-empty assertions in crates/engine/tests/integration.rs; its assert_ne!(value, "") suggestions preserve the checks and report the offending value. It also reports AtomicUsize::try_update as stable since Rust 1.95.0 while [workspace.package].rust-version is 1.89. I will align the declared MSRV to 1.95 so the required replacement API is accurate and clippy remains clean without an allow.
The six assert_is_empty failures are now changed to value equality assertions. The agent crate alone now declares Rust 1.95 because AtomicUsize::try_update requires 1.95; workspace package MSRV remains 1.89.
The next clippy pass found two
duration_suboptimal_unitsdiagnostics incrates/agent/src/lib.rsat lines 1712 and 2480. Both useDuration::from_secs(24 * 60 * 60); clippy recommends the equivalentDuration::from_hours(24).The two Duration constructor lints are now changed to Duration::from_hours(24). The next all-targets clippy pass found 11 additional assert_is_empty diagnostics: two in crates/server/tests/auth_http.rs, one in crates/server/src/storage/fence_tests.rs, two in crates/server/src/stream/watcher.rs, and six in crates/infraternal/tests/cli_mcp.rs. They are assertion-only; I will use assert_eq!/assert_ne! comparisons with empty values as clippy recommends.
Starting multitab-refresh on branch multitab-refresh; base origin/dev and HEAD
d4fee7ba2f. Investigating product vs browser-check races, preserving exactly one refresh, adding Vitest regression coverage and 30 loaded multi-tab runs, then scripts/verify fast. Only writer; no push.The prior 11 assert_is_empty assertions are now changed. The following clippy pass found four more: two in crates/server/tests/ssh_product.rs, one in crates/infraternal/src/login.rs, and one in crates/infraternal/src/http.rs. I am scanning the remaining Rust assertions for the same lint pattern before the next clippy run.
Product races reproduced with Vitest: four regression checks fail on base implementation (idle signed-in broadcast adoption, storage newer before recovery, timeout causing another refresh, sign-out during generation wait causing another refresh). No refresh timer exists in auth.svelte.ts: refresh is initialization/401 driven. The provided production log contains refresh/401 cascades involving the original tab as well as both cold tabs. Current fix adopts successors in idle signed-in tabs, compares generation to the held credential, and returns without rotating after a failed generation wait. Contract clarification is separate commit
b8b8cf1, for independent review before integration. Browser smoke now waits for two actual native lock requests rather than 200 ms, keeps the original poll active, counts every context refresh, and waits for all three pages to successfully use one public successor token ID. Sources: https://www.w3.org/TR/web-locks/ (lock held until callback settles); https://html.spec.whatwg.org/multipage/web-messaging.html#broadcasting-to-other-browsing-contexts (queued broadcast tasks); https://playwright.dev/docs/api/class-route#route-fetch (fetch without fulfilling). Four regressions pass after the minimal product fix; further simultaneous-tab coverage and loaded production checks pending.The first fast run with my misspelled run-ID variable failed its cargo test stage. Relevant output, quoted verbatim:
FAIL fast: cargo test (62.505s)
---- real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce stdout ----
thread 'real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce' (1035311) panicked at crates/infraternal/tests/agent_tunnel.rs:580:5:
remote inventory was not published
This exact test passed in the preceding cargo test --workspace run. I am rerunning it alone, then will rerun fast with the requested run ID.
The failed fast test passes alone. Relevant output, quoted verbatim:
test real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.85s
The same test also passed in the earlier cargo test --workspace run. I will rerun fast with one test thread and CARGO_BUILD_JOBS=1, using INFRATERNAL_VERIFY_RUN_ID=rust199.
Implementation committed: docs
b8b8cf1, product/regressionsdd0922b, deterministic browser smoke1eb1596. Auth regression suite: 18/18 pass; svelte-check: 0 errors/0 warnings. Initial gate prerequisites exposed a local sccache error (incremental compilation is prohibited), including after unsetting/zeroing CARGO_INCREMENTAL; running the same scripts/verify entry point with RUSTC_WRAPPER empty resolves it without repository tooling changes. Production release build is underway on the dev VM; stress and fast gates remain pending. Rust 1.99 stable also emits existing fetch_update deprecation warnings in provider runtime files, which may affect clippy; no unrelated source files modified.Finished rust199 (#48).
Branch: rust199
Base head:
d4fee7ba2fHead SHA:
57704632b5Commit: Fix Rust 1.99 lint failures
Warnings fixed:
Direct gates:
PASS fast: migrations are append-only (0.347s)
PASS fast: workspace package dependencies (0.308s)
PASS fast: JavaScript dependencies (0.020s)
PASS fast: advisory review policy tests (0.267s)
PASS fast: supply-chain checks (9.201s)
PASS fast: build web app (23.959s)
PASS fast: cargo fmt --check (1.374s)
PASS fast: cargo clippy (21.500s)
PASS fast: cargo test (240.728s)
PASS fast: svelte-check (28.018s)
PASS fast: UI package svelte-check (12.157s)
PASS fast: stream package typecheck (12.254s)
PASS fast: Vitest (7.136s)
PASS fast: UI package Vitest (17.512s)
PASS fast: stream package Vitest (1.288s)
PASS fast: OpenAPI and TypeScript client drift (150.905s)
PASS fast: terminal TypeScript (3.048s)
PASS fast: terminal Vitest (1.654s)
Total: 531.676s
The gate also reported the existing low-severity Bun cookie advisory as report-only. No push was performed.
Production browser gate passed on head
107e502(focused multi-tab app checks plus the required terminal fixture and production budget stages). Invocation: RUSTC_WRAPPER= CARGO_INCREMENTAL=0 INFRATERNAL_VERIFY_RUN_ID=multitab-refresh-browser INFRATERNAL_MULTITAB_ONLY=1 scripts/verify browser. The first run passed all stages but exited 2 after editing the running shell script; this stable rerun exited 0. Real gate output:VM screenshot artifacts inspected: /mnt/hdd/infraternal/artifacts/verify/multitab-refresh-browser/browser/multi-tab-supported-1440x900.png and multi-tab-supported-390x844.png. Existing fallback notices captured at both widths as well. Applied UI skill rules: Emil invisible correctness/instant feedback; better-accessibility accessible names and existing status live region; better-layout/typography narrow-width wrapping; better-ui/colors reuse existing tokens and surfaces; better-writing preserve consistent notice vocabulary; better-interface evidence scoped to this auth flow. No visual source changes or broad accessibility/color audit claimed; Safari/iOS remain owner checks.
Eight-worker stress-ng load is active for 30 targeted production multi-tab repetitions; fast gate follows after stopping it.
Loaded attempt 1 stopped after 8 complete repetitions: supported=9/9 exactly one refresh and sign-out=9/9 passed; locks-only=8 and neither=8 passed. Failure in repetition 9: locks-only second tab account probe returned 401 after first tab background poll issued its own refresh, so second tab correctly cleared its session. The unsupported fallback test claimed sequential activity but left both tabs polling. Fixing only this fixture: isolated BrowserContext with the real cookie carried forward, Playwright controlled clock pauses interval polls before navigation, real initial requests drain before the next tab rotates, and explicit stale GETs remain real/serialized. Supported scenario retains native timers, Web Locks, BroadcastChannel and the exact-one assertion. Clock API is context-wide per https://playwright.dev/docs/api/class-clock ; install must precede timer use per https://playwright.dev/docs/clock . Raw failed stress evidence: /home/kayg/.cache/infraternal-targets/multitab-refresh/multitab-stress-attempt-1.log. All eight stress-ng workers stopped cleanly. Restarting the entire 30-repeat run after fixture validation.
Final head
f47f6b2: 30 consecutive loaded production multi-tab repetitions completed with exit 0. Supported=30 (each exactly ONE refresh across all three active pages); sign-out=30; locks-only=30; neither=30; total 120 checks. Native supported-mode timers and coordination remain active. Fallback clocks are isolated and initial real authorization responses are observed before another tab rotates. Eight stress-ng CPU workers stayed active throughout and were stopped/reaped before fast verification.Actual browser output:
Actual load output:
Raw stress log: /home/kayg/.cache/infraternal-targets/multitab-refresh/multitab-stress.log. Screenshot artifacts: /mnt/hdd/infraternal/artifacts/verify/multitab-refresh-stress/browser/ (supported, locks-only, neither at 1440×900 and 390×844). Required INFRATERNAL_VERIFY_RUN_ID=multitab-refresh scripts/verify fast is now running, with the previously documented RUSTC_WRAPPER= and CARGO_INCREMENTAL=0 environment workaround. No Rust source changes; no performance-budget claim from this CPU-loaded VM test.
Initial fast run on
f47f6b2exited 101 on baseline Rust 1.99 lints. Actual failure lines:While the job ran, origin/dev advanced from
d4fee7ba2fto411c758719(Rust 1.99 pin and lint fixes, #48). Integrated those already-landed changes with non-destructive mergee650051149; no additional backend changes authored here. Branch delta against updated origin/dev remains only docs/contracts.md, auth.svelte.ts, auth.test.ts, browser-check.mjs and the one verify output filter. Auth/smoke content is identical to the 30/30 loaded run onf47f6b2. Required fast gate restarted one650051with the exact multitab-refresh run ID; final result pending.Finished multitab-refresh. Branch: multitab-refresh. Head:
e650051149. No push; issues remain open.Root cause: the product ignored session broadcasts in already signed-in tabs without a pending refresh, leaving revoked access credentials to start another rotation on polling 401s. Comparing the refresh baseline against the maximum stored generation also hid a completed rotation whose broadcast was still queued. Generation-wait timeout/sign-out could fall through to another refresh. The fix adopts successor credentials in signed-in tabs, compares against the generation of the held access credential, and fences waits/async completion against sign-out; observing a newer stored generation never falls through to another rotation.
Browser fixture races also existed: the fixed delay did not establish real lock contention, the existing signed-in tab's refreshes were uncounted, and unsupported-mode tabs could poll concurrently during supposedly sequential recovery. Supported-mode now waits for native held/pending refresh locks and successful use of the same credential by all three active tabs, and counts every context refresh. Unsupported-mode uses a context-wide paused browser clock and drains real requests between sequential operations. Exactly-one refresh assertion is unchanged.
Validation: 18 auth Vitest cases pass; four new product regressions failed against the original implementation. Production release browser stress passed 30 consecutive repetitions with stress-ng --cpu 8: supported=30, sign-out=30, locks-only=30, neither=30; every supported repetition issued exactly ONE refresh. Workers and test server were stopped. Stress was on
f47f6b2b5b; its auth/test/browser files are byte-identical to this head. Final head merges origin/dev 411c758's already-landed Rust 1.99 lint fixes; no additional backend changes were authored here.Final fast command: RUSTC_WRAPPER= CARGO_INCREMENTAL=0 INFRATERNAL_VERIFY_RUN_ID=multitab-refresh scripts/verify fast. Exit 0. The wrapper override avoids this VM's sccache incremental-compilation rejection. Full output quoted verbatim:
Production browser stress output quoted verbatim:
CPU load output quoted verbatim:
Artifacts: /mnt/hdd/infraternal/artifacts/verify/multitab-refresh/{fast,stress,stress-ng,auth-regressions,browser}.log. Desktop (1440x900) and mobile (390x844) screenshots: /mnt/hdd/infraternal/artifacts/verify/multitab-refresh-stress/browser/. Inspected signed-in supported desktop/mobile and unsupported mobile notice; no visual source changes.
Applied UI rules: preserve shared primitives/tokens and existing accessible notice/live-region behavior; verify native production browser behavior at desktop/mobile sizes. The change does not add motion or styling.
Mapping to calternal-port: carry over signed-in-or-eligible-pending broadcast adoption, credential-generation baseline, and epoch fences after awaits. Preserve its session_request sharing and BFCache reload logic. Its cold-start historical-generation/250ms-peer timeout must remain distinct from a rotation completed during this recovery; blindly applying the no-fallthrough rule to historical storage would strand cold tabs. Native-lock/shared-cookie regressions apply; adapt new shell locators.
Owner Safari/iOS checklist (no simulator used):
Starting batch3 integration. Branch: batch3. Base head:
f0bf5fa. Sole writer; integrating fix-sessions-pty, multitab-refresh, fix-scale-podman, cli-mcp-actions, embedded-list-registry in that order. Clippy after each merge; workspace, real-runtime, SDP audit and fast verification at the end. No push.Fast gate passed at
51cb880. Workspace tests, auth Vitest, frontend checks and generated API drift all passed. Full real-service integration now running.Head
ca3d71dab6: merge gate passed.Production-browser screenshots, desktop and 390×844: /mnt/hdd/infraternal/artifacts/verify/batch3-merge-final/browser/.
Batch3 integration complete
Head:
66b4eafe5b. Branch: batch3. Base:f0bf5fa. No push; issues remain open.c509e6fc5a833ee6500515c461a23cff6e4d7d1d051f2dac9acb0e9e2a627b8638706dBatch3 integration conflict notes
fix-sessions-pty: e2e/remote/README.md and operations.ts conflicted. Retained SSH changed-key/revocation and scope-denial rows/imports; added persistent DELETE row and CLI PTY import. First Clippy attempts blocked by sccache incremental policy and missing embedded bundle; disabled wrapper and built locked web dependencies.
multitab-refresh: clean textual merge; auth implementation, Vitest tests, browser-check and verify changes retained. Clippy passed.
fix-scale-podman: eight conflicts, all overlapping Rust 1.99 empty-value assertions already fixed differently on dev. Kept dev typed-empty-array assertions and existing security regression additions; preserved bounded collector, independent Podman discovery, coalesced invalidations and true resync-gap semantics.
cli-mcp-actions: six conflicted files. Kept dev assertion fixes, added the incoming server-tail/follow regression, and used the incoming tested MSRV-compatible atomic compare-exchange helper at both admission sites. Removed superseded deprecation comments/allowances. Retained scale event semantics and existing security tests.
embedded-list-registry: resolved 24 conflicted paths, including facade modify/delete conflicts. Removed the engine facade and server LocalInventory; moved the bounded event-batch primitive plus four regressions to crates/agent/src/provider_runtime/event_batch.rs. Kept scale ordinary invalidation vs true gap semantics, while forwarding each compact notice into the new bounded runtime ring. Native actor-less events now invalidate normally instead of fabricating a gap. Retained the new event normalization and adapted the existing secret-redaction regression to the provider envelope. Kept remote SSH directives, ownership, identity-stop deadlines, tunnel controls, confirmed/absent DELETE acknowledgements, CLI tail/follow tests, and dev assertion fixes. Adapted security sweep fixtures to LocalHost + HostRouter. Removed legacy RemoteInventory-only tests with that removed path; registry tests/real audit check cover the replacement. Preserved native collector table, updated provider and remote SSH contract text. Clippy caught duplicate imports and stale API/facade references; fixed them without restoring the old path.
Integration fixes beyond the merges:
51cb880: compact-ring test admits two ordinary notices before injecting actual overflow.1939d36: real stream fixtures wait for exact inventory readiness before OPEN or dispatch failure injection; authorization and cleanup deadlines stay unchanged.ca3d71d: failed/incomplete bounded scans retry after one second on the same native feed, including idle initial and post-resync failure. Healthy feeds do not poll or construct a timer. Seventeen provider-runtime tests and thirteen real stream regressions pass. docs/contracts.md records the policy.714fa1e: HTTP readiness helper retains typed ApiError responses, allowing the existing 200/503 readiness polling to run without decoding an error as a container list.66b4eaf: serialize real-client scenarios and isolate their temporary credential config paths; four combined CLI/MCP tests pass in one process.Validation on final head:
66b4eaf.66b4eaf.Parity and SDP evidence used
ca3d71dproduction code. Diff ca3d71d..66b4eaf changes only crates/server/tests/resources_http.rs and crates/infraternal/tests/cli_mcp.rs, so the production code is identical.Additional real tests: provider-podman/real 3 PASS (rootful, rootless, DOCKER_HOST isolation); provider-docker/real 1 PASS; provider_real 2 PASS (signed parity/reconnect and 305-container remote-runtime burst); real_sshd 15 PASS; ssh_product 4 PASS; agent_tunnel with Docker+liveness 1 PASS; remote_ssh 1 PASS.
Evidence:
Failed runs are retained in batch3-integration, batch3-integration-final, batch3-integration-recovery, batch3-integration-complete and batch3-remote artifact directories. They exposed startup-readiness races, idle-feed scan recovery, and test-fixture collisions corrected above. An embedded Podman inventory deadline and a persistent SSH replay exit deadline did not recur in targeted or full rechecks; their precise causes were not proven, and no deadlines or expected-failure policy were relaxed. Successful CLI schema validation prints caught alternate-schema assertion panics; its actual test result is 4 passed, 0 failed.
Measurements here are dev-VM diagnostics, not a new performance-VM A/B certification. Existing low cookie advisory remains report-only under repository policy. All commits carry the requested trailers. No generated caches, benchmark data, credentials or secrets were committed.
Working tree is clean; the temporary target symlink was removed. All five incoming heads are ancestors of HEAD; all ten first-parent integration commits have the requested trailers. No push and no issues closed.
Successful per-merge Clippy output tails (command: RUSTC_WRAPPER= CARGO_INCREMENTAL=0 cargo clippy --workspace --all-targets -- -D warnings):
fix-sessions-pty:
multitab-refresh:
fix-scale-podman:
cli-mcp-actions:
embedded-list-registry:
Required fast gate, output excerpt quoted verbatim:
Full integration gate, output excerpt quoted verbatim:
Merge gate, output excerpt quoted verbatim:
Browser evidence inspected at both sizes: desktop, 390×844.
Head
66b4eafe5b: completed report supplement.Owner Safari/iOS checklist (Chromium production-browser smoke passed; AI jobs do not drive iOS Simulator):
Starting read-mostly sdp-audit2 on detached origin/dev
2b405b3a92. Re-running every SSH/Docker/Podman flow and rough conditions on embedded and remote infraternal agents. One release binary; no product edits or pushes. Findings: /mnt/hdd/infraternal/jobs/sdp-audit2/findings.md. Corrected round-1 observers will be reused.Release binary at audited HEAD
2b405b3a92; product files unchanged. The explicit-binary rerun ofscripts/verify e2e-remotepassed all 60 cases in 78.014 s, with no expected failures. Remote SSH, both engines' streams, native persistent DELETE and revocation all pass. One release build, 562.419 s; initial gate binary-path defect documented separately. Extended CLI/MCP, 305-container, slow stop, long sessions and isolated fault tests are now running.Verbatim gate output:
Live round-2 finding on unchanged
2b405b3: Docker and rootless Podman cold-start recovery failed on the three completed routing pairs (embedded Docker, embedded Podman, remote Docker; remote Podman running). An owned Unix passthrough exists but refuses connections during infraternal startup. After allowing it to connect to the real engine, a new native container is created through the same endpoint, labelled infraternal.test=sdp-audit2. After 30 s, the API inventory still omits that exact ID. Embedded paths keep HTTP 200 empty inventories; remote Docker transitions from 503 host_offline to HTTP 200 empty. This differs from engine API restart after successful discovery, which recovered on all four pairs. P0 core recovery gap; evidence rough-isolated.json and rough.ts cold recovery loop.The 305-rootless-Podman test also made the remote host return 503 for both engines while its infraternal-agent process remained alive, with all 305 fixtures visible embedded. A controlled scale recheck is planned; later remote SSH/session errors in that run are treated as cascades, not independent product findings.
Round-2 findings on unchanged
2b405b3, release binary built once:scripts/verify e2e-remotewith explicit release binary: 60/60 pass. Remote SSH, stream parity and native session DELETE now work.--attachrejected exit 2); MCP only supports nonpersistent container_exec. MCP has existing SSH session attach/send/read but no fresh SSH open tool.Detailed findings and before/after synthesis: /mnt/hdd/infraternal/jobs/sdp-audit2/findings.md. Native fixture cleanup is recorded before finishing. No product files changed or pushed.
Completed read-mostly sdp-audit2 at unchanged detached origin/dev HEAD
2b405b3a92. No product edits/commits or pushes. One production release build, 562.419 s, nice 15 and three Cargo workers.60/60 strict e2e-remote parity cases pass. Tonight's remote SSH, streams, native persistent DELETE, Docker scale and CLI tail-follow fixes are verified. API/CLI/MCP 20-second graceful stops within 30-second grace pass on all four routes; SSH/Docker/Podman persistent replay survives a same-process real tunnel outage.
Top ten scoped jobs: 3 P0, 6 P1, 1 P2 (one P1 is verification; product-only 3 P0, 5 P1, 1 P2):
Full before/after table, safe repro excerpts, suspected source locations, small job scopes and acceptance tests:
/mnt/hdd/infraternal/jobs/sdp-audit2/findings.md. Original and corrected scale/outage observer evidence are retained; cascades and observer failures are not double-counted. Dev-VM times are diagnostic; no perf certification or full merge/nightly claim.Verbatim successful gate output:
Successful report:
jobs/sdp-audit2/evidence/verify/sdp-audit2-release/e2e/remote-sdp-audit2-release/report.json. Explicit release binary supplied after the default-path harness failure, with no rebuild.Cleanup verified: zero owned labelled Docker/Podman fixtures, zero owned services, zero private harness directories; unchanged HEAD and clean git status. Shared Docker/Podman services and demo containers were not action targets. Shared sshd owned public-key additions were removed; private SSH watchdog cleanup completed. Issues remain open.
Round-3 hands-on audit started. Worktree /mnt/hdd/infraternal/wt-lp/sdp-audit3, detached HEAD
ea776ea7cb. No product code edits/commits/push. Reusing round-2 corrected API/CLI/MCP and fault harnesses; real embedded + remote Docker/Podman/SSH flows. One lean release build (nice 15, CARGO_BUILD_JOBS=3, preset target directory), followed by scripts/verify e2e-remote and read-mostly diagnostics. Findings: /mnt/hdd/infraternal/jobs/sdp-audit3/findings.md.Round-3 audit finished at unchanged detached head
ea776ea7cb. No product edits, commits or pushes.Full report, round-2/round-3 flow table, exact repro/evidence, suspected files, regression assessment and ten scoped follow-ups:
/mnt/hdd/infraternal/jobs/sdp-audit3/findings.md
Readable summary: /mnt/hdd/infraternal/jobs/sdp-audit3/summary.md
Raw evidence and replay drivers remain in the same job directory.
Ranked fix-job counts: P0 3, P1 4, P2 2 (nine jobs; includes one P1 verification job). The tenth entry retains the historical unconfirmed close investigation and is excluded from current defect counts. Two intermittent P0 observations have unknown root causes and may overlap.
Top findings:
Full emitted provider cleanup traces and typed runtime diagnostics were captured for the second 70-iteration sweep; first sweep lacked provider stdout capture. Traces retain only inspected safe targets, bounded state/PID/timing/category fields; no credentials. See provider-traces.log and provider-http-timeline.jsonl.
Release binary built once successfully after supplying web assets, nice 15, three Cargo workers, preset target directory. Binary SHA-256: 213f3ee6f41d614beda2bedd7a1c48a27a8a99c3e8fef911b5178d37583a4819.
Verbatim scripts/verify e2e-remote output (exit 1):
Parity 62/62; availability 5 pass / 1 fail. Diagnostic mixed repeat, not a replacement gate:
No newly confirmed regression of merged round-2 fixes. Cold recovery, typed unavailable engines, additional SSH target ownership, fresh MCP SSH, CLI persistent exec and 1000-line default snapshots work. Newly covered fast completion/remote state gaps cannot be dated from the round-2 evidence. This is not a merge or release certification; full merge/nightly/browser/perf-test-VM tiers were not run.
Cleanup verified: both real engine label queries exit 0, zero infraternal.test=sdp-audit3 containers; no owned services/private harness directories/fault data remain; shared sshd exact audit key lines removed. No demo action targets. Git status clean at the unchanged head. Issues remain open.